Embedded Systems at Cargill
This is a report on a session at a Y2K conference about the method Cargill has developed to do I&A on embedded systems in small to medium size plants. Very informative on the subject. Note reference to TAVA in 4th paragraph.
Year2000.com Announcement List, Special Mailing,
July 15, 1998
by Jon Huntress
When you look at the numbers -- trillions of lines of computer code, billions of embedded chips and millions of dollars to find and fix everything, it is easy to get depressed, come to the conclusion that there is no way out, and that the gloom and doomers are right. Society as we know it is going into the toilet, or I should say the outhouse, because the toilets won't work anymore because the
pumps that move the water to the water towers will stop working because of the embedded chips on the automatic controls and valves. That's what a guy at the next table told me. How many of you out there remember the expression, "rougher than a cob?"
This was in my mind when I attended a session put on by Philip Hannay of the Cargill Company. The Cargill Company started in Iowa, just like I did, and they are well known for agricultural products and services. They would certainly benefit if it became necessary to go back to the time when a box full of corn cobs was a necessary accessory to every outhouse.
But Philip was telling the audience a different story. He said that while the problem was real, it wasn't all that bad. In the plant environment, many of the embedded system problems are minor if known about in advance, solutions are straightforward, work-arounds can save time and money, and that "wait and see" for non-critical applications is an option. A consultant in front of me was getting visibly upset at those conclusions. What Philip was recommending, he whispered to me, was too loose, too unstructured, too fast, to deal with the embedded systems most factories have.
I was coming to a different conclusion. I think Philip Hannay is a genius. So I warn you, there will be wide differences of opinion on the embedded systems assessment, inventory and remediation techniques recommended by Hannay. In a future report I will contrast this with TAVA's approach to the same problem with some major companies. They are quite different, and yet Ken Owen of TAVA brought Mr. Hanney to the conference, introduced him and sat behind him for the presentation. During the talk, Mr. Hanney told everyone to put off calling the big consulting companies until they had completed their own inventory and made their own plan. This is another case of people working together for the good of everyone, even though it seems to be counter to their own interests. TAVA needs to be congratulated for their forward thinking and ecumenical attitude. There are differences that make the both approaches worthwhile, but the Cargill approach hasn't been presented as a complete plan before.
The Cargill Company has 922 plants located around the world. While still primarily agricultural, they are also in recycling and steel, having mini-mills in six states. Cargill is not a publicly traded company. Even though they
don't have to answer to the SEC, Mr. Hannay pointed out that their response to the year 2000 problem is the same as everyone else. Delaying the fix would cause Cargill to lose market share which would impact the employees as directly as any other company. Cargill is also under all of the other federal regulations related to the environment and health and safety.
Cargill has a very loose corporate structure and the plants have a lot of autonomy. Most of the major decisions are made at the plant level and some at the division level. Philip, as head of their Y2k effort, didn't have the authority to order action in the plants. He came up with a plan to achieve compliance at the plant level quickly, inexpensively and in some cases even without the active support of management. Philip Hannay has created a simple and easy-to-
implement year 2000 embedded systems remediation plan for small and medium-size production facilities. It is a good plan for Cargill and other similar business but it is especially good for the 50% of the small and medium sized businesses I mentioned in the introduction to this conference coverage, that have not yet begun. For them, this could be a lifesaver. This is a blueprint for procrastinators, a plan for getting back on track. This is how they can avoid going out of business and this is about the last chance they will have because there just isn't enough time left for anything else.
Using a "Good News," "Bad News" approach to the presentation, Philip gave us some statistics. 10% of the plant components have a year 2000 issue. The scary statistic is that 50% of all the systems within a plant contain one or more of these components. With numbers like these, unless the systems are found and fixed, a plant wide shutdown is almost a certainty.
Let's get right to the plan.
The first "good news" was that getting started is easy. Just begin. It isn't necessary to have a meeting or a budget or a plan or even to get higher level corporate support, although at this stage, management participation should not be difficult to enlist. Philip gave several good arguments for getting management on board. He suggested the company lawyer review "due diligence" and "negligence" with them, and pointed out that there is a Linda Tripp in every company, waiting for an opportunity to make life difficult. He recommended not wasting time on meetings, and to bypass the big organizations, consultants and outsourcing. He pointed out that consultants often don't know much about plant systems, and after they are hired they have to be taught what the plant does and how it works. Outsourcing only makes sense if you already do it for all of the engineering. Philip said that deciding on definitions of compliance and debating ratings is a waste of time. Just get out in the plant and find out what you have. It is a good idea to skip system triage at this stage also. Triage and prioritizing are considered only when doing the remediation plan.
The next step is to decentralize responsibility to the plants by defining the basic steps everyone has to take. Cargill has two corporate people, forty plant people, and a local plant contact person. That is only three levels for the entire company. All the data is shared on a web site on the company intranet. Each plant gets a page and so does each vendor. Cargill has a very good intranet but Philip pointed out that having a company intranet is not necessary. Buying a web presence from a local provider is just as good and cheap, costing only $200 a month for a normal password site and $400 for one with advanced security. And he said forget about using fancy Excel spreadsheets or Access databases because they can inhibit the flow of information. The goal is to get as much reporting done as soon as possible. Everyone is going free format with a word processor using outline form. Cargill uses one document per plant. The plant information is divided into the areas of survey information, assessment, and the plan for each component in the plant.
Philip's first assertion is that nobody knows the plant better than the people who work there everyday. They might be taking their systems for granted, but they know what and where they are and what they do. They are familiar with outages, and the embedded systems problem at the plant level is a problem of outages. He recommended spending only eight hours on a plant wide survey as the first step, and taking no more than forty hours for the first assessment. Remember that this is only for small to medium sized plants, and Cargill's plants fit in this category.
First do a walk through of the plant and look at every production system. An electrician, an engineer, and one or two other people familiar with the plant should make this tour. On the first go through, just write down the make and model number of any piece of suspect equipment. A few weeks later, after the document process has started, do it again. One of the essential parts of this plan is that it demands a repetition of the assessment every six months right up
until 2000.
After the material has been written up it is posted on the plant's web page. Cargill's intranet is a private IP based network based on CISCO routers connecting LAN's at each office and plant site. TCP/IP can be done end-to-end anywhere, while limited routing of older protocols (SNA, DECNET, Novell) is also allowed on certain segments. Since TCP/IP can be used end-to-end, any software can be used that works on the Internet or on the internal intranet. Cargill's intranet is connected to the Internet using a standard firewall that passes e mail and certain other selected protocols like HTTP/web for identified Cargill users. Simple search engines have been put on top of the plant and
vendor sites, so that anyone can do a quick search and find out what other plants have problems with Allen Bradley units or what a particular vendor has to say about the compliance of his product. Philip stated that 70% of the vendors have web sites and many of them have posted year 2000 information about their products.
Along with their 922 plants worldwide, Cargill has 2 research sites, 4 aircraft, 2 barges, and 10 ocean vessels. So far they have found 637 vendors and 3122 products. Following are some of the problems they have encountered:
The ROM-based programs are difficult to view and change. PLC ladder logic programs don't have comments on the programming. DCS programs are written in proprietary script and needs an engineer experienced with the script. They have issues with the Fairbanks scales, Yokogawa recorders, WinSTAR timeclocks, MS Windows OS, Visual Basic compiler, Crisp-16 control, Intellution control and Wonderware control. They also have problems with their automatic Railcar Unloading using PDP-11/RSX11M/CRISP 16. Half of the systems in the plant contain one or more of the above components. There are also other controllers, weigh scales, recorders, phones and timeclocks scattered throughout all the plants.
But many of the fixes are easy when they are identified early. For Windows, the fix is to download a patch from Microsoft. For the PDP-11s, they are doing a workaround by archiving the old files and then using a date setback of 8 years to get to the same day. With the Yokogawa chart recorders, some of them are bad and some aren't. One of the fixes is to wait until 2000, and set back the date on the ones that don't work. With much of the suspect equipment, if it isn't in a critical system, "wait and see" is an option as long as there is a prescribed work-around. They are printing up the prescribed work-around and taping it to the machine so everybody can see it and will know what to do. Fixing the Intellution requires upgrading from 5.6 to 6.0. The Fairbanks scales need a ROM upgrade, WinSTAR will get an upgrade to 4.0, and some recorders will be replaced. For the PDP-11s, most will be setback to 1972 so the days of the week and the leap years will line up. The PC dryer controls will have their date set back to 1992 so the leap year lines up.
One of the problems Cargill has had is that many of the plants only shut down once a year. The fertilizer plants are always running flat out, no matter what the price of fertilizer is. If the assessment team misses the shut down date, it will be a year before it will happen again. In these cases the team needs to be ready to jump in any time there is an unscheduled shut down. And finally, there are the critical suppliers of natural gas, fuel, electricity, enzymes, chemicals and transportation, and the plant wide systems of telephones, intercoms and security.
This is a very loose, ongoing year 2000 remediation plan that many people, especially those in IT, probably won't like. Most of the IT people I have met are not comfortable with such a laissez faire attitude and want a much more thorough system of assessment and testing. But this plan will work because it is ongoing, and uses the simple expedient of having work-arounds for every important system. Hannay thinks that 90% compliance will get his company through, and I think he is right. 90% is usually good enough. Some say that our interdependencies require a much more total fix, very close to 100% in order to assure success. They may be right. I don't think so.
There are several other benefits to this plan. One is that the legal issues are addressed. There is an ongoing corporate wide plan that shows a consistency of action across the entire company, and consistency of effort is the best way to show due diligence. The consequences of not fixing it right can be serious and even fatal. Loss of systems can cause loss of income and market share, but it also can have environmental and safety consequences such as toxic spills or releases. Malfunctioning equipment can also cause injury or death to employees. This is a good opportunity to check all the fail points in a production process. Everything should fail in a safe way that doesn't threaten the employees or the environment.
Also, for the first time ever, Cargill will have a list of current equipment in all of their plants. Using the company intranet and simple editing procedures they will probably be able to sustain this into the future. This is an essential part of any disaster recovery plan, and will be an integral part of the business plan for after the millennium.
But what happens if they run into something they can't test, or that doesn't have a work around? If something comes up that is beyond the ability of the people at the plant -- such as the discovery that a vendor for a piece of equipment is no longer in business? Then the consultants and Y2K vendors can be called in. And when they do come in there won't be a long period of getting them up to speed; they will be helping fix a particular part of the plant and everyone will understand what needs to be done. The Cargill plan doesn't take away the need for the consultants or outsourcing or any other year 2000 solution. But it does put that effort into an enterprise wide plan that is ongoing at the grass roots level, where it needs to be.
The Cargill plan can be used by other organizations too. It could be adapted for municipal and community organizations because, like the Cargill, they are diverse with a lot of local autonomy. Philip mentioned one more thing he did that is helping the plan succeed. He asked his staff, "How can we make this more interesting for the people who are doing it?" Questions like this are important when you have to ask for people's support and help instead of just demanding it as a condition of their employment. For part of the answer he bought a bunch of digital cameras and began loaning them to the plants. The plants would take pictures of what they wanted to show about their plant and put them on their company web sites. It increased plant loyalty and effort, and it helped communication within the plant and among the plants, helping them learn from each other for the benefit of the whole company.
To summarize, the plan is to start immediately where you are with what you have. Do a quick walk through and assessment, (just make and model #) and write it up in outline form. (Eschew obfuscation!) While you are researching vendors, develop priorities and work-arounds. Use the internet or intranet for posting the information. Don't forget the supply chains. Communicate, communicate, communicate! Do it again six months later. Call for help if you need it.
If you have strong opinions on this method pro or con, let me know and I may include them in a future report.
Stay tuned for part three of our coverage of this conference next week.
Best practices,
Jon Huntress
jon@year2000.com
The Year 2000 Information Center |
