Read and enjoy <gggg>
datamation.com
---------------------------------
September 1998
---------------------------------
MANAGEMENT
The world's biggest
Easter egg hunt
The specter of Year 2000 problems in millions of embedded systems--from telephones to time clocks to security systems--looms large on the millennial horizon. In partnership with their facilities management colleagues, IT managers must identify and triage these systems now.
By Lauren Gibbons Paul
--------------------------------------------------------------------------------
Brad Pence, manager of nuclear process computing services for Omaha Public Power District, worries about cases where the vendor claimed its product was "Y2K compliant" or "Y2K ready," but OPPD's testing revealed a failure. With Pence is Jim Sojka, another member of the Y2K Core Team.
Photo: John Nollendorfs/SABA
--------------------------------------------------------------------------------
With all the sleepless nights spent worrying about being able to fix the Year 2000 bug in time, the last thing most IT managers fret about is their business grinding to a halt due to a Year 2000 failure in a phone system or an elevator. Yet these humble systems--which contain microchips that may be vulnerable to the Y2K problem--are staples of corporate and everyday life and are more fundamental to the functioning of business even than computers. (See "Lurking problems and what to check").
Brad Pence remembers that when his company began tackling the Y2K problem in 1992, systems containing microchips were the farthest thing from his mind. "We always thought it was a software problem. Embedded systems hadn't yet reared their ugly heads," says Pence, manager of nuclear process computing services for the $500 million Omaha Public Power District (OPPD) in Omaha, Neb. Since then, Pence has been heading up the effort to identify, prioritize, and replace the plant's problematic embedded systems. So far, he has cataloged 2,300 items (including plant control systems, distribution control systems, and continuous emissions monitoring systems, along with the more mundane telephones and elevators) that are Y2K suspect.
And when these systems are the foundation on which businesses run, 2,300 mundane items are no small matter. "What's the point in spending millions of dollars fixing your computer systems, if you don't have phones, elevators, or heat come January 1, 2000?" asks Michael Harden, president and CEO of Century Technology Services, a consulting company and vendor of Y2K remediation services in McLean, Va.
"This is a global Easter egg hunt, and you don't know how many eggs are out there so you'll never know if you've found all of them," says Brian Kishline, manager of systems engineering for Data Dimensions, a software vendor and consulting company in Bellevue, Wash.
Once you do sniff out these rotten eggs, the dubious payoff is the cumbersome process of contacting the vendor--if it's still in business--to determine the Y2K compliance status of the device (See "Caught in the Y2K time crunch? Compliance databases can help"). Then, if the device is not Y2K compliant, it's generally a matter of replacing or retiring the device altogether. Fixing the code is generally not an option, since usually you won't have access to the source code. With less than 500 days to go, time is too short for most companies to contemplate code repair for the myriad of distributed devices containing embedded systems at their premises.
The scope of the problem
It's easy to ignore embedded systems, since they are necessarily hidden from view (see "On the lookout: a 14-step methodology"). But, hard as it may be to fathom, experts say the Y2K problems inherent in embedded systems are much broader in scope--and potentially much more expensive to fix--than those of business computer systems. Like the Y2K issue in general, the embedded systems piece of the problem is fraught with uncertainty. No one can say for sure how many embedded systems are out there and how many will fail come 2000. Since it's impossible to determine the scope of the embedded systems problem, it's likewise impossible to specify how much it will cost U.S. businesses to fix the problem.
--------------------------------------------------------------------------------
AT A GLANCE:
Omaha Public Power District
The company: OPPD, based in Omaha, Neb., is a $500 million publicly owned electric utility--one of the largest in the United States. It serves more than 270,000 customers in 13 southeast Nebraska counties.
The problem: To identify, test, and remediate equipment (such as flow controllers and analyzers) containing embedded digital chips that are vulnerable to the Year 2000 problem.
The solution: OPPD has assigned two project teams, one that handles the mainframe software repair effort and the one that tackles hardware, desktop applications, and embedded systems, to identify vulnerable systems. If a system is susceptible to the Y2K problem, the vendor is contacted for compliance information and/or the responsible team conducts testing. The results of this testing will determine whether a system needs to be replaced or whether it can function adequately while displaying the wrong date.
The IT infrastructure: A mix of proprietary mainframe-based applications with client/server applications running on a TCP/IP WAN with 2,300 PCs. Main systems include the Novell NetWare 3.12 and Windows NT network operating systems and PeopleSoft's financials/general ledger.
--------------------------------------------------------------------------------
From interviews with the top logic chip manufacturers, Harden estimates that approximately 5 billion of the 70 billion chips produced since 1972 are subject to Y2K problems. "The question is, how do you go through the 70 billion to find the 5 billion that will have a problem? It's the quintessential needle in a haystack," he says.
Andrew Bochman of the Aberdeen Group thinks the problem rate is much higher than Harden puts it. "From what my clients are saying, I'm looking at a trouble rate of about 20%" of all devices containing embedded systems, says Bochman, senior analyst for Year 2000 services at Aberdeen, a Boston-based IT consulting company. And Aberdeen's manufacturing clients report spending three to four times as much on their embedded systems remediation efforts compared with their computer systems. Whatever that figure might be will only be determined in hindsight, but whatever the amount is, it's a lot of money.
Taking control of embedded systems
Of OPPD's 2,300 Y2K-suspect systems, Pence says only about 10% are "priority-one show stoppers. These are the ones that would cause us to have to shut the plant down and stop producing electricity if they failed," he says. For those systems, in addition to contacting the vendor for compliance information, Pence's group is doing its own testing to determine the extent and type of the Y2K failure. Ominously, there have been cases where the vendor claimed its product was "Y2K compliant" or "Y2K ready" where OPPD's testing revealed a failure.
"We've had some vendors say everything's good to go, but we've found out it's not. It's a little scary," says Pence. His group is now figuring out how much of their own testing they'll need to do. "It's a judgment call. If we get pressed for time, we may have to go with the vendor's word." (Ooooops!)
The results of OPPD's own testing will determine whether or not the system needs to be replaced. Some mission-critical systems may present a potential date problem, but can function adequately while displaying the wrong date. Determining how a system is used will involve working with the staff people who use the system.
Like many organizations, OPPD has split Y2K responsibility between two project teams, with one handling the mainframe software repair effort and the other tackling hardware, desktop applications, and embedded systems. In fact, despite his title, Pence is from the nuclear side of the organization, not IT. He says there are IT people on the team handling embedded systems, but the leadership comes from the business side.
Caught in the Y2K time crunch?
Compliance databases can help
Contacting vendors for Y2K status information on their products is the heart of the remediation exercise for embedded systems. But there's no need for companies to duplicate each other's efforts. Here's a sample of the many general and industry-specific Y2K databases that have sprung up to pool information on many of the products containing digital chips.
Century Technology Services' Century Process Remediation (CPR)
CPR services include an easy-to-use data repository that contains nearly one million items. If a requested item is not in the database, CTSI will automatically generate and send a letter to the vendor seeking Y2K status. Pricing varies.
(http://www.ctsi2000.com)
The Electric Power Research Institute's (EPRI) Year 2000 Issues for Embedded Systems Project
Membership in EPRI, a consortium for electric utilities, includes access to a Y2K device compliance database.
(http://year2000.epriweb.com/index.html)
InfoLiant's Year 2000 Network Advisor
This service culls publicly available year 2000 compliance information for roughly 15,000 products, from PCs to applications to routers, into a Lotus Notes database. The service costs approximately $2,800 for up to 200 Web inquiries and about $5,000 for 1,000 inquires.
(http://www.infoliant.com)
Tava Technologies' Compliance Y2kOne
Part of Tava's Plant Y2kOne service, the Compliance Y2k database contains information on approximately 50,000 plant floor devices and systems. Fees vary.
(http://www.tavatech.com)
Whose problem are they, anyway?
To date, most Y2K project managers from the IT side of the house have done a miserable job of identifying infrastructure and other devices that may contain susceptible embedded systems, says Century Technology Services' Harden. This is understandable, he adds, since they've been too tied up grappling with fixing Y2K problems in the business systems. IT managers are not used to the rigors of strolling the floors of the corporate offices and plants--from the bowels to the rafters--taking into account everything that could rely on date-sensitive data.
"IT people typically understand the information systems, how computers and code work, and how to manage a large project like this. But with embedded systems, the skillsets are very different. They need to crawl around on the floor and climb up on ladders. It's much more cumbersome. IT managers tend to ignore embedded systems because the skillset is so different," says Harden.
It's literally a dirty job, but someone's got to do it. Although facilities or operations personnel are often charged with the monstrous task of taking inventory of on-site devices containing embedded systems, IT managers must coordinate and support the effort if they have overall responsibility for the enterprise Year 2000 project. And that requires formulating and implementing a coherent embedded systems strategy to minimize risk (See sidebar "Y2K manager's road map to embedded systems").
In addition to someone from facilities or the shop floor who has intimate knowledge of the equipment, the team must also include some business managers. "There are business decisions to be made: Should this device be fixed or dumped? All embedded systems are not created equal," says Corinne Gregory, vice president of product marketing for Data Dimensions. In many cases, the right option will be to alter processes so the device is no longer used. For most companies, it will be too expensive and time-consuming to replace everything containing an embedded chip.
On the lookout:
a 14-step methodology
The following guidelines will help you identify devices with embedded systems that may cause year 2000 problems. If you answer "yes" to any of these questions for a specific device, it should be considered suspect.
1. Does the system display or print a date or time? This would indicate some type of date function is integral to the operation of the device.
2. Does the system produce regular reports? If reports are generated by the device, and dates are part of the report, there may be a problem.
3. Does the system store historical records? If dates are stored, they may also be manipulated and sorted.
4. Does the system time-stamp data? If a system date-stamps records, logos, or products, it will likely be dependent on utilizing a date that may not be able to handle the year 2000.
5. Does the system implement a timed sequence? If the system starts or stops a function based on date or time, it may have a problem.
6. Does the system perform an operation on a time or date basis? Systems that perform a function based on date or time, such as locking doors on weekends, depend on the correct date.
7. Does the system perform a calculation based on the differences between time or date? Systems that determine intervals, averages, or total times could be at risk for year 2000 problems.
8. Does the system request the date/time on start-up? When power is turned on, a system dependent on date may request it as input.
9. Does the system send date or time information to other systems?. If a system receives date information from other systems, it may have a date problem. Systems that must synchronize themselves with other systems will typically be dependent on knowing the exact date and time.
10. Does the system receive date information from other systems? If it doesn't have a date problem, it may be dependent on another system that does.
11. Does the system have a command that allows the date to be set? If the device or system allows a date to be input, there is likely a need for a correct date.
12. Does the system know which day of the week is based on a particular date? For example, if the system can tell that June 1, 1998 is a Monday, then some kind of calendaring function exists, and consequently a year 2000 problem is likely.
13. Does the system generate an alert based on some type of interval? If a system creates some kind of notification based on an elapsed period, an elapsed time counter may be involved, which has no date problem, but a real time clock may also be involved, which does. It is difficult to know which is being used, so these systems are suspect.
14. Does the system display or print data based on a time sequence? Logs or listings of events by date or time indicate a dependency upon knowing the correct date.
Source: "Millennium Minefields," Michael Harden, Century Technology Services
A whole different skillset
Many consultants and observers agree IT should not lead the search for embedded systems with Y2K problems. "If you have an IT person running this, the danger is that the person is going to unduly emphasize the business systems" at the expense of the embedded systems, says Aberdeen's Bochman. "Everyone is very late getting to this issue. IT may never get there."
"The IT manager is the worst person to [take on embedded systems]. You're talking about a whole different skillset. It's much more cumbersome, so it tends to go to bottom priority," says Century's Harden.
In many cases, IT managers are likely to try to get in and fix code, says Data Dimensions' Gregory. "But the goal with embedded systems right now is just to manage the risk, prioritize, and triage your systems."
IT people by themselves do not have enough perspective on the devices containing embedded systems, continues Gregory, whereas the operations people usually have contingency plans in place for those devices that are mission-critical. "Most embedded systems are expected to fail at least a couple of times over their lifetimes," she says. "Most operations people know what they can do to work around that." Of course, Y2K contingency planning must go well beyond run-of-the-mill contingency planning since demand may be too great for the usual alternatives (get someone to come in and fix the machine or go to a back-up site) to work. (See Karen Watterson's story, "When Y2K failure isn't an option," in the June 1998 issue of PlugIn Datamation.)
The buck stops here
There is an exception to the notion that IT is often best left out of the embedded systems Y2K odyssey. That's where the chief information officer, or another high-ranking IT official, is in charge of the enterprisewide Y2K effort. "Anything relating to Y2K is my responsibility," says Vin Lioce, CIO and vice president at Boston Mutual Life Insurance Co. in Canton, Mass.
Although he has no direct control over the operations department, Lioce is overseeing the facilities people's efforts to contact the elevator and telephone manufacturers for compliance status on their wares. "I'm keeping a file on everything. I consider all of this my domain," he says.
In fact, there's a real opportunity for enlightened IT professionals to educate the rest of the organization on infrastructure-related Y2K dangers, since they have lived with and understand the issue better than the other corporate sectors. The key, says Data Dimensions' Gregory, is that the business users help set priorities. "This is an enterprise issue. The stakeholders in those business operations need to be cognizant and involved."
The joys of analog
Of the many businesses affected by Year 2000-related problems, the nuclear power industry is one of the scariest industries to envision suffering infrastructure-related Y2K failures. Ironically, nuclear power plants are actually much farther ahead than any other type of utility, says Rick Nicholson, senior program director for energy information strategies at Meta Group, in Stamford, Conn. In fact, the nation's telephone systems are much more vulnerable than nuclear power reactors. Bellcore, for example, recently announced that 75% of its voice-networking equipment is date-sensitive.
"Nuclear power plants are the ones the public worries most about, but they're the ones that are the furthest along," says Nicholson. One reason is that most nukes' control systems are hard-wired and do not contain computer chips. The control systems are mostly analog, and thus not subject to Y2K worries.
That helps OPPD's Pence sleep at night, but he is ever vigilant. He has to be. Says Pence, "The subtleties of embedded systems are everywhere." //
Lauren Gibbons Paul is a freelance writer in Belmont, Mass., who writes frequently on year 2000 issues. She can be reached at laurenpaul@sprintmail.com.
Are there any embedded systems at your company that you identified as being Y2K suspect? E-mail us at letters@earthweb.com, and tell us what you found.
--------------------------------------------------------------------------------
Y2K manager's road map to embedded systems
Here's an easy shorthand for helping you identify whether an item will have a problem: Does it ask you to input a date at any point? Some devices, such as medical diagnostic equipment, require users to enter maintenance information. That's a red flag that the new century is likely to pose a problem, says Andrew Bochman of Boston-based Aberdeen Group Inc.
"A lot of self-diagnostic systems send a message they will stop operating until they've been maintained" and will require the user to enter the maintenance date, he says. That kind of system most likely will have to be replaced if you can't get adequate assurance from the manufacturer that it can handle four-digit years.
On the other hand, "A microwave oven doesn't care about the date," says Mike Wilkinson, president of Paragon Innovations, an electronics and software design company in Dallas. Wilkinson also points out that the vaunted example of new cars having over 50 embedded systems is misleading. "They do have lots of embedded systems, antilock brakes, fuel control, door controls, air bags. But they will not have Y2K problems as a rule," because they are in continuous operation and do not use date-based input.
After identifying suspect items, Y2K project managers who are responsible for embedded systems should follow these steps:
Form an embedded systems team. Make sure the team includes facilities or factory foremen who know the systems, as well as a high-level business manager who can set priorities.
Take inventory. This is by far the worst part of the process. "The inventory and assessment part are monstrous," says Aberdeen's Bochman. Get a blueprint of each building and go around with the team, recording make, model, serial number, and location--at a minimum--on a tape recorder. All is suspect. Don't overlook things like lighting systems, card-key readers, postage machines, copiers, refrigerators, and door locks.
Document your findings. Transcribe the tapes and create a spreadsheet with detailed information on all equipment containing embedded systems.
Assess/triage the systems. With time so short and source code usually unavailable, there should be no thought given to testing the devices for Y2K compliance. The only thing to do is to judge the impact on the business if the system fails (minimal, significant, or catastrophic) and then to prioritize vendor repair or replacement strategies.
Contact the vendors for Y2K compliance status. Consider using one of the many Y2K compliance databases from vendors such as InfoLiant and Tava Technologies or an industry alliance to share information and speed up the process.
Secure adequate funding from the financial people.
Repair/replace/retire the devices. Most top-priority devices with dubious Y2K status will have to be replaced.
Plan post-remediation support. Once the first few days of the year 2000 pass, other critical dates (such as leap year, February 29th, and the first change to daylight-savings time) will come up that may cause problems. Be ready with extra staff and contingency arrangements.
Keep in mind that not all systems with embedded chips will fail. Common sense should be your guide, at least to some degree. --Lauren Gibbons Paul
--------------------------------------------------------------------------------
Lurking problems
The following devices are among those likely to be vulnerable to Y2K problems due to their embedded systems:
Alarm clocks
Cameras
Door locks
Fax machines
Mobile telephones
Personal organizers
Time activated equipment
Time clocks
Traffic lights
VCRs
Vehicles
Voice mail systems
ATMs
Elevators
Environmental control systems
Fire detection and control systems
Refrigeration systems
Security systems
Telephone switchboards
Vaults and safes
Vending machines
Water heating systems
Lighting systems
Telephone systems
Ticket machines
Personal computers
Testing equipment
Copiers
Mail opening/sorting equipment
Postage machines
UPS systems
Electronic signs
Biomedical devices
Avionics
Point-of-sale terminals
Radar systems
Source: Century Technology Services
|
