Humbly report, from: msa.org
-----------------------------------------------------------------------
The Year 2000 Embedded Systems Threat to Core Infrastructure Services:
The Need for Discovery to be Done in the Current Months
by Roleigh Martin, M.A.
marti124@tc.umn.edu
ourworld.compuserve.com
The Daily Oklahoman, 10/29/1997, covered a press conference involving two
State Representatives, where they revealed the findings of a survey of power
utilities. The survey found one-third of all utility companies had not
started to correct the Year 2000 ("Y2k") electronics problem, and "another
third were severely behind." These legislators want Oklahoma to be the first
State with Y2k compliant infrastructures.
Two surveys conducted in early 1997 among public utility companies found
between 32% and 45% had not yet begun a Year 2000 analysis and repair
program and one survey found of those working on the Y2k problem, 37% were
behind schedule.
Four power utilities discovered that embedded electronic systems, critical
or significant for producing power, failed when they simulated the Year
2000.
Some of the companies in the utilities and related industries, who feel they
have solved their Y2k problems, have failed expert-hired Year 2000
compliance inspections.
The Electric Power Research Institute in their summary of their 9/10/1997
Y2k Embedded Systems Workshop (165 people from 54 utilities attended) says:
"Despite the industry's best attempts, there will probably still be problems
that will not be resolved necessitating some degree of emergency/disaster
planning." What about areas where a best effort is not made?
Do the utility engineers and managers know the full scope of the Y2k
embedded systems problem and the mistakes being made by some engineers and
consultants investigating Year 2000 issues in their equipment? Does top
management know how educated their own management and engineers are on this
issue? Is everyone aware of the longer than normal lead times due to
extraordinary difficulties many of which lay far outside the realm of normal
engineering expertise?
I have a web site at ourworld.compuserve.com
focusing on this problem and for awhile--until my campaign effort
embarrassed the utility industry I presume--the Electric Power Research
Institute's "Year 2000 Issues for Embedded Systems: Links to Y2k Sites" page
had an annotated link to the website. They noted: "Of particular interest is
a letter directed towards engineers at power utilities with a questionnaire
included."
The Gartner Group predicts that more than 50 million embedded system devices
will exhibit year 2000 date anomalies. The problem is determining which 50
million devices out of an estimated 25 billion devices that will be in
existence by the year 2000 and which of these devices are critical.
The 10/2/1997 issue of ComputerWeekly News reported on the magnitude of the
problem. Anthony Parish, director-general of the federation of Electronic
Industries said: "For every 1,000 embedded chips you look at, you'll find
two or three that need correction. But those two or three are the ones that
can close a blast furnace at the cost of œ1m a day or stop power
distribution. The problem is finding those two or three that are not
compliant." One petrochemical firm tested 150,000 embedded chips and found
100 not compliant.
The discovery effort for determining Y2k problems with embedded systems is
much harder than with computer software. With software, we normally know
where to look for problems, and we only have to modify the master source
code. With hardware, we do not necessarily know which of the 25 billion
chips have to be upgraded, which 50 projected million chips have to be
found. We have to test each copy of each suspect device to avoid the gotcha
that the pharmaceutical manufacturer Smith Kline Beecham found--see below.
We sometimes have to deal with devices that use embedded timing devices that
have no visual display of date/time, nor any means of input to see that a
date/time has been inputted at factory-creation time. With software, we can
discover and fix code up to midnight before 1/1/2000. With hardware, we have
to finish the discovery in time to get orders in, products delivered,
installed and verified. Last, there are fewer suppliers able to deliver the
replacements than there are programmers who can replace a line of code.
Smith Kline Beecham "bought two machines for monitoring and recording the
performance of drug production," says Guenier. "When they tested one, it
handled January 2000 very well, and they were very happy. But when they
tested the other -- same machine, identical chips -- it didn't." The scary
explanation for the anomaly, when the firm checked serial numbers with the
manufacturer, was that the chips had come from different makers, one of whom
had made them year 2000 compliant, while the other hadn't. Documentation
down to this level of detail is often not specified in the world of embedded
systems. And these were machines, notes Guenier, that had been made last
year."
Are all the enterprises that are testing their embedded systems aware of
this? Are they checking every single device that has time/date-sensitive
electronics in it or are they only testing a representative sample of such
devices? With the core infrastructures, we should not assume that the people
in charge know better because of the lack of historical precedence for Year
2000 upgrades.
The extent of embedded systems is vast -- below are examples of embedded
systems, both small and large, many of which might affect power utilities.
Not all of them are critical for generating power. Not all of them are known
yet to be significantly impacted by Y2k problems. Some are definite problems
at some utilities.
Examples of Embedded Systems (In Alphabetical Order)
Building Systems and Temperature Multi-Loop Control And Monitoring -
Controllers DCS, SCADA, Telemetry
Date Sensitive Computer Systems Are Panel Mounted Devices - Control,
Used For Forecasting Electrical Load Display, Recording And Operations
Demand On A Given Day, Hour, And
Minute Basis
Date Sensitive Systems Used For Programmable Logic Controls (PLC's)
And Embedded Date Sensitive Controls
Bidding Purposes In Wheeling And Power Exist In Transformers, Protective
Pools Relaying, And Breaker Control
Energy Metering Real Time Control Systems
Environmental Monitoring Equipment Remote Terminal Units
Field Devices - Measurement, Residential And Commercial Smart
Actuation, Recorders, Sensors Meters
Flow Controllers Simulators
Fossil Plant Boiler Control Systems Single Loop Controllers
Global Positioning System (GPS) Smart Instrumentation And Recorders
Problem
Heating, Cooling And Ventilating Substation Equipment Such As Scan Log
Systems And Alarm Recorders, Digital Fault
Recorders, Smart Relays And Breakers
Load Dispatch And Remote Switchyard Telephone Exchanges And Switches
Breaker Control For Power Plants
Monitoring and Signaling Systems Test Equipment Used To Program,
Maintain And Test Control Systems
ÿ Valve Actuators
The embedded systems problem is one that affects almost everyone, but most
heavily those in the infrastructure and manufacturing arena. Yet, many act
as if they have until 1/1/2000 to get things going. However, with embedded
electronic (hardware) chips, it is a race to get your manufacturing rework
job order into a manufacturer before thousands or millions of competing
orders get in front of yours. There are not that many job shops out there to
satisfy everyone by 1/1/2000.
The current months are critical for the utilities to conduct and finish
their discovery phase of exactly what equipment do they need to fix,
upgrade, or replace. To reinforce this urgency, at
epri.com, the Electric Power Research Institute
(EPRI) report on their Y2k Embedded Systems workshop held on September 9/10,
1997, which was attended by 165 people representing over 50 domestic
utilities. The EPRI writes that among the utility representatives at the
workshop, it was agreed that "time is critical given that the first
identified failure date for some systems is 1/1/99."
No utility can say they are on schedule to be Y2k compliant if they have not
completed the following phases. Until they are past these hurdles, the
phase-completion times are almost completely out of their hands:
ú The discovery phase of what equipment needs to be replaced or new parts
ordered, orders approved, vendors found that can deliver the items in the
time window desired, orders placed, and firm ship dates are secured.
ú Assuming the items arrive on time, the items must be tested in conjunction
with whatever existing or new equipment they are needed to be tested with.
Because Y2k compliancy can be done a multitude of ways, it can not be 100%
assured prior to testing that everything will work in sync. This also
assumes that everything is built to spec, and that the specs are correct.
ú If the testing reveals that nothing needs to be returned for modifications
or rejected and replaced with something else, then the remaining tasks of
installing all the received equipment in the field can perhaps be safely
scheduled and forecasted. Even this assumes that the original discovery
phase did not miss anything, that there is no loss of essential employees
(to better paying competitors, etc.), and that weather and natural disasters
do not slow down schedules.
Furthermore, no utility can say they are on schedule to be Y2k compliant if
they rely upon fossil fuel delivered by rail to their power generation
plants unless the railroads that service them from the source of the coal to
their plants are finished with their Y2k upgrading. I refer readers to the
recent shipping disasters taking place among customers of Union Pacific
Railroad because of the numerous computer problems following their
merged-buyout of the Santa Fe Railroad.
As for Nuclear-powered utilities, there are numerous issues including the
need to meet the Nuclear Regulatory Commission (NRC) Y2k guidelines, which
are covered in depth by Rick Cowles at his web site.
The above illustrates the problem in the power utility area, but what about
natural gas, gasoline, telephones, water, sewage, garbage, police and fire
services, health care, or a break-down in payment systems involving banks,
insurance companies, savings institutions, etc? There is also the need that
the scheduling software that services trucks can be depended upon to ensure
that at a least our grocery stores can be kept stocked. If some of these
disasters occur, life in Minnesota in January--a very cold month--will
become unbearable.
There are 126 municipal electric and 19 municipal gas utilities in
Minnesota. Living in the Twin Cities, I've tried to find out how NSP and
Minnegasco are doing with their Y2k embedded systems problem. Over a month
has passed and NSP and Minnegasco have not replied to my inquiry about the
Year 2000 problem. I wrote all the corporate officers, who were named in a
local Public Library, to no avail. I searched NSP's web site at
nspco.com but found nothing dealing with the Y2k software or
embedded systems problem. I have been interviewed by two newspapers, The St.
Paul Pioneer Press, and the Edina Sun Current. The latter spoke with Gaye
Jaenchen, a spokesperson for NSP. The reporter writes (11/5/1997): "She said
NSP addressed the point in its annual report to shareholders in 1996, and
that efforts to update NSP's equipment began a year and a half ago. 'We have
planned for this, we have budgeted for this,' she said. 'This is a
priority.'" To me it is more interesting what questions Jaenchen did not
answer:
ú Is NSP going to have an audit done by a professional Y2k engineering
consulting company experienced in doing Y2k embedded systems upgrades?
ú To the extent possible, is NSP going to do any partial or full system-wide
Year 2000 simulations of their equipment? The Knoxville Utilities Board
(KUB) announced in a press release printed in the 9/21/1997 Knoxville
News-Sentinel that in 1998 they will run computer system clocks forward on
Labor Day Weekend to ensure that the utility is Y2k compliant. KUB's Y2k
simulation is being done to prevent their "highly-computerized electrical,
gas and water service systems from turning into a technological pumpkin at
midnight on the first day of 2000." KUB says --industry wide-- that 2 to 4
percent of computers with embedded microchips could fail.
ú Is NSP finished with their discovery phase; have they placed their orders
for new equipment and have they received reliable receive-dates? If not, how
can they state they are on schedule? If yes, have they done subsequent
testing to ensure that everything new will work where need be with the
existing and other new equipment? If not, again, until the testing is done,
it is an unknown whether subsequent orders will have to be made.
ú Has NSP secured written guarantees from their railroad and fossil fuel
suppliers that they will maintain uninterrupted access to the fuel needed
for NSP's power plant?
ú Has NSP signed up for the Electric Power Research Institute Y2K embedded
systems program which costs $25,000 payable during the fourth quarter of
1997 and $50,000 for the 1998 program due the first quarter of 1998? This is
a new program explained at epri.com. The objectives of the
program is to provide a central clearinghouse for timely, Y2K information
including test data and test results, Y2K best practices, and contingency
planning. NSP knows about EPRI's program since Gene M. Heupel of NSP was
listed as attending the EPRI Year 2000 Embedded Systems Workshop September
9-10, 1997.
ú Has NSP surveyed and obtained satisfactory results from all of their key
vendors as to their Y2k upgrade plans?
If the answer to all of the above questions is "Yes," then one can probably
rest assured that NSP is really "on the ball" on this problem despite the
fact they have communicated nothing about this at their web site. I called
Ms Jaenchen on 11/6/1997 and asked her if NSP's discovery phase is completed
yet. She said she did not know. It is possible that others at NSP have not
let their own spokesperson know. However, since finishing the discovery
process would be cause for public celebration, it is more probable that NSP
has not finished their Y2k embedded systems discovery phase despite their
"efforts to update NSP's equipment [that] began a year and a half ago." If
it is taking NSP this long and they are not done, what about the other 125
electric utilities in Minnesota? If any of them have not started yet, will
they take as long as NSP is taking? That amount of time is not available to
them now.
The time to act and achieve positive results is now. The current months are
critical. In order to allow a reasonable period of testing after everything
new has been installed, to allow time for the orders to be approved,
ordered, received and installed, the critical time for discovery is the
period between now and mid-to-late 1998. Even then, there is no assurance
that things can be completed in time. I urge all of the readers of this
article to not only write their local core infrastructure services, but also
the public regulators in charge of the utilities, their local political
officials, and their newspapers.
Most of you have probably never felt the need to "go political" in the past.
But the Y2k problem, face it, is a technician-created problem, and
consequently technicians have to communicate this problem to the rest of
society. You can bet without such communication, the non-technicians who
manage everything will not understand the magnitude of the Y2k problem and a
disaster of historical proportions will befall some communities. We all owe
it to our families and society to speak out about this until we know that
all bases are covered and that triple auditing has taken place with
fallback/disaster plans in place. Yet so many working on the Y2k problem
walk around with horse blinders on, only focusing on their own systems. For
example, I called the Minnesota State Government Y2k head office and spoke
with the administrative assistant. She told me they were aware of the Y2k
embedded systems problem and that it involves core infrastructure
facilities, but she told me they have "too much on their plate" to take on
any additional work. If the people at the top refuse to do oversight on this
problem, then they must be reminded of their responsibilities by outside
political pressure.
It is depressing to visit every officially related web site located in
Minnesota and see nothing about the Y2k embedded systems threat to core
infrastructures and manufacturing facilities. Nothing at NSP. Nothing at the
State Government Y2k web page. Nothing at the Minnesota Municipal Utilities
Association web site. Nothing at the Minnesota League of Cities. Nothing at
the Hennepin and Ramsey County Government web sites. Nothing at the city of
Minneapolis and St. Paul Government web sites. Nothing at the Minnesota
Counties Association web site.
If the citizens of Knoxville and Oklahoma are intent on ensuring that their
core infrastructures are Y2k ready, certainly Minnesota, where it is much
colder, should be as responsible. We lead the nation in education but will
we lead the nation in showing that the education has taught us to recognize
a potential crisis and to proactively act responsibly?
There is much that is not covered here. Only the main points are introduced.
Please visit my web site to learn more. If you do not use a web browser,
email me at marti124@tc.umn.edu. I also recommend a book, Ed Yourdon's Time
Bomb 2000, forthcoming by Prentice-Hall, scheduled to be available by
12/20/1997. Personally, I'm ordering a large quantity of this book to hand
out to others, in the hope of waking up Minnesotans.
-----------------------------------------------------------------------
Svejk
(GL-15 applies: digiserve.com ;-) |
