Electrical Grid
greenspun.com
asked in the TimeBomb 2000 (Y2000) Q&A Forum
Here's a report I received via an e-mail list. Has anyone else
seen this? What do you think?
------------ The World Energy Conference: Houston
I'd been looking forward to the big World Energy Conference,
which Houston hosted Sept. 14-18. With 130 nations present,
it would be a good place to see how the rest of the world was
doing with fixing their computers and embedded systems for
their utilities - or so I thought.
I got my conference materials and began looking for the
seminars on the problem and couldn't find any. Then I checked
the white papers submitted to the conference. There were
about 250 papers presented on almost every conceivable topic
including nuclear fusion and methane hydrides on the ocean
floor but there was nothing about the year 2000 problem.
I went down to the vendor exhibits to see if there was a
different story there. EPRI (Electrical Power Research Institute)
was there and handed me the only Year 2000 brochure I found
on the whole vendor floor. The only energy problem these
people seemed to be interested in was the low price of oil.
Except for one notable exception, the year 2000 was not an
issue, not even a footnote, at the World Energy Conference.
The one exception was the release of the NERC report on the
state of U.S. electric utilities. (NERC stands for North
American Electric Reliability Council and includes the area of
Canada and a little of Mexico.) The report is generally
optimistic about the status of the electrical utilities in North
America and their ability to deal with the year 2000 problem. It
was announced at the conference on Thursday morning by
Deputy Secretary of Energy Elizabeth Moler, along with a panel
of people representing every aspect of the electrical industry. At
the same time, it was put on NERC's website as an 88-page
pdf document, which you can download at
nerc.com.
An outline of the report was discussed and the panel answered
questions from an audience I estimated at 75 to 150 people.
During the question time several people from the press wanted
to know if the level of year 2000 remediation work around the
country was uniform and who the laggards were and how could
we get them to move a little faster.
After the questions, the meeting ended and the panel went to
the pressroom for a briefing. Almost immediately, writers for
the news bureaus posted several articles on the report. Since
the report was released, several authors have criticized it for
being too optimistic and not detailing the reasons for optimism.
But the reasons were there. The NERC has responses from
75% of the industry, has measured their progress for two
months and based on this can make a prognosis of the speed of
the fix. They have concluded that there is enough time left for all
utilities to be Y2K ready by June 30, 1999.
"Y2K Ready" the report defines as, "...a system or component
(that) has been determined to be suitable for continued use into
the year 2000." As of now, 28% of the testing of components
has been completed. What they have found is that only 1% to
2% of the devices or components exhibit year 2000 anomalies
and few of those devices or components will shut down their
system. Also, there aren't thousands or even hundreds of
industry operating systems but only dozens.
The NERC did conclude that progress needs to be accelerated,
and there was also concern for the 25% of the industry that had
not yet responded. In many cases, we were told, this is because
their lawyers told them not to respond due to fear of litigation.
In fact, it was pointed out by Secretary Moler that the only
group to oppose the information sharing bill then before
Congress was the Trial Lawyers Association. Michael Gent,
President of the NERC, told me one of the most difficult things
they were dealing with was getting their findings out to the
public. He is hoping their web site will get a lot of traffic.
When they release their next report in December, the NERC
will list the companies that have responded. If you don't see
your utility there, it might be a good idea to call them up and
ask them why, and mention that you read the NERC report.
But the industry is more than a quarter finished, the susceptible
chip count is low and the failure rate for the systems they are
part of is even lower. On top of that, the NERC is making sure
there is enough fuel to run the plants, plus a surplus. They are
also checking into the Y2K status of the information systems of
the utilities so there won't be problems with billing or account
management. It should make you feel better that 88% of the
Y2K project managers report to a vice president. Executive
buy-in is the single most important factor in a successful year
2000 project.
The nuclear plants were in even better shape than the industry
as a whole, according to the NERC, taking its information from
The Nuclear Regulatory Commission and the Nuclear Energy
Institute. The nuclear power industry has been working at the
fix longer and they are much more highly regulated. It is also
harder to change anything in a nuclear plant when it comes to
upgrading it. I confirmed this down on the display floor.
I talked to people at the U.S. Atomic Energy Commission and
Framatome Technologies, which make and operate the French
nuclear reactors. The United States gets about a quarter of its
power from nuclear plants, but the French use nuclear power
for 70% of their energy needs and have been building reactors
all along.
At the AEC booth I was told that our reactors were safe and
the French reactors should be safe, too, because they were
based on an older, well proven Westinghouse design. The last
reactor to come on line in the U.S. started in 1995 with safety
and control systems designed 20 years before. The man from
the NRC told me they were very conservative when it came to
certifying new systems for nuclear power.
In the French exhibit it took me awhile to get to the technician
(the woman in charge didn't speak English), but he confirmed
what I had been told. The French reactors are of one design
that differs only in scale, coming in three output levels. I asked
him about embedded logic in the plants and he told me there
was almost none, nearly everything was analog. He told me that
it was not by plan but rather a fortuitous accident. They had
been slow in changing the design of their plants and just hadn't
gotten around to upgrading the systems with a lot of embedded
logic, when someone realized it would be to everyone's
advantage if they put off the upgrades until after 2000.
Surrounded by the opulence of the energy companies'
technology and ability helped me remember to keep Y2K in the
proper context. It is an important problem but it isn't the only
important problem in the world. Fixing it is expensive, estimated
to take 44% of the IT budgets for the next year, and while
that's a lot of money, for many of these companies it is a lot less
money than they spend on other things, such as exploration. I
would have felt better if they had spent a little more time and
money to tell people what they were doing about the year 2000
problem but other than myself and maybe one or two others,
nobody seemed to be asking. The tech people I talked to all
knew what their companies were doing about the problem and
that everything was on schedule. To the energy industry, Y2K
is just another business problem. I was reminded that the tail
doesn't wag the dog.
But later that morning at the luncheon, Patrick Wood, the head
of the Public Utilities Commission in Texas, told me he thought
the report was a whitewash, although he said he didn't know if
it was a deep whitewash or just a coat of paint. The situation in
Texas is a little different. Most people know we have an
Eastern and a Western power grid. But there is also a third
grid, called the ERCOT Interconnection, that actually covers
most of Texas. Mr. Wood doesn't like what he is hearing from
the disparate parts of his grid and is worried about getting them
all working together.
Who do you believe, the government, the people in the industry
or the consultants? The consultants are claiming you have to
find every single last potential problem and test it, which I think
may be overkill, even if there was enough time left. But if you
hire them and they miss one, guess who is liable? I hope it
doesn't come as a surprise to anyone, but the government has
been known to claim things were better than they subsequently
turned out to be. I don't know what the situation is in Texas, but
I knew some people in other parts of the country and I spent
the weekend calling electrical engineers.
What I found was that the NERC report is probably right on
the money for most of the country. There is enough time left,
and what they are finding is not that disturbing. The people I
talked to just aren't finding any problems in chips where the
clock function isn't used, and they are finding very few
problems with the systems that do use dates. Very soon now
someone is going to say out loud that you don't really have to
check every chip or system if a hundred others just like it have
been checked and nothing has turned up.
Electric people have a lot of experience dealing with outages. A
former engineer for Pacificorp told me that when there are five
generators in a plant, all are independent of each other so that
one or more can be down at any time. If they have identical
systems and the first two-test fine, do you really have to test the
other three? If time is pressing and if your consultant budget is a
little thin and your plant people tell you they have a
work-around, you might just put checking those three units on
the back burner.
Electric plants are like the railroads. They are old technology. I
have checked the RR switches on the main line near me and
none of them have chips, just big bronze padlocks, probably
the same ones they used when my father was a boy. I could, all
by myself, switch a train on a manual or computer-controlled
switch in 15 minutes with a sledge hammer. (The switch would
continue to be operational, too.) Give me an hour and I could
figure out a way to do it without using the hammer. To make a
rail switch work you only have to move two rails (designed to
be movable) three inches.
Peter de Jager used a similar example at the SPG conference in
San Francisco. He asked, "what would you do if you owned a
small business and all your mainframe programs died?" Do you
just lock the doors? No, you run down to the computer store
and buy 50 copies of Quicken and you set up your core
functions again. You can be up and taking money in a few
hours.
This is why buying a year's worth of food is probably a bad
investment unless you're really into camping. The findings that
are starting to come out now just don't warrant that level of
anxiety. The December report from the NERC and the January
look-forward failures will give us a much better picture of
where we really stand.
Best practices, Jon Huntress jon@year2000.com
The Year 2000 Information Center year2000.com
This coverage is Copyright 1998 Year2000.com Partnership
Asked by Libby Alexander (libbyalex@aol.com) on October 18, 1998.
Answers
What do I think?
I think this is one of the most rational, thoughtful, and thorough
investigations of the grid I've read. The author of this calm,
investigate, first-hand report has done us all a great favor by
injecting reason and facts into an increasingly frenzied debate
whose one dominant note seems to be a particularly nasty
brand of fear, to wit, I'll get mine, and to H with all the rest of
you.
Thanks very much for posting it. We needed this antidote.
Answered by Jeffry Lichten (Jef2870a3@halcyon.com) on October 18,
1998.
sorry, this is a long post... and it's the response to this "report"
from Rick Cowles posted on his site at euy2k.com in the forum
area at:
greenspun.com
______________________________________________________________________
All,
The following is my reply to Jon Huntress' recent mailing to the
Year2000.com Announcement List (from year2000.com)
regarding the recent World Energy Conference. Jon's entire
report is not reproduced here, but you can view it on the CBN
Y2k website at:
cbn.org
--- Jon,
Given the subject of your recent mailing, I was obviously
interested in your personal take on the World Energy
Conference. After reading your report, I was somewhat
dismayed.
I was dismayed by two things: the inaccuracies in the report,
and your apparent acceptance of the information given to you
verbatim. My comments follow.
On Fri, 16 Oct 1998 08:35:37 -0500 (CDT), you wrote:
>Year2000.com Announcement List, Special Mailing,
>October 16, 1998
>After the questions, the meeting ended and the panel went to
>the pressroom for a briefing. Almost immediately, writers for
>the news bureaus posted several articles on the report. Since
>the report was released, several authors have criticized it >for
being too optimistic and not detailing the reasons for >optimism.
But the reasons were there.
You make the statement, "But the reasons were there", and
then present a couple of very weak arguments. I provided some
very strong arguments for viewing the report as whitewash
bullshit, which is exactly what it was. If you haven't seen my
rebuttal to the NERC report at y2ktoday.com, I'm attaching a
copy of it to this email.
>The NERC has responses >from 75% of the industry, has
measured their progress for two >months and based on this can
make a prognosis of the speed >of the fix.
75%, Jon. You've been dealing with Y2k for quite awhile. Let
that number roll off of your tongue. It's October, 1998.
>They have concluded that there is enough time >left for all
utilities to be Y2K ready by June 30, 1999.
The report did NOT conclude this. The report reiterated
NERC's deadline, and said it was and aggressive goal.
"Aggressive" is an understatement.
>"Y2K Ready" the report defines as, "...a system or
component >(that) has been determined to be suitable for
continued use >into the year 2000." As of now, 28% of the
testing of >components has been completed. What they have
found is that >only 1% to 2% of the devices or components
exhibit year 2000 >anomalies and few of those devices or
components will shut >down their system. Also, there aren't
thousands or even >hundreds of industry operating systems but
only dozens.
First, I didn't find this anywhere in the report (other than the
28% figure). Second, if the industry is only finding 1 to 2
percent of the devices having "Year 2000 anomalies", why does
my company's database, with over 50K discrete components
and control systems cataloged, find an "anomaly" rate of 15 to
20%, with about 5% being showstoppers? Third, while you (or
NERC) are probably right about "dozens of operating systems",
consider that there are only a few dozen major operating
systems - Windows 95/98, Unix, MVS, DOS, VMS, etc. etc.
This paragraph belies a fundamental misunderstanding, on your
part, of the nature of the problem. The O/S's for mainframes,
networks, and desktops are not necessarily the issue (but in
some cases are, for older versions). It's the programs behind
the operating systems!
>The NERC did conclude that progress needs to be
accelerated, >and there was also concern for the 25% of the
industry that >had not yet responded. In many cases, we were
told, this is >because their lawyers told them not to respond
due to fear of >litigation.
And you believed this?
>When they release their next report in December, the NERC
>will list the companies that have responded. If you don't see
>your utility there, it might be a good idea to call them up >and
ask them why, and mention that you read the NERC report.
>But the industry is more than a quarter finished, the
>susceptible chip count is low and the failure rate for the
>systems they are part of is even lower.
Read this paragraph again. "the industry is more than a quarter
finished..." Jon, again, it's October, 1998. Not October, 1996.
I'm a 'glass half full' type of guy, Jon, but in this case, the glass is
3/4 empty.
>The nuclear plants were in even better shape than the
>industry as a whole, according to the NERC, taking its
>information from The Nuclear Regulatory Commission and the
>Nuclear Energy Institute. The nuclear power industry has
>been working at the fix longer and they are much more highly
>regulated. It is also harder to change anything in a nuclear
>plant when it comes to upgrading it. I confirmed this down
>on the display floor.
>I talked to people at the U.S. Atomic Energy Commission and
>Framatome Technologies, which make and operate the
French >nuclear reactors.
The A.E.C. was replaced in the 1970's by the NRC. There is
no A.E.C. There is a Canadian AECL - perhaps that's what
you were alluding to.
>The United States gets about a quarter of >its power from
nuclear plants, but the French use nuclear >power for 70% of
their energy needs and have been building >reactors all along. >
>At the AEC booth I was told that our reactors were safe and
>the French reactors should be safe, too, because they were
>based on an older, well proven Westinghouse design.
This might be just a poorly worded paragraph. 50% or so of
U.S. plants are Westinghouse design (PWR's). 40% are
General Electric BWR's. The remaining 10% are B&W or CE
PWR design. Frammatome is a quasi-governmental
architectural / engineering design agency in France that used the
basic 2 and 4 loop Westinghouse PWR designs in all of their
plants. French plants are standardized, to a large degree. Not
so in the U.S. Every plant in the U.S. is like a fingerprint - no
two are alike. This is because of two things: the demographics
of the age of the U.S. nuclear plant fleet, and the number of
architectural / engineering firms that implemented the
aforementioned designs. The above paragraph gives the
impression that all U.S. reactors are safe because they are
based on an older, well proven Westinghouse design. So was
Three Mile Island. Design of the plants, plant operating
characteristics, or 'safe shutdown systems' have nothing to do
with the Y2k problem. And that seems to be what everyone in
the industry is to be blindly focusing on. Process monitoring and
control, and event logging have *everything* to do with the
Y2k problem in the nuclear industry. You, of all people, should
know this, Jon. Last word on nuclear power: The NRC just
released the results of its first Y2k audit of a nuclear facility, the
Monticello plant in Minnesota. Monticello just kicked off a
formal Y2k program in July, 1998. But the NRC, again, is
"cautiously optimistic" that Monticello can make the NRC's
deadline of June, 1999. Again, I'll let you figure this one out.
>But later that morning at the luncheon, Patrick Wood, the
>head of the Public Utilities Commission in Texas, told me he
>thought the report was a whitewash, although he said he
>didn't know if it was a deep whitewash or just a coat of
>paint. The situation in Texas is a little different. Most >people
know we have an Eastern and a Western power grid. But
>there is also a third grid, called the ERCOT Interconnection,
>that actually covers most of Texas. Mr. Wood doesn't like
>what he is hearing from the disparate parts of his grid and >is
worried about getting them all working together. > >Who do
you believe, the government, the people in the >industry or the
consultants?
I'll ask two simple questions and let you ponder the answers:
who has more expertise at working with the problem? Which
group is focusing specifically on what the problem is all about
about? You said it best in a previous paragraph: this is merely
another business issue to this industry (and a business issue that
they, for the most part, haven't got much of a clue on how to
deal with).
>What I found was that the NERC report is probably right on
>the money for most of the country. There is enough time left,
>and what they are finding is not that disturbing. The people >I
talked to just aren't finding any problems in chips where >the
clock function isn't used, and they are finding very few
>problems with the systems that do use dates. Very soon now
>someone is going to say out loud that you don't really have
>to check every chip or system if a hundred others just like >it
have been checked and nothing has turned up. > >Electric
people have a lot of experience dealing with >outages. A
former engineer for Pacificorp told me that when >there are five
generators in a plant, all are independent of >each other so that
one or more can be down at any time. If >they have identical
systems and the first two-test fine, do >you really have to test
the other three? If time is pressing >and if your consultant
budget is a little thin and your plant >people tell you they have a
work-around, you might just put >checking those three units on
the back burner. > >Electric plants are like the railroads. They
are old >technology. I have checked the RR switches on the
main line >near me and none of them have chips, just big
bronze padlocks, >probably the same ones they used when my
father was a boy. >I could, all by myself, switch a train on a
manual or >computer-controlled switch in 15 minutes with a
sledge hammer. >(The switch would continue to be operational,
too.) Give me >an hour and I could figure out a way to do it
without using >the hammer. To make a rail switch work you
only have to move >two rails (designed to be movable) three
inches. > >Peter de Jager used a similar example at the SPG
conference >in San Francisco. He asked, "what would you do
if you owned a >small business and all your mainframe
programs died?" Do you >just lock the doors? No, you run
down to the computer store >and buy 50 copies of Quicken
and you set up your core >functions again. You can be up and
taking money in a few >hours. > >This is why buying a year's
worth of food is probably a bad >investment unless you're
really into camping. The findings >that are starting to come out
now just don't warrant that >level of anxiety. The December
report from the NERC and the >January look-forward failures
will give us a much better >picture of where we really stand.
I can't even bring myself to dissect the above five paragraphs.
Jon, I am really distressed by the last few paragraphs in your
report. I'm distressed for a variety of reasons. This is the most
non-critical and non-researched view of the Y2k issue I've ever
read, and I am truly surprised that it made the cut with Cliff and
Peter.
1. You apparently bought the industry party line, hook, line and
sinker, with absolutely no critical analysis of the information
presented to you.
2. It's very clear to me that you don't understand the Y2k issues
in the electric industry, or in fact the industry itself, and so you
reported the hook, line, and sinker. I've also come to the
conclusion that, even with your background at Tenagra over the
past year or so, that you fundamentally don't understand the
Y2k problem (or you've been "Charlie Reubenized"). You were
clearly out of your depth in reporting on this conference.
3. In the last paragraph in the report, you flippantly told people
that personal advance Y2k preparations are a waste of time
and money. Your article, posted on the year2000.com, and via
the year2000.com mail list, reaches hundreds of thousands of
decision makers, and individuals like myself. Now, I'm most
assuredly not one of the 'apocalyptic' doomsayers - but I'm
responsible enough to tell people that, given the nature of the
issue, that some low cost, advance preparation for some
lifestyle disruptions (no matter how remote the possibility), just
makes sense.
Given the powerful podium that you speak from on the Y2k
issue (year2000.com), I expect critical thinking and justification
for significant optimism such as written in your report. If there's
a reason for optimism, then support it with some independent
verification of your own. Your report did a major disservice to
the cause, as a whole.
...Rick Cowles (rcowles@waterw.com) on October 17, 1998.
__________________________________________________ |
