Good write up on TAVA
Now I will address what one company (Tava) is finding as
they work on the non-IT assessments of a number of large
businesses. Bill Heerman of Tava's Denver office presented
the session in Chicago. Mr. Heerman reported on four of the
companies his company was helping. They are working with a
major food processing company, a major automaker, a major
soft drink company and a major pharmaceutical manufacturer.
Because of contract agreements, Mr. Heerman couldn't name
the companies. Mr. Heerman began by using the illustration
of an iceberg and comparing the non-IT area of manufacturing
to be like the big (underwater) part of the iceberg -- an
apt metaphor for titanic times. He gave a couple of
examples of IT personnel at companies saying, "That's not my
job," when asked to help with embedded systems issues. Then
he made the comment that while traditional IT counts the
money, it is non-IT that MAKES the money. Finding the
embedded systems is everyone's job because everyone's job
depends on finding and fixing the systems.
Brainstorm devoted several days of the Chicago conference to
the non-IT aspects of the year 2000 problem. I am not going
to attempt to cover the methodology and tools used for
non-IT assessment and remediation. If you want to know how
they do it, just call some of the vendors or go to the next
Brainstorm conference in New York (Aug. 31-Sept. 2) What I
will cover in this issue are the numbers that Mr. Heerman
released on the overall difficulty of each project, what
they think it will cost and how long they think it will take
for four different major companies.
------------ Case Study #1: The pharmaceutical company
The pharmaceutical company has global operations in 39
countries. In addition to their manufacturing systems, they
have laboratory systems, along with their facility systems
(and their external dependencies). One fully integrated
complex was done. 4,457 items were inventoried and of these
2,618 were unique. 392 were in the Lab, 980 were in the
facility, 837 were in manufacturing, and 409 were found in
external relations.
The results were that 18% of the more than 4000 items were
found to be non-compliant and 17% would cause a plant
shutdown or would affect production.
In the lab, 18% of the hardware was non-compliant and 27% of
the software needed to be fixed. 36% of the custom code was
non-compliant. 8% of the suppliers of these non-compliant
systems were no longer in business and 11% of the suppliers
would not supply an upgrade except with the purchase of a
new model.
Of the manufacturing systems, 16% of the hardware and 22% of
the software was non-compliant. 41% of the custom code
wasn't compliant either. 9% of the suppliers were out of
business and 18% would not provide upgrades except through
new models.
In the company's facilities, 18% of the hardware and 33% of
the software was non-compliant. 29% of the custom code was
non-compliant. 11% of the suppliers were no longer in
business and 21% wouldn't upgrade except for new models.
In the external dependencies, 12% of the hardware, 25% of
the software, and 27% of the custom code was non-compliant.
5% of the suppliers were out of business and 9% wouldn't
upgrade except through new models.
The chance of these facilities failing was 70% for the lab,
80% for manufacturing and facilities, and 90% for the
external dependencies. This means that without any
remediation at all, total plant failure is a very good bet.
It was estimated that seven out of nine manufacturing lines
would stop running within the first 3-4 days, and the network
and telecommunications equipment wouldn't work. Security
systems would admit anyone with any credit card and the
in-plant power substation would fail along with municipal
water.
For just one plant, the inventory is expected to take eight
weeks and the analysis is expected to take five weeks. Risk
assessment is expected to take two weeks. Inventory and
analysis would cost $487,000. The estimated time to fix is
43 weeks and the estimated cost is $1,200,000. For all 125
plants, the inventory and analysis would take 39 weeks and
the cost would be $11.5 million. The fix would take 31
weeks and the cost was estimated at 54.8 million.
------------ Case Study #2: The major beverage company
The major beverage company has global operations that
include 47 plants in 21 countries. The company has
manufacturing systems and laboratory systems along with
facility systems. There is also a lot of Eastern European
and Russian equipment in the plants. Nine plants were done
in a pilot project. 3,819 items were inventoried and 894
were unique. 210 were in the lab, 304 were in facilities
and 380 were in manufacturing. 24% were non-compliant and
28% of them would affect production.
In the lab, 34 % of the hardware, 26% of the software and
39% of the custom code was non-compliant. 16% of the
suppliers were out of business and 34% of the suppliers of
non-compliant products did not offer upgrades except through
new models. 35% of the equipment would have to be replaced.
In manufacturing, 26% of the hardware, 23% of the software
and 33% of the custom code wasn't compliant. 15% of the
suppliers were no longer in business and 21% offered no
upgrades except for new equipment.
In the facilities, 9% of the hardware and 6% of the software
was not compliant. There was no custom code. 19% of the
suppliers were out of business but 40% of the technology was
old and manually operated.
The inventory and assessment took 20 weeks and cost
$893,000. Time to fix was estimated at 35 weeks and the
cost at $2.9 million.
For all 47 plants, the inventory and assessment was
estimated to take 25 weeks and cost $4.9 million with the
fix taking 29 weeks and the cost estimated to be $23.4
million.
------------ Case Study #3: A major automotive manufacturer
The major automotive manufacturer has global operations with
162 plants in 41 countries. Two plants were done in a pilot
project. 1,389 items were inventoried with 435 of them
being unique. 131 items were in the facility, 254 in
manufacturing and 50 in external dependencies. 16% were
found to be non-compliant and 6% could cause a serious plant
shutdown.
In the manufacturing systems, 11% of the hardware, 19% of
the software and 37% of the custom code was non-compliant,
but this time all the suppliers were still in business. 18%
of the suppliers would not provide an upgrade except through
a new model.
In the facilities, 17% of the hardware, 19% of the software,
and 42% of the custom code was non-compliant. 7% of the
suppliers were no longer in business and 21% would not
provide an upgrade except through a new model.
The greatest threat to this client was the supply chain with
more than 60,000 suppliers. Of these, 7,500 suppliers are
critical to the manufacturing process. 30 suppliers were
investigated and only 6 had active and defined year 2000
programs. The other 24 maintained that they were already
compliant. After testing, 46% of these were projected as
non-compliant.
The inventory and analysis of the two plants took nine weeks
and cost $221,000. The fix was estimated to take 12 weeks
and cost $1.1 million. For all 162 plants, the inventory
and analysis was estimated to take 31 weeks and cost $13.6
million, with the fix taking 58 weeks and costing $72.2
million. The risk assessment for the plant projected that
3 out of 4 of the manufacturing lines would stop running in
the first 2 days. The networks and communications would
fail, and the security systems would refuse to admit anyone.
One of the two power substations would fail and the city's
sewer removal system would fail and could not be upgraded.
------------ Case Study #4: The petroleum company
The final large client was a global petroleum company with
18 plants all located in the United States. The systems
identified were quality systems, manufacturing systems and
energy management systems. The pilot project was an
inventory and assessment of a catalytic cracker and the
co-generation plant. The tie-on system with the local grid
was also included.
1,035 items were inventoried and 514 of these were unique.
365 were in manufacturing and 149 in external dependencies.
21% were not compliant and 6% would cause serious plant
shutdowns or affect production negatively.
The results for the manufacturing systems were that 11% of
the hardware, 14% of the software and 23% of the custom code
was non-compliant. 2% of the suppliers were no longer in
business and 12% of the suppliers would not supply an
upgrade except through purchase of new models.
For the quality systems, 24% of the hardware, 29% of the
software and 29% of the custom code was found to be
non-compliant. All suppliers were still in business but 16%
would not provide upgrades except through new models. The
risk of failure was put at 60%.
For the co-generation plant's results, 19% of the hardware,
36% of the software, and 24% of the custom code wasn't
compliant. 13% of the products were no longer made. In
this case, the energy management system tied into the local
grid that was non-compliant and couldn't be upgraded. Some
of the more interesting findings were that the catalytic
cracker would fail and the refinery could no longer make
gasoline. One of the more troublesome findings was that the
analyzers would continue to work but would send erroneous
data. The proprietary networks from the control systems to
the analyzers would fail. The inventory and analysis would
take 7 weeks and cost $122,000. The conversion for two
units would take an estimated 15 weeks and cost $760,000.
------------ In Summary
In an overview of all these facilities, problems were found
in the sensors & analyzers, lab equipment, programmable
Control Systems, embedded systems, SCADA systems
(Supervisory Control And Data Acquisition), distributed
control systems, human-to-machine-interfaces (HMI's),
networks, computers, third party applications, and Operating
Systems.
Mr. Heerman also noted that they had found that some of the
vendor compliance statements were incorrect. Some pieces of
equipment the vendor had claimed to be compliant had failed.
One piece of equipment successfully made the January 1, 2000
transition and was allowed to continue. Just over a month
later, when checked again, the date on the equipment was
January 34! This points out the need for continued testing,
especially at the system level.
Mr. Heerman also indicated that Tava has found that the
single most important thing that can make a year 2000
project successful is the degree to which executive
management is involved.
The projected risk levels for failure of all the units of
these companies was between 60% and 90% if the non-IT parts
of the business were not found and fixed. I think the main
difference between the Cargill and Tava plans is the level
of thoroughness in the inventory and assessment phase.
Also, a company like Tava has access to a large database of
compliant and non-compliant devices that might not be
available to individual companies.
An open question is the same one I posed in my coverage of
Cargill's approach. Is 90% assessment and inventory with
work-arounds for the rest good enough? One person wrote me
and said it was similar to a house losing 10% of its nails.
I agreed with him because I used to make houses. If a house
loses 10% of its nails over the entire house, it won't fall
down. Other than some more squeaks, probably no one would
notice any difference. That's because most houses are
overbuilt by a factor of at least 50%. But what if the 10%
of nails were all in one corner of the house, or just in
the floor joists? Then the house would fall down, or become
uninhabitable. Some companies will fail because of
this. For them, 90% won't be good enough because the nails
they lost were the ones that held the place together.
Usually, one nail doesn't make that much difference. But
sometimes it does. I'm reminded of the quote: "For want of
a nail, the shoe was lost. For want of a shoe, the horse was
lost. For want of a horse, the rider was lost. For want of
a rider, the battle was lost."
It will be really interesting to see what ultimately comes
out of this.
Best practices,
Jon Huntress
jon@year2000.com
The Year 2000 Information Center |
