Gerry Docherty ged@rtel.demon.co.uk
Embedded systems and real-time problems
Thanks to all who have recently contacted me on the above topic.
Apologies for the slow response - it's been a bit hectic recently!
The following is a general chat on the problems peculiar to embedded
systems. If you're working with such systems just now, you should
recognise most of them. I've enclosed a copy of the presentation
slides I've been using recently - I can't claim that they are
entirely self-explanatory, but I hope they help.
There's been some renewed discussion recently about Year2000
problems as they relate to real-time and embedded systems in
production, manufacturing and engineering environments. We're
working on resolving the Year2000 problems in these areas already,
delivering the Real Time 2000 Programme to a variety of high-
profile clients. I hope the following is a helpful contribution.
The simple fact to grab and hold onto is that embedded systems
underpin the whole of the world's manufacturing and engineering
base. The world's energy supplies (oil, gas, coal, nuclear) depend
on embedded systems. Planes fly, and ships sail, based on embedded
systems. Pharmaceutical industries use embedded systems to create
the world's drug supply. The food we eat, the drink we consume,
primarily comes from processes which depend on embedded systems.
Not to mention clean water. And, of course, defence of the realm is
heavily based on embedded systems. And car manufacture. And railway
networks. And broadcast media. And communications. And so on.
So real-time and embedded systems are prime components of global
infrastructure. They are also the commercial building blocks of
engineering and manufacturing worldwide. So addressing the Year
2000 problems for these systems is at least as important as doing
it for banking and financial institutions. Probably more so. And
fixing the problems is more complicated.
It's true that all of the problems which exist in traditional big
IT applications also exist in real-time and embedded systems. So
we can have problems arising at processor level, or from operating
systems, packages/tools, and bespoke applications. The technical
solutions are also much the same - some replacements, some
modifications, some workarounds.
But the big difference is the culture which surrounds real-time or
embedded systems in production environments. Real-time systems can
be very complex, and they are used to control or monitor very high-
value processes. Typically, a large installation (e.g., a petrochemical
refinery, oil/gas platform, power station) will have scores of
real-time systems. They have been bought for different reasons by
different people over the years, usually mirroring the gradual
development of the installation. The production processes are now
dependent on the successful continuous operation of the real-time
systems.
Because the production processes are so valuable, production managers
and engineering staff fear the failure of real-time systems. When
real-time systems fail, high-value processes shut down, and the costs
of unexpected shutdowns can be enormous. For oil platforms,
pharmaceutical manufacturers or power stations, the cost of an
unexpected shutdown can be hundreds of thousands of pounds. Even for
small manufacturing companies, the costs are crucial, because the
production process is their only true source of income. The pressure
to keep the production process running is great. As a result,
production managers resist changes to embedded systems on the
if it ain't broke, don't fix it basis.
This means that when the next version of the operating system comes
along, it is not automatically installed. If improved functionality
could be achieved by upgrading bespoke software, it is not acted upon.
Hardware which is no longer supported by the manufacturer remains in use.
The result is a bunch of ageing systems, based on languages, packages
and processors for which the skills are gradually being lost. Because
of this culture, fixing the Year2000 problems is more complicated than
for banking or a dministrative applications. The systems are more
difficult to audit, because some are so old that the information about
them has literally been lost. Systems dating from the late 70s and early
80s are pretty common. Doing the triage is complicated, because there is
a risk that taking the system through a mock millennium change will cause
the process to fail, with huge cost penalties. Applying the fixes is
fraught, again because of the potential to cause a production failure.
So to fix the problems, you need people who understand embedded systems
technology, the production processes, and the commercial impact of
mistakes in a manufacturing environment. These people are very, very
thin on the ground. There are not many companies who specialise in
real-time and embedded systems.
From what we can see, few manufacturing companies have recognised the
scale of the problem yet. Systems are not yet failing, because
real-time systems tend to have a lookahead of less than a month. So
the failures will come late in 1999. Nonetheless, from our work over
the past six months in this area, we know that the likelihood of
failure of embedded systems is high. The companies we are working
with are in the vanguard. The big organisations might be able to sort
themselves out by throwing money at the problem, though resources will
be very scarce. The small manufacturers are in trouble - most of them
don't know they have a potential problem, and when they find out,
they'll find it very difficult to compete with the big boys for decent
skilled staff. Remember, of course, that around the office, embedded
systems are widespread - personnel tracking systems, PABX and Fax
machines, security access, heating and air conditioning, etc. Outside
the office environment, humble domestic appliances, alarms systems,
video recorders and the like also use embedded systems. For the most
part, the failure of these systems will have nuisance value, and I
don't worry too much about that. Between here and the Year 2000, we
only have time for the important problems.
Gerry Docherty, Real Time Engineering Ltd., Academy House, Academy Park,
Glasgow G51 1PR, United Kingdom. +44 141 427 4142. ged@rtel.co.uk |
