'Enterprise systems, real world (long, technical and scary)
asked in the TimeBomb 2000 (Y2000) Q&A Forum
--------------------------------------------------------------------------------
I have taken the liberty of cross-posting from csy2k, because I think this is an important message and emphatically worth reading. It's quite technical, but I think even the non-technical people here will get the gist of it. To me, one report like this is worth 100 surveys by Taskforce 2000. Check it out
-----------
I am what Cory would describe as an Assembler gear-head. Big-iron, 20 years system software development, VTAM, NCP, blood'n guts machine code programming on three continents. And, believe it or not, before that I was a tax accountant.
For the cause (read; money), I've spent the last two years working on Y2K projects.
I feel like I know many of you here. My daily routine includes checking out Paul's informative, fascinating and often hilarious posts, and chuckling at BKS and Don Scott's predictable rebuttals. And Cory's commentary will one day make for a good play-by-play coverage of those crazy days leading up to the turn of the century when food was plentiful and rates were high. (Cory, forget the farm, get youself stranded on Oahu for the duration. You like fruit and fish too, don't ya? - and no woodstove!)
For those of you who must know if I am a Pollyanna or a Doomsayer, let's get that out of the way right up front; I have no idea how bad it will be. Neither do you. My personal experience leads me to believe it will be catastrophic. But I am not an embedded systems engineer, neither do I know much about utilities other than what we have all read. Those of you who do know about those things probably know very little about my side of things.
What I do know about is large-enterprise business systems, and how dependent business, and thus, the global economies are on those systems. I also know how difficult it is to remediate those systems for Y2K.
(By the way, have you ever noticed how spell-checkers trip over the word 'remediate' yet we use it all the time?)
Let me describe a typical Y2K project from my experiences - I'd like to know if this is true elsewhere - let me know.
Let's start with a large government department - they have multiple mainframe legacy systems going back 20 years or so, as well as every platform-of-the-day application you can think of. They decide its time to think about Y2K and after going through the usual bid process and ashort pilot project, (read; after one year) they pick a consulting firm to take on responsibility for all aspects; legacy code, embedded systems, building systems, etc. The chosen vendor has a solid reputation as a supplier of system migration consulting services, specializing in COBOL.
So,its spring 1998, the contract is signed, and the project is underway. That is, as soon as the vendor can hire the necessary skills. (I'm not kidding.)
The main requirement is for MVS COBOL skills. Well, just COBOL will do - its virtually the same on any platform right? So they get a headhunter on the job and go shopping for programmers. Now keep in mind that by this time, Cory has already started ranting about rates, and good legacy skills are almost nowhere to be found. (Calm down, I said **good** legacy skills.)
This where I come in. Fortunately for them, I had just arrived back in town after doing some contract work in Europe and the USA, and was tired of being on the road (I live in Canada). So what the hell, I needed a rest and some time with my family so I signed on as their first employee on the project. I would be the senior MVS guy who could provide guidance to those less experienced.
On my first day, I was introduced to the other two hirelings, a nice lady who had come out of retirement to refresh her COBOL from 15 years ago, and a fellow from the middle-east who was still struggling with english. He also knew COBOL but wasn't sure what MVS was all about.
The initial task at hand, was to learn how to use an ISPF based scanning tool to perform impact analysis, provide line-counts, and identify the potential trouble-spots in the COBOL code. Sounds simple enough right? I said ISPF-based. (You hit PF3 to get out. No, now you've split the screen, hit PF2 again. No, now we're back at the primary option menu, you must have hit PF4. No, I'll teach about split screens another day. AARGH! OK, that screen submits the scanning job. A job? well that's how MVS performs batch processing - JCL stands for Job Control Language - What? no, you can't write JCL in COBOL...)
By some miracle, we have actually fixed and tested a number of systems and gotten them back into production. Will they function after 31/12/99? I hope so, but I wouldn't bet the farm on it.
Our crew today consists of the two I mentioned, half of South Africa, India, Taiwan, and the old folk's home down the street. I think we have a couple of Canadians too. Don't get me wrong, most of these are good people who are working very hard on an operating system they know little about in a language they do not yet speak or understand clearly. My point is that considering the urgency of the project, and the uncertainty regarding how long it will take, perhaps it should have been staffed up differently. (But that would have been expensive!)
Six months into the project, I decided I had had enough so I resigned. I got a call at 4:00am from the owner of the company who was vacationing in Florida. Please don't go. The project is over if you go. We can't replace you.
Hmmm, here's financial opportunity if I ever saw one. I stayed. As a subcontractor.
It turns out that staffing was the least of our problems. Trying to get the application areas to let go of the code and let us fix it is darn near impossible. We are still screaming at them to let us work on mission critical stuff. "But its not available yet - we have too many outstanding urgent service requests from the users."
At the moment, we are working on an ancient, mission-critical COBOL financial application consisting of about 500,000 lines of code with absolutely no documentation. Anyone who ever knew anything about the application has long fled the government for jobs that pay a fair wage.
That's the batch side of it. The online side is written in ADF. The IMS database segments are hard-coded everywere, and to prevent having to restructure the database, they have been redefining parts of segments with overlays for the past 20 years. Those parts of the overlay not referenced by a particular program are defined therein as fillers, but may contain serious data. The batch cycle starts with a database extract to flat files, containing multiple record layouts which are referenced in the code something like:
01 INPUT-REC. 05 FILLER PIC X(86). 05 FIELD-I-WANT-TO-USE PIC X(2). 05 FILLER PIC X(327).
Of course the fillers contain all kinds of 6-digit dates and the Year 2000 "tool" in use cannot identify them.
Are we going to find them all? You be the judge. My original estimate for completion of this particular application was 31/10/98, then 31/01/99. I will no longer give a firm date. The fiscal year-end is 31/03/99 at which time it **will** fail. Joanne is right.
This post rambles a lot and for that I apologize. But this is therapy for me (twitch-twitch)- so read on or move on.
Perhaps I am involved in a particularly whacko project, but what concerns me is that in most other aspects, this shop is typical of all those I have worked in around the world. So I assume their Y2K project is also typical, and if that's true, we are in trouble.
I was involved in the start-up of another Y2K project for a large multi-national bank in London. I can't identify them but their initials are CITIBANK. I was supposed to be their main Assembler guy but I ended up setting up CICS 4.1 regions for them for their 6 European 'branches'.
I did some initial assessment of their assembler code and it was ugly. Very ugly. Unfortunately, I was forced to cut the project short and come home due to a death in the family so I didn't see the project through. In all fairness, perhaps they have since completed their project successfully. But I didn't come away with a good feeling. They were having a hell of a time finding skilled people. They had to go to Canada to get me!
You're going to love this one. Around 1990, I was involved in the development of a massive EDI client-server communications system in Seoul, South Korea. The server is an MVS mainframe, which deals which all of their suppliers and customers across an SNA/NPSI/X.25 connection to primitive PC apps at remote sites. The underlying messaging protocol is X.400.
The X.400 code was originally written in C for UNIX. We took that code and ported it to MVS using the Waterloo C compiler. Instead of using CICS or IMS, it was decided (not by me) to use a home-grown TP-monitor to handle all transaction processing and SNA communications. Now this platform was a monster, written entirely in Assembler/370 out of necessity, it was understood only by those who had developed it in the first place. You know where I'm going with this.
The company who developed the system is no longer in business. I went back to Seoul independently in 1995 with a friend and overhawled the system from top to bottom for performance reasons, but Y2K was not an issue for the client at the time.
The original UNIX code, as well as the transaction processor are not Y2K compliant and will cease to function at the end of this year. I don't know if the system is still in use, I presume it is. It is a proprietary system, there are no shrink-wrapped replacements they can put in its place. They don't have the source (I do.) Their economy is in a mess. I havn't heard from them.
They are the 3rd largest steel producer in the free world.
So, enough said. I don't know if its TEOTWAWKI, but based on my view of things, I think we are in for an economic event such as the world has never seen. Perhaps that will trigger Infomagic's spiral, perhaps it won't. Either way it won't be pretty.
I can't do the Milne thing. My children are grown and no-one will come. Fortunately, I live in an area of mild climate. I am stocking up on as much as I can to cover the extended family for as long as possible. I'll know how long when I can no longer buy supplies.
As for Pollyannas and Doomsayers? I can understand the motives of most Doomsayers - if you believe Y2K will be bad there is perhaps a moral obligation to warn others in as loud a voice as you can muster - whether or not you end up being mistaken. What motivates the Pollyannas? I have no idea. If you think Y2K is nonsense, why are you wasting so much energy proclaiming it? Ignore it and it will go away soon enough.
-- Flint (flintc@mindspring.com), January 22, 1999
Answers
Most of the report went over this layman's head, but one section did strike me as being very similar to Alan Greenspan's testimony before Congress (which I believe is available through links on Sen. Bennett's site) in which Greenspan recounted his days as a programmer and the fact that he (Greenspan) would have great difficulty in going back now and remediating code that *he* wrote in the 1970's because proper documentation no longer exists.
-- Puddintame (dit@dot.com), January 22, 1999.
--------------------------------------------------------------------------------
Thank you for this one, Flint. I gave up monitoring csy2k long ago.
-- No Spam Please (anon@ymous.com), January 22, 1999.
--------------------------------------------------------------------------------
Flint, why do your still have the source?
MoVe Immediate
-- MVI (vtoc@aol.com), January 22, 1999.
--------------------------------------------------------------------------------
MVI: its a repost from c.s.y2k gearhead (flint is mr. embed. sys.)
Flint: a good pull. This is what I see also. For instance managers that do not understand that when a senior programmer leaves, they're not just losing a good coder, they're losing part of the users manual to their software. All those little things in the code that someone somewhere used to know so well now have to be painstakingly relearned one at a time. It's comparable to someone coming out of a coma and having to learn how to talk again. Most managers I have dealt with have the technical inclination of a turnip seed.
-- a (a@a.a), January 22, 1999.
--------------------------------------------------------------------------------
Thanks Flint.
This guy reminds me of myself, I have also worked around the world, Uk, France, Germany, Saudia Arabia, East/middle/west coast of US - from what I have seen on those projects those companies are at a very real risk of being burnt toast next January. Things are better in the US and the UK, but not much better.
-- Andy (2000EOD@prodigy.net), January 22, 1999.
--------------------------------------------------------------------------------
If you got this far and your sides don't hurt from the laughter, and you aren't revamping your shopping by X2, doesn't understand teh tech background for Y2K. Flint, this is one of the worst horror stories I've had the pleasure/pain to read. this guy knows where some bodies are buried, and the owners don't even know they've died!
Chuck
-- Chuck, night driver (rienzoo@en.com), January 22, 1999.
--------------------------------------------------------------------------------
I have to agree with Chuck- I sat here bursting out laughing (while not almost crying internally) at some points in this post. Sure has the ring of truth to me.
-- Drew Parkhill/CBN News (y2k@cbn.org), January 23, 1999.
--------------------------------------------------------------------------------
Wow. I need to be checking to NG more frequently. This is the exact type of post I've been looking for!
Code on!
I'm a-codin' too...gotta get that woodstove installed soon...
-- Delete (del@dos.com), January 23, 1999.
--------------------------------------------------------------------------------
I especially liked Cory's response to Toast;
**You have pushed me another notch over to the dark side. I'm sorry but your uncertainty makes it that much certain.
Your uncertainty confirms much of my sense of this. There is a wall at December 31, 1999. On the other side of the wall, there be dragons...
Toast, I would like to rerun an expanded version of this article in a future WRP. This is the scariest one I've seen in a while.
cory hamasaki 345 Days, 8,283 Hours
more Y2K stuff at kiyoinc.com**
-- c (c@c.c), January 23, 1999.
--------------------------------------------------------------------------------
Contribute an answer to "Enterprise systems, real world (long, technical and scary)"
--------------------------------------------------------------------------------
timebomb2000@yourdon.com |
