For those who aren't satisfied with "what" but need to know "why" and "how". From Energis' website:
<<<
High capacity SDH networks for new operators
Using the national electricity distribution network.
Alistair M Henderson, Energis Communications, Carmelite House, 50 Victoria Embankment, London, UK.
Abstract
This paper describes the stages of development of Energis as a national telecom operator, based on the infrastructure of the UK electricity distribution network. The development over the first 4 years is covered, and future strategies considered.
The strategic impacts of SDH.
SDH equipment became available from an increasing range of suppliers during the early 90's, promising increased reliability, greater flexibility, reduced cost, and improved management systems. Initially, many operators throughout Europe were reluctant to deploy SDH, as the established PDH infrastructure had been a heavy investment, and was providing a perfectly adequate service. New PDH products were still being developed by many manufacturers offering sophisticated features, and ATM was "only a couple of years away anyhow".
Within the UK, the major operators saw the opportunity and began the lengthy process of study, and the writing of detailed requirement specifications. Throughout Europe, operators unveiled plans to use SDH in critical parts of their networks, largely to complement or integrate existing PDH. Some worked from the core out, with 4/4 cross connects for fast restoration, and high capacity STM16 bit haulers to replace 565Mbit line systems. Others concentrated on the access networks, with STM1 ADMs, rings and 4/1 cross connects.
At about the same time within the UK, the deregulation of the Telecoms industry was creating exciting opportunities for new entrants. Cable TV companies, (mostly subsidiaries of the USA RBOC's) commenced the deployment of dense regional networks in franchise areas. These new operators were licensed to carry both entertainment and telephony services within the area, and focused on the residential user.
It was also felt that there were opportunities for new national operators to compete with the established duopoly of BT and Mercury. The availability of the new technology SDH was seen as a key factor, offering the opportunity to construct a flexible highly reliable national network for the business community.
New technology was however not enough. The greatest problem in building a national network, is obtaining the wayleaves, and laying the fibre. During 1992/93, operators based on canals, railway lines, electricity pylons, water pipes and many other ingenious solutions were proposed, and business cases prepared.
The development of Energis.
One such proposal for a new operator was set up by the National electricity distribution company (NGG). At the time, the NGG was owned jointly by the regional electricity distribution companies (RECs), and carried power between the generators and the RECs. The NGG has since been floated as an independent PLC. Initially named Telecom Electric, the licence to operate in the UK was first applied for in May 1992.
The proposal was exciting:
The NGG network offered a fast and secure way of building a fibre core across the UK. Nearly 7000km of pylon routes mesh the UK, covering every major centre of population and industry. The technology was ready, and would allow the construction of the first 100% SDH national network. The National Grid had the financial resource and skilled staff (5000 employees) to make it happen.
Despite such advantages, projects can sometimes degenerate into endless planning loops. A major contract awarded by the BBC at the beginning however ensured that this project was for real. The contract involved a national network taking 2.5Gbit/sec SDH onto the customer site to provide video and radio contribution and distribution capabilities. The time scales were tight, and the demands of broadcast video technically exacting.
The first steps
The design and build of a new national telecoms network, by a company with no previous history as an operator, raises critical skill resource problems. The BBC contract also had critical time constraints, resulting in the need to compress activities into about 18 months. Substantial scarce skilled resources were required, firstly in network planning, then in build, installation, network integration, and network maintenance. Once each phase is complete however, the skills are no longer required in the initial volumes. A major problem for any new operator is the skill mix change in the early years, as described below.
Stage 1 The best SDH planners and designers are needed to ensure optimum network design, using the best products in the market. There is no time for extensive study. Stage 2 The building begins. A large and experienced installation planning team, logistics managers, and an even larger field force must be marshalled. They must be experienced with SDH. There is no time for training. Stage 3 The network must be fully integrated and tested, not as a series of individual products, but as a network. This skill is even scarcer to find! Stage 4 When the network is built and operational, a small core team of designers, planners, logistics and integration remains for core network growth. Stage 5 Operations, maintenance, and customer connection skills become the priority, as the new company fill the network with customers. Stage 6 Product marketing and new services design skills are required to differentiate the customer offering and raise the margins. Stage 7 Access network development skills are required to increase network density, and reduce dependence on other operators.
The Energis team had to rise rapidly to nearly 1500 skilled resources for 12 months, then fall back to a maintainable size, coupled with a skill mix change! Clearly this could only be done with assistance from someone in the business, and no existing UK (or European) operator was willing to make that offer.
Energis began the urgent search for a supplier partner, capable of providing access to the significant range and depth of SDH and management skills. Such a partner had to be able to work as member of a fully integrated team together with Energis staff, initially as leader, subsequently giving way to the permanent team as it built up. A further requirement was that the team had to provide access to a full range of SDH equipment, taking the best from the market as required by Energis.
Selecting a supplier
Selecting a network development partner did not necessarily mean we had selected a product supplier! Energis had to balance the need for competitive product supply with the problems of multiple source equipment integration. The chosen strategy initially was:
Maintain competition by ensuring more than one supplier is always able to meet requirements. Minimise the complications by limiting the number of supplier interfaces. Use the network development partner. Ensure supplier has sufficient business prospect to make it worthwhile. Ensure relationships once begun are long term to maintain supplier interest in current job!
With SDH the problem is in some ways simplified, and in others greatly complicated. Of course SDH is designed to allow product interoperability, and indeed as long as the more esoteric functions of SDH are not required, most products do in practice inter work well at the traffic level. The problem comes when you try to pull it all together, and provide a common management system.
To meet the target of enabling competition, it was decided that Energis must maintain the ability to buy from two suppliers. We could split the network by geography, contract phase, product type, or by service. If any supplier had been able to offer a fully integrated SDH system managed from a single common platform system, we might have tried a Geographic split of suppliers. As we could find nobody offering this, the Geographic split meant doubling the trouble, for no perceptible advantage.
Supply by contract phase is all very well for fully interchangeable product, but SDH is not at that stage. (Will it ever be?). This does not meet the requirement of maintaining suppler opportunity either.
Supply by product type was eventually chosen, as it allows for two major supplier relationships, with both suppliers having a strong position in a well defined part of the network technology. It was essential though to maintain the competitive element by ensuring that both were able to supply in the others area of strength if we were being let down.
Nortel was chosen to supply PKI STM-16 line systems, and DSC 4/4 cross connects. Essentially, Nortel was to build and integrate the VC4 level of the network. GPT was chosen to supply the ADM155/ADM622 products, together with access multiplexers, effectively supplying Energis with the VC12 and below network.
The supplier strategy is fundamental, as it leads naturally to constraints in the future on network operations.. By splitting the product suppliers into the SDH levels of VC4 and VC12, care must be taken to ensure that customer end to end path management over the VC4 and VC12 layers is maintained.
Planning the network
Having chosen partners and suppliers, the real task began. A core network of fibre was planned over the NGG network, taking us to the initial target population centres. An ambitious plan covering 4000 km of the NGG 7000 km power network was agreed, taking us within 8km of over 70% of the population of England and Wales. Access points were also planned for the 20 largest population centres in the first stage.
For initial construction, time was critical in order to maximise return on investment. The options available at the time included OPGW, where the fibre is buried inside a replacement earth wire, and 24 fibre wrap, where the fibre is wrapped around the existing earth wire. The NGG developed techniques for delivering both solutions on a live network, however wrapping techniques offered the fastest route to market. A wrapped network was planned for the majority of locations, with some OPGW, mainly in places where the earth wire was already due for replacement. Within London, extensive use was made of London underground tunnels to plan the routes.
The basic transmission interconnection between sites was planned at STM16 across the entire network. The Optical reach between sites is of vital importance to a new operator, as premises for repeaters are expensive, and have to be minimised. To get the reach necessary, a 28dB optical budget was essential. Indeed if Optical amplification had been more readily available in 1993, it would certainly have eased the planning task greatly.
Network resilience can be significantly improved with an SDH network. A number of options are available, each offering various advantages and penalties. For the high capacity high integrity services such as the BBC broadcast video service, two diverse routes were planned across the network, each taking a physically separate route from source to destination. Path protection mechanisms ensure the first level of resilience. The penalty here is cost, which cannot always be justified for all service types.
To enable shared protection, major node flexibility points are required. The Energis network was planned with a series of 4/4 cross connects, allowing VC4 service provisioning and restoration. A number of network configurations were planned to cover various time of day requirements, and disaster fallback scenarios. Current management capabilities restrict the speed at which such plans can be implemented in practice, so this remains a secondary protection mechanism for the near future.
One interesting feature of television broadcast is the unidirectional nature of traffic. SDH is entirely appropriate for carrying unidirectional traffic, however care must be taken with any bi-directional path level features! For example, an STM-1 terminating in Bristol may carry an incoming VC4 from London, with the opposite direction being routed to Glasgow, and so on, forming a complex thread of interlinking paths. You do not want far end receive fail (FERF) alarms in Glasgow, alerting a failure on the link from London to Bristol! As new path features are provided in SDH, recognition of the special nature of unidirectional traffic must be maintained.
The access network design was based on establishing equipment rooms in the major population centres, each linked back to a core network equipment housing (EAM) at the nearest point on the national grid network. These sites (commonly called POP's or points of presence) were called SNAPS (Synchronous network access points) in Energis, to emphasise the 100% SDH nature of the network. Each was linked to the main network with STM16 capacity, and equipped with ADM155, and access mux banks.
In practice, there are a number of parallel networks to be planned and built. The traffic network was one of the simplest! Others are:
The sync network.
The DCN network.
The Management system network.
The switch signalling network.
The environmental alarm network.
Each network has its individual peculiarities, however each interrelates with the other in a complex manner. Care must be taken to avoid nepotism, e.g. routing of alarm traffic over paths reporting the alarm etc.
The sync network gives particular planning problems. For any given network failure, restoration of traffic may be trivial. The sync trail restoration however will require careful pre planning and implementation, perhaps requiring sync source reversal down an entire chain of line systems.
One of the "features" of SDH is the vast amount of information available. It sometimes seems that each network element has been designed to tell you everything it ever knew (except what you want to know) at the slightest excuse - and they also seem to do it in crowds. This requires a substantial data communication network, built for resilience and capacity. Alarm floods and capacity choke points need to be carefully calculated.
The planning of the management system can be the poor relation. In practice the integrity of the entire network is dependant on the ability of operators to rapidly pinpoint failures and take corrective action. Great care was taken in the planning of the Energis network management system, allowing each of the features of SDH to be used to maximum effect. The substantial complications of managing mixed SDH/PDH networks were avoided.
Build and test
Building of the network commenced before the design was compete. The truth is, the design was not complete until some time after it was all built and working! In practice, parallel design and build is becoming more common as the time scale pressures of our competitive environment increase.
A major aspect of the build programme was the fibre wrapping. This was carried out for us by the National Grid contracts division. Fibre digs supplemented the wrapping, being carried out in all conditions, often in remote locations throughout the UK. The fibre testing produced a mass of data for analyses, indicating fibre condition. A point to watch is to ensure all contractors use the same OTDR equipment, to enable analyses and comparison.
For a new operator, premises are a big issue. Nortel provided pre assembled units called EAM's (Equipment accommodation modules) which could be built and tested at the factory before being shipped to site. Each EAM was positioned on pre-prepared sites, and hooked up to the fibre. This method enabled very rapid build and installation.
Unfortunately, build never occurs as planned. Unforeseen difficulties delay sections, whilst others prove easy. With a virtual workforce of 1500 people, working to very tight time scales, logistical control was a major problem. It was vital that the NGC, Nortel, Energis and other contractors worked as a seamless team, with a common strong point of control. (Our war room!). As sections were linked, network integration could begin.
The integration process for the network involved a series of steps:
Fibre test. (loss and OTDR)
EAM installation test. (Power, environment)
Section tests. A series of tests carried out between each Network Element and its neighbour.
Span tests. A series of tests between network node points.
Ring tests. Full ring testing of each STM16 subnetwok.
Inter-ring tests.
Network wide feature testing.
War Gaming!
Each test step logically built on the proceeding one, gradually bringing in testing of DCN, sync and environmental alarms at the appropriate stage. At one stage over 5000 separate exceptions and bugs of varying seriousness were being tracked. Many were exciting, requiring much lateral thinking to solve. Often problems in one part of the network caused strange effects in totally unrelated parts. One thing became clear - the various alarms and events currently defined in SDH are by no means comprehensive!
Having been through the process, there was much scope for making SDH roll out more efficient in the future. Analyses of failures picked up at each test stage indicates that much testing effort detected little or no failures, and a more elementary test sequence has been developed as we trend towards "plug and play". Much can be learnt from this process in terms of next generation equipment requirements!
Operations
The final stage of build and test was war gaming. This involved teams of field staff, complete with a collection of failed cards from integration test, attempting to beat the network operators! Of course, such fun can rarely be had with a live network. A serious competition developed, as more obscure faults were simulated, with multiple fault scenarios, and a final climax involving everything the field force could throw at them.. After going through that, the operators were somewhat disappointed when the network went live - and nothing much happened. As with all networks, the vast majority of faults and problems are caused by field activity, and finger trouble. Left alone, the network works with failures a rarity.
In October 1994, the Energis SDH network went live, initially for basic telephony traffic, and in January 1995 for broadcast video.
Subsequent developments.
The Energis network has evolved rapidly, with growth rates of up to 20% per month at one point. There are currently 8 DMS100 switches operational, with 3 further on the way. Traffic has risen from 0 to nearly 4 million minutes per day, and is continuing to rise rapidly as more customers take up the service. Applications running on the network include VPN services, Frame relay, ATM, leased line, broadcast video etc. One interesting feature of the traffic profile is the high degree of traffic during the evening and weekend, due to the capture of over 50% of the UK internet inbound business! All this traffic has created an ever increasing demand for bandwidth, and a high degree of build churn. SDH with centralised management has proved its worth many times over as network growth has progressed.
During this period, the ranges of SDH equipment have evolved rapidly. Some features will be more critical to Energis than others. Key drivers will as always be, will it cost less to buy or will it cost less to run. or will it bring in additional revenue.
Substantial savings have already been made over the costs of running an equivalent PDH network, but much remains to be done in the field of network management. For example, currently, the routing of a private circuit involves data entry into numerous different systems, and substantial time delay. The interlinking of different suppliers management systems will be an important step in reducing operational costs. The next step will be in convincing customers to manage their own networks, (and charging them for the feature).
SDH developments of interest include optical amplifiers, WDM, STM64 systems, optical switches, STM16 ADM, and a VC12 flexibility layer.
Clearly optical amplifiers are of great interest to new operators. Each equipment accommodation module is an expensive investment, and repeaters in a thin network contribute little. Optical amplification should allow greater reach between flexibility and SNAP points, obviating the need for many current EAM sites. (Which can then be re-deployed to expand the network in other areas).
Capacity will always grow. Many will remember forecasts which predicted a total of eleven 565Mbit line systems as the total UK market! (Another operators forecast). The Energis network, with 24 fibres capable of carrying STM16 should have no problems for a long time! In practice however, there seems no limit for the UK appetite for raw bandwidth! Energis will require higher capacity systems as soon as they become available. The debate between WDM and STM64 is intense, with trials of both in advanced planning.
Network availability and restoration are key issues. The greatest source of path failures has proven to be the fibre, and not the equipment. (By a substantial margin). This applies for dig, wrap and OPGW. As capacities on a single fibre increase, and consequences of failure become intolerable, it becomes clear that rings are insufficient for the core network. A minimum of three diverse routes are required between any two places. Consequently, Energis is in the process of lighting third routes across the network. SDH is fundamentally designed on the concept of rings, and this has caused complex network design issues to arise.
In practice however, as failures occur mostly at the fibre level, it may be more cost effective and efficient, to carry out restoration at the optical layer of the network, rather than at the SDH layer in future. Clearly optical switches offer one alternative. No suitable high capacity high reliability device has yet been identified for this purpose however. An alternative, which may be preferable, would be the use of WDM for optical layer traffic restoration. Particularly on routes where optical amplification has been deployed between network restoration nodes, each lambda could be fed to a number of route alternatives if carefully planned. Fast traffic recovery would take place at the SDH layer, with WDM optical layer recovery increasing resilience again at a slightly slower pace.
Such a scheme would be more easily managed in a network which implemented a consistent interface between the SDH and optical layers, i.e. all at STM16 2.5Gbit/sec. The deployment of STM64 in such a network scenario could be a complicating factor better avoided.
STM16 ADM systems are another design factor. In practice, the bulk of traffic on a given route transits directly between flexibility nodes composed of 4/4 cross connects. Only limited traffic needs to drop off at intermediate SNAP sites. This implies that the majority of the market may remain as low cost, simple bulk bit haulers between nodes. That said, there is a smaller, but still significant requirement for STM16 ADM at SNAP sites, allowing protection, flexibility and increased capacity.
A further trend emerging from the Energis network is the degree of network interconnect between operators. As the UK operators SDH networks expand, it is becoming increasingly common for circuits to transit a number of operators. This has only been possible with the definition and agreement between UK operators of a common interconnect recommendation. The Public Network Operators transport interconnect group (PNO-TIG) is a working group managed by the public network operators group on behalf of the Network Interfaces Co-ordination committee. (NICC). This group has published recommendations for the technical, operations, and maintenance aspects of interconnecting VC12, VC3 and VC4 over STM-1/4/16 optical links between.
A series of recommendations have been published giving guidelines for the technical issues of interconnect, the practicalities of shared link installation and commissioning, and the process for joint operations and maintenance. Copies of these recommendations are available on request to the author.
One interesting problem to arise from the studies within PNO-TIG is the utilisation of path identification. Energis already delivers SDH directly to customers, for connection to SDH interfaces on their equipment. Such customers will often have multiple operators connected to their premises, and circuits may transit a number of different operators in the UK, and other countries. If path identification codes are to be implemented, who will administer the allocation of the unique codes? Customers will clearly own the codes for their own equipment, and will demand SDH number portability between operators. Operators will use identification codes which must be unique nationally and internationally, and must allow for route switching.
Perhaps the future is a single network in the UK, owned and managed by different operators, but all providing a common flexible platform for service providers. If that can be achieved, SDH will have more than justified its existence.
For Energis, it now seems inconceivable that anything other than SDH could be used to provide a service platform. The most particular of customers seem to be notice the improvement. Broadcast video engineers are not easily satisfied, and the rising tide of Internet service providers are aware of every bit error. As customer awareness of the advantages of SDH becomes more universal, and the volume of special and directly connected services increases, it will become increasingly difficult for operators to provide service on any medium other than SDH.>>> |
