Filling in the Gaps
Cable MSOs battle labor and resource shortages,
network complexities as they rush to add new services
cedmagazine.com
By Roger Brown, Editorial Director and Jeff Baumgartner, Senior Editor
When it comes to deploying new services over a cable TV network, the lion?s share of the work falls on the shoulders of regional engineers who must do what it takes to make a company?s vision come to life. To get a feel for what life is like on the front lines, CED magazine Editorial Director Roger Brown and Senior Editor Jeff Baumgartner convened a telephone panel of four experts. On the call were: Mark Scott of Cox Communications; Jeff Stamm of Insight Communications; Jim Brown of Buckeye Cable; and John Santangelo of Charter Communications. What follows is an edited transcript.
CED: On a day-to-day basis, what are the burning things you?re working on?
Stamm: That?s kind of a mixed bag, because we?re working on putting together the master headend, working on fiber and upgrades. We?re also in the process of bringing in some additional systems to our district, including some [old] Marcus systems. We?re trying to get those folks operationally migrated into Insight. We?re also working on new service launches, and getting the equipment installed to do that.
Brown: Our biggest one this year is to get a digital tier launched. If all stays on plan, we?re targeting to get that up and running by fourth quarter of this year. Other things that we?ve got on the agenda include video-on-demand, and we?re looking at those product platforms currently. Certainly rebuilds, as with Jeff, it?s a priority for us to get those rebuilds completed. The Toledo system is being rebuilt right now [and] is scheduled to be complete the early part of next year. We?re about 50 percent complete right now. A second priority is the interconnects at two other headends, which will be combined with the Toledo master headend. Further out on the timeline is residential telephone.
Santangelo: Getting all the upgrades executed is paramount to us here, because in the next three years, this region will upgrade about 13,200 miles. Second to that, of course, is that we?ve been in a very aggressive acquisition mode. We?re dealing with a tremendous amount of integration and challenges with all of the new acquisitions. Certainly, the activation of certified two-way plant is a priority.
That surfaces a number of issues. Noise abatement is probably on the top of the list. There are two ways that we approach that?dealing with noise abatement as you go through the upgrade, and then post-upgrade and making sure it supports these new services. So, we spend a tremendous amount of time trialing different methods [for] return plant monitoring maintenance. We?re highly focused on a very aggressive deployment of digital services, as well as modems. We have a regional NOC (network operations center) here that?s brand new to us. We?re learning everyday about what a NOC needs to do.
--------------------------------------------------------------------------------
CED: Regarding the issues as they surround the launch of digital services and the challenges in getting your networks upgraded and ready to go, what have you learned along the way?
Stamm: Several of our systems already have a digital format with a telephone return path, and, as we activate those systems with two-way, we?re faced with the challenges of dealing with that return path on the digital DCT (manufactured by Motorola) boxes coming back. Some of our systems already have two-way @Home service. We?re just now launching the digital service in those markets, and that?s an advantage because a lot of noise compression problems and other things have already been taken care of. Like in other Insight systems in Rockford, Ill. and in Evansville, Ind. and Columbus, Ohio, we?re launching a video-on-demand and an interactive channel at the same time. So that puts more demand on the return path.
--------------------------------------------------------------------------------
CED: Are those VOD launches or trials?
Stamm: In Rockford, Evansville, and Columbus, they?re currently using DIVA. We?ll be rolling that out from this master headend in Kokomo throughout the month of May and June throughout various systems here in North central Indiana. The interactive service (Source Media and the Interactive Channel) is a really great product for us, because it has a tremendous amount of local content that the customer can get access to such as Web sites, cable mail and other things.
--------------------------------------------------------------------------------
CED: Is the return path portion of that the toughest nut to crack, or are there other issues that you have to work your way through as you?re going along?
Stamm: If you get the return path to where it?s working properly for Internet services, the digital box seems to be robust enough. My biggest concern is [when] we install digital set-tops in homes [without] Internet service. Every time you do that, you?re opening up the return path on those installations, which is one of the major causes of return path problems. That?s going to complicate the preventive maintenance on that return path by having more installs out there without high-pass filters on them.
Santangelo: If you?re going [with] national control [of those launches], there?s a whole set of issues. If you?re going local control, there?s complexity. And if you?re doing any national-to-local conversions, that?s very, very complex. We?re doing all of the above.
We?re [installing] about 4,000 digital set-tops per month. Training is a huge, huge issue, and it starts right at the headend with people [needing] to become more familiar with local control issues. You have to shore-up your training for your return path, which has been ignored for years. In-house contractors are contributing to solutions, and they can?t be overlooked with training that you need to provide. So it impacts every component of the operation. Don?t even count the upgrade associated issues when you look at the product deployments.
Stamm: We just went through a national controller to local back[-office] conversion in all of our systems in central Indiana. That was one of the biggest nightmares I?ve been through recently?to be able to get the talent that you need at the headend. And we?re putting together our digital product from many different angles, because we?re using GI equipment to pick up services and cherry-picking things from others using S-A, and trying to combine all of those. Out of this one back[-office], we?re supporting multiple channel lineups at various systems, because of the differences on must-carries. It gets extremely complicated to get that all put together in one package so it works properly in each market.
Santangelo: Every system has different layers of complexity. You have interim lineups, final lineups. You?re dealing with changing possibly HRC to standard frequency plans. You have multiple types of convertors, both one-way and two-way. The margin for error is huge. You miss a ton of stuff no matter how diligent you are. The more you try to become customer-friendly, the more complicated it becomes.
--------------------------------------------------------------------------------
CED: How did any of those nightmares manifest themselves, or were they transparent to the customers?
Santangelo: I?d like to say the latter, but I?m going to be honest. Some of them are transparent and some aren?t. First of all, it slows down your ability to deploy two-way digital in any given area that is upgraded. So it might have been upgraded and ready, but, because of the back issues, it delayed deployment. That had a customer impact, because people are waiting for this. We?ve had some problems with boxes where . . . they would shut down or various functions would be lost.
Brown: We launched cable modems almost a year ago now. We also launched WorldGate on our advanced analog platform about four months ago. Through both of those products, we?ve learned a lot of things related to return path issues. We?ve found that we had to tighten up all of our specs for return path qualifications. We?re confidently hoping that those are going to take away some of the issues [related to] the digital launch.
We?ve got some other issues, too. We?ve got a billing system integration. We?re also looking at 64- versus 256-QAM, how the plant will accommodate that. That?s going to be a big issue. With regard to digital launch, we?re going with local control. But the channel lineup is going to require a lot of local encoding and a lot of statistical multiplexing. We?re also integrating all of our cable platform product and monitoring systems into our NOC so we can monitor not only the telephony side of our business, but the cable side. We?re monitoring our optical nodes or power nodes, as well as some of our cable modem products.
--------------------------------------------------------------------------------
CED: For those of you who have either deployed cable modems or are planning to do so, how much does retail play into your plans? And to what degree are you trying to automate the whole deployment process?
Scott: We?re still involved today with our DOCSIS installation [for] the most part through retail. We are starting to take on retail partners and trialing and working on customer self-install. I?d say today that we?re not there yet; there are still some methods and procedures that are being worked out between ourselves and our corporate folks. A typical installation runs about an hour and a half on average.
Santangelo: That?s about what we?re finding, as well. On the retail side, we?ve had some deals with regional retail outlets since June of last year. However, it?s not really a huge market at this point in time. As for auto-provisioning, I agree with Mark. It?s kind of difficult to get there right away. We talked earlier about getting the noise abatement issues addressed properly so that it becomes more a part of the normal culture of everybody who?s working on the plant, from the headend down to the installer putting in a modem.
We have A-plus certified in-house technicians who do both the RF side and the modem installations. We also utilize new A-plus certified people who are contracted out and come in immediately after our RF technician. Another [option] is when they actually ride together. Ideally, you?d like to have one person do it all. I think I?m leaning toward having an RF and A-plus certified experienced person who handles the modems.
Brown: We?ve done a lot of the same types of things. We deployed DOCSIS about a year ago. We deployed a telco return product in areas that were not rebuilt yet. That deployment created some issues for us. We found a lot more problems with the telco-return product than we did the two-way.
We?ve seriously looked at self-provisioning, though we haven?t gone there just yet. We?re looking at products from a couple of vendors for that. We also deployed a help desk primarily for cable modems, but they?re also handing WorldGate calls and will handle both of our advanced service platform calls. We are hiring computer technicians who go out to the home to help resolve issues. They are apart from our normal service technician group. Our service tech or line tech group handles problems that are return related. The other ones handle in-home or computer related problems primarily.
Stamm: My concerns with the commercial roll out of the modem installs are that most of the two-way plants are stuffed with high-pass filters. You?re going to have a truck roll anyway when you go out and have to remove that and harden the installation so that you don?t mess up your return path. I?m not really sure logistically how that?s going to be deployed. It would be great to be able to have most of that work done by the customer, but I?m still not sure how you?re going to get by without rolling a truck.
--------------------------------------------------------------------------------
CED: The other unstated part of this issue is related to the wiring that?s actually inside the house. What kinds of things are you finding there?
Brown: We?re running into issues there, certainly. Most of the noise comes out of the home or drop part of the plant, which we all know. We?re using a window filter. We?re thinking about moving that window filter back to the splitter on the house and isolate all those ports on that splitter that are not related to the cable modem so that we can still try to keep the noise level at the home down somewhat.
Stamm: That?s exactly what we?re doing, but again it?s going to cost you a truck roll every time a person goes to a retail store and buys a modem.
Scott: What we?re experiencing is that 60 percent of all of [our installs] require some re-work of that drop network, [because] splitters are configured incorrectly, they have a bad piece of cable or bad connectors. When you talk about auto-provisioning, I think we?re some years away before we [can do that].
Stamm: You?re going to have the same issue with the digital product, too. [With the digital products we?ve deployed], a customer is able to pick the box up and self-install it with a telco-return. That has helped us eliminate truck rolls and get better penetration, but with the digital, two-way interactive stuff, we?re going to have the same issue with the high-pass filter. It?s going to increase truck rolls to get those installs done and the install hardened for the return.
Brown: We?ve run into some issues with customer concern about security on the cable modem network. One of the things we?ve considered doing is providing what I?ll call a ?software firewall? product to our cable modem customers. What have you guys experienced with DSL competition in your markets? The DSL competitor we have in this market happens to be Ameritech, and it?s touting a product that?s supposed to come out mid-year that?s 768 (kilobits per second) down, 128 kbps up.
Santangelo: Southwestern Bell certainly offers a DSL product here, and it?s kind of adversarial right now. [The security issue] forced us to do something on our own to deal with that negative kind of press. Bottom line, they [DSL] are no more secure than we are, but security has been a big deal here for us probably since early this year.
Scott: SBC in our area is rolling out a similar product, and SNET is doing 640 downstream. From a pure competitive nature, I think the cable modem technology is reliable and the DOCSIS product just speaks for itself. With full QoS initiatives, bandwidth availability and proper network management, cable modem technology is superior.
Brown: Are you increasing download speeds to meet the marketing side of the competition? That?s our big issue. Our marketing people are driving us to raise their ceiling of 768 downstream.
Scott: We?re meeting that today. With QoS, I don?t see that as a problem. We?re getting pushed there, because our current level of service is performing at that.
Brown: Are you putting a ceiling on the modem [throughput speeds]?
Scott: Yes, we are, to meet the basic residential demand. We have a different commercial product that?s available for the virtual private networks that have different rate levels available. The two are not really interfering.
Santangelo: We sell three packages: bronze, silver and gold. Bronze is 256 kbps downstream and 64 kbps upstream; the silver is 512 and 128; and the gold is 768 and 256. We?re finding that about 40 percent choose the silver package. On the residential side we?re finding more and more that we?re being forced to deal with networking in the home, and that has created some challenges.
Santangelo: When you?re 16 or 18 months into deployment, you?re still getting the high-profile users?people who are very familiar with the Internet and cable modems. For many of them, it?s not uncommon that they want to put in the hub in their house. They want to network between four to five devices in their house, and some of them are literally using them as a business. We?re not fully redundant, whether it?s through the electronics or the circuits or the paths. They?re calling us up, complaining about speed, for example, and they may have bought the 256 package. No matter how hard you try to avoid it, there are a lot of unnecessary truck rolls, but then, who else is going to roll them? We have not established a policy that clearly depicts here?s what you get, here?s what you can do and here?s what we can support. We thought we did, but we?re learning we didn?t. It has blown us away over the last 45 to 60 days, the number of people who are putting on multiple components or devices. It really has created some problems for us.
Stamm: We get hit with that a lot over at our Lafayette system with Purdue University. There are a lot of folks over there who want to do networking. And probably like everybody else, most of our data installers are not Windows NT network specialists, and they get into a whole host of problems trying to work with that customer to make it compatible.
Santangelo: It?s certainly a business that we want to be in, but it just caught us off-guard. We need to figure out how we provision that for the customer, and how to service them. We?re leaning toward having a third party handle those configurations. We?ve talked to a number of vendors, but we?re just not sure we can scale up in-house.
Brown: We?ve had some discussion on partnering with somebody in town that does high-end entertainment video/audio/data, and work with them to provide an in-home network.
Scott: Home networking is becoming a bigger and bigger issue, and I think we all need to look at multiple options for our customers. With Cox@Home, we do a lot of multiple IP addresses. Cox, as well as Charter and others, I think, are looking at opportunities that are within the smart house environment.
--------------------------------------------------------------------------------
CED: What are some of the overriding lessons you?ve learned over the last six to nine months?
Scott: I think one of the early lessons, particularly in the Long Island area where we?re approaching 80 percent of our plant upgraded to two-way/750, is to get the plant built and two-way activated and filtered and cleaned up before you launch anything. That?s the first critical step. From there, you can build with success. With having multiple, two-way services running in many areas with telephone, data and even digital having its own return path with impulse pay-per-view, it?s critical that you get that first part done right.
Santangelo: I would most certainly concur. Certification of those upgraded nodes is paramount. The trick is that you have to make sure that you have enough trained people to continue to maintain it. The fundamental stuff, such as normal leakage monitoring, becomes crucial. Once you have your headend issues solved, digital is fantastic. Where you learn a lot is just making sure you?re well prepared across all disciplines in the business. Your ops people have to be fully acclimated to what?s going on and how they respond to a customer. If you plan all of those elements in concert and everybody?s communicating, deployments have gone very smoothly.
It?s extremely hard. We?re all challenged with a lack of resources, you can?t find enough qualified people, and you can?t train quickly enough. So, where we had strength in numbers and skill, we were very successful. Where we didn?t, and we?re taking on the integration issues and new acquisitions, those have been a struggle.
--------------------------------------------------------------------------------
CED: If the talent pool of these skilled individuals is diminishing, what can you do right now to help offset that? What options do you have?
Santangelo: Compensation usually comes up.
Brown: Resources are a huge issue and it?s been really difficult to grow staff. It is not easy to hire contract resources for some of this stuff, which creates some complications in launching product. Sometimes the desire to get new services out there is bigger than the ability to get them there. Those are the two biggest issues?training and resources.
Scott: I agree, resources and training are critical to this. The other part of this is the commitment to the resources that it takes to run this as a communications business. In fact, in the New England area, we have 24-by-seven coverage on our plant maintenance.
Santangelo: There are two things I heard there. First off, when we?re talking about getting these products to market so quickly, the engineering element of the business doesn?t stand up and say, wait a minute, you can?t do this right now until these other things are in place. I think more courage needs to be displayed by engineers to say that to the marketing folks. Cash flow is king, I understand, but you all know the impact that upgrades have on our customers, so it doesn?t make sense to throw on top of it a poorly organized deployment. Secondly, you really have to support these people. This industry historically has had technicians try to be everything. Certainly compensation is an issue, but we?ve never afforded them enough time to do the kinds of things that we really should have been doing.
Now, we?ve had to make that commitment. You need to certify the drop. That impacts quota. That impacts a whole bunch of things, including increased resources and contract labor. But we have to make that commitment to be successful.
Brown: I do not push deployments back as diligently as I should. We have run into issues, and pushing product to the street has created some launch problems and some issues that our customers are seeing that they probably shouldn?t see.
Scott: One of the things that we found critical is making sure you get the right tools to the people doing the job through back office support systems and network monitoring performance systems. The people issue is paramount?making sure you have people trained and making sure they have the resources to do the job.
--------------------------------------------------------------------------------
CED: Cable networks typically have never really been monitored. Secondly, networks historically haven?t been backed up with stand-by power. In your networks, has that changed?
Stamm: I think it?s almost happening by demand. In [one] network here we?re using centralized 90-volt power. We?re also [preparing] for generators in each location so that we can roll into local telephony service and not have to rely on commercial power. Currently, we?re using batteries at those locations until we roll out residential telephony.
Scott: We are deploying what we call network reliability units at a rate of 10 per day, trying to [provision] our two-way areas with backup capability. We do currently [achieve] 99.98 percent reliability. But in order to maintain it, you?ve got to have all of these things we?re talking [about] in order to meet those kinds of numbers. We?re trying to combine the frequency levels and the different service levels and looking at troubles that may occur in combining that information into an integrated report so that the right person is dispatched at the right time. Putting those pieces together into a common information center is probably the most difficult part of supporting three services.
Brown: We?ve deployed Cheetah for our status monitoring for our optical power net, but through our CLEC and our cable side of the world we?ve tried to consolidate them all onto one platform.
Scott: When we talk about the reliability and the competition, the clear winner will be [based on the] service levels that are provided. I believe that if this industry continues with its efforts, that we will win that game.
|
