Storewidth Intensity: The Dynamics of Storage and Bandwidth Growth
by Fred Moore
New data reveal that global bandwidth demand is growing at very significant rates, but not as fast as storage demand. The relative slowing of the demand for bandwidth results from the persistent problem of network latency-the minimal time it takes to transfer data over the network regardless of how much network bandwidth is available. An increasingly popular solution to this problem is network caching technology which stores frequently accessed data on enterprise and carrier networks while reducing the number of round-trips that data must make over the Internet and private networks by as much as 35%. At the same time that network caching decreases demand for bandwidth it increases demand for storage. Yet another factor responsible for the different demand dynamics of storage and bandwidth is the emergence of applications that are more storage-intensive than bandwidth-intensive.
What is the best way for IT managers to view and measure the relationships between bandwidth and storage? The answer is Storewidth Intensity. This new metric is a ratio describing the relationships between annual storage demand and annual bandwidth demand. Worldwide, Storewidth Intensity will increase from .98 in 2000 to 1.71 in 2004 as the annual growth rates for both grow significantly but also diverge. Storage demand will grow faster than bandwidth demand because all applications consume storage, but not all applications consume bandwidth at the same rate. As staggering as bandwidth demand has been and will continue to be, storage demand will be even greater if both remain abundant (Table 1).
Engines Of The Digital Era
Storage, bandwidth, and processing power are all becoming abundant, and with no immediate technological barriers in sight. They are the three key technologies of the digital era for creating, storing, retrieving, and moving information. The relationships among the three are significant, and each has an impact on the others. Computing technology reduces the time required to process information; storage technology minimizes the space necessary for containing and harnessing information; and bandwidth technology eliminates distances.
The price per unit for processing power is approaching zero.
Corollary 1 - The density of transistors doubles every 18-24 months.
Corollary 2 - The cost of processing power falls between 25-35% per year.
Corollary 3 - The performance of computers is improving at 30-35% per year.
The price per unit for storage is basically approaching zero.
Corollary 1 - Storage density increases between 60-100% every year.
Corollary 2 - The cost of storage hardware falls between 30-40% per year, but the total cost of managing storage is increasing and is up to 10 times greater than the cost of storage hardware.
Corollary 3 - The value of data is growing exponentially.
Bandwidth is on a pricing slope that will see the price per unit approach zero.
Corollary 1 - The cost of bandwidth falls between 20-30% per year.
Corollary 2 - Optical data streams transfer at 10Gbps, and electronic switches deliver data streams up to 2.5Gbps. In the near future, light becomes the most effective way to move data.
Corollary 3 - Bandwidth outside the computer (optical) is growing much faster than bandwidth inside the computer (electronic). The network is now faster than the computer, and the gap will increase.
From the above facts and figures we can conclude that the technologies driving the information age-its supply side-continue to demonstrate remarkable strength. But what about the demand side?
The Aggregate Supply And Demand Of Bandwidth And Storage
The 21st century began with both storage capacity and bandwidth capacity in abundant supply. The United States and Europe have an abundance of installed bandwidth capacity compared with bandwidth demand.
The installed capacity of bandwidth in the United States exceeded demand for bandwidth by a factor of 1.3 in 2000 and is expected to grow to a ratio of 2.3 to 1 in 2001 before the gap begins to shrink. The amount of unlit or dark fiber in the United States is projected to exceed demand for the foreseeable future because the rate of bandwidth consumption has slowed compared with the past few years, which were marked by explosive growth.
Bandwidth supply outpaces bandwidth demand in Europe even more than in the United States. Supply exceeded demand by a factor of 1.9 to 1 in 2000 and is expected to grow to 3.5 by 2001 before the gap begins to shrink. European bandwidth usage also has been less than originally predicted.
The supply of Trans-Atlantic bandwidth compared to demand is even more pronounced than that of Europe or the United States. Clearly, excess capacity has been installed to mitigate the higher costs of laying undersea fiber. Trans-Atlantic Internet demand has proven to be about one-third of what was originally anticipated further contributing to the lower overall demand. In 2001, supply of bandwidth will exceed trans-Atlantic demand by a factor of 8.6 to 1.
Why will the demand for bandwidth continue to fall short of expectations? The following section will attempt to explain this trend by looking at the factors driving demand for bandwidth and storage.
Demand Drivers For Bandwidth And Storage
To provide further insight into the current demand dynamics of bandwidth and storage, a random sample of 100 network administrators was surveyed by Protem Partners. The findings reveal the constraints on bandwidth expansion, the fact that existing bandwidth capacity is not fully utilized, and that a majority of network managers view latency as a bigger problem than network bandwidth.
Peak bandwidth demand.
Regarding the peak bandwidth demand as a percentage of installed bandwidth, the survey indicates that 86.7% of the companies in the sample do not reach the limit of installed bandwidth during peak usage periods.
- 64.3% use more than 50% of their installed bandwidth.
- 49% use more than 70% of their installed bandwidth.
- 13.3% use 100% of their network bandwidth capacity during peak usage.
Bandwidth expansion constraints.
Three major factors were cited as the primary constraint that the businesses surveyed faced in expanding bandwidth.
Constraint Factor Percentage Citing:
- The cost of the investment 53%
- Outside vendor availability 10%
- Lack of technical capabilities 8%
Latency.
Latency-or the total time it takes to get data from the sender to the receiver-was considered a bigger problem than the actual transfer rate or bandwidth of the network itself. When asked which is the bigger problem, the respondents said:
Latency 53%
Transfer rate 45%
Regarding latency, 44% indicated that it would become an even bigger problem in the future. The majority of businesses, 77.3%, indicated they plan to deal with the growing latency problem by adding redundant processing power and/or storage capacity:
- 40.9% will add additional server capacity, including server-attached storage.
- 36.4% will add additional network storage capacity.
Network latency looms as the primary performance challenge for global networks today. The number of hops (or switches) that an Internet packet takes to reach its destination is estimated to be between 17 and 20. The average latency delays per hop ranges from 250 milliseconds to more than 600 milliseconds, and it is growing. Adding text, video, or audio attachments to the packet further adds to latency delays. With heavy congestion and rerouting, traces show some messages can take more than 400 hops and several hours to arrive at their destination. The survey results indicate that the solution favored by most network administrators is to add redundant storage (with or without a server) to reduce the number of hops and improve service times.
Looking beyond business networks to the emerging (and potentially much bigger) market for networking all the world's households-the promise behind the laying of a significant portion of existing fiber-latency poses an even greater problem. The network can operate no faster than the speed of the slowest link on the data path. Cable and DSL are catching on but more slowly than expected. This well-documented "last mile" problem creates a tremendous opportunity for storage growth. According to a study conducted by the School of Information Management and Systems at UC Berkeley, individuals already produce a significant portion of the world's new information. It is estimated that 2,700 photographs are taken every second around the world, generating annually the digital equivalent of 800,000 terabytes. As photos and videos move to digital format, households will have to manage terabytes of data and struggle to do it effectively.
In addition to the latency problem, another factor produces different demand dynamics for storage and bandwidth. For most applications, bandwidth and storage chase each other in an upward spiral with each pushing demand for the other higher and higher. It is important to note, however, that some new applications are more storage-intensive than bandwidth-intensive. Two examples of such emerging storage consumption are eHealthcare and e-mail.
Generally lacking any robust storage management or measurement tools since its inception, e-mail was the first "killer application" for the Internet and now consumes multiple terabytes of disk storage and archival storage at many locations. E-mail messages are often kept "forever" with little focus on deleting data that is no longer valuable. A recent study by the Midrange Performance Group indicated that the average size of an e-mail message, including any attachments, has now exceeded 50 kilobytes. In the near future, voice-mail and video-mail will merge with e-mail, pushing the requirements of e-mail storage far beyond current levels.
The emergence of electronic medicine is supported by the growing awareness that advancements in medical knowledge and improved treatment cannot be sustained by paper-based methods. Recent studies suggest that 85 to 90 percent of all healthcare information is stored on paper or film. A typical X-ray consumes 12 megabytes of storage. If a hospital performs 200 X-rays per bed per year, a 500-bed hospital will generate 100,000 X-rays per year, resulting in 1.2 terabytes of storage. Backing up this data doubles the storage requirement. A discharged patient's X-rays may seldom if ever be accessed again, but a lifelong archive of the data would still be required.
With plenty of bandwidth and storage, not much consideration has been given to understanding the demand-creation aspects for each component. One thing is clear: The emergence of storage-intensive applications and the persistent latency problem will change the way networks are deployed and have long-lasting effects on both the storage and bandwidth markets.
Storewidth Intensity
George Gilder defines Storewidth as the conversion of abundant bandwidth and heterogeneous petabytes into accessible information. With exabytes (1018th) and soon zettabytes (1021st) and yottabytes (1024th) of data stored in digital format connected to hundreds of terabits per second of global bandwidth, it is important to keep track of the relationship between and the growth trajectories of storage and bandwidth.
The overall annual demand for disk storage systems, according to IDC, is expected to grow from 302,550 terabytes in 2000 to 2,934,412 terabytes in 2004, representing a compound annual growth rate of 76.5%. With the emergence of more storage-intensive applications, this annual growth projection may soon be revised upwards. The overall annual demand for bandwidth (U.S. and Europe) is expected to grow from 11.9 terabits per second in 2000 to 80.2 terabits per second by 2004. However, these projections indicate a yearly slowdown in annual growth rates between 2000 and 2004, ranging from 79.8% in 2000 to 44.7% in 2004, a compounded annual growth rate of 61.6%. Much of this slowdown is attributed to the latency problem, which decreases bandwidth demand while increasing storage demand.
IT managers should measure and track the new relationships between bandwidth and storage by creating their own Storewidth Intensity chart, plotting the projected growth of storage and bandwidth in their IT operations.
The bandwidth revolution will continue as the Internet further links the world together and builds the foundation to achieve the reality of the global database. Even more intense is the storage revolution that not only benefits symbiotically from the abundance of bandwidth but also grows independent of bandwidth availability. Time becomes the next prize in the fast-paced storage-networking evolution. With the Internet as the framework, the key challenge shifts to providing timely access to the reservoirs of corporate data and ultimately rich content and individual information stored digitally. Clearly, the journey to convert abundant bandwidth and heterogeneous petabytes into accessible information delivered instantaneously has just begun.
wwpi.com |
