OT, re: SANs. Could this be a hotel in Las Vegas? The SANs?
Or... was it only a Mirage?
Enjoy, Frank Coluccio
------------
"A Bet Worth Taking"
By James E. Gaskin
September 7, 1999
INTERNETWEEK via NewsEdge Corporation :
Here's why the Mirage Resorts Inc. in Las Vegas
needs a storage area network: The busiest time of
the day for the $1.5 billion company is between 10
p.m. and 2 a.m., which means the staff has four
hours to back up more than 570 GB of data to tape
before the day's first shift starts at 6 a.m.
On top of its backup needs, the company is making
a push toward a Web-based paperless office, which
means managers need access to reports 24 x 7.
And, Mirage also has to accommodate the
requirements of Nevada Gaming Board auditors who
demand complete access to every transaction
during the most recent period under scrutiny. Tack
on that the company's Las Vegas hotels run more
than 5,000 desktops and servers, another 5,000
network printers, and track more than 6,000 slot
machines daily and you can understand why the
hotels needed a powerful, flexible storage system.
Glenn Bonner, CIO at Mirage Resorts, says all of
this network traffic generates 3.5 terabytes of online
tape storage a week-a data onslaught he says the
company couldn't possibly handle without a SAN.
The SAN offers Mirage Resorts something traditional
storage doesn't: It moves all the storage
components to a separate network. A dedicated
server controls all storage details for the storage
devices, and the storage network is attached to the
main computer network through one or more links.
High-performance SANs like the one Mirage Resorts
uses have multiple links between their client/server
network and the SAN, all controlled by
high-throughput switches. Bonner needs the fastest
link possible between disk storage and tape
libraries, and the high-speed SAN supports both
devices, keeping back-up traffic away from the
operational networks.
"There's a small window between the slowdown of
the casino transaction volume and the ramp up of
transactions for room service, starting about 6 a.m.,
" says Bonner. "We fit the start of the back-up
procedure and night auditing into our available
window only because the SAN lets us back up
nearly 600 GB without impacting real-time Web
transactions and our other systems.
"The largest volume comes from our hotel
systems-phones, keylocks, front desk operations
and the like," adds Bonner. "The next largest set of
transactions comes from the casinos, with slots
leading the way, closely followed by the casino
credit system."
The Mirage Resorts SAN supports all of these
transactions at three of Mirage Resorts' 3,000-plus
room hotels. All three are within a mile and a half of
one another. Each of the hotels is a world-class
facility already famous as a Las Vegas landmark.
The Mirage is the resort with the volcano erupting
out front every 15 minutes. Treasure Island has a
complete pirate ship outside. And, the Bellagio
boasts a world-class art gallery inside and
thousands of fountains in the lake outside. Other
"smaller" properties (each with about 2,000 rooms
and still among the world's largest hotels) include
Golden Nugget hotels in Las Vegas and Laughlin,
Nev., the Beau Rivage in Biloxi, Miss., and the Les
Jardine in Atlantic City, N.J. All these sites are
connected to the SAN via a T1 connection to the
corporate WAN.
Here's how the Mirage Resorts SAN works: Data
from hundreds of servers flows to one of two
StorageTek 9310 PowderHorn tape libraries. The
9310s are separated by a Fibre Channel link about
one and a half miles long between IS headquarters,
based in a separate building apart from the hotels,
and the computer room in the Bellagio.
Mirage Resorts uses two T1 links to provide Web
and Internet services, both to customers and
business partners. Its online reservations system,
which handles tickets to concerts and special
performances, ties directly to the hotel system.
Business-to-business Web activity includes group
reservations and links to an extranet bid
management portal so hotel purchasing agents can
buy perishable food like meat, fruit and vegetables.
All transactions flow into the hotel management
system, one of hundreds of homegrown applications
developed by Mirage Resorts.
One of the SAN's most important features is
backup, because it's the best way to keep massive,
streaming amounts of traffic from clogging a
network. Walt Hinton, chief strategist for
StorageTek, says that by building a separate
network for storage and backup, the production
network is free to support server-to-client and
server-to-server transactions. Separate networks let
Mirage shovel gigabytes per minute of data between
disk and tape on the SAN without slowing down the
transactions for the front desk operations or casino
on the production network.
Only The Best
Bonner assigned his best engineer to research
storage options three years ago, when the company
realized it would need a new system to
accommodate traffic increases at the
then-forthcoming Bellagio. Finding a method to
execute backup in the time allowed was critical, but
so was the stability of the vendor. Local service and
support was non-negotiable. Few businesses
generate revenue every minute of the day, but
casinos do. Taking thousands of slot machines
offline for backup was never an option.
StorageTek made the short list of vendors that had
the right equipment and local support. Bonner visited
CitiBank's nearby credit card clearinghouse.
CitiBank has eight 9310 automated cartridge
systems online, and StorageTek already had local
support in place. Bonner felt that StorageTek
provided the scalability and reliability he needed,
and had the company's 9310 installed in September
1997.
Planning for the Bellagio's opening in September
1998 pushed Bonner into purchasing the second
tape storage unit. The company had made capacity
projections based on the Mirage Resort, but the
Bellagio's success almost overwhelmed its
projections.
"The Bellagio, in the first month, did twice our
previous total transaction volume," says Bonner. The
volume came from the 3,000-plus rooms, sales of
show tickets and retail sales from the adjacent mall,
which includes Chanel, Giorgio Armani, Gucci,
Tiffany & Co. and others.
The main IT headquarters is in the Mirage Resorts
Design Center, which shares a building with the
company's construction arm. A second large data
center was added in the Bellagio, a mile and a half
away.
"We decided to try the Fibre Channel connection
because we didn't have room in the Design Center
for a second 9310 tape unit," Bonner explains.
Mirage Resorts has dark fibre between each of its
four Las Vegas properties, and runs all remote
computer rooms dark as well. A T1 links the
company's new Beau Rivage property in Biloxi,
Miss. "We use Web cameras to monitor anyone
coming in or out of the remote computer rooms,"
Bonner adds.
High-Speed Upgrade
A Sun 6500 with eight processors controls the SAN;
it was upgraded from a Sun 3000 when the second
tape library was added. The company runs a Sonet
OC-3 (155 Mbps) data connection between the
Design Center and the Bellagio that will be upgraded
to OC-12 (622 Mbps) before the crush of New Year's
business.
The SAN supports 21 operating systems, which
includes all the normal desktop, server and
mainframe systems, plus specialized systems for
doors, keylocks, slot machines, casino banking and
credit, and security, among others. Three Tandem
systems with a total of 30 processors are spread
across three of the properties; redundant 12-way
systems at the Bellagio and Treasure Island, and a
six-way system at Beau Rivage in Mississippi.
These handle the casino credit accounts, among
other functions, acting as a bank for guests.
There are three AS/400 systems, with a redundant
system for backup. Fifteen IBM RS/6000s, including
an eight-way R50 model, handle the slot machine
transactions. Windows NT servers abound, and
most are from Dell Computer. Five Dell Quad Xeon
servers running Windows NT Terminal Server
software support 500 Wyse WinTerm desktops.
Since Mirage Resorts has an excellent working
relationship with Dell, it also has been testing a new
Dell eight-way server to drive thin clients.
Bonner wants to move away from discrete RAID
arrays to a so-called "wall " of disks, planning to
save money with the replacements over time. The
Mirage Resorts SAN is controlled by a new
StorageTek Access Hub that uses a loop switch
technology. This technology lets any server on any
network loop reach any tape drive on any other loop.
The new Access Hub supports up to 32 Fibre
Channel Arbitrated Loop connections, all without
blocking traffic from one loop to the other. Each
Arbitrated Loop can support up to 126 devices at
speeds of 100 megabytes per second for a distance
of 10 kilometers. The new Access Hub eliminates
one of the problems with basic Arbitrated Loop. In
the past, when any one unit on a loop failed, it took
down the entire loop.
Dell's Fibre Channel RAID technology helps Bonner
stay ahead of the demand curve. For performance,
Bonner converted one of the earlier Clariion disk
units to a 600 GB cache for the tape libraries. Both
the Design Center data site and the Bellagio data
center have 1 TB Fibre Channel arrays for online
disk storage and tape library cache.
Bonner installed the second 9310 tape silo for
several important reasons. First, Mirage Resorts
was looking for a way to reduce paper consumption
and reliance on printed reports. The second 9310's
added capacity lets Bonner offer
information-previously printed as reports-in real time.
The quality of the information increased as well,
since online reports are always up-to-date. Using
Computer Output To Tape made the paper reduction
possible by printing reports in an easily viewable
online format and increased security since sensitive
information couldn't go out in the trash by accident.
Quick access to all information is necessary for the
Nevada Gaming Board auditors. In addition, the
extra "nearline" tape storage eliminates the need to
archive old tapes to an outside group. This saves
Mirage Resorts more than $250,000 per year.
Shared storage under centralized control also adds
peace of mind. Each of the six data centers beyond
the Design Center are controlled remotely, which
also keeps the 195 Mirage Resort IT employees
centralized. Between 40 and 100 servers of various
types are in each property, yet the SAN provides
online storage or backup for all sites and servers.
"About three-quarters of our systems are front-ended
by Sun systems," says StorageTek's Hinton.
To formalize that relationship, Sun and StorageTek
announced a partnership in mid-August that lets
Sun sell tape libraries from StorageTek, while
StorageTek will offer disk subsystems from Sun.
Bonner's network storage is all Clariion disk
subsystems marketed by StorageTek, but that
relationship may change since StorageTek's main
competitor, EMC, just bought Data General, parent
company of Clariion.
Of course, whatever happens with Data General,
StorageTek will be part of Bonner's network for quite
a while.
"We have high confidence in their products and
service," says Bonner. "With two tape silos, I can
add up to 400 terabytes of storage. There are
options to move up to higher-throughput storage if
necessary, all with StorageTek."
Which is why for Bonner and the IT staff at Mirage
Resorts, installing a SAN is as close as you can get
to a sure bet in Las Vegas. And it certainly has
been a bet worth taking.
James E. Gaskin is a freelance computer journalist
based in Mesquite, Texas. He can be reached at
james@gaskin.com.
---
SAN LAUNDRY LIST
Use this checklist to determine if your company is
ready for a storage area network:
The company is starting on its second terabyte of
disk storage. Typically, 50 percent of storage is
added after installation-build an infrastructure to
support storage growth.
The backup process interferes with production
processes. SANs separate the disk to tape traffic
from the rest of the corporate network.
The back-end needs more bandwidth between
servers and storage devices. Multiple paths between
servers and storage will speed processing,
especially for OLTP systems.
It makes financial sense to amortize new storage
costs over all servers and operating systems. SANs
support multiple servers concurrently, so every
server benefits from new storage.
Corporate budgeted for the project. StorageTek's
Hinton says a multiple-terabyte SAN with room to
grow starts at around $100,000.
Copyright c 1999 CMP Media Inc.
|
