Facebook Shifts Switch to 100G
SAN JOSE, Calif. — Facebook announced it is working on a 100 Gbit/second top-of-rack Ethernet switch for its next-generation data centers. The news is another example of how big Internet companies are designing their own systems, chomping at the heels of leading-edge computing and semiconductor technology.
The Wedge 100 is a 32 x 100G switch said to use Broadcom’s latest Tomahawk switch chip with a 3.2 Tbit/s aggregate maximum throughput. As with all its designs, Facebook will make the hardware open source for others to make and use. It is expected to run a variant of open source Linux-based software called FBOSS Facebook currently uses on a 40G switch.
In August, a Facebook optics expert described plans to drive 100G costs to $1/Gbyte using an 100G transceiver with single-mode fiber. The design relaxed distance requirements to 500 meters down from 2km and eased specs on operating temperature and product lifetime. It is based on a QSFP28 pluggable form factor which uses four 25Gbit/second lanes.
Facebook did not provide details on when it expects to deploy the Wedge 100 or how it will handle aggregating data from the 100G TOR switches.
Less than 18 months ago, the Web giant announced its work on a 40Gbit/s TOR Ethernet switch, the first generation Wedge. The design emerged after meetings with switch chip makers in late 2013.
Facebook re-purposed that design into an aggregation switch called 6-Pack. The systems are key components of a new data center fabric it announced a year ago.
In March, the company showed its 16 x 40G Wedge based on a Broadcom Trident II chip. It then repurposed the design as a line card and a dual-socket fabric card to create the 128 x 40GE 6-Pack aggregation switch, based on 10G channels that can be upgraded to 25G for the 100G interfaces.
At the time Facebook engineers said there were working on a 32 x 100GE switch and had plans to use 100 and 400G optical links.
“We now use thousands [of Wedge 40 TOR switches] in production and expect to continue using it for a quite a while,” a Facebook engineer said via email.
“We're still in the early phases with Wedge 100 before it will be ready for our next generation of data centers,” the engineer said. “Eventually, our goal is to use Wedge for all our top-of-rack switches throughout all of our data centers,” he added.
Next page: 50G channels coming, challenges described
Facebook released its 40G Wedge design as open source. (Graphic: Facebook)
Facebook’s design presumably provides lower cost and tighter optimization for its data center architecture than 100G switches using the same silicon now available from vendors such as Arista and Dell. Like other Web giants, Facebook designs its systems to fit into a warehouse-sized computer network complex.
The big data centers are running close to the limits of computer networking technology. A crop of 400G switches based on 50G components is still in the lab and a standard for 800G Ethernet was only recently inked
“The 16x25 Gbit/s flavor of 400G Ethernet doesn’t provide much value, so 50 Gbit/s signaling is the key to” next generation systems, said Bob Wheeler, a senior analyst with The Linley Group. “Several vendors are now sampling 56 Gbit/s PAM4 PHYs and optical modules should reach production in 2017,” Wheeler said.
Facebook announced its Wedge 100 work in a blog post. It provided some details on the current availability of the 40G switch as well as its challenges using it.
“An often-seen failure mode in switches occurs when the CPU is overloaded to the point where it falls behind on its control plane duties like sending routing protocol messages or programming the ASIC, which is when traffic outages can appear,” the blog said.
“Early on during our testing, we ran into several scenarios that showed just how important it is to protect the CPU and the control plane,” the blog said. “We now have configured hard limits on the amount of traffic that gets sent to CPU from the switch ASIC. Further, we prioritize control plane traffic to the switch in order to ensure that the control plane of the fabric is up as much as possible,” it added
http://www.eetimes.com/document.asp?doc_id=1328303&page_number=2 |
