Chinese Tech Giant Tencent Teams with Mellanox and IBM to Smash Big Data Analytics Record
November 10, 2016 John Biebelhausen
mellanox.com
Tencent Inc., Mellanox Technologies and IBM today announced new world record-breaking performance in select data analytics categories of the TeraSort Benchmark (sortbenchmark.org) using Mellanox 100GB Ethernet interconnect technology and OpenPOWER-based server technology.
Long considered to be the gold standard for measuring scalability and efficiency of general purpose distributed computing systems – or, many small servers that are interconnected and tuned to behave as one massive system, the GraySort and MinuteSort benchmarks are among the industry’s most widely cited benchmarks. GraySort, named after the pioneering computer scientist Jim Gray, has evolved over the years into a benchmark that measures how rapidly 100 Terabytes of data can be sorted. The MinuteSort focuses on sorting as much data as possible in one minute. The GraySort and MinuteSort benchmarks are each measured in two categories, Indy for speed and Daytona for general purpose performance.
Tencent has set four new records in the 2016 Sort Benchmark’s GraySort and MinuteSort categories improving last year’s best overall results by up to five times and achieving more than one Terabyte per second of sort performance. Additionally, per system results improved by up to 33 times per node.
To achieve these record-breaking results, Tencent deployed a cluster of 512 new and finely tuned OpenPOWER-based servers. That means they used 85 percent fewer than the 3,377 servers used by last year’s winner of the Sort benchmarks. Combining these new servers with Mellanox 100Gb Ethernet interconnect technology enabled Tencent to excel in this year’s contest. These servers and network infrastructure were collaboratively designed to meet the requirements of intense transactional and real-time data analytics workloads at scale, including those found in the Terasort Benchmark contest, and in clients’ data centers.
Achieving these world record breaking results required a deep collaboration with Tencent and IBM. Real-time-analytics and Big Data environments are extremely demanding and the network is critical in linking together the high performance IBM POWER based servers to crunch the massive amounts of data that Tencent generates and consumes. Tencent developed and tuned an optimized hardware/software platform to achieve more than one Terabyte per second of SORT performance. This truly remarkable result demonstrates that Mellanox’s 100Gb/s Ethernet technology including ConnectX®-4 network adapters, Spectrum™ Switches, and LinkX® cables deliver total infrastructure efficiency and improve application performance. This makes them ideal for Big Data applications as they have demonstrated the ability to keep the massive compute engines of the IBM POWER-based servers supplied with data while continuously processing. |
