ZeroDailyGet it on Google Play

LinkedIn Slashes Spark Shuffle Failures 97% with Apache Celeborn, Boosting Data Processing

Original headline: LinkedIn Cuts Daily Spark Shuffle Fetch Failures from 750 to 20 with Apache Celeborn — infoq.com

More in Programming and developer news

LinkedIn has successfully deployed Apache Celeborn, a new remote shuffle platform, to handle the massive data workloads across its Spark and Flink systems. The company runs over 300,000 Spark jobs daily, processing around 200 petabytes of data. Shuffling—the process of moving intermediate data between processing stages—was a major bottleneck, causing about 750 daily failures and slowing things down.

The new system separates data storage from the compute infrastructure using dedicated workers, which reduced daily shuffle failures from 750 to just 20. This also cut compute costs by 10% and improved application runtimes by 14%. The key takeaway is that by decoupling storage from computation, LinkedIn achieved major reliability and performance gains for its critical data pipelines, showcasing a significant infrastructure optimization.

Read the original at infoq.com

Satirical commentary on real tech news, summarised with AI. Read the original source for the facts. How ZeroDaily works

Open in the appGet it on Google Play

More stories

Add ZeroDaily on Google