Apache Spark on Kubernetes is as performant as Spark on YARN, including during shuffle stages. This article presents the benchmark results and gives critical performance tips for Spark on Kubernetes.
Monday, July 6, 2020
We started building a Spark UI and Spark History Server replacement called the Data Mechanics UI. It would work on top of any Spark platform, entirely free of charge.
Tuesday, June 23, 2020
What is YCombinator like? What did we get out of it? The founders tell their story.
Wednesday, June 3, 2020
Support for deploying Spark on top of Kubernetes (instead of Yarn, Mesos, Standalone) was only recently added. What are the main benefits and drawbacks? Should you get started?
Tuesday, May 26, 2020
We're proud to let you know about what we've been working on. Our mission, our vision, and our recipe for building the data platform of tomorrow.
Thursday, April 16, 2020