Setting up, Managing & Monitoring Spark on Kubernetes
Earlier this year at Spark + AI Summit, we went over the best practices and pitfalls of running Apache Spark on Kubernetes. We’d like to expand on that and give you a comprehensive overview of how you can get started with Spark on k8s, optimize performance & costs, monitor your Spark applications, and the future of Spark on k8s!
Monday, September 21, 2020
How We Built A Serverless Spark Platform On Kubernetes - Video Tour Of Data Mechanics
In this video, we give you a product tour of our serverless Spark platform and its core features: connecting a Jupyter notebook, submitting apps programmatically, monitoring their logs and metrics, tracking their costs and performance over time.
Tuesday, September 8, 2020
Apache Spark Performance Benchmarks show Kubernetes has caught up with YARN
Apache Spark on Kubernetes is as performant as Spark on YARN, including during shuffle stages. This article presents the benchmark results and gives critical performance tips for Spark on Kubernetes.
Monday, July 6, 2020
We're building a better Spark UI
We started building a Spark UI and Spark History Server replacement called the Data Mechanics UI. It would work on top of any Spark platform, entirely free of charge.
Tuesday, June 23, 2020
Our Experience Going Through YCombinator
What is YCombinator like? What did we get out of it? The founders tell their story.
Wednesday, June 3, 2020
The Pros and Cons of Running Apache Spark on Kubernetes
Support for deploying Spark on top of Kubernetes (instead of Yarn, Mesos, Standalone) was only recently added. What are the main benefits and drawbacks? Should you get started?
Tuesday, May 26, 2020
Introducing Data Mechanics
We're proud to let you know about what we've been working on. Our mission, our vision, and our recipe for building the data platform of tomorrow.
Thursday, April 16, 2020