Blog

Setting up, Managing & Monitoring Spark on Kubernetes

Earlier this year at Spark + AI Summit, we went over the best practices and pitfalls of running Apache Spark on Kubernetes. We’d like to expand on that and give you a comprehensive overview of how you can get started with Spark on k8s, optimize performance & costs, monitor your Spark applications, and the future of Spark on k8s!

Monday, September 21, 2020

How We Built A Serverless Spark Platform On Kubernetes - Video Tour Of Data Mechanics

In this video, we give you a product tour of our serverless Spark platform and its core features: connecting a Jupyter notebook, submitting apps programmatically, monitoring their logs and metrics, tracking their costs and performance over time.

Tuesday, September 8, 2020

Apache Spark Performance Benchmarks show Kubernetes has caught up with YARN

Apache Spark on Kubernetes is as performant as Spark on YARN, including during shuffle stages. This article presents the benchmark results and gives critical performance tips for Spark on Kubernetes.

Monday, July 6, 2020

We're building a better Spark UI

We started building a Spark UI and Spark History Server replacement called the Data Mechanics UI. It would work on top of any Spark platform, entirely free of charge.

Tuesday, June 23, 2020

Our Experience Going Through YCombinator

What is YCombinator like? What did we get out of it? The founders tell their story.

Wednesday, June 3, 2020

The Pros and Cons of Running Apache Spark on Kubernetes

Support for deploying Spark on top of Kubernetes (instead of Yarn, Mesos, Standalone) was only recently added. What are the main benefits and drawbacks? Should you get started?

Tuesday, May 26, 2020

Introducing Data Mechanics

We're proud to let you know about what we've been working on. Our mission, our vision, and our recipe for building the data platform of tomorrow.

Thursday, April 16, 2020

They trust us

Ready to get started?

🍪 We use cookies to optimize your user experience. By browsing our website, you agree to the use of cookies.
close
30