Blog

Data + AI Summit Europe 2020 Highlights

Data + AI Summit 2020 Highlights: What’s new for the Apache Spark community? In this article we’ll go over the highlights of the conference, focusing on the new developments which were recently added to Apache Spark or are coming up in the coming months: Spark on Kubernetes, Koalas, Project Zen.

Tuesday, November 24, 2020

We’re releasing a free, cross-platform Spark UI and Spark History Server

Today we’re releasing a web-based Spark UI and Spark History Server which work on top of any Spark platform, whether it’s on-premise or in the cloud, over Kubernetes or YARN, with a commercial service or using open-source Apache Spark. This is our first step towards building Data Mechanics Delight - the new and improved Spark UI.

Monday, November 16, 2020

Spark on Kubernetes Made Easy: How Data Mechanics Improves On The Open-Source Version

How Is Data Mechanics different than running Spark on Kubernetes open-source? In this article, we explain how our platform extends and improves on Spark on Kubernetes to make it easy-to-use, flexible, and cost-effective. We'll go over our intuitive user interfaces, dynamic optimizations, and custom integrations

Tuesday, November 10, 2020

How to be successful with Apache Spark in 2021

Apache Spark is the leading technology for data engineering at scale. But making Spark easy-to-use, stable, and cost-efficient remains challenging. In this article, the AI & Data consulting firm Quantmetry and Data Mechanics team up to give you their best practices to ensure you're successful with Spark in 2021.

Monday, November 2, 2020

Spark and Docker: Your Spark development cycle just got 10x faster !

Native support for Docker is in fact one of the main reasons companies choose to deploy Spark on top of Kubernetes instead of YARN. In this article, we will illustrate the benefits of Docker for Apache Spark by going through the end-to-end development cycle used by many of our users at Data Mechanics.

Tuesday, October 13, 2020

Setting up, Managing & Monitoring Spark on Kubernetes

Earlier this year at Spark + AI Summit, we went over the best practices and pitfalls of running Apache Spark on Kubernetes. We’d like to expand on that and give you a comprehensive overview of how you can get started with Spark on k8s, optimize performance & costs, monitor your Spark applications, and the future of Spark on k8s!

Monday, September 21, 2020

How We Built A Serverless Spark Platform On Kubernetes - Video Tour Of Data Mechanics

In this video, we give you a product tour of our serverless Spark platform and its core features: connecting a Jupyter notebook, submitting apps programmatically, monitoring their logs and metrics, tracking their costs and performance over time.

Tuesday, September 8, 2020

Apache Spark Performance Benchmarks show Kubernetes has caught up with YARN

Apache Spark on Kubernetes is as performant as Spark on YARN, including during shuffle stages. This article presents the benchmark results and gives critical performance tips for Spark on Kubernetes.

Monday, July 6, 2020

Data Mechanics Delight - We're building a better Spark UI

Data Mechanics Delight is a Spark UI and Spark History Server replacement with new metrics and visualizations that will delight you. It works on top of any Spark platform and is entirely free.

Tuesday, June 23, 2020

Our Experience Going Through YCombinator

What is YCombinator like? What did we get out of it? The founders tell their story.

Wednesday, June 3, 2020

The Pros and Cons of Running Apache Spark on Kubernetes

Support for deploying Spark on top of Kubernetes (instead of Yarn, Mesos, Standalone) was only recently added. What are the main benefits and drawbacks? Should you get started?

Tuesday, May 26, 2020

Introducing Data Mechanics

We're proud to let you know about what we've been working on. Our mission, our vision, and our recipe for building the data platform of tomorrow.

Thursday, April 16, 2020

They trust us

Ready to get started?

🍪 We use cookies to optimize your user experience. By browsing our website, you agree to the use of cookies.
close
30