Cost-Effective Weather Analytics At Scale with Cloud-Native Apache Spark
Customer Story: Weather2020 is a predictive weather analytics company. In 3 weeks, their data engineering team built Apache Spark pipelines ingesting terabytes of weather data to power their core product. Data Mechanics performance optimizations and pricing model lowered their costs by 60% compared to Databricks, the main alternative they considered.
Wednesday, January 13, 2021
Data + AI Summit Europe 2020 Highlights
Data + AI Summit 2020 Highlights: What’s new for the Apache Spark community? In this article we’ll go over the highlights of the conference, focusing on the new developments which were recently added to Apache Spark or are coming up in the coming months: Spark on Kubernetes, Koalas, Project Zen.
Tuesday, November 24, 2020
Released: Free Cross-platform Spark UI & Spark History Server
Today we’re releasing a web-based Spark UI and Spark History Server which work on top of any Spark platform, whether it’s on-premise or in the cloud, over Kubernetes or YARN, with a commercial service or using open-source Apache Spark. This is our first step towards building Data Mechanics Delight - the new and improved Spark UI.
Monday, November 16, 2020
Spark on Kubernetes Made Easy: How Data Mechanics Improves On The Open-Source Version
How Is Data Mechanics different than running Spark on Kubernetes open-source? In this article, we explain how our platform extends and improves on Spark on Kubernetes to make it easy-to-use, flexible, and cost-effective. We'll go over our intuitive user interfaces, dynamic optimizations, and custom integrations
Tuesday, November 10, 2020
How to be successful with Apache Spark in 2021
Apache Spark is the leading technology for data engineering at scale. But making Spark easy-to-use, stable, and cost-efficient remains challenging. In this article, the AI & Data consulting firm Quantmetry and Data Mechanics team up to give you their best practices to ensure you're successful with Spark in 2021.
Monday, November 2, 2020
Spark and Docker: Your Spark development cycle just got 10x faster !
Native support for Docker is in fact one of the main reasons companies choose to deploy Spark on top of Kubernetes instead of YARN. In this article, we will illustrate the benefits of Docker for Apache Spark by going through the end-to-end development cycle used by many of our users at Data Mechanics.
Tuesday, October 13, 2020
Setting up, Managing & Monitoring Spark on Kubernetes
Earlier this year at Spark + AI Summit, we went over the best practices and pitfalls of running Apache Spark on Kubernetes. We’d like to expand on that and give you a comprehensive overview of how you can get started with Spark on k8s, optimize performance & costs, monitor your Spark applications, and the future of Spark on k8s!
Monday, September 21, 2020
How We Built A Serverless Spark Platform On Kubernetes - Video Tour Of Data Mechanics
In this video, we give you a product tour of our serverless Spark platform and its core features: connecting a Jupyter notebook, submitting apps programmatically, monitoring their logs and metrics, tracking their costs and performance over time.
Tuesday, September 8, 2020
Apache Spark Performance Benchmarks show Kubernetes has caught up with YARN
Apache Spark on Kubernetes is as performant as Spark on YARN, including during shuffle stages. This article presents the benchmark results and gives critical performance tips for Spark on Kubernetes.
Monday, July 6, 2020
Data Mechanics Delight - We're building a better Spark UI
Data Mechanics Delight is a Spark UI and Spark History Server replacement with new metrics and visualizations that will delight you. It works on top of any Spark platform and is entirely free.
Tuesday, June 23, 2020
Our Experience Going Through YCombinator
What is YCombinator like? What did we get out of it? The founders tell their story.
Wednesday, June 3, 2020
The Pros and Cons of Running Apache Spark on Kubernetes
Support for deploying Spark on top of Kubernetes (instead of Yarn, Mesos, Standalone) was only recently added. What are the main benefits and drawbacks? Should you get started?
Tuesday, May 26, 2020
Introducing Data Mechanics
We're proud to let you know about what we've been working on. Our mission, our vision, and our recipe for building the data platform of tomorrow.
Thursday, April 16, 2020