Apache Spark and Kubernetes logos
Apache Spark on Kubernetes Architecture

A Cloud-Native architecture

Apache Spark on Kubernetes

A Kubernetes cluster consists of a set of nodes on which you can run containerized Apache Spark applications (as well any other containerized workloads). Each Spark app is fully isolated from the others and packages its own version of Spark and dependencies within a Docker image.

When you submit a Spark app, it starts a Spark driver pod (a Docker container, to put it simply) on the Kubernetes cluster. The driver pod and Kubernetes directly talk to each other to start Spark executor pods. The start and removal of  executors is automated based on load when dynamic allocation is enabled.

A thriving DEVELOPER community

Spark-on-Kubernetes now Generally Available

Since its initial release in Spark 2.3, the Spark-on-Kubernetes community has been thriving behind both starts and enterprises adoptions. The community led the development of key features such as volume mounts, dynamic allocation, and graceful handling of node shutdown.

As a result of these features, the Spark-on-Kubernetes project will officially be marked as Generally Available and production ready as of Spark 3.1. 

Timeline of improvements to Spark on Kubernetes

Why Choose Spark on Kubernetes?

The advantages of working with Apache Spark on Kubernetes

Spark on k8s offers many advantages for companies big and small who want to simplify and speed up their development workflows in addition to dramatically reducing their costs.

Person in front of different files

1. Containerized

Use Docker to package your dependencies.

Build it once, and run it everywhere: locally or in the cloud, in dev or in prod environments. Your application is portable and fully isolated from other workloads running at the same time.

Person in front of a screen

2. Developer-friendly

Kubernetes offers a rich ecosystem of developer tools and solutions that simplify your data operations on a day-to-day basis.

You won't need to learn about the obscure intricacies of Hadoop YARN, or of an opaque proprietary vendor solutions. 

Piggy bank

3. Cost Effective

Keep your infrastructure costs down by running all your apps on a shared cluster. 

The cluster can automatically scale up and down based on load, and use a mix of spot/on-demand nodes and heterogeneous instance types to reduce your cloud bill.

Rocket

4. Fast

Speed up your iteration cycle by 10X.

Kubernetes can start and scale applications in a matter of seconds. You won't need to wait for slow virtual machine setup, for YARN overhead, or for slow bootstrap scripts to run.

Person sitting on the floor with a laptop on their knees.

5. Reliable & Secure

Bring in the DevOps best practices in your data teams.

Kubernetes enable repeatable and reliable workflows, and enables gold-standard security best practices around networking, data access permissions, and user ACLs.

Person in front of world globe with multiple locations.

6. Cloud and Vendor Agnostic

Build on top of a standard open-source technology and avoid lock in.

Kubernetes lets you easily switch between cloud and on-premise platforms, and avoid expensive alternatives by vendors who will lock you in with proprietary technologies.

What does Data Mechanics bring to the table?

Data Mechanics is a fully managed Spark on Kubernetes platform deployed in your cloud account. We handle the setup, the maintenance, and the scaling of the cluster. We add intuitive UIs, key integrations, and powerful optimizations on top, to make Spark easy-to-use and cost-effective for data teams.

Monitoring dashboard in the Data Mechanics platform.

Transparent performance monitoring

An Intuitive Monitoring UI

Our dashboard gives you visibility over the metrics, logs, and Spark UI for each of your applications. Our Jobs UI gives you a historical graph of your pipeline's key metrics and costs, ensuring your team can stay on top of your production pipelines.

In addition, we're building a new and improved Spark UI called Data Mechanics Delight.

Dynamic Optimizations

History-Based Automated Tuning of Spark Pipelines

You probably run Spark pipelines on a periodic basis (weekly, daily, hourly). But Spark does not have a memory - it re-discovers how to run your applications every time. 

Our Optimization Engine learns from your pipelines historical runs to optimize your infrastructure parameters and Spark configurations. This will save you from tedious manual tuning work, and can reduce your infrastructure costs by 50%.

Architecture of history-based automated tuning.
Logos of various technologies: Airflow, Kubernetes, Docker, Jupyter, Data Mechanics.

Key Ecosystem Integrations

Integrations with the tools you already use & love

Data Mechanics has integrations with notebooks services like Jupyter, JupyterLab, JupyterHub as well as scheduler/workflows services like Airflow.

The full Docker & Kubernetes ecosystem is at your fingertips, and since we're deployed in your cloud account and in your VPC, you can also easily connect homemade tools.

They trust us

Ready to get started?

🍪 We use cookies to optimize your user experience. By browsing our website, you agree to the use of cookies.
close
30