Apache Spark and Kubernetes logos
Apache Spark on Kubernetes Architecture

A Cloud-Native architecture

Apache Spark on Kubernetes

A Kubernetes cluster consists of a set of nodes on which you can run containerized Apache Spark applications (as well any other containerized workloads). Each Spark app is fully isolated from the others and packages its own version of Spark and dependencies within a Docker image.

When you submit a Spark app, it starts a Spark driver pod (a Docker container, to put it simply) on the Kubernetes cluster. The driver pod and Kubernetes directly talk to each other to start Spark executor pods. The start and removal of  executors is automated based on load when dynamic allocation is enabled.

A thriving DEVELOPER community

Spark-on-Kubernetes now Generally Available

Since its initial release in Spark 2.3, the Spark-on-Kubernetes community has been thriving behind both starts and enterprises adoptions. The community led the development of key features such as volume mounts, dynamic allocation, and graceful handling of node shutdown.

As a result of these features, the Spark-on-Kubernetes project will officially be marked as Generally Available and production ready as of Spark 3.1. 

Timeline of improvements to Spark on Kubernetes

Why Choose Spark on Kubernetes?

The advantages of working with Apache Spark on Kubernetes

Spark on k8s offers many advantages for companies big and small who want to simplify and speed up their development workflows in addition to dramatically reducing their costs.

Person in front of different files

1. Containerized

Use Docker to package your dependencies.

Build it once, and run it everywhere: locally or in the cloud, in dev or in prod environments. Your application is portable and fully isolated from other workloads running at the same time.

Person in front of a screen

2. Developer-friendly

Kubernetes offers a rich ecosystem of developer tools and solutions that simplify your data operations on a day-to-day basis.

You won't need to learn about the obscure intricacies of Hadoop YARN, or of an opaque proprietary vendor solutions. 

Piggy bank

3. Cost Effective

Keep your infrastructure costs down by running all your apps on a shared cluster. 

The cluster can automatically scale up and down based on load, and use a mix of spot/on-demand nodes and heterogeneous instance types to reduce your cloud bill.


4. Fast

Speed up your iteration cycle by 10X.

Kubernetes can start and scale applications in a matter of seconds. You won't need to wait for slow virtual machine setup, for YARN overhead, or for slow bootstrap scripts to run.

Person sitting on the floor with a laptop on their knees.

5. Reliable & Secure

Bring in the DevOps best practices in your data teams.

Kubernetes enable repeatable and reliable workflows, and enables gold-standard security best practices around networking, data access permissions, and user ACLs.

Person in front of world globe with multiple locations.

6. Cloud and Vendor Agnostic

Build on top of a standard open-source technology and avoid lock in.

Kubernetes lets you easily switch between cloud and on-premise platforms, and avoid expensive alternatives by vendors who will lock you in with proprietary technologies.

🍪 We use cookies to optimize your user experience. By browsing our website, you agree to the use of cookies.