The New & Improved Spark UI

We've released the first milestone of Delight: a free, hosted, and cross-platform Spark History Server.
The full release of Delight will be in January 2021!

Spark History Server screenshot

Installation instructions

How It Works

Delight consists of an open-source agent running inside your Spark applications, streaming metrics to our backend. Just install our agent and you're ready to go!

Data Mechanics Logo

1. Sign Up

Create an account on our website, then head to Settings to retrieve your access token.

Apache Spark logo in a cube.

2. Install the open-sourced agent

Follow the installation instructions specific to your Spark platform on our GitHub page.

A green check logo.

3. You’re all set

Your Spark applications will show up on our dashboard as soon as they're completed.

Data Mechanics Delight

Frequently Asked Questions

If you have an infrequently asked question, feel free to ask us on the live chat interface.

Is Delight really free?

Yes, Delight is entirely free of charge.

Is Delight open-source?

Data Mechanics Delight consists of two main components:

1. An agent which runs within your Spark applications (SparkListener) and streams metrics in real-time to our backend. This agent is open-sourced. This is a matter of trust: we want he Spark community to audit the information collected by the agent.

2. A backend system responsible of collecting, storing, and serving the metrics necessary to Delight, as well as authentication. We don't have plans to open source the server yet.

What data does Delight collect? Is Delight Secure?

The open-sourced agent running inside your Spark application collects Spark event logs. This is non-sensitive information about the metadata of your Spark application (for example, for each Spark task there is metadata on memory usage, CPU usage, network traffic). Delight does not record any sensitive information (like the data that Spark operates on). 

This data is encrypted with your access token and sent over HTTPS to the Data Mechanics control panel. Your access token guarantees that the metrics collected will only be visible to yourself, and to your colleagues from your Google organization if you signed up with your company's Google account. 

This data is automatically deleted 30 days its collection, and it is not shared with any third party.

When will Delight be released?

As of November 2020, we've reached a first milestone in Delight implementation and decided to release part of the project. There are no new metrics or visualizations yet, but it consists of a dashboard listing all your Spark applications, and a link to the Spark UI for each of them. So it is a free hosted Spark History Server.

The next release of Delight is planned for January 2021. It will consist of an overview screen with some of the new metrics and visualizations you've been waiting for. 

Can I run Delight over other platforms?

Yes, Delight works on top of any Spark platform whether it's on premise or in the cloud, a commercial platform or your own open-source setup. All it needs is the ability to make outbound internet calls to stream metrics to our backend. Our Github page has instructions for installing Delight on top of Databricks, EMR, Dataproc, Spark-Submit, Spark-on-Kubernetes operator, Apache Livy, and more setup.

Is Delight accessible while the Spark app is running ?

No, at this time Delight is only accessible about a minute after a Spark application has completed. Making Delight accessible in real-time for live applications is on our roadmap.

How can I report a bug or request a feature?

Please submit a Github issue. We'd love to have your feedback!

Which Google account should I use to signup? What if I don't have a Google account ?

At this time, the only way to sign up is using a Google account. We will add more authentication mechanisms in the future.

You can sign up with a personal Google account or your company's Google account. We recommend the latter, as it means the Delight dashboard will be shared with your colleagues and give you a global view of your company's applications.

Get Started

Our first milestone is live: Dashboard + Hosted Spark History Server
🍪 We use cookies to optimize your user experience. By browsing our website, you agree to the use of cookies.