Delight consists of an open-source agent running inside your Spark applications, streaming metrics to our backend. Just install our agent and you're ready to go!
Create an account on our website, then head to Settings to c your access token.
Follow the installation instructions specific to your Spark platform on our GitHub page.
Your Spark applications will show up on our dashboard as soon as they're completed.
If you have an infrequently asked question, feel free to ask us on the live chat interface.
Data Mechanics Delight consists of two main components:
1. An agent which runs within your Spark applications (SparkListener) and streams metrics in real-time to our backend. This agent is open-sourced. This is a matter of trust: we want he Spark community to audit the information collected by the agent.
2. A backend system responsible of collecting, storing, and serving the metrics necessary to Delight, as well as authentication. We don't have plans to open source the server yet.
The open-sourced agent running inside your Spark application collects Spark event logs. This is non-sensitive information about the metadata of your Spark application (for example, for each Spark task there is metadata on memory usage, CPU usage, network traffic). Delight does not record any sensitive information (like the data that Spark operates on).
This data is encrypted with your access token and sent over HTTPS to the Data Mechanics control panel. Your access token guarantees that the metrics collected will only be visible to yourself, and to your colleagues from your Google organization if you signed up with your company's Google account.
This data is automatically deleted 30 days its collection, and it is not shared with any third party.
The efficiency ratio is calculated as the sum of the duration of all the Spark tasks, divided by the sum of the core uptime of your Spark executors.
An efficiency score of 75% means that on average, your Spark executor cores are running Spark tasks three quarter of the time. A low efficiency score means that you are wasting a lot of your compute resources. The Data Mechanics platform automatically tunes your Spark application configurations to make them more efficient!
Yes, Delight works on top of any Spark platform whether it's on premise or in the cloud, a commercial platform or your own open-source setup. All it needs is the ability to make outbound internet calls to stream metrics to our backend. Our Github page has instructions for installing Delight on top of Databricks, EMR, Dataproc, Spark-Submit, Spark-on-Kubernetes operator, Apache Livy, and more!
No, at this time Delight is only accessible about a minute after a Spark application has completed. Making Delight accessible in real-time for live applications is on our roadmap.
The timeline and roadmap of the project are on Github. Please submit a Github issue to report bugs or request improvements. We'd love to have your feedback!
At this time, the only way to sign up is using a Google account. We will add more authentication mechanisms in the future.
You can sign up with a personal Google account or your company's Google account. We recommend the latter, as it means the Delight dashboard will be shared with your colleagues and give you a global view of your company's applications.
