Skip to content

Alerting in Databricks

This document details the setup process for alerts being sent from
Databricks to a Microsoft Teams channel.

Prerequisites

Note: this only needs to be done once for each destination Teams channel per
Databricks workspace. For the 'AIML Alerting' channel against the Prod
workspace, this has already been done.

In order for Databricks alerts to be sent to a Teams channel, it needs
to be set up as a 'Notification Destination' in Databricks. This
has to be done by the Workspace Admin. To do so, follow these steps:

  1. Pick (or create) a Teams Channel.
  2. On the channel, click ... and select Workflows.
  3. Search for the template Send webhook alerts to a channel.
  4. This will generate a Webhook URL, copy this.
  5. Reach out to the Databricks Workspace admin (at the time of writing this is David Torres) and ask them to add the channel as a notification destination. Provide the channel name and the webhook URL.

Once this is set up, the channel will be available in the dropdown for notifications in Databricks. The Webhook URL will
allow anyone with the URL to send messages to the channel so be wary about where this is exposed.

Setting up a job alert

In the Databricks UI

Within Databricks, navigate to the job you want to create alerts for under Jobs & Pipelines.
Note that you will need MANAGE permissions on the job in question.

To set up an alert do the following:

  1. In the sidebar of the selected job, navigate to the Job notifications section and click Edit notifications.
  2. Click Add notification and select the desired Teams channel in the destination dropdown.
  3. Specify desired notification events, likely Failure and possibly Duration warning (the duration
    warning will require setting up a metric threshold in the Add metric thresholds menu within Job notifications).
  4. Click Save.

And that's it. The desired alerts will appear in Teams when triggered.

In an asset bundle

If you are using asset bundle management, adding an equivalent notification is done via the webhook_notification
section, e.g.:

webhook_notifications:
  on_failure:
    - id: <webhook-id>

The webhook-id identifies the particular alert destination. In the case of the AIML Alerting Teams
channel from our production Databricks workspace this is 665ec34f-c343-408d-897e-8ece7d8cf92d. Other notification
options include on_success, on_start, on_duration_warning_threshold_exceeded, etc. To have the duration warning
you need to additionally configure a health metric:

webhook_notifications:
  on_failure:
    - id: <webhook-id>
  on_duration_warning_threshold_exceeded:
    - id: <webhook-id>
health:
  rules:
    - metric: RUN_DURATION_SECONDS
      op: GREATER_THAN
      value: 3600