topic Re: How to write event_log destination into DLT Settings JSON via Asset Bundles in Data Engineering

How to write event_log destination into DLT Settings JSON via Asset Bundles

susanne — Wed, 19 Mar 2025 09:06:41 GMT

Hi all,

I would like to publish the event_log of my DLT Pipeline to a specific schema in Unity Catalog.
Following this article (https://docs.databricks.com/gcp/en/dlt/observability#query-the-event-log) this can be done by writing this into the DLTs settings JSON:

  "event_log": {
    "catalog": "catalog_name",
    "schema": "schema_name",
    "name": "event_log_table_name"
  }

I want to incorporate this setting in my Databricks Asset Bundle Deployment, so it doesnt need to be manually added.
Is there a way to do this?

Thanks and best regards,
Susanne

Re: How to write event_log destination into DLT Settings JSON via Asset Bundles

jorperort — Sat, 22 Mar 2025 12:51:27 GMT

Hi @susanne

I’m sharing an option in case you find it useful. In my case, I was testing how to publish these tables using Databricks Asset Bundles, but I couldn’t get it to work. I'm not sure if I posted a question about this. Let’s see if we’re lucky and someone provides the corresponding property, if it exists.

From what I’ve seen, it seems that this property doesn’t exist. So, I decided to create a Python script to handle this process. The script communicates directly with the Databricks API of the corresponding workspace and, based on the ID of several pipelines, retrieves the configuration using a GET method. Then, it updates the configuration by adding the event_log and finally performs a PUT to update it.

Additionally, you can automate the execution of this job using Databricks Asset Bundles.

I hope this option helps you.

Re: How to write event_log destination into DLT Settings JSON via Asset Bundles

ashraf1395 — Sun, 30 Mar 2025 09:52:33 GMT

Hi @susanne , indeed , i tried to create it using dabs as well. This feature is not available using dabs I guess, maybe they will add it once event_logs moves to ga from public preview.

databricks API will be a good alternative but if you try it using databricks SDK , it will be more secure.
doc link : https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/pipelines.html#databricks.sdk.service.pipelines.PipelineSpec