Configuring DLT _delta_logs with Log Analytics Workspace on Job Clusters

[mkEngineer]

Hi,

How do I configure my DLT (Delta Live Table pipeline notebook) _delta_logs with my Azure Log Analytics workspace? I'm encountering issues because the pipeline runs on a job cluster, which doesn't allow me to specify the destination of the log files under Advanced Options when creating the cluster.

Additionally, when running the pipeline on the job cluster, I encounter errors with commands like:
dbutils.secrets.get

This function is meant to retrieve secrets and configure logging. How can I ensure that my Log Analytics workspace ID and Primary Key—stored in my Azure Key Vault and saved within a Databricks scope—are correctly configured?

Thank you!

[Alberto_Umana]

Hi @mkEngineer,

Since we need to ensure that secrets from Azure Vault are retrieved, what is the error you are getting when running dbutils.secrets.get command?

[mkEngineer]

Hi @Alberto_Umana ,

The error I received was related to cells not being connected to the DLT pipeline, as mentioned in my other post, "Cannot run a cell when connected to the pipeline Databricks." However, after browsing the web, I realized that there were some necessary libraries that had to be downloaded and configured in order to set up a pipeline/job cluster. Though, I'm a bit unsure how to configure those libraries when editing the job compute. There is no "Library" tab on the job compute like there is for all-purpose for example. Currently comparing similar functions in Datarbicks due to the hassle of configuring to Log Analtyics. 
https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs