Databricks Community

We were not able to find a clean solution for this, so what we ended up doing is referencing the common lib like this in every notebook it is needed.%pip install ../../../artifacts/.internal/common-0.1-py3-none-any.whl 

There might be few possibilities : can you check this items ? 1. Is there any s3 bucket policy configured like within timeframe file deletion or file validity configured ? 2. check the autload configuration once again to validate the option of cleanu...

Hi @sivaram_mandepu,In the first screenshot, the input must be a valid JSON array, so instead of using {{mvname: "mv_......"}}, update it to [ { "mvname": "mv_......." } ].In the third screenshot, the SQL error likely comes from a newline or extra sp...

@dipudot, yes the permission are OK. I can see them in SQL Serber Management using the same account.I have read somewhere that the some characters might not be supported. The views have all the pattern <tenant>$<table_name>.I will retest with a small...

I came across the same issue and the file causing problems needed the csv option "multiline" set back to the default "false" to read the file:df = spark.read.option("multiline", "false").csv("CSV_PATH") If this approach eliminates the error above, I ...

1. no such command exists2. probably not - we tend to dump old clusters and create new ones (for new sets of data) fairly frequently and (I think) use the latest stable DBR when creating3. I did find a workaround.  unminimize has been added to apt so...

Yes, you can display the current git branch name from a Databricks notebook in several ways: Using the Databricks UI The simplest method is to use the Databricks UI, which already shows the current branch name:- In a notebook, look for the button nex...

The issue with your DLT pipeline is that you've defined the table and schema correctly, but you haven't actually implemented the data loading logic in your `ingest_from_storage()` function. While you've created the function, you're not calling it any...

In Unity Catalog standard access mode (formerly shared access mode) with Databricks Runtime 15.4 LTS, direct access to `sparkSession.sparkContext` is restricted as part of the security limitations. However, there are still ways to access the Hadoop c...

Not sure, but Databricks may default to AWS-style paths if the configurations are incomplete. Try setting the MinIO endpoint by configuring spark.hadoop.fs.s3a.endpoint to your MinIO server's URL. If MinIO uses HTTP, disable SSL by setting spark.hado...

Hi @Malthe, refreshing is automatically handled during pipeline runs in here. To implement effective role segregation, you should define separate DLT pipelines for deployment and execution, each with its own set of roles and permissions. This approac...