Databricks Community

Mathias_Peters · 10-09-2024

Hi, we are using asset bundles to deploy artefacts in Databricks. We use a service account and github workflows for that. Currently, we deploy in dev mode whenever a PR is created against the main branch of our gh repo. So far so good. However, we ar...

Mathias_Peters · 08-15-2024

Hi, we are managing our Databricks Workspaces entirely using terraform. As of today, the module to read the current clusters fails with a timeout. We use this:data "databricks_clusters" "this" { }This has worked flawlessly before. However, now we are...

Mathias_Peters · 07-10-2024

We have implemented an asset bundle (DAB) that creates a wheel. During DAB deployment, the wheel is built and stored in the folder of the service principal running the deployment via GH workflow. The full path is/Workspace/Users/SERVICE-PRINCIPAL-ID/...

Mathias_Peters · 05-26-2024

Hi, I implemented a job that should incrementally read all the available data from a Kinesis Data Stream and terminate afterwards. I schedule the job daily. The data retention period of the data stream is 7 days, i.e., there should be enough time to ...

Mathias_Peters · 05-15-2024

Hi, I have a job consisting of three tasks: tasks: - task_key: Kinesis_to_S3_new spark_python_task: python_file: ../src/kinesis.py parameters: ["${var.stream_region}", "${var.s3_base_path}"] j...

Mathias_Peters

Hi, I had a similar problem today. I changed the way, that we deploy our main bundle using pull requests and in order to play around with it locally, I copied python and dbt code into the databricks src dir (that is normally done during a github work...

Mathias_Peters · 08-15-2024

Correction of my previous post: The error occurs when bumping the provider version to 1.50.0. It works fine on 1.49.0

Mathias_Peters · 07-10-2024

Update: When moving the whl to the shared workspace folder, the installation on the cluster works.

Mathias_Peters · 05-16-2024

Hi @Ajay-Pandey ,thank you for the hints. I will try to recreate the job via UI. I ran the tasks in a Github workflow. The file locations are mixed: the first two tasks (python and dlt) are located in the databricks/src folder. The dbt files come fro...

Mathias_Peters · 05-10-2024

Ok, it seems that the default language of the notebook and the language of a particular cell can clash. If the default is set to Python, switching a cell to SQL won't work in DLT and vice versa. This is super unintuitive tbh.

Databricks Community

User Stats

User Activity

Is there a way to use name prefixes for asset bundle deployments to prevent collisions

Databricks Terraform Module for reading the current clusters times out

Service principal seemingly cannot access its own workspace folder

How to properly implement incremental batching from Kinesis Data Streams

Asset Bundles: Adding project_directory in DBT task breaks previous python task

Re: Asset bundle doesn't sync files to workspace

Re: Databricks Terraform Module for reading the current clusters times out

Re: Service principal seemingly cannot access its own workspace folder

Re: Asset Bundles: Adding project_directory in DBT task breaks previous python task

Re: DLT table not picked in python notebook