I've created a completely fresh project with a completely empty workspaceLocally I have the databricks CLI version 0.230.0 installedI rundatabricks bundle init default-pythonI have auth set up with a PAT generated by an account which has workspace ad...
I have a materialized view that always does a "COMPLETE_RECOMPUTE", but I can't figure out why.I found how I can get the logs: SELECT * FROM event_log(pipeline_id) WHERE event_type = 'planning_information' ORDER BY timestamp desc; And for my table...
I split up materialized view in 3 separate ones:step1:@Dlt.table(name="step1", table_properties={"delta.enableRowTracking": "true"}) def step1(): isolate_names = dlt.read("soruce").select("Name").groupBy("Name").count() return isolate_namesst...
Hello there, i have a temp table where i want to remove a null / empty values ( see below )if there are no rows to delete, then shouldn't it just say Zero rows affected?
@RobDineen This should answer your question: https://community.databricks.com/t5/get-started-discussions/how-to-create-temporary-table-in-databricks/m-p/67774/highlight/true#M2956Long story short, don't use it.
In a scenario where multiple teams access overlapping but not identical datasets from a shared data lake, is it better to create separate datamarts for each team (despite data redundancy) or to maintain a single datamart and use views for team-specif...
Hi,We have a workflow which calls 3 notebooks and when we run this workflow using shared cluster it runs fine, but when ran with job cluster , one of the notebooks fail.This notebook uses sql function Lpad and looks like it errors because of it. Has ...
Hi community,Since recently (2 days ago) we have been receiving the following error when validating and deploying our DAB (Databricks Asset Bundle):"Error: error downloading Terraform: Get "https://releases.hashicorp.com/terraform/1.5.5/index.json": ...
@FabianGutierrez From where are you deploying DAB? The error states that there is definitely some kind of connectivity issue.
When deploying multiple jobs using the `Databricks.yml` file via the asset bundle, the process either overwrites the same job or renames it, instead of creating separate, distinct jobs.
@filipniziol my requirements is to just deploy notebooks in databricks workspace. I don’t not wana create any job. Can I still uses databricks asset bundle
I'm trying to use the foreachBatch method of a Spark Streaming DataFrame with databricks-connect. Given that spark connect supported was added to `foreachBatch` in 3.5.0, I was expecting this to work.Configuration:- DBR 15.4 (Spark 3.5.0)- databrick...
@olivier-soucy Are you sure that you're using DBR 15.4 and databricks-connect 15.4.2?I've seen this issue when using databricks-connect 15.4.x with DBR 14.3LTS.Anyway, I've just tested that with the same versions you've provided and it works on my en...
I have created a recipient using the data bricks in the console, also, I have created data bricks using the console, Now I want to map the existing recipient to the existing delta share, Is there a way to do this using Terraform
While reading csv file using spark and listing the files under a folder using data bricks utils, I am getting below error:[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501File <comma...
@Tamizh035 , If your file in dbfs or external location or local folder ?Use dbutils.fs.ls to verify that the path exists and you have access:files = dbutils.fs.ls("dbfs:/path_to_your_file/") display(files)
HiQuestion probably around best practices, but curious if someone else has dealt with a similar situation. I have 2 Databricks workspaces - one for Dev and one for Prod. Had to be two workspaces because Azure Landing Zones had to be air gapped from e...
Hi @RangaSarangan ,We have faced same issue and solved using databricks workflow API and json file for job metadata that consist job and thier respective status for each env.You can create azure devops that run after your cicd pipeline and change the...
Hello, My team and I are experimenting with bundles, we follow the pattern of having one main file Databricks.yml and each job definition specified in a separate yaml for modularization. We wonder if it is possible to select from the main Databricks....
"This include array can appear only as a top-level mapping." - you can't use include inside targets. You can use sync - exclude to exclude the yml files, but if they are in the include the workflows are going to be created anyway, even if the yml fil...
Hello Databricks Community!I'm working on a project where I need to deploy jobs in a specific sequence using Databricks Asset Bundles. Some of my jobs (let's call them coordination jobs) depend on other jobs (base jobs) and need to look up their job ...
Hey Steph, Have you found anything here please ? I'm currently stuck here, trying to achieve the same thing
Hello,I have a remote azure sql warehouse serverless instance that I can access using databricks-sql-connector. I can read/write/update tables no problem.But, I'm also trying to read/write/update tables using local pyspark + jdbc drivers. But when I ...
I added `UseNativeQuery=0` in url. It works for me.
I have a pretty large SQL query that has the following stats from the query profiler:Tasks total time: 1.93sExecuting: 27sBased on the information in the query profiler this can be due to tasks waiting for available nodes.How should I approach this t...
Hi nengenYou may have more infos to share, so we can help you?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group| User | Count |
|---|---|
| 1604 | |
| 758 | |
| 344 | |
| 285 | |
| 250 |
