- 72 Views
- 3 replies
- 1 kudos
GIT automate DEV Databricks instance to PROD instance
Hello,Would like to use GIT to automate the process of syncing between a DEV Databricks instance and a PROD Databricks instance. Something like :On GIT console pull changes/sync with DEV Databricks.Have some kind of approval process in GIT (like some...
- 72 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @adrianhernandez ,This can be easily achieved using Databricks Assets Bundles in combination with Azure Devops Pipeline (or github actions).So, typical CI/CD workflows looks something like this:Store: Store your Databricks code and notebooks in a ...
- 1 kudos
- 62 Views
- 1 replies
- 0 kudos
DAB + DLT destroy fails due to ownership/permissions mismatch
Hi all,We are running into an issue with Databricks Asset Bundles (DAB) when trying to destroy a DLT pipeline. Setup is as follows:Two separate service principals:Deployment SP: used by Azure DevOps for deploying bundles.Run_as SP: used for running t...
- 62 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mikvaar ,I know this is really confusing but this is expected behaviour. It's well described in documentation. So,as they stated below, run as can be even use in situations where original user who created the pipeline has been deactivated - for ...
- 0 kudos
- 194 Views
- 5 replies
- 0 kudos
Serverless Compute - Spark - Jobs failing with Max iterations (1000) reached for batch Resolution
Hello Community,We have been trying to migrate our jobs from Classic Compute to Serverless Compute. As part of this process, we face several challenges, and this is one of them.When we try to execute the existing jobs with Serverless Compute, if the ...
- 194 Views
- 5 replies
- 0 kudos
- 0 kudos
@K_Anudeep FYI: Serverless Compute doesn't support spark.sql.planChangeLog.level.If we try to set up, the job will fail with [CONFIG_NOT_AVAILABLE] Configuration spark.sql.planChangeLog.level is not available. SQLSTATE: 42K0I error.Classic Compute su...
- 0 kudos
- 28 Views
- 1 replies
- 1 kudos
Resolved! To change ownership of a materialized view
working in a Unity Catalog-enabled Databricks workspace, and we have several materialized views (MVs) that were created through a Delta Live Tables (DLT) / Lakeflow pipeline.Currently, the original owner of the pipeline has moved out of the project,...
- 28 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Data_NXT ,You can change the owner of a materialized view if you are a both a metastore admin and a workspace admin.Use the following steps to change a materialized views owner:Open the materialized view in Catalog Explorer, then on the Overview ...
- 1 kudos
- 119 Views
- 19 replies
- 2 kudos
No Longer Able to Create DeltaTables in ADLS Gen 2
Hi Community,Up until recently I was happily deleting deltaTables in ADLS Gen with their associated _delta_log table, and subsequently recreating the same table with a new _delta_log table.Now, after deleting a table with its associated _delta_log ta...
- 119 Views
- 19 replies
- 2 kudos
- 2 kudos
Hi @szymon_dybczak , you must have a special version of Databricks Community Edition as I don't have those options after I select settings Shall I try from my premium paid for version of Databricks?
- 2 kudos
- 45 Views
- 4 replies
- 3 kudos
Workspace Catalog Connection to 'on-prem' MS SQL Server
Hi, first post/question here ...I'm trying to add a MS SQL Connection to my workspace (us-west-2) Catalog. Connection type is SQL Server, Host equal the public IP of that Server, port is the default 1433. (Server is a AWS EC2 instance)All attempts to...
- 45 Views
- 4 replies
- 3 kudos
- 3 kudos
Thank You to all responses.I've added 44.234.192.32/28 and 52.27.216.188/32 as a (for now) All traffic inbound rule.But no success ...I'll keep looking and trying.We're currently evaluating with a Serverless Cloud workspace, but at the end might go t...
- 3 kudos
- 52 Views
- 6 replies
- 1 kudos
Notebook Dashboard refreshes for all users when one user refreshes
Hi Team ,I have created a dashboard using the dashboard functionality of the notebook.But when one user refreshes the dashboard it refreshes for all the users with whom the dashboard is shared.What are the ways this issue can be resolved ?
- 52 Views
- 6 replies
- 1 kudos
- 126 Views
- 7 replies
- 11 kudos
Workflows now harder to find old failed runs
Some time in the past few weeks I think there was an update to databricks workflows. Previously you could:run a workflowit failsrepair the workflowclick into the workflowview past runs before that failed via a dropdown bar (like in the screenshot bel...
- 126 Views
- 7 replies
- 11 kudos
- 11 kudos
@szymon_dybczak did you see this post yesterday about Choropleth maps not working?: https://community.databricks.com/t5/warehousing-analytics/choropleth-maps-stopped-working/td-p/131991 I wonder if a similar fix can happen for this situation, as it c...
- 11 kudos
- 112 Views
- 1 replies
- 0 kudos
delta live tables
Hi . i have a source table that is a delta live streaming table created using dlt.auto_cdc logic and now i want to create another sreaming table that filters the record from that table as per the client but it also should have auto cdc logic for the...
- 112 Views
- 1 replies
- 0 kudos
- 0 kudos
@tenzinpro have you looked into the documentation Incremental-refreshes for materalised views with streaming tables yet? https://docs.databricks.com/aws/en/optimizations/incremental-refresh .. there's a section in there which jumped out to me:I'd sug...
- 0 kudos
- 786 Views
- 3 replies
- 0 kudos
Delta live tables - ignore updates on some columns
Hello Team,I have scenario where in apply_changes, i want to ignore updates on 1 column. Is there any way we can achieve this in Delta live tables?
- 786 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi there @Anish_2 , Yes you can do that Here is the doc link : https://docs.databricks.com/aws/en/dlt/cdc?language=PythonFor python you can simply add an attribute except_columns_list like thisdlt.apply_changes( target = "target", source = "users...
- 0 kudos
- 50 Views
- 1 replies
- 1 kudos
Serverless Workspace Observability
I’m setting up observability for a Databricks serverless workspace on AWS and need some guidance.I know we can configure audit logs for S3 delivery, but I’m unsure if that alone is sufficient.For a complete observability setup especially when integra...
- 50 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @APJESK , Audit Logs, are critical logs for security, they track user actions, such as who accessed what data, who ran a job, and who changed permissions. Delivery of these to an S3 bucket is a standard, must-have practice. Additionally, you have...
- 1 kudos
- 22096 Views
- 7 replies
- 2 kudos
Resolved! Install ODBC driver by init script
Hello,I want to install ODBC driver (for pyodbc).I have tried to do it using terraform, however I think it is impossible.So I want to do it with Init Script in my cluster. I have the code from the internet and it works when it is on the beginning of ...
- 22096 Views
- 7 replies
- 2 kudos
- 2 kudos
Actually found this article and using this to migrate my shell script to workspace.Cluster-named and cluster-scoped init script migration notebook - Databricks
- 2 kudos
- 29 Views
- 2 replies
- 2 kudos
Connecting SQL Server From Databricks
Hi Team,Good Day!Iam trying to connect to sql server from databricks using pyodbc , However iam not able to connect , And I have tried many ways like adding the init script in the cluster configuration etc,But it is showing me error,I want to know ea...
- 29 Views
- 2 replies
- 2 kudos
- 2 kudos
@GowthamR supplying errors, if possible, given that they aren't leaking credentials (please mask those in the screenshots) will be really useful for helping us to debug what's happening. All the best,BS
- 2 kudos
- 96 Views
- 4 replies
- 5 kudos
Azure Databricks Job Run Failed with Error - Could not reach driver of cluster
Hello Community,I am facing an intermittent issue while running a Databricks job. The job fails with the following error message:Run failed with error message:Could not reach driver of cluster <cluster-id>.Here are some additional details:Cluster Typ...
- 96 Views
- 4 replies
- 5 kudos
- 5 kudos
Hello Anudeep,Thank you for your detailed response and the helpful recommendations.I would like to provide some additional context:For our jobs, we are running only one notebook at a time, not multiple notebooks or tasks concurrently.The issue occurs...
- 5 kudos
- 203 Views
- 7 replies
- 1 kudos
Upgrading Drivers and Authentication Method for Snowflake Integration
Hello Databricks Support Team,I am reaching out to request assistance with upgrading the drivers and configuring authentication methods for our Snowflake–Databricks integration.We would like to explore and implement one of the recommended secure auth...
- 203 Views
- 7 replies
- 1 kudos
- 1 kudos
@Khaja_Zaffer ,Need assistance on upgrading Snowflake drivers on Cluster. We installed Snowflake package on cluster, How to upgrade Snowflake library? For Authentication, I will reach out to Azure team.Thanks for the details.
- 1 kudos
| User | Count |
|---|---|
| 1616 | |
| 786 | |
| 349 | |
| 347 | |
| 287 |
