cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Eibraao
by New Contributor II
  • 1562 Views
  • 7 replies
  • 0 kudos

Disable the dashboard sharing field for dashboard creators

"How can I disable the dashboard sharing field for dashboard creators who are not admins? I tried changing the creator’s permission from 'CAN_MANAGE' to 'CAN_READ', but it had no effect — the creator still retains the 'CAN_MANAGE' permission

  • 1562 Views
  • 7 replies
  • 0 kudos
Latest Reply
charl-p-botha
Databricks Partner
  • 0 kudos

Any movement on this topic?We too need to restrict sharing, in our case ideally with only other users of the same workspace. As it stands, any user who creates a dashboard can share that dashboard with any account user, across all of our workspaces. ...

  • 0 kudos
6 More Replies
tasa
by Visitor
  • 84 Views
  • 2 replies
  • 0 kudos

lakeflow connect postgres sql

Hi can anyone tell me how to enable tc LakeFlow Connect for Direct Cdc Managed Ingestion Pipeline. it shows that this mode is not available for this workspace for the selected connector.

  • 84 Views
  • 2 replies
  • 0 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 0 kudos

Hi @tasa,   Just responded to your other post on the same topic. It looks like your workspace is not currently enabled for Direct CDC/integrated CDC for that connector. In Databricks, this commonly surfaces as the DIRECT_CDC_NOT_ENABLED valid...

  • 0 kudos
1 More Replies
tasa
by Visitor
  • 98 Views
  • 2 replies
  • 0 kudos

lakeflow connect postgres sql in uae north

Hi can anyone tell me how to enable tc LakeFlow Connect for Direct Cdc Managed Ingestion Pipeline. it shows that this mode is not available for this workspace for the selected connector.

  • 98 Views
  • 2 replies
  • 0 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 0 kudos

Hi @tasa, It looks like your workspace is not currently enabled for Direct CDC/integrated CDC for that connector. In Databricks, this commonly surfaces as the DIRECT_CDC_NOT_ENABLED validation error, which means the feature is not enabled for the org...

  • 0 kudos
1 More Replies
Rameshw
by New Contributor
  • 79 Views
  • 1 replies
  • 0 kudos

Header Alignment in databricks dashboard

Hi Expert,when global filter page is added then visual header alignement gets missedThoughts: The user is reporting a UI/layout issue with the Lakeview dashboard — when a global filter page is added, the visual header alignment gets misaligned. This ...

  • 79 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 0 kudos

Hi @Rameshw, As you have pointed out, what you’re seeing is expected behaviour when a filter is added above an existing grid layout. Widgets on the canvas retain their existing positions, so adding a new row at the top does not automatically reflow t...

  • 0 kudos
freddyT
by New Contributor
  • 171 Views
  • 3 replies
  • 0 kudos

DBR 17.3 – sporadic import failures from bundle

Hi all,Since two days ago we've been getting sporadic Python import failures on DBR 17.3, with no change on our side. The errors are either "cannot import name ..." or "no module named ...", and they come and go: the same code fails on one run and pa...

  • 171 Views
  • 3 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

Hi, I think the accepted answer is pointing in the right direction. Given that DBR 17.3 is an LTS runtime, this is more likely to be a packaging or deployment issue than a runtime regression. One common cause is wheel caching. If your wheel version d...

  • 0 kudos
2 More Replies
aditi_mokashi
by New Contributor
  • 161 Views
  • 2 replies
  • 0 kudos

Urgent: Installing Lakebridge on Databricks

Hi,I want to install Databricks Lakebridge on my Databricks environment and use the analyze and transpile commands through a python script.The usecase is that we need to create an automated pipeline that will migrate the existing scripts from snowfla...

  • 161 Views
  • 2 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

Hi, the error you're seeing is consistent with the execution environment rather than with Lakebridge itself. Lakebridge is a Databricks CLI Labs extension (databricks labs install lakebridge) designed to run wherever the Databricks CLI is installed—y...

  • 0 kudos
1 More Replies
JTBS
by New Contributor II
  • 145 Views
  • 2 replies
  • 0 kudos

StatusCode.UNIMPLEMENTED error: DatabricksConnect library using AKS/PySpark to calling Spark cluster

I am running PySpark application in AKS/Pythgon container/pod:Using Databricks 18.2.1 library with Databricks Spark cluster 18.2Once a while I am getting below error: InactiveRpcError of RPC that terminated with: status = StatusCode.UNIMPLEMENTED det...

  • 145 Views
  • 2 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

Short answer: this looks more like an intermittent Spark Connect transport/routing issue than a Spark job logic issue. Databricks Connect uses gRPC over HTTP/2, and the specific InactiveRpcError ... UNIMPLEMENTED ... Received http2 header with status...

  • 0 kudos
1 More Replies
Malthe
by Valued Contributor II
  • 132 Views
  • 2 replies
  • 0 kudos

Resolved! NOT NULL constraint violated for column during OPTIMIZE

We're running an optimize on a delta table with a VARIANT column that has a NOT NULL constraint.Now, there are no NULL entries in this column. And yet, OPTIMIZE gives this error:[DELTA_NOT_NULL_CONSTRAINT_VIOLATED] NOT NULL constraint violated for co...

  • 132 Views
  • 2 replies
  • 0 kudos
Latest Reply
Malthe
Valued Contributor II
  • 0 kudos

So funny story on this: the column in question does not match on "col is null", but it does match on "col::string is null".Perhaps because in the Parquet-file which corresponds to the rows that have the problem, that column is entirely missing.Still,...

  • 0 kudos
1 More Replies
Abdel_GBAL
by New Contributor
  • 159 Views
  • 1 replies
  • 0 kudos

Resolved! Lakebase CDF databricks error synced

Hello,I have defined Lakebase Change Data Feed (CDF) in order to sync some lakebase native tables into my catalog.schema, i can see that the tables are created with a naming convention like this: lb_*_history.sometimes the status is persisting in sna...

Data Engineering
Apps
CDF
databricks lakebase
lakebase
lakebase sync
  • 159 Views
  • 1 replies
  • 0 kudos
Latest Reply
yogeshsingh
Databricks Employee
  • 0 kudos

Thanks for sharing the screenshots. Assuming this is Lakebase Change Data Feed (formerly Lakehouse Sync) from Lakebase Postgres into Unity Catalog lb_*_history tables, this does not look like expected steady-state behavior. A few things stand out: O...

  • 0 kudos
Sam500
by New Contributor III
  • 148 Views
  • 1 replies
  • 0 kudos

Auto Loader duplicate tracking

Hi experts, I read an article about auto loader duplicates handling that got me bit confused. It is checkpoint that tracks what is being processed , and upon confirmation it process only the new incoming records. But, let's say I am reloading whole b...

  • 148 Views
  • 1 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

Hi, Auto Loader's tracking is at the file level, not the record level, and that distinction is exactly what's tripping you up here. The checkpoint keeps a RocksDB-backed record of every file it has already discovered and ingested, keyed by things lik...

  • 0 kudos
Dolly0503
by New Contributor III
  • 70 Views
  • 1 replies
  • 0 kudos

Automation for any script changes in databricks and bit bucket

If any code changes are done and moved to bit bucket based on env now i want to fetch all the code changes and move all these changes to a table in adls/unity catalog table.Anyone please help with the approaches we are using bit bucket for repository

  • 70 Views
  • 1 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

Hi, couple ways to do this depending on how tightly you want it wired in. If Bitbucket is just your source repo and you want a row per change, the simplest path is a Bitbucket Pipeline that runs on push. In that pipeline step, pull the commit metada...

  • 0 kudos
KriStromNAF
by New Contributor
  • 139 Views
  • 1 replies
  • 1 kudos

Bug? Genie Space benchmark evaluates "Score reason" wrongly as "Empty result" on agent mode

First some context: I'm setting up a Genie Space and have multiple benchmark questions with business validated SQL ground truths defined. Currently no configuration defined, apart from metadata and SQL joins.I haven't found any other way of reporting...

  • 139 Views
  • 1 replies
  • 1 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 1 kudos

Hi, this looks like a real quirk in agent mode benchmark scoring, not something you're doing wrong on the setup side. Chat mode benchmarks grade by comparing Genie's generated SQL result set against your SQL ground truth, so "Empty result" there mea...

  • 1 kudos
GabeMatch
by New Contributor
  • 256 Views
  • 5 replies
  • 0 kudos

Lakeflow connect Native connectors (tik, meta ads, Google Ads) - one table per account

We want to leverage these connectors to pull in marketing spend data. But the docs seem to say that the destination must be unique based on accounts. For Tik, we have a hundred accounts... each account will have a destination table for each object.  ...

  • 256 Views
  • 5 replies
  • 0 kudos
Latest Reply
prafullkatiyar
New Contributor
  • 0 kudos

You can name a destination table when you create or edit your managed ingestion pipeline using Declarative Automation Bundles, notebooks, or the Databricks CLI, have a look at below doc. This should help.  Name a destination table | Databricks on AWS

  • 0 kudos
4 More Replies
hanifmusa
by New Contributor
  • 76 Views
  • 1 replies
  • 0 kudos

Getting error hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: The specified blob d

I am exporting parquet files (partitioned by id) in append mode. However, I encounter errors occasionally, while other times the job completes successfully.Apache Spark Exception: Exception thrown in awaitResult: hadoop_azure_shaded.com.microsoft.azu...

  • 76 Views
  • 1 replies
  • 0 kudos
Latest Reply
balajij8
Contributor III
  • 0 kudos

Its generally due to race conditions when Spark checks for existing partition files before writing combined with Azure Blob Storage's eventual consistency mode.You can follow below1. Switch to Delta Lake - You can use Delta Lake format instead of Par...

  • 0 kudos
Labels