cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ruicarvalho_de
by New Contributor III
  • 1541 Views
  • 8 replies
  • 0 kudos

Resolved! Databricks API - Get Dashboard Owner?

Hi all!I'm trying to identify the owner of a dashboard using the API.Here's a code snippet as an example:import json dashboard_id = "XXXXXXXXXXXXXXXXXXXXXXXXXX" url = f"{workspace_url}/api/2.0/lakeview/dashboards/{dashboard_id}" headers = {"Authoriz...

  • 1541 Views
  • 8 replies
  • 0 kudos
Latest Reply
ruicarvalho_de
New Contributor III
  • 0 kudos

Hello, meanwhile, I've found a workaround to get the owner, or more likely the creator of the dashboard.SELECT request_params.dashboard_id AS dashboard_id, FIRST(user_identity.email) AS owner_email FROM system.access....

  • 0 kudos
7 More Replies
maikel
by Contributor III
  • 920 Views
  • 3 replies
  • 2 kudos

Resolved! The best way to write and debug python notebooks

Hello Community,do you have a suggestion, what is the best way to develop python notebooks in databricks? I know that it can be done directly in databricks UI but for more complex projects where I would like to split notebook and import functions fro...

  • 920 Views
  • 3 replies
  • 2 kudos
Latest Reply
saurabh18cs
Honored Contributor III
  • 2 kudos

Hi @maikel for us databricks extension on vs code to deploy workflow using asset bundles works like a charm. if plugin is not helping out then i suggest you to install databricks CLI separately or point to the one installed together with databricks e...

  • 2 kudos
2 More Replies
liquibricks
by Databricks Partner
  • 900 Views
  • 2 replies
  • 1 kudos

Resolved! SDP: Your table schema requires manually enablement of the following table feature(s): timestampNtz.

Trying to create a Spark Declarative Pipeline with a Materialized View based on a Streaming Table. The Streaming Table works fine, but when trying to add a Materialized View the pipeline fails to initialize with:Your table schema requires manually en...

  • 900 Views
  • 2 replies
  • 1 kudos
Latest Reply
stbjelcevic
Databricks Employee
  • 1 kudos

Hi @liquibricks , I see you said that the materialized view does not exist, so you can't run the ALTER TABLE command. Try to add TBLPROPERTIES ('delta.feature.timestampNtz'='supported') to the MV CREATE statement (or pass it via table_properties in y...

  • 1 kudos
1 More Replies
alesventus
by Contributor
  • 3192 Views
  • 4 replies
  • 3 kudos

Resolved! How to handle load of 300 tables to delta lake

My task is to sync 300 tables from on prem sql server to delta lake. I will load CDC from Raw. First step is to move CDC data to bronze with autoloader. Then using delta stream get changes from bronze, make simple datatype changes and merge this data...

  • 3192 Views
  • 4 replies
  • 3 kudos
Latest Reply
Marc_Gibson96
Contributor
  • 3 kudos

Hi Alesventus,The approach suggested by syzmon_dybczak is definitely the same direction I would go in. To supplement their answer, have a look at this Databricks Labs repository that you may find useful as inspiration (DLT-Meta): https://github.com/d...

  • 3 kudos
3 More Replies
bomb_omb
by New Contributor II
  • 1412 Views
  • 1 replies
  • 0 kudos

Resolved! Implementing Ruff linter in Gitlab CI/CD for Databrick notebooks

Hi all, I am trying to implement ruff linting in Gitlab CI/CD where I am only trying to catch undefined name errors ["F821"]. However it is picking up my magic commands like %sql and %pip as syntax errors and I am unable to exclude it in the pyprojec...

  • 1412 Views
  • 1 replies
  • 0 kudos
Latest Reply
saurabh18cs
Honored Contributor III
  • 0 kudos

you're using ruff for linting. The issue with Databricks magic commands (%sql, %pip, %run) is that they're Databricks-specific syntax, not standard Python, so ruff treats them as errors. you should exclude databricks notebook from ruff.

  • 0 kudos
liquibricks
by Databricks Partner
  • 1996 Views
  • 11 replies
  • 7 kudos

Resolved! Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

We have a Lakeflow Spark Declarative Pipeline using the new PySpark Pipelines API. This was working fine until about 7am (Central European) this morning when the pipeline started failing with a PYTHON.NAME_ERROR: name 'kdf' is not defined. Did you me...

  • 1996 Views
  • 11 replies
  • 7 kudos
Latest Reply
yassine_eal
New Contributor III
  • 7 kudos

 Hi, I ran into the same error starting late last week.The issue was caused by dependency version conflicts between my custom package and the libraries preinstalled on the Databricks serverless environment.I fixed it by pinning all my package depende...

  • 7 kudos
10 More Replies
soumitradutta
by New Contributor
  • 512 Views
  • 1 replies
  • 1 kudos

Resolved! Soumitra Dutta : How do I debug exceptions in Databricks notebooks?

Hello,My name is Soumitra Dutta,I’m currently in the USA and running into some exceptions in my Databricks notebook. For anyone who has dealt with similar issues, what steps do you usually take to debug them? I’d really appreciate your suggestions.” 

  • 512 Views
  • 1 replies
  • 1 kudos
Latest Reply
CerberusByte
Databricks Employee
  • 1 kudos

Hi Soumitra, this is quite a broad question as it depends on what kind of error you are seeing. If there is something specific, please reply here for something more tailored. In general, when debugging I usually start with the Databricks Assistant as...

  • 1 kudos
Indika_debnath
by New Contributor II
  • 7351 Views
  • 10 replies
  • 0 kudos

Databricks Certification voucher not received

Hello team,I have attended the webinar Databricks Certification Overview Series- Data Engineer on Jan 17Completed the Databricks Lakehouse fundamentals accreditation and Completed the survey.As per communication it is expected that I will receive Dat...

  • 7351 Views
  • 10 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Indika Debnath​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
9 More Replies
AJ270990
by Contributor III
  • 1227 Views
  • 8 replies
  • 4 kudos

Resolved! Transfer of Jobs/ETL Pipelines/Workflows/Workspace Notebooks from One Subscription to another sub

We need to transfer the Jobs/ETL Pipelines/Workflows/Workspace Notebooks from One Azure Subscription to another Azure subscription. Manual way of exporting the notebook and jobs is not feasible as we have around 100s of notebook and workflows. Sugges...

  • 1227 Views
  • 8 replies
  • 4 kudos
Latest Reply
pradeep_singh
Contributor III
  • 4 kudos

If you dont have these resource in dabs already writing/testing configuration might be a good amount of work . with terraform exporter utility you can export all the resource from one workspace as terraform code and deploy it to new workspace quite e...

  • 4 kudos
7 More Replies
hietpas
by Databricks Partner
  • 1262 Views
  • 3 replies
  • 2 kudos

Resolved! Serverless: SparkConnectGrpcException: 403 Forbidden: 403: Invalid access token.

I am running a Databricks job with a runas principal using Serverless compute. The job has a single notebook task. The notebook runs successfully on a job compute or when I run (as me) in the Workspace (still using serverless compute). However, I get...

hietpas_0-1769632818999.png
  • 1262 Views
  • 3 replies
  • 2 kudos
Latest Reply
hietpas
Databricks Partner
  • 2 kudos

Thanks @MoJaMa!Originally, I attempted the grant using a SQL Warehouse and it failed. However, I ran again in a notebook using serverless compute and it succeeded. This does appear to resolve the original issue.On a side note, the original error was ...

  • 2 kudos
2 More Replies
IMDT
by New Contributor
  • 1669 Views
  • 2 replies
  • 0 kudos

"Socket Closed" Can't Connect to Oracle DB from DBX Connection

I want to create a LakeHouse Federation to query data to a remote oracle DB.  I did the test connection to an Oracle DB to create the foreign catalog but keep running into socket closed error using compute.  Which is best Compute or WH?In Compute usi...

  • 1669 Views
  • 2 replies
  • 0 kudos
Latest Reply
luketl2
Contributor
  • 0 kudos

Would also like to know this as I am facing the same issue and there's really no troubleshooting I can do myself as a third-party owns the RDS instance. When testing the connection it just loads for a while and then says "socket closed". From my unde...

  • 0 kudos
1 More Replies
fly_high_five
by Contributor
  • 2429 Views
  • 5 replies
  • 7 kudos

Resolved! Unable to retrieve catalog, schema, tables using JDBC endpoint of SQL Warehouse

Hi,I am connecting to SQL Warehouse in UC using its JDBC endpoint via DBeaver. However, it doesn't list any catalogs, schemas and tables. I checked the permission of SQL WH by logging to ADB Workspace and queried the table (attached a dummy table exa...

fly_high_five_0-1764770250626.png fly_high_five_1-1764770371607.png fly_high_five_2-1764770788643.png
  • 2429 Views
  • 5 replies
  • 7 kudos
Latest Reply
Commitchell
Databricks Employee
  • 7 kudos

Strange! That sounds like it might be a DBeaver limitation. I'm glad you got it working and found a workaround!

  • 7 kudos
4 More Replies
Kirankumarbs
by Contributor III
  • 556 Views
  • 2 replies
  • 2 kudos

Resolved! any open-source library or init script for exporting Databricks job metrics to Prometheus

Hello Community,I’m exploring ways to export the default metrics from Databricks Jobs/Workflows to Prometheus and visualize them in Grafana.There’s a nice option to use Databricks as a data source in Grafana, but that’s available only in the Enterpri...

  • 556 Views
  • 2 replies
  • 2 kudos
Latest Reply
Raman_Unifeye
Honored Contributor III
  • 2 kudos

@Kirankumarbs If you jst want to track Job status Success/Failure, Run Duration, and Costs, use exporter that talks to the Databricks Workspace API or System Tables.grafana/databricks-prometheus-exporter - This is an open-source exporter. It queries ...

  • 2 kudos
1 More Replies
NotCuriosAtAll
by New Contributor III
  • 607 Views
  • 2 replies
  • 4 kudos

Resolved! Non deterministic behavior from the cluster

I asked this question a while ago where I explain the cluster that my team uses on databricks. To save you some time, we use an all-purpose Standard D2ads v6 with 8 gigs of ram and 2 cores cluster. We are facing an issue with the memory, which is pin...

  • 607 Views
  • 2 replies
  • 4 kudos
Latest Reply
pradeep_singh
Contributor III
  • 4 kudos

How to explain it to the client - The job is operating at the resource ceiling of a very small driver. Tiny, normal day‑to‑day differences (file layout, plan choice, GC timing) sometimes push it over the limit, which is why restarts occasionally “fix...

  • 4 kudos
1 More Replies
luketl2
by Contributor
  • 1529 Views
  • 6 replies
  • 4 kudos

Resolved! CSV Ingestion using Autoloader with single variant column

I've been working on ingesting csv files with varying schemas using Autoloader. Goal is to take the csvs and ingest them into a bronze table that writes each record as a key-value mapping with only the relevant fields for that record. I also want to ...

  • 1529 Views
  • 6 replies
  • 4 kudos
Latest Reply
pradeep_singh
Contributor III
  • 4 kudos

Cheers !! 

  • 4 kudos
5 More Replies
Labels