cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
orso
by • New Contributor III
  • 9896 Views
  • 1 replies
  • 0 kudos

Resolved! Java - FAILED_WITH_ERROR when saving to snowflake

I'm trying to move data from database A to B on Snowflake. There's no permission issue since using the Python package snowflake.connector  works Databricks runtime version: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)Insert into database B fail...

Data Engineering
Reply
  • 9896 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
orso
New Contributor III
  • 0 kudos

Found the problem. The sub-roles didn't have grants to the warehouse.I hope it will help someone one day

  • 0 kudos
sanjay
by • Valued Contributor II
  • 2571 Views
  • 1 replies
  • 0 kudos

Trigger Events in data pipeline

Hi,I am running datapipeline in databrick using matillion architecture. I am facing inconsistent events in silver to gold layer in case any row deleted/updated from a partition. Let me explain with example.e.g. I have data in silver layer with partit...

Data Engineering
Reply
  • 2571 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sanjay
Valued Contributor II
  • 0 kudos

Thank you Kaniz. Further queries on this.1. If I have nested partitions e.g. on department & date, finance->09, finance->10 and if am updating one record in finance->09 then will then updates partition finance->10 as well2. Is it good idea to have sm...

  • 0 kudos
erigaud
by • Honored Contributor
  • 6138 Views
  • 4 replies
  • 5 kudos

Resolved! DLT overwrite part of the table

Hello !We're currently building a pipeline of file ingestion using a Delta Live Tables pipeline and autoloader. The bronze tables are pretty much the following schema : file_name | file_upload_date | colA | colB (Well, there are actually 250+ columns...

Data Engineering
Reply
  • 6138 Views
  • 4 replies
  • 5 kudos
View Replies
Latest Reply
Tharun-Kumar
Databricks Employee
  • 5 kudos

@erigaud  Using jobs/workflows would be the right choice for this.

  • 5 kudos
3 More Replies
Gilg
by • Contributor II
  • 2839 Views
  • 3 replies
  • 1 kudos

DLT: Autoloader Perf

Hi Team,I am looking for some advice to perf tune my bronze layer using DLT.I have the following code very simple and yet very effective. @dlt.create_table(name="bronze_events", comment = "New raw data ingested from storage account ...

Gilg_0-1696561163925.png
Data Engineering
Reply
  • 2839 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Tharun-Kumar
Databricks Employee
  • 1 kudos

Hi @Gilg  You mentioned that micro-batch time is around 12 minutes recently. Do we also see jobs/stages with 12 minutes in the spark ui. If that is the case, then the processing of the file itself takes 12 minutes. If not, the 12 minutes is spent on ...

  • 1 kudos
2 More Replies
Gilg
by • Contributor II
  • 2250 Views
  • 1 replies
  • 1 kudos

APPLY_CHANGES late arriving data

Hi Team,I have a DLT pipeline that uses APPLY_CHANGES to our Silver tables. I am using Id as keys and timestamp to know the sequence of the incoming data. Question: How does APPLY_CHANGES handles late arriving data?i.e., for silver_table_1, the data ...

Data Engineering
Reply
  • 2250 Views
  • 1 replies
  • 1 kudos
View Replies
Monika_Bagyal
by • New Contributor
  • 15659 Views
  • 0 replies
  • 0 kudos

Access denied error while reading file from S3 to spark

I'm seeing the access denied error from spark cluster while reading s3 file into notebook.Running on personal single user compute with LTS 13.3 ML.configs setup looks like this:spark.conf.set("spark.hadoop.fs.s3a.access.key", access_id)spark.conf.set...

Data Engineering
Reply
  • 15659 Views
  • 0 replies
  • 0 kudos
PradyumnJoshi
by • New Contributor
  • 1850 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks Academy - Advanced Data Engineering - Notebook Error while loading configurations

Hi Databricks Academy team,I am getting below errors while running classroom setup command in Databricks Academy - Advanced data engineering course Notebooks in  databricks community edition. Please help me resolve it. #databricksacademy #advanceddat...

PradyumnJoshi_1-1696489013813.png PradyumnJoshi_0-1696488924057.png
Data Engineering
Reply
  • 1850 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16847923431
Databricks Employee
  • 0 kudos

Hi, all. Our apologies - the Advanced Data Engineering with Databricks course will not run on Databricks Community Edition. If you would like a lab environment to run this course on, please see the new paid lab subscription available via the Databric...

  • 0 kudos
RiyuLite
by • New Contributor III
  • 3172 Views
  • 3 replies
  • 2 kudos

Where do I get Account level logs after enabling diagnostic logs for Azure databricks?

I need to retrieve the accountBillage usage from Audit logsI have enabled Diagnostic logs, and it's been 36 hours. While enabling the logs , I selected every possible logs in this image. But still i am not able to see the containers for account level...

RiyuLite_3-1696492947786.png RiyuLite_2-1696492911080.png
Data Engineering
Reply
  • 3172 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
RiyuLite
New Contributor III
  • 2 kudos

Hi @Retired_mod , I checked Azure Monitoring and log delivery documentations, The log delivery is same as workspace level.What is the procedure to enable account level service in audit logs for Azure ? 

  • 2 kudos
2 More Replies
naga_databricks
by • Contributor
  • 3690 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks asset bundles deployment to development

Hi All,I am using Databricks Asset Bundles to deploy my code on github to databricks workspace. I have written out the Github Action as provided on databricks documentation.I have setup the personal access token for the service principal I want to us...

Data Engineering
asset_bundles
Reply
  • 3690 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
naga_databricks
Contributor
  • 0 kudos

Finally, i was able to identify the missing piece. This was setting up the environment identifier for the runner.  name: "Deploy bundle" runs-on: ubuntu-latest environment: ${{github.event.inputs.Environment}}With this, the action was able...

  • 0 kudos
N_M
by • Contributor
  • 2224 Views
  • 0 replies
  • 0 kudos

Unzip multipart files

Hi all,Due to file size and file transfer limitation, we are receiving huge files compressed and split, in the format    FILE.z01, FILE.z02,...,FILE.zipHowever, I can't find a way to unzip multipart files using databricks.I tried already some of the ...

Data Engineering
bash
unzip
Reply
  • 2224 Views
  • 0 replies
  • 0 kudos
Kaviana
by • New Contributor III
  • 2211 Views
  • 0 replies
  • 0 kudos

internal server error when creating workspace

I tried to create a workspace and it is not generated either automatically or manually. The strange thing is that it stopped working after a certain time. It seems like an internal Databricks error but it is not known if it is like that or a bug, wha...

Data Engineering
Reply
  • 2211 Views
  • 0 replies
  • 0 kudos
dbickshammer
by • New Contributor II
  • 5135 Views
  • 2 replies
  • 4 kudos

Resolved! how can I export dashboard to HTML?

Now I can successfully export notebook view to HTML using job api (run export).However how can I export dashboard view which is generated by the tab of 'show in dashboard view' to HTML? the tab is on the right top of the corner in the cell.I want an ...

Data Engineering
Reply
  • 5135 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
kyxam
New Contributor II
  • 4 kudos

Hi @Retired_mod ! I am wondering how to export a dashboard tab from a notebook and I found this old topic.I am not able to find the "views_to_export" parameter that @Hubert-Dudek refers to in docs. May the docs have been updated and now the parameter...

  • 4 kudos
1 More Replies
Divs23
by • New Contributor II
  • 1888 Views
  • 2 replies
  • 0 kudos

Resolved! databricks certified data engineer associate exam got suspended

Hi TeamI have given Databricks certified data engineer associate exam, but it got suspended before completion. I showed my exam area to proctor accordingly. I got message by proctor that my exam got suspended due to your failure to meet certain envir...

Data Engineering
Reply
  • 1888 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Cert-Team
Databricks Employee
  • 0 kudos

@Divs23 The support team is working on your ticket now, please be sure to check your spam folder. 

  • 0 kudos
1 More Replies
Sanjay96m
by • New Contributor
  • 1411 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks Certification exam Suspended. Need Assistance

I was taking online exam for Databricks Certified Data Analyst Associate on 06-Oct-2023 1:45PM. In between, they paused it and wanted to survey my whole room which they did, told me to clear the table of water bottle and laptop charger and then asked...

Data Engineering
Reply
  • 1411 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Cert-Team
Databricks Employee
  • 0 kudos

@Sanjay96m Thank you for your patience, the support team is working through support tickets and will reach out to you shortlly.

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels