- 3294 Views
- 1 replies
- 0 kudos
Trigger Events in data pipeline
Hi,I am running datapipeline in databrick using matillion architecture. I am facing inconsistent events in silver to gold layer in case any row deleted/updated from a partition. Let me explain with example.e.g. I have data in silver layer with partit...
- 3294 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you Kaniz. Further queries on this.1. If I have nested partitions e.g. on department & date, finance->09, finance->10 and if am updating one record in finance->09 then will then updates partition finance->10 as well2. Is it good idea to have sm...
- 0 kudos
- 8840 Views
- 4 replies
- 5 kudos
Resolved! DLT overwrite part of the table
Hello !We're currently building a pipeline of file ingestion using a Delta Live Tables pipeline and autoloader. The bronze tables are pretty much the following schema : file_name | file_upload_date | colA | colB (Well, there are actually 250+ columns...
- 8840 Views
- 4 replies
- 5 kudos
- 4017 Views
- 3 replies
- 1 kudos
DLT: Autoloader Perf
Hi Team,I am looking for some advice to perf tune my bronze layer using DLT.I have the following code very simple and yet very effective. @dlt.create_table(name="bronze_events", comment = "New raw data ingested from storage account ...
- 4017 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Gilg You mentioned that micro-batch time is around 12 minutes recently. Do we also see jobs/stages with 12 minutes in the spark ui. If that is the case, then the processing of the file itself takes 12 minutes. If not, the 12 minutes is spent on ...
- 1 kudos
- 4565 Views
- 1 replies
- 1 kudos
APPLY_CHANGES late arriving data
Hi Team,I have a DLT pipeline that uses APPLY_CHANGES to our Silver tables. I am using Id as keys and timestamp to know the sequence of the incoming data. Question: How does APPLY_CHANGES handles late arriving data?i.e., for silver_table_1, the data ...
- 4565 Views
- 1 replies
- 1 kudos
- 17704 Views
- 0 replies
- 0 kudos
Access denied error while reading file from S3 to spark
I'm seeing the access denied error from spark cluster while reading s3 file into notebook.Running on personal single user compute with LTS 13.3 ML.configs setup looks like this:spark.conf.set("spark.hadoop.fs.s3a.access.key", access_id)spark.conf.set...
- 17704 Views
- 0 replies
- 0 kudos
- 2652 Views
- 1 replies
- 0 kudos
Resolved! Databricks Academy - Advanced Data Engineering - Notebook Error while loading configurations
Hi Databricks Academy team,I am getting below errors while running classroom setup command in Databricks Academy - Advanced data engineering course Notebooks in databricks community edition. Please help me resolve it. #databricksacademy #advanceddat...
- 2652 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, all. Our apologies - the Advanced Data Engineering with Databricks course will not run on Databricks Community Edition. If you would like a lab environment to run this course on, please see the new paid lab subscription available via the Databric...
- 0 kudos
- 4607 Views
- 3 replies
- 2 kudos
Where do I get Account level logs after enabling diagnostic logs for Azure databricks?
I need to retrieve the accountBillage usage from Audit logsI have enabled Diagnostic logs, and it's been 36 hours. While enabling the logs , I selected every possible logs in this image. But still i am not able to see the containers for account level...
- 4607 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Retired_mod , I checked Azure Monitoring and log delivery documentations, The log delivery is same as workspace level.What is the procedure to enable account level service in audit logs for Azure ?
- 2 kudos
- 5593 Views
- 1 replies
- 0 kudos
Resolved! Databricks asset bundles deployment to development
Hi All,I am using Databricks Asset Bundles to deploy my code on github to databricks workspace. I have written out the Github Action as provided on databricks documentation.I have setup the personal access token for the service principal I want to us...
- 5593 Views
- 1 replies
- 0 kudos
- 0 kudos
Finally, i was able to identify the missing piece. This was setting up the environment identifier for the runner. name: "Deploy bundle" runs-on: ubuntu-latest environment: ${{github.event.inputs.Environment}}With this, the action was able...
- 0 kudos
- 3316 Views
- 0 replies
- 0 kudos
Unzip multipart files
Hi all,Due to file size and file transfer limitation, we are receiving huge files compressed and split, in the format FILE.z01, FILE.z02,...,FILE.zipHowever, I can't find a way to unzip multipart files using databricks.I tried already some of the ...
- 3316 Views
- 0 replies
- 0 kudos
- 2998 Views
- 0 replies
- 0 kudos
internal server error when creating workspace
I tried to create a workspace and it is not generated either automatically or manually. The strange thing is that it stopped working after a certain time. It seems like an internal Databricks error but it is not known if it is like that or a bug, wha...
- 2998 Views
- 0 replies
- 0 kudos
- 7261 Views
- 2 replies
- 4 kudos
Resolved! how can I export dashboard to HTML?
Now I can successfully export notebook view to HTML using job api (run export).However how can I export dashboard view which is generated by the tab of 'show in dashboard view' to HTML? the tab is on the right top of the corner in the cell.I want an ...
- 7261 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Retired_mod ! I am wondering how to export a dashboard tab from a notebook and I found this old topic.I am not able to find the "views_to_export" parameter that @Hubert-Dudek refers to in docs. May the docs have been updated and now the parameter...
- 4 kudos
- 3171 Views
- 2 replies
- 0 kudos
Resolved! databricks certified data engineer associate exam got suspended
Hi TeamI have given Databricks certified data engineer associate exam, but it got suspended before completion. I showed my exam area to proctor accordingly. I got message by proctor that my exam got suspended due to your failure to meet certain envir...
- 3171 Views
- 2 replies
- 0 kudos
- 0 kudos
@Divs23 The support team is working on your ticket now, please be sure to check your spam folder.
- 0 kudos
- 2146 Views
- 1 replies
- 0 kudos
Resolved! Databricks Certification exam Suspended. Need Assistance
I was taking online exam for Databricks Certified Data Analyst Associate on 06-Oct-2023 1:45PM. In between, they paused it and wanted to survey my whole room which they did, told me to clear the table of water bottle and laptop charger and then asked...
- 2146 Views
- 1 replies
- 0 kudos
- 0 kudos
@Sanjay96m Thank you for your patience, the support team is working through support tickets and will reach out to you shortlly.
- 0 kudos
- 2366 Views
- 0 replies
- 0 kudos
Change Data Feed - Selected Columns
Is there a way to configure CDF in delta table but only for selected columns and not all?
- 2366 Views
- 0 replies
- 0 kudos
- 4183 Views
- 1 replies
- 1 kudos
Resolved! Databricks 13.3 Runtime Unable to use Python OS library to access mounted GCP bucket
In Databricks 12.2, this Python code block lists out the contents of our ad_hoc folder in our mounted GCP bucket. import os os.listdir('/dbfs/mnt/hlm/ad_hoc/') For some reason in 13.3 this same code block throws a " No such file or directory" error.
- 4183 Views
- 1 replies
- 1 kudos
- 1 kudos
I've discovered the cause of this issue. The path is functional, but what I'm actually running into is a restriction due to the cluster set to "Shared" access mode:Cannot use R, RDD APIs, or clients that directly read the data from cloud storage, suc...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
