I'm trying to move data from database A to B on Snowflake. There's no permission issue since using the Python package snowflake.connector works Databricks runtime version: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)Insert into database B fail...
Hello !We're currently building a pipeline of file ingestion using a Delta Live Tables pipeline and autoloader. The bronze tables are pretty much the following schema : file_name | file_upload_date | colA | colB (Well, there are actually 250+ columns...
Hi Team,I am looking for some advice to perf tune my bronze layer using DLT.I have the following code very simple and yet very effective. @dlt.create_table(name="bronze_events",
comment = "New raw data ingested from storage account ...
Hi @Gilg
You mentioned that micro-batch time is around 12 minutes recently. Do we also see jobs/stages with 12 minutes in the spark ui. If that is the case, then the processing of the file itself takes 12 minutes. If not, the 12 minutes is spent on ...
I tried to create a workspace and it is not generated either automatically or manually. The strange thing is that it stopped working after a certain time. It seems like an internal Databricks error but it is not known if it is like that or a bug, wha...
Hi @Kaviana , Thank you for posting your question in our community! We are happy to assist you.
To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
Hi all,Due to file size and file transfer limitation, we are receiving huge files compressed and split, in the format FILE.z01, FILE.z02,...,FILE.zipHowever, I can't find a way to unzip multipart files using databricks.I tried already some of the ...
Hi @N_M , Thank you for posting your question in our community! We are happy to assist you.
To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your qu...
When running a databricks notebook connected to an s3 cluster I randomly but frequently experience the following error: java.nio.file.AccessDeniedException: s3://mybucket: getFileStatus on s3://mybucket: com.amazonaws.services.s3.model.AmazonS3Except...
Hi @phoebe_dt , Thank you for posting your question in our community! We are happy to assist you.
To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
I'm seeing the access denied error from spark cluster while reading s3 file into notebook.Running on personal single user compute with LTS 13.3 ML.configs setup looks like this:spark.conf.set("spark.hadoop.fs.s3a.access.key", access_id)spark.conf.set...
Hi @Monika_Bagyal , The "Access Denied" error you are seeing is likely due to insufficient permissions to read the S3 bucket.
The configurations you've set up are correct for accessing S3 using temporary AWS credentials, but the credentials themse...
Hi Team,I have a DLT pipeline that uses APPLY_CHANGES to our Silver tables. I am using Id as keys and timestamp to know the sequence of the incoming data. Question: How does APPLY_CHANGES handles late arriving data?i.e., for silver_table_1, the data ...
Hi @Gilg , The APPLY_CHANGES function in Databricks Delta Live Tables handles late arriving data using a specified SEQUENCE BY column, which in your case is the timestamp. It uses this column to propagate appropriate sequencing values to the __START_...
Hi Databricks Academy team,I am getting below errors while running classroom setup command in Databricks Academy - Advanced data engineering course Notebooks in databricks community edition. Please help me resolve it. #databricksacademy #advanceddat...
Hi, all. Our apologies - the Advanced Data Engineering with Databricks course will not run on Databricks Community Edition. If you would like a lab environment to run this course on, please see the new paid lab subscription available via the Databric...
Hi everyone, I've been stuck for the past two days on this issue with my Databricks JDBC driver and I'm hoping someone can give me more insight into how to troubleshoot. I am using the Databricks JDBC driver in RStudio and the connection was working ...
@Debbie Ng​ From your message I see there was a windows update and this failure started. based on the conversation you tried latest version of the driver and still you face the problem. I believe this is something related to the Java version compatib...
I've been struggling with figuring out how to delete a managed delta live table. If I run a drop command in Databricks SQL I get:[STREAMING_TABLE_OPERATION_NOT_ALLOWED.DROP_DELTA_LIVE_TABLE] The operation DROP is not allowed: The operation does not a...
Hi @dbdude, To completely remove the underlying data of a Delta Live Table (DLT), you need to manually delete the data stored in the path. When you drop a table, only the metadata gets dropped and the underlying data remains untouched. The DROP TABLE...
Hi All,I am using Databricks Asset Bundles to deploy my code on github to databricks workspace. I have written out the Github Action as provided on databricks documentation.I have setup the personal access token for the service principal I want to us...
Finally, i was able to identify the missing piece. This was setting up the environment identifier for the runner. name: "Deploy bundle"
runs-on: ubuntu-latest
environment: ${{github.event.inputs.Environment}}With this, the action was able...
I am trying to ingest data into Databricks with Kafka. I have Kafka installed in a Virtual Machine where I already have the data I need in a Kafka Topic stored as json. In Databricks, I have the following code:```df = (spark.readStream
.format("kaf...
Hi @Pbarbosa154, We can build a thriving shared knowledge and insights community. Come back and accept the solution to contribute to our ongoing pursuit of excellence.
Now I can successfully export notebook view to HTML using job api (run export).However how can I export dashboard view which is generated by the tab of 'show in dashboard view' to HTML? the tab is on the right top of the corner in the cell.I want an ...
Hi @Kaniz ! I am wondering how to export a dashboard tab from a notebook and I found this old topic.I am not able to find the "views_to_export" parameter that @Hubert-Dudek refers to in docs. May the docs have been updated and now the parameter is ca...
Hi TeamI have given Databricks certified data engineer associate exam, but it got suspended before completion. I showed my exam area to proctor accordingly. I got message by proctor that my exam got suspended due to your failure to meet certain envir...
