Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
I have a pipeline that generates two DLT streaming tables: a Bronze table and a Silver table. I need to delete specific records from both tables. I've read an article (https://www.databricks.com/blog/handling-right-be-forgotten-gdpr-and-ccpa-using-de...
Remove records using the DELETE operation in both Bronze & Silver tables.After doing each delete step, you can Optimize the table which rewrites the parquet files for that table behind the scenes to improve the data layout (Read more about optimize h...
I have this shell unzip that I use to unzip files %shsudo apt-get updatesudo apt-get install -y p7zip-full But when it comes to new workspace, I get error sudo: a terminal is required to read the password; either use the -S option to read from standa...
First, you can read the ZIP file in a binary format [ spark.read.format("binaryFile") ], then use the zipfile Python package to unzip and extract all the files from the zipped file and store them in a Volume.
Hi @mmceld1,
Autoloader is for cloud storage files. You can achieve similar functionality by using Delta Lake and its capabilities for handling slowly changing dimensions (SCD Type 2) and change data capture (CDC)
I have a job to move notebook from Volume to workspace, then execute it with dbutils.notebook.run(). Instead of directly running the notebook, i want to append some logic (i.e. Save results to a certain able) at the end of the notebook, is there a su...
Hi @lauraxyz,
Currently, there is no built-in feature in Databricks that directly supports appending logic to a notebook before execution, so treating the notebook as a regular file and modifying its content is a practical solution.
I'm encountering an issue with incomplete Spark event logs. When I am running the local Spark History Server using the cluster logs, my application appears as "incomplete". Sometime I also see few queries listed as still running, even though the appl...
Thanks for your question!
I believe Databricks has its own SHS implementation, so it's not expected to work with the vanilla SHS. Regarding the queries marked as still running, we can also find this when there are event logs which were not properly c...
I dont have the complete context of the issue.But Here it is what I know, a friend of mine facing this""I am fetching data from Oracle data in databricks using python.But every time i do it the schema gets changesso if the column is of type decimal f...
Thanks for your question!To address schema issues when fetching Oracle data in Databricks, use JDBC schema inference to define data types programmatically or batch-cast columns dynamically after loading. For performance, enable predicate pushdown and...
I need to increase the stack size (from the default of 16384) to run a subprocess that requires a larger stack size.I tried following this: https://community.databricks.com/t5/data-engineering/increase-stack-size-databricks/td-p/71492And this: https:...
Facing issues while Importing dlt library in Databricks Runtime 14.3 LTS. Previously while using the Runtime 13.1 The `import dlt` was working fine but now when updating the Runtime version to 14.3 LTS it is giving me error.
While DLT has some powerful features, I found myself doing a double-take when I realized it doesn’t natively support hard deletes. Instead, it leans on a delete flag identifier to manage these in the source table. A bit surprising for a tool of its c...
Thanks for your feedback !
I believe, Delta Live Tables (DLT) does not natively support hard deletes and instead uses a delete flag identifier to manage deletions, a design choice rooted in ensuring compliance with regulations like GDPR and CCPA. Thi...
I am experiencing performance issues when loading a table with 50 million rows into Delta Lake on AWS using Databricks. Despite successfully handling other larger tables, this especific table/process takes hours and doesn't finish. Here's the command...
Thank you for your question! To optimize your Delta Lake write process:
Disable Overhead Options: Avoid overwriteSchema and mergeSchema unless necessary. Use:
df.write.format("delta").mode("overwrite").save(sink)
Increase Parallelism: Use repartition...
Hello,I have a job that should run every six hours. I need to set up an alert for the case if this doesn't start (for example, someone paused it). How do I configure such an alert using Databricks native alerts?Theoretically, this may be done using s...
Thank you for your question! Here’s a concise workflow to set up an alert for missed job runs in Databricks:
Write a Query: Use system tables to identify jobs that haven’t started on time.Save the Query: Save this query in Databricks SQL as a named q...
Is it possible to install our own binaries (lib or exec) in Databricks clusters and use JNI to execute them?I guess that Photon is native code as far as I could read so it must use a similar technic.
Thanks for your question!
I believe it should be possible, although Photon itself is not extensible by users. Are you currently facing any issues while installing and using your own libraries, and JNI to execute them?
Is Databricks S3 Commit Service enabled by default if Unity Catalog is not enabled and the compute resources run in our AWS account (classic compute plane)? If not, how can it be enabled?This service seems to resolve the limitations with multi-cluste...
No, the Databricks S3 commit service is not guaranteed to be enabled by default in the AWS classic compute plane. The configuration may vary based on your specific workspace setup.
How can it be enabled?
To enable the Databricks S3 commit service, ...
I've the filename as below and I want to extract the datetime values and convert to datetime data type. This_is_new_file_2024_12_06T11_00_49_AM.csvHere I want to extract only '2024_12_06T11_00_49' and convert to datetime value in new field. I tried S...
I'm new to Databricks and, I'm looking to get data from an external database into Databricks and keep it synchronized when changes occur in the source tables. It seems like I may be able to some form of change data capture and the delta live tables. ...
To synchronize data from an external database into Databricks with change data capture (CDC), you can use Delta Live Tables (DLT). Start by configuring a JDBC connection to your source database and use a CDC tool (like Debezium or database-native CDC...
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
