- 9350 Views
- 13 replies
- 10 kudos
Suddenly can't find the option to uplaod files into Databricks Community Edition
Hi everyone,I am suddenly unable to find the option to upload my files into Databricks Community Edition today. Please find the same in the screenshot attached. Is anyone else also facing this issue?
- 9350 Views
- 13 replies
- 10 kudos
- 10 kudos
GChandra, did it get removed again? I just signed up for Community free edition, and I don't see an option to Upload files. Please advise.
- 10 kudos
- 2594 Views
- 3 replies
- 1 kudos
Deleting Records from DLT Bronze and Silver Tables
I have a pipeline that generates two DLT streaming tables: a Bronze table and a Silver table. I need to delete specific records from both tables. I've read an article (https://www.databricks.com/blog/handling-right-be-forgotten-gdpr-and-ccpa-using-de...
- 2594 Views
- 3 replies
- 1 kudos
- 1 kudos
Remove records using the DELETE operation in both Bronze & Silver tables.After doing each delete step, you can Optimize the table which rewrites the parquet files for that table behind the scenes to improve the data layout (Read more about optimize h...
- 1 kudos
- 5602 Views
- 2 replies
- 0 kudos
Volumes unzip files
I have this shell unzip that I use to unzip files %shsudo apt-get updatesudo apt-get install -y p7zip-full But when it comes to new workspace, I get error sudo: a terminal is required to read the password; either use the -S option to read from standa...
- 5602 Views
- 2 replies
- 0 kudos
- 0 kudos
First, you can read the ZIP file in a binary format [ spark.read.format("binaryFile") ], then use the zipfile Python package to unzip and extract all the files from the zipped file and store them in a Volume.
- 0 kudos
- 954 Views
- 1 replies
- 1 kudos
Resolved! Does Autoloader Detect New Records in a Snowflake Table or Only Work With Files?
The only thing I can find with autoloader is picking up new files, nothing about new records in an existing snowflake table.
- 954 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @mmceld1, Autoloader is for cloud storage files. You can achieve similar functionality by using Delta Lake and its capabilities for handling slowly changing dimensions (SCD Type 2) and change data capture (CDC)
- 1 kudos
- 1108 Views
- 1 replies
- 1 kudos
Resolved! Programmatically edit notebook
I have a job to move notebook from Volume to workspace, then execute it with dbutils.notebook.run(). Instead of directly running the notebook, i want to append some logic (i.e. Save results to a certain able) at the end of the notebook, is there a su...
- 1108 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @lauraxyz, Currently, there is no built-in feature in Databricks that directly supports appending logic to a notebook before execution, so treating the notebook as a regular file and modifying its content is a practical solution.
- 1 kudos
- 5415 Views
- 1 replies
- 1 kudos
Not able to get spark application in Spark History server using cluster eventlogs
I'm encountering an issue with incomplete Spark event logs. When I am running the local Spark History Server using the cluster logs, my application appears as "incomplete". Sometime I also see few queries listed as still running, even though the appl...
- 5415 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for your question! I believe Databricks has its own SHS implementation, so it's not expected to work with the vanilla SHS. Regarding the queries marked as still running, we can also find this when there are event logs which were not properly c...
- 1 kudos
- 4367 Views
- 1 replies
- 0 kudos
Schema issue while fetching data from oracle
I dont have the complete context of the issue.But Here it is what I know, a friend of mine facing this""I am fetching data from Oracle data in databricks using python.But every time i do it the schema gets changesso if the column is of type decimal f...
- 4367 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your question!To address schema issues when fetching Oracle data in Databricks, use JDBC schema inference to define data types programmatically or batch-cast columns dynamically after loading. For performance, enable predicate pushdown and...
- 0 kudos
- 4441 Views
- 1 replies
- 0 kudos
Increase Stack Size for Python Subprocess
I need to increase the stack size (from the default of 16384) to run a subprocess that requires a larger stack size.I tried following this: https://community.databricks.com/t5/data-engineering/increase-stack-size-databricks/td-p/71492And this: https:...
- 4441 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your question!Are you referring to a Java stack size (-Xss) or a Python subprocess (ulimit -s)?
- 0 kudos
- 4471 Views
- 1 replies
- 0 kudos
Facing Issue in "import dlt" using Databricks Runtime 14.3 LTS version
Facing issues while Importing dlt library in Databricks Runtime 14.3 LTS. Previously while using the Runtime 13.1 The `import dlt` was working fine but now when updating the Runtime version to 14.3 LTS it is giving me error.
- 4471 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your question! Unfortunately, this is actually a known limitation with Spark Connect clusters.
- 0 kudos
- 4544 Views
- 1 replies
- 1 kudos
A Surprise Findings in Delta Live Table
While DLT has some powerful features, I found myself doing a double-take when I realized it doesn’t natively support hard deletes. Instead, it leans on a delete flag identifier to manage these in the source table. A bit surprising for a tool of its c...
- 4544 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for your feedback ! I believe, Delta Live Tables (DLT) does not natively support hard deletes and instead uses a delete flag identifier to manage deletions, a design choice rooted in ensuring compliance with regulations like GDPR and CCPA. Thi...
- 1 kudos
- 4588 Views
- 1 replies
- 0 kudos
Infinity load execution
I am experiencing performance issues when loading a table with 50 million rows into Delta Lake on AWS using Databricks. Despite successfully handling other larger tables, this especific table/process takes hours and doesn't finish. Here's the command...
- 4588 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you for your question! To optimize your Delta Lake write process: Disable Overhead Options: Avoid overwriteSchema and mergeSchema unless necessary. Use: df.write.format("delta").mode("overwrite").save(sink) Increase Parallelism: Use repartition...
- 0 kudos
- 4815 Views
- 1 replies
- 0 kudos
Job scheduled run fail alert
Hello,I have a job that should run every six hours. I need to set up an alert for the case if this doesn't start (for example, someone paused it). How do I configure such an alert using Databricks native alerts?Theoretically, this may be done using s...
- 4815 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you for your question! Here’s a concise workflow to set up an alert for missed job runs in Databricks: Write a Query: Use system tables to identify jobs that haven’t started on time.Save the Query: Save this query in Databricks SQL as a named q...
- 0 kudos
- 4372 Views
- 1 replies
- 0 kudos
Native code in Databricks clusters
Is it possible to install our own binaries (lib or exec) in Databricks clusters and use JNI to execute them?I guess that Photon is native code as far as I could read so it must use a similar technic.
- 4372 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your question! I believe it should be possible, although Photon itself is not extensible by users. Are you currently facing any issues while installing and using your own libraries, and JNI to execute them?
- 0 kudos
- 4554 Views
- 1 replies
- 1 kudos
Databricks S3 Commit Service
Is Databricks S3 Commit Service enabled by default if Unity Catalog is not enabled and the compute resources run in our AWS account (classic compute plane)? If not, how can it be enabled?This service seems to resolve the limitations with multi-cluste...
- 4554 Views
- 1 replies
- 1 kudos
- 1 kudos
No, the Databricks S3 commit service is not guaranteed to be enabled by default in the AWS classic compute plane. The configuration may vary based on your specific workspace setup. How can it be enabled? To enable the Databricks S3 commit service, ...
- 1 kudos
- 2565 Views
- 8 replies
- 3 kudos
Extract datetime value from the file name
I've the filename as below and I want to extract the datetime values and convert to datetime data type. This_is_new_file_2024_12_06T11_00_49_AM.csvHere I want to extract only '2024_12_06T11_00_49' and convert to datetime value in new field. I tried S...
- 2565 Views
- 8 replies
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 792 | |
| 553 | |
| 349 | |
| 287 |
