- 6521 Views
- 4 replies
- 3 kudos
Performance Issues with Unity Catalog
Hey I need some help / suggestions troubleshooting this, I have two DataBricks Workspaces Common and Lakehouse. There difference between them is: Major Differences:- Lakehouse is using Unity Catalog- Lakehouse is using External Locations whereas cre...
- 6521 Views
- 4 replies
- 3 kudos
- 3 kudos
- Listing directories in common is at least 4-8 times faster than Lakehouse environment. Are you able to replicate the issue using simple a dbutils list operation (dbutils.fs.ls) or by performing a sample file (say 100 MB file) copy using dbutils.f...
- 3 kudos
- 7302 Views
- 2 replies
- 1 kudos
Resolved! How to disable all cache
Hi, I'm trying to test some SQL perf. I run below firstspark.conf.set('spark.databricks.io.cache.enabled', False) However, the 2nd run for the same query is still way faster than the first time run. Is there a way to make the query start from a clean...
- 7302 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks @VZLA . How to runspark.sparkContext.getPersistentRDDs.values.foreach(_.unpersist())from databricks notebook?
- 1 kudos
- 7534 Views
- 5 replies
- 1 kudos
Shapley Progressbar
The shapley progress bar or tqdm progress bar in general doesn't show in notebooks. Do I need to set something special to get this or any other similar widgets to work?
- 7534 Views
- 5 replies
- 1 kudos
- 1 kudos
I think you're looking for tqdm.notebookfrom time import sleepfrom tqdm.notebook import tqdmfor _ in tqdm(range(20)): sleep(5)
- 1 kudos
- 6514 Views
- 1 replies
- 1 kudos
Resolved! Where can I find those delta table log files?
I created a delta table with SQL command CREATE TABLE, and inserted several records into with INSERT statements. Now it can be seen from the catalog.But I want to understand how delta works, and would like to see where are those log files stored.Even...
- 6514 Views
- 1 replies
- 1 kudos
- 1 kudos
To locate the log files for your Delta table, please first note that Delta Lake stores its transaction log files in a specific directory within the table's storage location. These log files are for maintaining the ACID properties and enabling feature...
- 1 kudos
- 2016 Views
- 1 replies
- 0 kudos
Azure Databricks - Terraform errors while using workspace level provider
Hello All,I have a question about deploying Azure Databricks with Terraform. Does Databricks have any API call limits? I can deploy external location, a storage credential and it's tested and confirmed working. But when i try to deploy 2 additional e...
- 2016 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Terraformuser, Could you try to enable debug on while applying your terraform, that would give you more context on the failure. TF_LOG=DEBUG DATABRICKS_DEBUG_TRUNCATE_BYTES=250000 terraform apply -no-color 2>&1 |tee tf-debug.log
- 0 kudos
- 3788 Views
- 1 replies
- 1 kudos
TIME data type
Our business does a LOT of reporting and analysis by time-of-day and clock times, independent of day or date. Databricks does not seem to support the TIME data type, that I can see. If I attempt to import data recorded as a time (eg., 02:59:59.000)...
- 3788 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @TamD ,Basically, it's just like you've written. There is no TIME data type, so you have 2 options which you already mentioned:- you can use Timestamp data type and ignore its date part- store it as string and do conversion each time you need it
- 1 kudos
- 3223 Views
- 2 replies
- 0 kudos
Code Review tools
Could you kindly recommend any Code Review tools that would be suitable for our Databricks tech stack?
- 3223 Views
- 2 replies
- 0 kudos
- 5430 Views
- 3 replies
- 1 kudos
Resolved! Retention window from DLT created Delta tables
Hi guysI am working with data ingested from Azure EventHub using Delta Live Tables in databricks. Our data architecture includes the medallion approach. Our current requirement is to retain only the most recent 14 days of data in the silver layer. To...
- 5430 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @MuthuLakshmi Thank you for sharing the configurations. Here is a bit more clarity on our current workflow.DELETE and VACUUM WorkflowOur workflow involves the following:1. DELETE Operation:We delete records matching a specific predicate to mark th...
- 1 kudos
- 8185 Views
- 9 replies
- 4 kudos
Resolved! Trigger queries to SQL warehouse from Databricks notebook
Hello, I am trying to explore triggering for sql queries from Databricks notebook to serverless sql warehouse along with nest-asyncio module.Both the above are very new for me and need help on the same.For triggering the API from notebook, I am using...
- 8185 Views
- 9 replies
- 4 kudos
- 5653 Views
- 2 replies
- 1 kudos
Best practices for code organization in large-scale Databricks ETL projects: Modular vs. Scripted
I’m curious about Data Engineering best practices for a large-scale data engineering project using Databricks to build a Lakehouse architecture (Bronze -> Silver -> Gold layers).I’m presently comparing two approaches of code writing to engineer the s...
- 5653 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ashap551 ,I would vote for modular approach which lets you reuse code and write unit test in simpler manner. Notebooks are for me only "clients" of these shared modules. You can take a look at official documentation where they're following simila...
- 1 kudos
- 1649 Views
- 3 replies
- 0 kudos
How to read zip folder that contains 4 .csv files
Hello Community, I have uploaded one zip folder "dbfs:/FileStore/tables/bike_sharing.zip" I was trying to unzip the folder and read the 4 .csv files. I was unable to do it. Any help from your side will really be grateful!
- 1649 Views
- 3 replies
- 0 kudos
- 0 kudos
Hope this link will help. You can use cell command within a notebook to unzip (assuming you have the path access where do you want to unzip the file).https://stackoverflow.com/questions/74196011/databricks-reading-from-a-zip-file
- 0 kudos
- 20071 Views
- 2 replies
- 0 kudos
- 20071 Views
- 2 replies
- 0 kudos
- 0 kudos
Is the limit per "table/dataframe" or for all tables/dataframes put together?The driver collects the data from all executors (which are having the respective table or dataframe) and distributes to all executors. When will the memory be released in bo...
- 0 kudos
- 12769 Views
- 4 replies
- 4 kudos
Resolved! Spark SQL: USING JSON to create VIEWS/TABLES with existing schema file
Hi there,I'm trying to understand if there's an easy way to create VIEWS and TABLES (I'm interested in both) *WITH* a provided schema file. For example, I understand that via dataframes I can accomplish this via something like this:df = spark.read.sc...
- 12769 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @ChristianRRL ,(A) You're not missing anything, there's no such an option as of today for SQL API. (B) It would be much better for you to just use pyspark, but if you have to stick to just SQL API you can use following aproach. Define your schema...
- 4 kudos
- 1239 Views
- 1 replies
- 0 kudos
Is there any impact on data ingestion and data extract while REORG TABLE is in progress
While using deltalake for eventing system, with repeated updates and merges etc, we are using deletion vector to improve performance. With that comes "REORG TABLE" maintenance task.My question is in a ingestion and extract heavy system, when we condu...
- 1239 Views
- 1 replies
- 0 kudos
- 0 kudos
It is advisable to schedule REORG TABLE operations during periods of low activity to minimize disruptions to both data ingestion and extraction processes.This can potentially affect ongoing data ingestion processes because the table's underlying file...
- 0 kudos
- 1421 Views
- 1 replies
- 0 kudos
Unable to install kafka in community edition
Hi,I am trying to install kafka in databricks community edition after downloading. Using below command in notebook.. %sh cd kafka_2.12-3.8.1/ls -ltr ./bin/zookeeper-server-start.sh config/zookeeper.properties Below is the error log.Kindly help.
- 1421 Views
- 1 replies
- 0 kudos
- 0 kudos
It seems like the issue might be related to a port conflict, as indicated by the java.net.BindException: Address already in use error. You might want to check if another instance of ZooKeeper or another service is using the same port
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
