- 5866 Views
- 2 replies
- 3 kudos
Resolved! How to get previous version of the table in databricks sql dynamically
hello, im trying to get the previous version of a delta table using timestamp but databricks sql does not allow to use variables the only thing i can do is use TIMESTAMP AS OF CURRENT_DATE() -1 if i have refreshed the table today.please let me know i...
- 5866 Views
- 2 replies
- 3 kudos
- 3 kudos
thank you @Vidhi_Khaitan .Is there an upgrade or use case in works where we can pass parameters via workflow while triggering a databricks dashboard?
- 3 kudos
- 2808 Views
- 1 replies
- 0 kudos
Update databricks job parameter with CLI
Use Case:Updating a Databricks job with multiple tasks can be time-consuming and error-prone when changes (such as adding new parameters) need to be applied to each task manually.Possible Solutions:1. Using Databricks CLI – jobs reset commandYou can ...
- 2808 Views
- 1 replies
- 0 kudos
- 0 kudos
hello Divya, Could you also try YAML and update your task accordingly and deploy it as a part of asset bundles? let me know if you feel both are same? Regards,Anil.
- 0 kudos
- 1588 Views
- 1 replies
- 0 kudos
Get the total amount of S3 storage used per user
In Databricks is it possible to get the total amount of delta lake storage being used in the parquet format per user? Subsequently, what are the best practices on making sure that users saving delta files are not taking up storage unnecessarily, for ...
- 1588 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @zach, can you expand on why you need to know the total storage per user?Best practices - If you use Databricks managed tables, optimization is taken care of. https://docs.databricks.com/aws/en/optimizations/predictive-optimization
- 0 kudos
- 4491 Views
- 1 replies
- 0 kudos
DB to Snowflake connection error
We are getting below mentioned error after upgrading Databricks server to 15 version - error while making connection to Snowflake in Notebook " Bad request; operation not supported." .Has any one got this error ? Any pointers how to fix it ?
- 4491 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Abhay, Here are some general troubleshooting steps and pointers to help you resolve this issue: Ensure that you are using the correct connection configurations for Snowflake. Configuration mismatches can sometimes lead to operation errors. Using ...
- 0 kudos
- 5570 Views
- 2 replies
- 2 kudos
Resolved! Small Files Persist After OPTIMIZE with Target File Size Set to 100MB – Seeking Possible Reasons
I'm currently working on optimizing a Delta table in Databricks. As part of this, I’ve increased the target file size from the (~33MB) to 100MB using the OPTIMIZE command. However, after running the OPTIMIZE operation, I still observe a large number ...
- 5570 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi pooja_bhumandla,Great question! How are you doing today? Even after running the OPTIMIZE command with a higher target file size like 100MB, it’s common to still see some small files in your Delta table—especially in partitions with very little dat...
- 2 kudos
- 2356 Views
- 2 replies
- 1 kudos
How to fast sync large files (> 100GB)
I want to sync large files (>100GB) from my local system to a DBX Volume. I see 2 Options with different problems, do you have suggestions?Option 1: Needs to open the file completely -> Memory issues with open(local_file_path, 'rb') as file: ...
- 2356 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi PeSe,How are you doing today? As per my understanding, You're absolutely right to think through both options carefully. Option 1 runs into memory issues because it's trying to read the whole large file into memory at once, which doesn't work well ...
- 1 kudos
- 4573 Views
- 6 replies
- 2 kudos
Clarification Needed: COMPUTE STATISTICS vs COMPUTE DELTA STATISTICS on Delta Tables
Hi everyone,I’m trying to understand the difference between the two commands in Databricks:ANALYZE TABLE <table_name> COMPUTE STATISTICSANALYZE TABLE <table_name> COMPUTE DELTA STATISTICSSpecifically:What exactly does each command do, and how do they...
- 4573 Views
- 6 replies
- 2 kudos
- 2 kudos
If you want to know more about query optimization I suggest you look in Spark's Catalyst Optimizer and Adaptive Query Execution (AQE). You should alway run Analyze table compute statistics becuase this will help Spark's query optimization converge o...
- 2 kudos
- 1111 Views
- 1 replies
- 0 kudos
COMPUTE STATISTICS- QUERY OPTIMIZER
I ran ANALYZE TABLE <table_name> COMPUTE DELTA STATISTICS on my tables to calculate column-level statistics at the file level for each Parquet file inside the Delta table. Now, I’m wondering which command is better to run next: ANALYZE TABLE <table_n...
- 1111 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sainath368 ,From what I understand, COMPUTE DELTA STATISTICS generates Delta statistics that are mainly used for data skipping, helping speed up table scans by avoiding unnecessary file reads. However, these stats aren't used by the query optimiz...
- 0 kudos
- 4063 Views
- 8 replies
- 3 kudos
Resolved! Unity catalog with pool cluster
We are trying to execute notebook recently migrated to unity catalog through Pool cluster from synapse pipeline. But it giving error as "UC not enabled".Could you please suggest on this.
- 4063 Views
- 8 replies
- 3 kudos
- 8507 Views
- 7 replies
- 4 kudos
Resolved! All-purpose cluster upsize failures
Hello,For one of my clients, we are using an all-purpose cluster to run some Databricks notebooks. We noticed in the logs of the cluster some Azure Quota Exceptions, from which we would like to know more.As you can see attached, the cluster always su...
- 8507 Views
- 7 replies
- 4 kudos
- 4 kudos
@szymon_dybczak That makes sense thank you ! However, do you have any idea of why it shows "Current usage: 0" ? And also why does the cluster still manage to reach the max number of nodes even with these "failures" ? Sacha
- 4 kudos
- 3964 Views
- 1 replies
- 1 kudos
[UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE] Missing Credential Scope. Unity Credential Scope id not foun
Hi,I'm having a problem while trying to write to a delta table from a org.apache.spark.sql.util.QueryExecutionListenerCode:val df = SparkSession.active().createDataFrame(batch, MyClass::class.java)df.write().mode("append").format("delta").saveAsTable...
- 3964 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @thari To answer this issue you are facing: What’s actually happening here is that when you try to write to a Delta table managed by Unity Catalog within a QueryExecutionListener’s callback, Spark’s security context isn’t set up correctly. Tha...
- 1 kudos
- 5199 Views
- 3 replies
- 2 kudos
Resolved! Problem using dbutils to access local files
Greetings, total newbie here, please advise if this is an inappropriate forum. I'm trying to learn some basics.I have tried to copy a file from the local file system onto dbfs, and while it fails in python, it succeeds in the command line. The error ...
- 5199 Views
- 3 replies
- 2 kudos
- 2 kudos
Well, it was indeed a very fundamental misunderstanding. In the UI, I had not connected to my own compute cluster, it was still running "serverless": So, I guess it's not surprising the process didn't have any privileges, even though the terminal tha...
- 2 kudos
- 3005 Views
- 3 replies
- 0 kudos
autoloader running task batch
Hello Guys, I run task in batch mode with autoloader i enable option trigger (available now true). So, when my script finish, the continue running. Someone know who's happen? Cordially ;
- 3005 Views
- 3 replies
- 0 kudos
- 0 kudos
this is my script I enable this options when i read files on Volumes before write on delta table(reader_stream.option("cloudFiles.format", self.file_format) .option("cloudFiles.schemaLocation", self.schema_location) ...
- 0 kudos
- 9609 Views
- 3 replies
- 1 kudos
Resolved! Spark in not executing any tasks
I have an issue where Spark in not submiting any task, on any worksapce or cluster, even SQLWarehouse.Even for very simple code it hangs forever.Anyone ever faced something similar? Our infra is AWS.
- 9609 Views
- 3 replies
- 1 kudos
- 1 kudos
Found the solution: Turned out to be an issue with the Security Groups. The internal security group communication was not open to all ports for TCP and UDP. After fixing that the jobs ran fine. Seems like we did require more workers too.
- 1 kudos
- 1310 Views
- 2 replies
- 0 kudos
Automate data bricks batch which copy data from file/database and put it in some other file/database
Hi,I have created a data bricks batch which copy data from a file/database table and put it in some other file/database table.I want to automate the functional test cases of this data brick batch.Can someone suggest, is there any tool available for t...
- 1310 Views
- 2 replies
- 0 kudos
- 0 kudos
Create a workflow for your batch load notebook and then schedule it.https://docs.databricks.com/gcp/en/jobs/jobs-quickstart
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
