- 939 Views
- 0 replies
- 0 kudos
docs.databricks.com
Best practices | Databricks on AWSLearn best practices when using or administering Databricks.https://docs.databricks.com/best-practices-index.html
- 939 Views
- 0 replies
- 0 kudos
Forum Posts
- 4311 Views
- 1 replies
- 0 kudos
Resolved! How to prevent Delta Lake checkpoints to be removed in Databricks?
I am seeing with new commits the old checkpoints are getting removed and i can time travel only last 10 versions , Is there any way I can prevent it so that delat checkpoints are not removed I'm using Azure Databricks 7.3 LTS ML.
- 4311 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
If you want to keep your checkpoints X days, you can set delta.checkpointRetentionDuration to X days this way:spark.sql(f""" ALTER TABLE delta.`path` SET TBLPROPERTIES ( delta.checkpointRetentionDuration = 'X days'...
- 0 kudos
- 2518 Views
- 1 replies
- 0 kudos
Resolved! How to track the progress of a VACUUM command.
My VACCUM command is stuck. I am not sure if it's deleting any files.
- 2518 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
There is no direct way to track the progress of the VACUUM command. One easy workaround is to run a DRY RUN from another notebook which will give the estimate of files to be deleted at that point in time. This will give a rough estimate of files to b...
- 0 kudos
- 4030 Views
- 1 replies
- 0 kudos
Resolved! Can auto-loader process the overwritten file
I have a directory where I get files with the same multiple times. Will Auto-loader process all the files or will it process the first and ignore the rest
- 4030 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Autoloader has an option - "cloudFiles. allowOverwrites". This determines whether to allow input directory file changes to overwrite existing data. This option is available in Databricks Runtime 7.6 and above.
- 0 kudos
- 4833 Views
- 1 replies
- 0 kudos
Truncate delta table in Databricks
I cannot find how to truncate table using pyspark or python commnd , I need to truncate delta table using python
- 4833 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Not everything is exposed as a function for Python or Java/Scala. Some operations are SQL-only, like spark.sql("TRUNCATE TABLE delta.`<path>`")
- 0 kudos
- 1419 Views
- 0 replies
- 0 kudos
How do I toggle between reading encrypted and writing unencrypted
If we want to read from a kms encrypted s3 bucket, but write out unencrypted, Do we use the global init script?I am wondering how to “toggle” btw reading encrypted, and writing unencrypted
- 1419 Views
- 0 replies
- 0 kudos
- 1871 Views
- 1 replies
- 0 kudos
Prevent Duplicate Entries to enter to delta lake Storage
I have a data frame and I write this data frame to adls table, next day I get an updated data frame which has some records from the past also and i want to update the delta table without creating duplicate
- 1871 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
This is a task for Merge command - you define condition for merge (your unique column) and then actions.MERGE INTO target USING src ON target.column = source.column WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *could be your dataf...
- 0 kudos
- 2590 Views
- 1 replies
- 0 kudos
Resolved! When should I run the FSCK REPAIR command on my Delta table
Is it a good practice to run the FSCK REPAIR command on a regular basis? I have Optimize and VACUUM commands scheduled to run every day.
- 2590 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Unlike OPTIMIZE and VACUUM, FSCK REPAIR is not an operational command that has to be executed on a regular basis. FSCK REPAIR is useful to repair the Delta metadata and remove the reference of the files from the metadata that are no longer accessible...
- 0 kudos
- 2388 Views
- 1 replies
- 0 kudos
Resolved! How to update table using merge from value rather than from a table
My question is how can we do an upsert directly, that is, without using a source table. I would like to give the values myself directly.s there a simple way to do that for Delta tables?
- 2388 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
A source table can be a subquery so the following should give you what you're after.MERGE INTO events USING (VALUES(...)) // round brackets are required to denote a subquery ON false // an artificial merge condition WHEN NOT MATCHED ...
- 0 kudos
- 2650 Views
- 1 replies
- 0 kudos
Resolved! Unable to drop a table
I have a Table which and I do not have access to the underlying data any longer. We do not need this dataset anymore, but unable to drop the table
- 2650 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Use the below code snippet to forcefully drop the table:package org.apache.spark.sql.hive { import org.apache.spark.sql.hive.HiveUtils import org.apache.spark.SparkContext object utils { def dropTable(sc: SparkContext, dbName: String, tableName...
- 0 kudos
- 2407 Views
- 1 replies
- 0 kudos
Delta Table to Spark Streaming to Synapse Table in azure databricks
Is there a way to keep my synapse database always in sync with latest data from delta table, My synapse database I believe doesn't support the stream as sink, can i get any workaround
- 2407 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
You could try to keep the data in sync by appending the new data dataframe in a forEachBatch on your write stream, this method allows for arbitrary ways to write data, you can connect to the Datawarehouse with jdbc if necessary:with your batch functi...
- 0 kudos
- 2635 Views
- 2 replies
- 0 kudos
Resolved! Unable to run any commands on the cluster.
All the commands get canceled. even 1+1 is failing, the cluster is completely unusable.
- 2635 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
More details on similar issues here: https://kb.databricks.com/python/python-command-cancelled.html
- 0 kudos
- 3140 Views
- 1 replies
- 0 kudos
- 3140 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
1. AnalysisThe first phase of Spark SQL optimization is the analysis. Spark SQL starts with a relationship to be processed that can be in two ways. A serious form from an AST (abstract syntax tree) returned by an SQL parser, and on the other hand fro...
- 0 kudos
- 1729 Views
- 0 replies
- 0 kudos
What types of artifact files can I save using MLflow?
I know that it is possible to log models as artifacts during an MLflow run. Can I save other types of files as well?
- 1729 Views
- 0 replies
- 0 kudos
- 3386 Views
- 1 replies
- 0 kudos
- 3386 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
setting the parameter ‘spark.cleaner.ttl’ or by dividing the long running jobs into different batches and writing the intermediary results to the disk.
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
