- 7344 Views
- 1 replies
- 6 kudos
How can I simplify my data ingestion by processing the data as it arrives in cloud storage?
This post will help you simplify your data ingestion by utilizing Auto Loader, Delta Optimized Writes, Delta Write Jobs, and Delta Live Tables. Pre-Req: You are using JSON data and Delta Writes commandsStep 1: Simplify ingestion with Auto Loader Delt...
- 7344 Views
- 1 replies
- 6 kudos
- 6 kudos
This post will help you simplify your data ingestion by utilizing Auto Loader, Delta Optimized Writes, Delta Write Jobs, and Delta Live Tables.Pre-Req: You are using JSON data and Delta Writes commandsStep 1: Simplify ingestion with Auto Loader Delta...
- 6 kudos
- 4418 Views
- 5 replies
- 3 kudos
Resolved! AutoOptimize, OPTIMIZE command and Vacuum command : Order, production implementation best practices
So databricks gives us great toolkit in the form optimization and vacuum. But, in terms of operationaling them, I am really confused on the best practice.Should we enable "optimized writes" by setting the following at a workspace level?spark.conf.set...
- 4418 Views
- 5 replies
- 3 kudos
- 3 kudos
@AKSHAY PALLERLA​ Just checking in to see if you got a solution to the issue you shared above. Let us know!Thanks to @Werner Stinckens​ for jumping in, as always!
- 3 kudos
- 984 Views
- 0 replies
- 0 kudos
Verify auto-optimize from delta history
How can I verify if auto-optimize is activated from Delta history for the two scenarios below? Will the DESC history show the details in both the cases? 1). Auto-optimize set on the table properties2). Auto-optimize enabled in spark sessionP.S. - I'm...
- 984 Views
- 0 replies
- 0 kudos