- 1657 Views
- 1 replies
- 2 kudos
Few things you should not do in Databricks!
Few things you should not do in Databricks!
- 1657 Views
- 1 replies
- 2 kudos
- 2 kudos
Compared to OSS Spark, these are few things the users don't have to worry about when running the same job on Databricks. Memory management: Databricks use an internal formula to allocate the Driver and executor heap based on the size of the instance....
- 2 kudos
- 3583 Views
- 1 replies
- 0 kudos
- 3583 Views
- 1 replies
- 0 kudos
- 0 kudos
Although not a hard limit, it's recommended to keep the number of cells in the notebook less than 100 for better UI experience as well as code readability. Having a really large block of code in a cell defeats the purpose of notebook execution and al...
- 0 kudos
- 23053 Views
- 1 replies
- 0 kudos
Resolved! Can I download files from DBFS to my local machine? I see only the Upload option in the Web UI.
- 23053 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, it's possible to download files from DBFS. To download the filesFiles stored in /FileStore are accessible in your web browser at https://<databricks-instance-name>.cloud.databricks.com/files/. For example, the file you stored in /FileStore/my-da...
- 0 kudos
- 2005 Views
- 2 replies
- 0 kudos
What is the best way to convert a very large parquet table to delta ? possibly without downtime!
What is the best way to convert a very large parquet table to delta ? possibly without downtime!
- 2005 Views
- 2 replies
- 0 kudos
- 0 kudos
I vouch for Sajith's answer. The main advantage with "CONVERT TO DELTA" is that operations are metadata centric which means we are not reading the full data for the conversion. For any other file format conversion, it's necessary to read the data com...
- 0 kudos
- 2156 Views
- 2 replies
- 0 kudos
Why should I move to Auto-loader?
I have a streaming workload using the S3-SQS Connector. The streaming job is running fine within the SLA. Should I migrate my job to use the auto-loader? If Yes, what are the benefits? who should migrate and who should not?
- 2156 Views
- 2 replies
- 0 kudos
- 0 kudos
That makes sense @Anand Ladda​ ! One major improvement that will have a direct impact on the performance is the architectural difference. S3-SQS uses an internal implementation of the Delta table to store the checkpoint details about the source files...
- 0 kudos
- 3580 Views
- 1 replies
- 0 kudos
- 3580 Views
- 1 replies
- 0 kudos
- 0 kudos
Stats collected on a Delta column are either using for Partitioning Pruning, Data Skipping. See here - https://docs.databricks.com/delta/optimizations/file-mgmt.html#delta-data-skipping for detailsIn additional stats are also used for Metadata only q...
- 0 kudos
- 1881 Views
- 0 replies
- 0 kudos
What are the recommendations around collecting stats on long strings in a Delta Table
It is best to avoid collecting stats on long strings. You typically want to collect stats on column that are used in filter, where clauses, joins and on which you tend to performance aggregations - typically numerical valuesYou can avoid collecting s...
- 1881 Views
- 0 replies
- 0 kudos
- 1991 Views
- 2 replies
- 0 kudos
Delta Optimistic Transactions Resolution and Exceptions
What is the best way to deal with concurrent exceptions in Delta when you have multiple writers on the same delta table ?
- 1991 Views
- 2 replies
- 0 kudos
- 0 kudos
While you can try-catch-retry , it would be expensive to retry as the underlying table snapshot would have changed. So the best approach is to avoid conflicts using partitioning and disjoint command conditions as much as possible.
- 0 kudos
- 6775 Views
- 1 replies
- 0 kudos
- 6775 Views
- 1 replies
- 0 kudos
- 0 kudos
by default a delta table has stats collected on the first 32 columns. This setting can be configured using the following.set spark.databricks.delta.properties.defaults.dataSkippingNumIndexedCols = 3However there's a time trade-off to having a large n...
- 0 kudos
- 1379 Views
- 1 replies
- 0 kudos
- 1379 Views
- 1 replies
- 0 kudos
- 0 kudos
Its typically a good idea to run optimize aligned with the frequency of updates to the Delta Table. However you also don't want to over do as there's a cost/performance trade-off. Unless there are very frequent updates to the table that can cause sma...
- 0 kudos
- 1845 Views
- 1 replies
- 0 kudos
- 1845 Views
- 1 replies
- 0 kudos
- 0 kudos
Optimize merges small files into larger ones and can involve shuffling and creation of large in-memory partitions. Thus its recommended to use a memory optimized executor configuration to prevent spilling to disk. IN additional use of autoscaling wil...
- 0 kudos
- 2126 Views
- 1 replies
- 0 kudos
- 2126 Views
- 1 replies
- 0 kudos
- 0 kudos
Z-ordering is generally effective on up to 3-4 columns and New clustering algorithm in DBR 7.6 can even go upto 5 columns. However, the key is to Z-order on columns that are typically used in filters/where predicates and joins.
- 0 kudos
- 2270 Views
- 1 replies
- 0 kudos
- 2270 Views
- 1 replies
- 0 kudos
- 0 kudos
This is typically caused by not have SSO enabled on the token with your Git Provider. If you have SSO, you need to authorize your token for the same
- 0 kudos
- 4381 Views
- 1 replies
- 0 kudos
- 4381 Views
- 1 replies
- 0 kudos
- 0 kudos
gzip format is not splittable so the load process is sequential and thus slower. You can either try to split the CSV up into parts, gzip those separately and load them. Alternatively bzip is a splittable zip format that is better to work withOr you c...
- 0 kudos
- 1841 Views
- 1 replies
- 0 kudos
- 1841 Views
- 1 replies
- 0 kudos
- 0 kudos
Courtesy of my colleague Sri, here's some sample library code to execute on a databricks cluster with a short SLAimport logging import textwrap import time from typing import Text from databricks_cli.sdk import ApiClient, ClusterService # Create a cu...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 486 | |
| 349 | |
| 287 |
