- 1397 Views
- 1 replies
- 0 kudos
- 1397 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks notebooks can be exported and stored in S3 or any other object storage. The internal storage of the databricks notebook cannot be changed or configured. The implementation is internal to Databicks control plane and not user configurable.
- 0 kudos
- 3656 Views
- 1 replies
- 0 kudos
- 3656 Views
- 1 replies
- 0 kudos
- 0 kudos
The below code snippet is useful to get the modification time of files. %scala import scala.util.Try import org.apache.hadoop.conf.Configuration import org.apache.hadoop.fs.{FileSystem, Path} import org.apache.hadoop.io.IOUtils import java.io.IOExcep...
- 0 kudos
- 7236 Views
- 2 replies
- 0 kudos
Resolved! How and when to capture the thread dump of the Spark driver?
What is the best way to capture the thread dump of the Spark driver process. Also, when should I capture the thread dump?
- 7236 Views
- 2 replies
- 0 kudos
- 0 kudos
For Spark driver the process is the same. Choose the driver from the Executor page and view the thread dump. A thread dump is the footprints of the JVM they are very useful in debugging issues where the JVM process is stuck or making extremely slow p...
- 0 kudos
- 2919 Views
- 2 replies
- 0 kudos
Resolved! Autoloader: How to identify the backlog in RocksDB
With S3-SQS it was easier to identify the backlog ( the messages that are fetched from SQS and not consumed by the streaming job) How to find the same with Auto-loader
- 2919 Views
- 2 replies
- 0 kudos
- 0 kudos
For DBR 8.2 or later, the backlog details are captured in the Streaming metricsEg:
- 0 kudos
- 3388 Views
- 1 replies
- 0 kudos
Resolved! How to ensure that a Databricks Run Submit run invoked from Airflow only runs one time?
I am running jobs on Databricks using the Run Submit API with Airflow. I have noticed that rarely, a particular run is run more than one time at once. Why?
- 3388 Views
- 1 replies
- 0 kudos
- 0 kudos
Idempotency can be ensured by providing the idempotency token. It's easy to pass the same through REST API as mentioned in the below doc:https://kb.databricks.com/jobs/jobs-idempotency.htmlThe primary reason for multiple runs is the client submits t...
- 0 kudos
- 1629 Views
- 1 replies
- 0 kudos
Resolved! Performance improvement after running VACUUM commands
How often should I run VACUUM commands? Will running the VACUUM command on a Delta table improve my read/write performance or is it just the storage benefits.
- 1629 Views
- 1 replies
- 0 kudos
- 0 kudos
VACUUM removes uncommitted/stale files from the Storage. The primary benefit is to save the storage cost. Ideally running VACUUM should not show any performance improvement as Delta does not list the storage directories but rather access the files di...
- 0 kudos
- 2159 Views
- 1 replies
- 1 kudos
Does running OPTIMIZE on a delta table destroy the transaction history of table?
If I run OPTIMIZE on a Delta Lake table, will it prevent me from time travelling to a version before OPTIMIZE was run?
- 2159 Views
- 1 replies
- 1 kudos
- 1 kudos
No, you will still be able to time travel to versions previous to the OPTIMIZE command. OPTIMIZE is just another transaction like MERGE, UPDATE, etc. Check out these docs to learn more about retention periods and the VACUUM command.
- 1 kudos
- 2577 Views
- 1 replies
- 0 kudos
How do I find the users in workspaces
Looking to pull a list of all the users in their workspaces (including the ones who have never done anything), is there a way to do that? This for AWS
- 2577 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use the SKIM APIs. Endpoint: https://docs.databricks.com/dev-tools/api/latest/scim/scim-users.html#get-users Or you can use the Workspace API. The workspace API does not have a direct list users command, but you can use the workspace API to l...
- 0 kudos
- 14962 Views
- 2 replies
- 4 kudos
Resolved! Max Columns for Delta table
Is there an upper limit/recommended max value for no. of columns for Delta table?
- 14962 Views
- 2 replies
- 4 kudos
- 4 kudos
Original answer posted by @Gray Gwizdz​ This was a fun question to try and find the answer to! Thank you for that I reviewed some of the most recent issues/bugs reported with Delta Lake and was able to find a similar issue where a user was running i...
- 4 kudos
- 2219 Views
- 0 replies
- 1 kudos
Databricks Autoloader Best practice
Databricks Autoloader is a popular mechanism for ingesting data/files from cloud storage into Delta; for a very high throughput source, what are the best practices to be following while scaling up an autoloader based pipeline to the tune of millions ...
- 2219 Views
- 0 replies
- 1 kudos
- 2531 Views
- 3 replies
- 0 kudos
Resolved! How to resuse Pandas code in PySpark?
I have single threaded Pandas code that is both not yet supported by Koalas nor easy to reimplement in PySpark. I would like to distribute this workload using Spark without rewriting all my Pandas code - is this possible?
- 2531 Views
- 3 replies
- 0 kudos
- 0 kudos
This is for a specific scenario where the code is not yet supported by Koalas. One approach to consider is using a Pandas UDF, and splitting up the work in a way that allows your processing to move forward. This notebook is a great example of taking ...
- 0 kudos
- 8574 Views
- 2 replies
- 0 kudos
Trigger.once mode recommendation
When is it recommended to use Trigger.once mode compared to fixed processing intervals with micro batches?
- 8574 Views
- 2 replies
- 0 kudos
- 0 kudos
Also note, the configurations like maxFilesPerTrigger, maxBytesPerTrigger are ignored with Trigger.Once. Streaming queries with significantly less throughput can switch to Trigger.Once to avoid the continuous execution of the job checking the availab...
- 0 kudos
- 3672 Views
- 2 replies
- 0 kudos
VACUUM during read/write
Is it safe to run VACUUM on a Delta Lake table while data is being added to it at the same time? Will it impact the job result/performance?
- 3672 Views
- 2 replies
- 0 kudos
- 0 kudos
In the vast majority of cases, yes, it is safe to run VACUUM while data is concurrently being appended or updated to the same table. This is because VACUUM deletes data files no longer referenced by a Delta table's transaction log and does not effect...
- 0 kudos
- 3963 Views
- 2 replies
- 0 kudos
How does running VACUUM on Delta Lake tables effect read/write performance?
If I don't run VACUUM on a Delta Lake table, will that make my read performance slower?
- 3963 Views
- 2 replies
- 0 kudos
- 0 kudos
VACUUM has no effect on read/write performance to that table. Never running VACUUM on a table will not make read/write performance to a Delta Lake table any slower.If you run VACUUM very infrequently, your VACUUM runtimes themselves may be pretty hig...
- 0 kudos
- 1869 Views
- 1 replies
- 1 kudos
Can I have a Databricks Cluster that is only 1 node?
Yes you can create a "Single Node" Cluster, https://docs.databricks.com/clusters/single-node.html . It is currently not recommended to use "Single Node" cluster for streaming workloads
- 1869 Views
- 1 replies
- 1 kudos
- 1 kudos
Single Node clusters should not be used for production workloads involving streaming queries, or complex computations. The intention here is to bring up the Spark cluster for all kinds of workloads
- 1 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 485 | |
| 349 | |
| 287 |
