- 808 Views
- 1 replies
- 0 kudos
I keep it a point to use the latest DBR versions for my workloads and mostly we leverage those new features. But I have 300 jobs on the Databricks workspace and updating the DBR versions for each job manually is difficult to do. Any quick hack
- 808 Views
- 1 replies
- 0 kudos
Latest Reply
Below code snippet can be helpful if you are using Databricks CLIfor jobid in `databricks jobs list | awk '{print $1}'`;
do databricks jobs get --job-id $jobid | jq .settings > /tmp/jobs/$jobid.json; done
sed -i 's/"spark_version": ".*"/"spark_ver...
- 746 Views
- 1 replies
- 0 kudos
I do not want to upgrade my cluster every one month. I am looking for stability over new features.
- 746 Views
- 1 replies
- 0 kudos
Latest Reply
The strong recommendation is not to use an unsupported version of DBR on your cluster. For production workloads where you don't welcome newer versions, then check the Databricks LTS DBR versions. if using an unsupported version then you don't receiv...
- 1028 Views
- 1 replies
- 0 kudos
I am reading data from S3 from a Databricks cluster and the read operation seldom fails with 403 permission errors. Restarting the cluster fixes my issue.
- 1028 Views
- 1 replies
- 0 kudos
Latest Reply
The main reason for this behavior is : AWS keys are used in addition to the IAM role. Using global init scripts to set the AWS keys can cause this behavior.The IAM role has the required permission to access the S3 data, but AWS keys are set in the Sp...
- 1017 Views
- 1 replies
- 0 kudos
I have a Spark structured streaming job reading data from Kafka and loading it to the Delta table. I have some transformations and aggregations on the streaming data before writing to Delta table
- 1017 Views
- 1 replies
- 0 kudos
Latest Reply
The typical reason for data loss on a Structured streaming application is having an incorrect value set for watermarking. The watermarking is done to ensure the application does not develop the state for a long period, However, it should be ensured ...
- 472 Views
- 1 replies
- 0 kudos
I have used Ranger in Apache Hadoop and it works fine for my use case. Now that I am migrating my workloads from Apache Hadoop to Databricks
- 472 Views
- 1 replies
- 0 kudos
Latest Reply
Currently, Table ACL does not support column-level security. There are several tools like Privcera which has better integration with Databricks.
- 3995 Views
- 1 replies
- 0 kudos
I've seen .cache() and .checkpoint() used similarly in some workflows I've come across. What's the difference, and when should I use one over the other?
- 3995 Views
- 1 replies
- 0 kudos
Latest Reply
Caching is extremely useful than checkpointing when you have lot of available memory to store your RDD or Dataframes if they are massive.Caching will maintain the result of your transformations so that those transformations will not have to be recomp...
- 586 Views
- 1 replies
- 1 kudos
Does Databricks have a data processing agreement?
- 586 Views
- 1 replies
- 1 kudos
Latest Reply
Databricks offers a standalone data processing agreement to comply with certain data protection laws that contains our contractual commitments with respect to applicable data protection and privacy law. If your company determines that you require ter...
- 2498 Views
- 1 replies
- 0 kudos
Do login sessions into Databricks have an idle timeout?
- 2498 Views
- 1 replies
- 0 kudos
Latest Reply
Short Answer:YesDetailed Answer:User sessions automatically timeout after six hours of idle time. This timeout is not configurable. User sessions are terminated if the user is removed from the workspace. To trigger session end for users who were remo...