- 15780 Views
- 10 replies
- 3 kudos
- 15780 Views
- 10 replies
- 3 kudos
Latest Reply
Can you not use a No Isolation Shared cluster with Table access controls enabled on workspace level?
9 More Replies
- 2446 Views
- 3 replies
- 4 kudos
py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.streaming.DataStreamReader org.apache.spark.sql.SQLContext.readStream() is not whitelisted on class class org.apache.spark.sql.SQLContextI already disable acl for cluster using "...
- 2446 Views
- 3 replies
- 4 kudos
Latest Reply
Hi @Ravi Teja,Just a friendly follow-up. Do you still need help? if you do, please share more details, like DBR version, standard or High concurrency cluster? etc
2 More Replies
- 2093 Views
- 0 replies
- 0 kudos
I'm working on a task where I transform a dataset and re-save it to an S3 bucket. This involves joining the dataset to two others, dropping fields from the initial dataset which overlapped with fields from the other two, hashing certain fields with p...
- 2093 Views
- 0 replies
- 0 kudos
- 3269 Views
- 1 replies
- 1 kudos
We are adapting the multi-tasks workflow example from dbx documentation for our pipelines https://dbx.readthedocs.io/en/latest/examples/python_multitask_deployment_example.html. As a part of configuration we specify cluster configuration and provide ...
- 3269 Views
- 1 replies
- 1 kudos
Latest Reply
Tasks within the same multi task job can reuse the clusters. A shared job cluster allows multiple tasks in the same job to use the cluster. The cluster is created and started when the first task using the cluster starts and terminates after the last ...
- 4287 Views
- 4 replies
- 0 kudos
wondering if this is to parameterize the azure storage account name part in the spark cluster config in Databricks?I have a working example where the values are referencing secret scopes:spark.hadoop.fs.azure.account.oauth2.client.id.<azurestorageacc...
- 4287 Views
- 4 replies
- 0 kudos
by
jpwp
• New Contributor III
- 5355 Views
- 0 replies
- 0 kudos
Why does adding a dependent library to a Job task also permanently add it to the entire cluster?I am using python wheels, and even when I remove the dependent library from a Job task, the wheel is still part of the cluster configuration.If I then upd...
- 5355 Views
- 0 replies
- 0 kudos
- 1530 Views
- 0 replies
- 0 kudos
Attached to this post we have added an ADLS Gen2 access recommendation to have the ideal security and governance over your data. The best practice involves leveraging Cluster ACLs, cluster configuration, and secret ACLs to handle user access over you...
- 1530 Views
- 0 replies
- 0 kudos
- 523 Views
- 0 replies
- 0 kudos
Best practices: Cluster configuration | Databricks on AWSLearn best practices when creating and configuring Databricks clusters.https://docs.databricks.com/clusters/cluster-config-best-practices.html
- 523 Views
- 0 replies
- 0 kudos
- 3669 Views
- 1 replies
- 0 kudos
I am new to using Databricks and want to create a cluster, but there are many different worker types to choose from. How do I know which worker type is the right type for my use case?
- 3669 Views
- 1 replies
- 0 kudos
Latest Reply
For delta workloads, where you could benefit from caching it is recommended to use storage optimized instances that come with NVMe SSDs. For other workloads, it would be a good idea to check Ganglia metrics to see whether your workload is Cpu/Memory ...
