- 1694 Views
- 0 replies
- 0 kudos
ADLS Gen2 access with folder ACL
How to access data in ADLS Gen2 from Databricks cluster with only folder level ACL?
- 1694 Views
- 0 replies
- 0 kudos
Forum Posts
- 2248 Views
- 1 replies
- 0 kudos
How can I prevent users from consuming excessive costs for jobs?
If users are allowed to create clusters, how can an operations team prevent them from consuming excessive costs?
- 2248 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Cluster policies can be used to constrain the node types that are available to users to create clusters, the number of nodes they can use to create clusters and the max DBU consumption they can use.The following resources provide further information:...
- 0 kudos
- 1436 Views
- 0 replies
- 0 kudos
What are the applicable limits when using airflow to co-ordinate execution of jobs
When using Airflow to co-ordinate execution of jobs, what are the applicable limits?
- 1436 Views
- 0 replies
- 0 kudos
- 1646 Views
- 1 replies
- 0 kudos
Stream is not getting started from kafka after 2 hours of cluster statrt
Hi Team I am setting up the Kafka cluster on databricks to ingest the data on delta, but it seems like the cluster is running from last 2 hours but still, the stream is not started and I am not seeing any failure also.
- 1646 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
This Type of issue happens if you have firewall on cloud account and your ip is not whitelisted, so pleaae whitelist the ip and issue will resolve
- 0 kudos
- 3698 Views
- 1 replies
- 0 kudos
Databricks notebook command gets cancelled:Generally when cluster is having init scripts or lib issues while starting cluster. Exact error can be look...
Databricks notebook command gets cancelled:Generally when cluster is having init scripts or lib issues while starting cluster. Exact error can be looked into driver logs.
- 3698 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Awsome Knowledge
- 0 kudos
- 3842 Views
- 2 replies
- 2 kudos
- 3842 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
You can tag your cluster and that tags get propagated to the billing management and there you can see it the cost
- 2 kudos
- 2303 Views
- 1 replies
- 0 kudos
Azure Databricks with Storage Account as data layer and DBFS understanding
What is the difference between ADLS mounted ON DataBricks and dbfs does the Mount of ADLS on databricks make gives any performance benefit , is the mounted ADLS still behave as object storage or it become simple storage
- 2303 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
DBFS is just an abstraction on cloud storage By default when you create a workspace, you get an instance of DBFS - so-called DBFS Root. Plus you can mount additional storage accounts under the /mnt folder. Data written to mount point paths (/mnt) is...
- 0 kudos
- 7207 Views
- 1 replies
- 0 kudos
How to conver Dataframe into JSON on Databricks?
Can I convert my jdbc Dataframe into JSON ? Because when I tried it, it got an error. I'm using a script as Pandas DataFrame function df.to_json()
- 7207 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
df.toJSON()
- 0 kudos
- 8202 Views
- 3 replies
- 1 kudos
- 8202 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
The answer varies depending on the cloud provider (as of June 2021) . In GCP, since the architecture is based on GKE , there are additional ip requirements. For more details see
- 1 kudos
- 1610 Views
- 0 replies
- 0 kudos
Escaped quotes mess up table records
When table content is dumped from the RDBMS (e.g. Oracle), some column values may contain escaped double quotes (\") in the column values, which may cause the values from multiple columns to be concatenated into one value and result in corrupted reco...
- 1610 Views
- 0 replies
- 0 kudos
- 53995 Views
- 6 replies
- 0 kudos
Resolved! Job fails with "The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached."
No other output is available, not even output from cells that did run successfully. Also, I'm unable to connect to spark ui or view the logs. It makes an attempt to load each of them, but after some time an error message appears saying it's unable ...
- 53995 Views
- 6 replies
- 0 kudos
Latest Reply
- 0 kudos
most of the time it's out of memory on driver node. check over all the drive log, data node log in Spark UI. And check if u r collecting huge data to drive node, e.g. collect()
- 0 kudos
- 2631 Views
- 1 replies
- 0 kudos
Delta - open source?
Delta is open source but certain features such as OPTIMIZE, ZORDER are only available on managed DBR. So how open sourced is it really?
- 2631 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Some of the feature is exclusively added by datbricks on top of delta not by comunity so comapny has right whether it wants to open source or not
- 0 kudos
- 8111 Views
- 2 replies
- 1 kudos
- 8111 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Bucketing is physical partition of the the table but the Zordering is arrangement of records in a file , in most optimal manner
- 1 kudos
- 2216 Views
- 1 replies
- 0 kudos
- 2216 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
There isn’t a problem purging old data. When using auto loader it’ll take into account new data being added.
- 0 kudos
- 2096 Views
- 1 replies
- 2 kudos
What is the best practice for generating jobs in an automated fashion?
What is the best practice for generating jobs in an automated fashion?
- 2096 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
There are several approaches here. You can write an automation script that programmatically accesses Databricks API’s to generate configured jobs. You can also utilize the Databricks Terraform provider. The benefit of the latter approach is that Terr...
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
