- 2594 Views
- 1 replies
- 1 kudos
Resolved! spark.sparkContext.setCheckpointDir - External Azure Storage
Is it possible to direct spark.sparkContext.setCheckpointDir to an external Azure Storage Container location (instead of DBFS) & if so how, there's very little documentation on that.
- 2594 Views
- 1 replies
- 1 kudos
- 1 kudos
yes,the directory must be an HDFS path if running on a cluster.All you need to do is provide the correct path.
- 1 kudos
- 12878 Views
- 8 replies
- 1 kudos
Resolved! Pass Typesafe config file to the Spark Submit Job
Hello everyone ! I am trying to pass a Typesafe config file to the spark submit task and print the details in the config file. Code: import org.slf4j.{Logger, LoggerFactory} import com.typesafe.config.{Config, ConfigFactory} import org.apache.spa...
- 12878 Views
- 8 replies
- 1 kudos
- 1 kudos
I've experenced similar issues; please help to answer how to get this working;I've tried using below to be either /dbfs/mnt/blah path or dbfs:/mnt/blah pathin either spark_submit_task or spark_jar_task (via cluster spark_conf for java optinos); no su...
- 1 kudos
- 1336 Views
- 0 replies
- 1 kudos
Hi Everyone, As we continue to see the rise of AI, particularly with technologies such as ChatGPT, it's important to consider how this will impact...
Hi Everyone,As we continue to see the rise of AI, particularly with technologies such as ChatGPT, it's important to consider how this will impact the future of our workplaces and long-term career goals. With that in mind, I would love to hear your th...
- 1336 Views
- 0 replies
- 1 kudos
- 10677 Views
- 2 replies
- 1 kudos
Unity catlog replication or Disaster recovery implementation
I am working on Disaster recovery implementation on databricks on aws.I am not able to find how to implement with unity catalog. I am planning to create two workspaces in two different regions, one would be primary workspace which will be active and ...
- 10677 Views
- 2 replies
- 1 kudos
- 1 kudos
@Suram Vinay​ From my end i have not implemented this, but just cheked this blog previously. terraform script will help for DR setup. https://www.databricks.com/blog/2022/07/18/disaster-recovery-automation-and-tooling-for-a-databricks-workspace.htmlc...
- 1 kudos
- 5356 Views
- 2 replies
- 1 kudos
readStream query throws exception if there's no data in delta location.
Hi,I have a scenario where writeStream query writes the stream data to bronze location and I have to read from bronze, do some processing and finally write it to silver. I use S3 location for delta tablesBut for the very first execution , readStream ...
- 5356 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Pranathi Girish​,Hope all is well!Checking in. If @Suteja Kanuri​'s answer helped, would you let us know and mark the answer as best? If not, would you be happy to give us more information?We'd love to hear from you.Thanks!
- 1 kudos
- 5288 Views
- 2 replies
- 9 kudos
databricks Photon is a next-generation engine on the Databricks Lakehouse Platform that provides speedy query performance at a low cost.- Its function...
databricks Photon is a next-generation engine on the Databricks Lakehouse Platform that provides speedy query performance at a low cost.- Its function coverage is growing, and UDF under Photon is coming, which can bring significant improvements in us...
- 5288 Views
- 2 replies
- 9 kudos
- 9504 Views
- 6 replies
- 3 kudos
Resolved! Migrate notebooks and workflows and others .
I want to move notebooks , workflows , data from one users to another user in Azure Databricks. We move have access to that databricks. Is it possible? If, yes. How to move it.
- 9504 Views
- 6 replies
- 3 kudos
- 2590 Views
- 1 replies
- 1 kudos
java.lang.IllegalArgumentException: 0 - Could not create directory structure: /databricks/spark/work/app-20230224111412-0000/5/exID_5/16463034998783127067093113462199/079A4589-F569-5076-9142-B9B71CC0C7C5/16384469418283127081089124359741/DIM_643/v1/23B76BA
We are getting these error can someone help why these hapeens and what would be the root cause
- 2590 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Parvez Mushaf Maniyar​,Hope everything is going well.Does @Kaniz Fatma​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos
- 2060 Views
- 2 replies
- 2 kudos
workflow cluster was create error
I set the workflow to run at 12:00 every day in the workflow, but the workflow failed with the error message below, and I don't know why.Run result unavailable: run failed with error message Unexpected failure while waiting for the cluster (0506-0233...
- 2060 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @Sangwoo Lee​ ,As mentioned by vignesh, it seems like an infra related issue. > Does the user (which executes the job) has access to start a cluster?> Incase if it is not an access issue and Incase if you are starting a lot of workflow jobs tog...
- 2 kudos
- 2500 Views
- 1 replies
- 1 kudos
Current_Database() function unexpected results when queried with PowerBI
Hi,I`m creating some views to be queried by PowerBI. In our delta tables we have a column called database name which contains the source systems database name. What I`m doing is using this to filter data WHERE databaseName = current_database(). No...
- 2500 Views
- 1 replies
- 1 kudos
- 9917 Views
- 9 replies
- 1 kudos
Unable to launch notebook
I have created a 12.2 LTS cluster and trying to launch notebook attached to this cluster. But unable to launch, it is not giving any error instead it is still showing the same home page.
- 9917 Views
- 9 replies
- 1 kudos
- 1 kudos
Hi @Muskan Bansal​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 1 kudos
- 5574 Views
- 3 replies
- 1 kudos
databricks cluster creation error
Hi Databricks Community,My Databricks workspace created on Azure pay-as-go subscription.I am facing two sided challenges.First>> I am not able to create Delta-Live-table pipeline or any other all-purpose multi-node cluster as it is throwing below e...
- 5574 Views
- 3 replies
- 1 kudos
- 1 kudos
HI All, Thanks for your reply. Just to update you, I am now able to to create DLT pipeline as well as all purpose multi-node cluster with minimum resources. This is due to Quota limit and I was able to increase it. But observation is, if I try to us...
- 1 kudos
- 5005 Views
- 6 replies
- 1 kudos
Resolved! How can I get the list of downstream dashboards of a query
I want to deprecate a query from our workspace. However, I don't know if there is any downstream dashboard needs it.In case there is downstream dependency, then I can't just delete the query.I tried the search by query name, but it only returns the ...
- 5005 Views
- 6 replies
- 1 kudos
- 1 kudos
Databricks support suggests me to check the query history, to find out any history of the query, then check which dashboard is using it.Not an ideal solution but it works for me.
- 1 kudos
- 7658 Views
- 2 replies
- 1 kudos
Authorised views in databricks?
In GCP you can give a user access to a view, and then the view itself access to the underlying object, meaning you don't have to give end users access to the tables themselves.Is there a similar way of managing these permissions in databricks? The vi...
- 7658 Views
- 2 replies
- 1 kudos
- 9936 Views
- 7 replies
- 4 kudos
Seem to have outdated Account Console
Hello, We have been testing for a long time with Databricks and are now going to run it in production. Our tests were done over Databricks for AWS using the Standard plan and have since upgraded to the Premium plan. One of the aims to upgrade plans w...
- 9936 Views
- 7 replies
- 4 kudos
- 4 kudos
Hello, Does anyone have a proper way of contacting support ? As explained in some answers on this thread, we aren't able to create a support ticket in the help centre. We have contacted our account executive 10 days ago, to try to understand why we c...
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
