- 1045 Views
- 2 replies
- 0 kudos
Hi All,I am new to databricks need some understanding for my requirement .our requirement:a: we have zip file in azure blob storage and we are bringing that file to dbfs and unzip that file and executing our transformations in multiple steps (3 steps...
- 1045 Views
- 2 replies
- 0 kudos
Latest Reply
Hi @praveen rajak​ Does @Debayan Mukherjee​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
1 More Replies
by
kjoth
• Contributor II
- 8038 Views
- 10 replies
- 5 kudos
I'm running a scheduled job on Job clusters. I didnt mention the log location for the cluster. Where can we get the stored logs location. Yes, I can see the logs in the runs, but i need the logs location.
- 8038 Views
- 10 replies
- 5 kudos
Latest Reply
Hi @Sai Kalyani P​ , Yes it helped. Thanks
9 More Replies
by
dmayi
• New Contributor
- 1926 Views
- 2 replies
- 1 kudos
Hi i want to set up custom tags on an all-purpose cluster for purposes of cost break down and chargebacks. What: specifically, i want to capture JobName, JobID, UserId who ran jobI can set other custom tags such as Business Unit, Owner... However,...
- 1926 Views
- 2 replies
- 1 kudos
Latest Reply
Hey there @DIEUDONNE MAYI​ Does @Kaniz Fatma​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
1 More Replies
- 6412 Views
- 8 replies
- 5 kudos
I have a job running with no issues in Databricks runtime 7.3 LTS. When I upgraded to 8.3 it fails with error An exception was thrown from a UDF: 'pyspark.serializers.SerializationError'... SparkContext should only be created and accessed on the driv...
- 6412 Views
- 8 replies
- 5 kudos
Latest Reply
Adding to @Sean Owen​ comments, The only reason this is working is that the optimizer is evaluating this locally rather than creating a context on executors and evaluating it.
7 More Replies
- 2356 Views
- 3 replies
- 1 kudos
I have an always-on job cluster triggering Spark Streaming jobs. I would like to stop this streaming job once a week to run table maintenance. I was looking to leverage the foreachBatch function to check a condition and stop the job accordingly.
- 2356 Views
- 3 replies
- 1 kudos
Latest Reply
Hi @Nolan Lavender​, How is it going?Were you able to resolve your problem?
2 More Replies
- 3591 Views
- 4 replies
- 0 kudos
Continuing the above case, does that mean if i have several like 5 ADF pipelines scheduled regularly at the same time, its better to use an existing cluster as all of the ADF pipelines would share the same cluster and hence the cost will be lower?
- 3591 Views
- 4 replies
- 0 kudos
Latest Reply
Atanu
Esteemed Contributor
for adf or job run we always prefer job cluster. but for streaming, you may consider using interactive cluster . but anyway you need to monitor the cluster load, if loads are high there will be chance to job slowness as well as failure. also data siz...
3 More Replies
by
Junee
• New Contributor III
- 2794 Views
- 7 replies
- 3 kudos
I am using Databeicks Job Api 2.1 to trigger and run my jobs. "jobs/runs/submit" this API helps in starting the cluster, as well as create the job and run it. This API works great for normal jobs as it also cleans the cluster once job is finished suc...
- 2794 Views
- 7 replies
- 3 kudos
Latest Reply
@Junee, Anytime! It is crisply mentioned in the doc too. https://docs.databricks.com/clusters/index.html
6 More Replies
- 3029 Views
- 3 replies
- 4 kudos
When I run a job enabling using spot instances , I would like to know how many number of workers are using spot and how many number of workers are using on demand instances for a given job run In order to identify the spot instances we got for any...
- 3029 Views
- 3 replies
- 4 kudos
Latest Reply
You can. do it on Azure Portal using the Virtual Machines list:You will filter by either JobId tag or RunName tag (job name) and group by azure spot eviction policy or azure spot eviction type, the vm's under Stop/Deallocate and Capacity (using the 2...
2 More Replies
- 1064 Views
- 1 replies
- 0 kudos
I would like to make it so users can only create job clusters and not interactive clusters. Is it possible to do this in a workspace?
- 1064 Views
- 1 replies
- 0 kudos
Latest Reply
This can be accomplished with cluster policies. You can use a policy similar to this example to restrict certain users or groups to only have permission to create job clusters.
