cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Rahul_Samant
by • Contributor
  • 4951 Views
  • 8 replies
  • 1 kudos

Mounting File Share in init script of cluster

we have a flow where we have to process chunk of files from file share. currently we are moving the files first to storage account and then post processing move files back to file share again. this is adding to the execution time for moving files bac...

Data Engineering
Reply
  • 4951 Views
  • 8 replies
  • 1 kudos
View Replies
Latest Reply
Samirshaikh
New Contributor II
  • 1 kudos

Hi @Rahul Samant is this issue solved Please help we are also facing same issues

  • 1 kudos
7 More Replies
tatekeller
by • New Contributor
  • 1460 Views
  • 1 replies
  • 0 kudos

Can you access a repo file in an init script?

I'd like to configure a cluster with python libraries as defined in a requirements file. I have a pip requirements.txt file in a private repo which I have integrated on Databricks (and I can access it through the UI and view it on Databricks). I upda...

Data Engineering
Reply
  • 1460 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sher
Valued Contributor II
  • 0 kudos

you can install in a cluster

  • 0 kudos
Jfoxyyc
by • Valued Contributor
  • 1520 Views
  • 2 replies
  • 2 kudos

How to use partial_parse.msgpack with workflow dbt task?

I'm looking for direction on how to get the dbt task in workflows to use the partial_parse.msgpack file to skip parsing files that haven't changed. I'm downloading my artifacts after each run and the partial_parse file is being saved back to adls.Wha...

Data Engineering
Reply
  • 1520 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 2 kudos

Hi, Could you please confirm what will be your expectation and the used case? Do you want the file to be saved somewhere else?

  • 2 kudos
1 More Replies
Tahseen0354
by • Valued Contributor
  • 2778 Views
  • 3 replies
  • 1 kudos

Resolved! Can I add custom cluster tag from init script ?

Hi, is it possible to add custom tags from init script during cluster initialization ? We would like to automatically add custom tags whenever someone creates a new cluster in databricks.

Data Engineering
Reply
  • 2778 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Prabakar
Esteemed Contributor III
  • 1 kudos

Hi @Md Tahseen Anam​ I don't think there is a possibility to use an init script for cust tags. But the easiest way is to use cluster policies. You can mention a list of custom tags in the policy so that you can simply add the policy to the cluster wh...

  • 1 kudos
2 More Replies
flachboard
by • New Contributor
  • 3782 Views
  • 4 replies
  • 1 kudos

How do you install R packages?

I've tried this, but it doesn't appear to be working: https://community.databricks.com/s/question/0D53f00001GHVX1CAP/unable-to-install-sf-and-rgeos-r-packages-on-the-clusterWhen I run the following after that init script, I receive an error.library(r...

Data Engineering
Reply
  • 3782 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey there @Christopher Flach​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...

  • 1 kudos
3 More Replies
Prabakar
by • Esteemed Contributor III
  • 6234 Views
  • 2 replies
  • 7 kudos

Resolved! Library installation fails with mirror sync issue

While trying to install ffmpeg package using an init script on Databricks cluster, it fails with the below error.Init script:#! /bin/bash set -e sudo apt-get update sudo apt-get -y install ffmpegError message:E: Failed to fetch http://security.ubuntu...

Data Engineering
Reply
  • 6234 Views
  • 2 replies
  • 7 kudos
View Replies
Latest Reply
Prabakar
Esteemed Contributor III
  • 7 kudos

Cause: The VMs are pointing to the cached old mirror which is not up-to-date. Hence there is a problem with downloading the package and it's failing. Workaround: Use the below init script to install the package "ffmpeg". To revert to the original lis...

  • 7 kudos
1 More Replies
Serhii
by • Contributor
  • 3371 Views
  • 5 replies
  • 8 kudos

Resolved! init_script error during cluster creation - 101: Network is unreachable

When I run the init_script during cluster creationapt-get update && apt-get install -y ffmpeg libsndfile1-devI get an error in cluster logs E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/o/openal-soft/libopenal1_1.19.1-1_amd64.deb ...

Data Engineering
Reply
  • 3371 Views
  • 5 replies
  • 8 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 8 kudos

Hi @Sergii Ivakhno​ , Just a friendly follow-up. Do you still need help, or @Pratik Bhawsar​ 's response help you to find the solution? Please let us know.

  • 8 kudos
4 More Replies
MarcoData01
by • New Contributor III
  • 2368 Views
  • 6 replies
  • 4 kudos

Resolved! Is there the possibility to protect Init script folder on DBFS

Hi everyone,We are looking for a way to protect the folder where init script is hosted from editing.This because we have implemented inside init script a parameter that blocks the download file from R Studio APP Emulator and we would like to avoid th...

Data Engineering
Reply
  • 2368 Views
  • 6 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Marco Data​ Thank you for sending in your question. It is awesome that you found a solution. Would you like to mark the answer as best so others can find the solution quickly?Cheers!

  • 4 kudos
5 More Replies
fermin_vicente
by • New Contributor III
  • 4414 Views
  • 7 replies
  • 4 kudos

Resolved! Can secrets be retrieved only for the scope of an init script?

Hi there, if I set any secret in an env var to be used by a cluster-scoped init script, it remains available for the users attaching any notebook to the cluster and easily extracted with a print.There's some hint in the documentation about the secret...

Data Engineering
Reply
  • 4414 Views
  • 7 replies
  • 4 kudos
View Replies
Latest Reply
pavan_kumar
Contributor
  • 4 kudos

@Fermin Vicente​ good to know that this approach is working well. but please make sure that you use this approach at the end of your init script only

  • 4 kudos
6 More Replies
GoldenTuna
by • New Contributor II
  • 3334 Views
  • 5 replies
  • 2 kudos

Resolved! Mounting an Azure Storage Account in a cluster init script?

We are trying to configure our environment so when our cluster starts up, it checks to see if we have mounted our Azure storage account container and if is not, mount it. We can do this fine in a notebook however have no luck doing this through an in...

Data Engineering
Reply
  • 3334 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@David Kruetzkamp​ - Would you be happy to mark whichever answer helped the most as best? That will help other members find the solution more quickly.

  • 2 kudos
4 More Replies
User16752239289
by • Valued Contributor
  • 1415 Views
  • 1 replies
  • 1 kudos

Resolved! Failed to add S3 init script in job cluster

I use below payload to submit my job that include am init script saved on S3. The instance profile and init script worked on interactive cluster. But when I move to job cluster the init script cannot be configure. { "new_cluster": { "spar...

Data Engineering
Reply
  • 1415 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
User16752239289
Valued Contributor
  • 1 kudos

It is due to the region is missing. For init script saved in S3, the region field is required. The init script section should be like below :"init_scripts": [ { "s3": { "destination": "s3://<my bucket>...

  • 1 kudos
Labels