cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

digui
by New Contributor
  • 2085 Views
  • 4 replies
  • 0 kudos

Issues when trying to modify log4j.properties

Hi y'all.​I'm trying to export metrics and logs to AWS cloudwatch, but while following their tutorial to do so, I ended up facing this error when trying to initialize my cluster with an init script they provided.This is the part where the script fail...

  • 2085 Views
  • 4 replies
  • 0 kudos
Latest Reply
cool_cool_cool
New Contributor II
  • 0 kudos

@digui Did you figure out what to do? We're facing the same issue, the script works for the executors.I was thinking on adding an if that checks if there is log4j.properties and modify it only if it exists

  • 0 kudos
3 More Replies
LukaszJ
by Contributor III
  • 10962 Views
  • 6 replies
  • 2 kudos

Resolved! Install ODBC driver by init script

Hello,I want to install ODBC driver (for pyodbc).I have tried to do it using terraform, however I think it is impossible.So I want to do it with Init Script in my cluster. I have the code from the internet and it works when it is on the beginning of ...

  • 10962 Views
  • 6 replies
  • 2 kudos
Latest Reply
MayaBakh_80151
New Contributor II
  • 2 kudos

Actually found this article and using this to migrate my shell script to workspace.Cluster-named and cluster-scoped init script migration notebook - Databricks 

  • 2 kudos
5 More Replies
niklas
by Contributor
  • 1537 Views
  • 2 replies
  • 1 kudos

Resolved! How can I specify a custom CRAN mirror to be used permanently by default when installing packages within R Notebooks?

When installing Notebook-scoped R libraries I don't want to manually specify the custom CRAN mirror each time like this:install.packages("diffdf", repos="my_custom_cran_url'')Instead I want to take the custom CRAN mirror URL by default so that I don'...

  • 1537 Views
  • 2 replies
  • 1 kudos
Latest Reply
niklas
Contributor
  • 1 kudos

Got solution on Stack Overflow for this problem: https://stackoverflow.com/a/76777228/18082636

  • 1 kudos
1 More Replies
Chaitanya_Raju
by Honored Contributor
  • 2755 Views
  • 7 replies
  • 0 kudos
  • 2755 Views
  • 7 replies
  • 0 kudos
Latest Reply
Vartika
Moderator
  • 0 kudos

Hi @Ratna Chaitanya Raju Bandaru​Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best? If not, please tell us so we can help you.Thanks!

  • 0 kudos
6 More Replies
glebex
by New Contributor II
  • 5499 Views
  • 8 replies
  • 8 kudos

Resolved! Accessing workspace files within cluster init script

Greetings all!I am currently facing an issue while accessing workspace files from the init script.As it was explained in the documentation, it is possible to place init script inside workspace files (link). This works perfectly fine and init script i...

  • 5499 Views
  • 8 replies
  • 8 kudos
Latest Reply
jacob_hill_prof
New Contributor II
  • 8 kudos

@Gleb Smolnik​ You might also want to try cloning a github repo in your init script and then storing dependencies like requirements.txt files and other init scripts there. By doing this you can pull a whole slew of init scripts to be utilized in your...

  • 8 kudos
7 More Replies
ossinova
by Contributor II
  • 1048 Views
  • 1 replies
  • 0 kudos

Creating cluster from ADF linked service with Workspace init script

Similar issue: https://stackoverflow.com/questions/76220211/create-new-databricks-cluster-from-adf-linked-service-with-initscripts-from-abfsI am trying to create clusters using ADF linked service where the cluster is configured with a init script. As...

  • 1048 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Oscar Dyremyhr​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
kfoster
by Contributor
  • 1144 Views
  • 2 replies
  • 3 kudos

Terraform Global Init Script base64encoding

I am working on converting manual global init scripts into a terraform IaC process for multiple environments. Within terraform, we are using the resource "databricks_global_init_script" and set the content_base64 with the following:base64encoded(<<-...

  • 1144 Views
  • 2 replies
  • 3 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 3 kudos

I am looking into it @Kristian Foster​ Are you able to get it working?

  • 3 kudos
1 More Replies
FRG96
by New Contributor III
  • 1340 Views
  • 0 replies
  • 0 kudos

How to set the ABFSS URL for Azure Databricks Init Scripts that have spaces in directory names?

I want to use an Init Script on ADLS Gen2 location for my Azure Databricks 11.3 and 12.2 clusters. The init_script.sh is placed in a directory that has spaces in it:https://storageaccount1.blob.core.windows.net/container1/directory%20with%20spaces/su...

  • 1340 Views
  • 0 replies
  • 0 kudos
Sulfikkar
by Contributor
  • 3212 Views
  • 8 replies
  • 4 kudos

Cluster Scoped init script through pulumi

I am trying to run a cluster-scoped init script through Pulumi. I have referred to this documentation https://learn.microsoft.com/en-us/azure/databricks/clusters/configure#spark-configuration However, looks like the documentation is not very clear.I ...

  • 3212 Views
  • 8 replies
  • 4 kudos
Latest Reply
Vivian_Wilfred
Honored Contributor
  • 4 kudos

Hi @Sulfikkar Basheer Shylaja​ , Why don't you store the init-script on DBFS and just pass the dbfs:/ path of the init script in Pulumi? You could just run this code on a notebook-%python dbutils.fs.put("/databricks/init-scripts/set-private-pip-repos...

  • 4 kudos
7 More Replies
yopbibo
by Contributor II
  • 1587 Views
  • 2 replies
  • 0 kudos

pip install in cluster using web UI and extra index

In an init script or a notebook, we can:pip install --index-url=<our private pypi url> --extra-index-url=https://pypi.org/simple <a module>In the cluster web UI (libraries -> install library), we can give only the url of our private repository, but n...

  • 1587 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Philippe CRAVE​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 0 kudos
1 More Replies
Rahul2025
by New Contributor III
  • 2152 Views
  • 4 replies
  • 4 kudos

Make environment variables defined in init script available to Spark JVM job?

Hi,We're using Databricks Runtime version 11.3LTS and executing a Spark Java Job using a Job Cluster. To automate the execution of this job, we need to define (source in from bash config files) some environment variables through an init script (clust...

  • 2152 Views
  • 4 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Rahul K​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 4 kudos
3 More Replies
Rahul2025
by New Contributor III
  • 2646 Views
  • 11 replies
  • 1 kudos

Limitation on size of init script

Hi,We're using Databricks Runtime version 11.3LTS and executing a Spark Java Job using a Job Cluster. To automate the execution of this job, we need to define (source in from bash config files) some environment variables through an init script (clust...

  • 2646 Views
  • 11 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Rahul K​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your ...

  • 1 kudos
10 More Replies
repcak
by New Contributor III
  • 1572 Views
  • 1 replies
  • 2 kudos

Init Scripts with mounted azure data lake storage gen2

I'm trying to access init script which is stored on mounted azure data lake storage gen2 to dbfsI mounted storage to dbfs:/mnt/storage/container/script.shand when i try to access it i got an error:Cluster scoped init script dbfs:/mnt/storage/containe...

  • 1572 Views
  • 1 replies
  • 2 kudos
Latest Reply
User16752239289
Valued Contributor
  • 2 kudos

I do not think the init script saved under mount point work and we do not suggest that. If you specify abfss , then the cluster need to be configured so that the cluster can authenticate and access the adls gen2 folder. Otherwise, the cluster will no...

  • 2 kudos
User16869510359
by Esteemed Contributor
  • 8353 Views
  • 2 replies
  • 0 kudos
  • 8353 Views
  • 2 replies
  • 0 kudos
Latest Reply
Mooune_DBU
Valued Contributor
  • 0 kudos

It's set as an environment variable called `DATABRICKS_RUNTIME_VERSION`In your init scripts, you just need to add a line to display or save the info (see python example below):import os print("DATABRICKS_RUNTIME_VERSION:",os.environ.get('DATABRICKS_R...

  • 0 kudos
1 More Replies
killjoy
by New Contributor III
  • 4083 Views
  • 7 replies
  • 0 kudos

Resolved! Pipeline failed while calling Databricks Notebook - Cluster Terminated

Hello,We have an Azure Data Factory pipeline running during the night, and one of the activities calls a Databricks Notebook with dynamic DatabricksInstancePoolId, ClusterVersion and Workers. Yesterday, it failed with with the following error:Cluster...

  • 4083 Views
  • 7 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

Hi @Rita Fernandes​,What are you trying to install in your init script? only the ODBC driver or some other libraries/dependencies?

  • 0 kudos
6 More Replies
Labels