cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Error using curl within a job

mb1234
New Contributor

I have a notebook that, as a first step, needs to download and install some drivers. The actual code is this:

%sh

# Install gdebi command line tool

apt-get -y install gdebi-core

# Install Posit professional drivers

curl -LO https://cdn.rstudio.com/drivers/7C152C12/installer/rstudio-drivers_2024.03.0_amd64.deb

The code works fine in a notebook.

But now I am trying to run this notebook as a job. The problem is that, when run as a job, this curl command cannot download the file, and so the rest of the script fails.

As a debugging step, I set the job to run as me and use the same all-purpose cluster that I was using for interactive development. It still can't download the file.

Is it possible to have this code execute in a job?

1 REPLY 1

szymon_dybczak
Contributor III

Hi @mb1234 ,

What error did you get?

Edit: I've checked and it worked in job 

Slash_0-1721675720471.png

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group