Error using curl within a job

mb1234 — Mon, 22 Jul 2024 17:13:42 GMT

I have a notebook that, as a first step, needs to download and install some drivers. The actual code is this:

%sh

# Install gdebi command line tool

apt-get -y install gdebi-core

# Install Posit professional drivers

curl -LO https://cdn.rstudio.com/drivers/7C152C12/installer/rstudio-drivers_2024.03.0_amd64.deb

The code works fine in a notebook.

But now I am trying to run this notebook as a job. The problem is that, when run as a job, this curl command cannot download the file, and so the rest of the script fails.

As a debugging step, I set the job to run as me and use the same all-purpose cluster that I was using for interactive development. It still can't download the file.

Is it possible to have this code execute in a job?

Re: Error using curl within a job

szymon_dybczak — Mon, 22 Jul 2024 19:15:42 GMT

Hi @mb1234 ,

What error did you get?

Edit: I've checked and it worked in job

topic Error using curl within a job in Data Engineering

Error using curl within a job

Re: Error using curl within a job