cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Library installation fails with mirror sync issue

Prabakar
Esteemed Contributor III

While trying to install ffmpeg package using an init script on Databricks cluster, it fails with the below error.

Init script:

#! /bin/bash
set -e
sudo apt-get update
sudo apt-get -y install ffmpeg

Error message:

E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libc/libcaca/libcaca0_0.99.beta19-2.1ubuntu1.20.04.1_amd...  404  Not Found [IP: 91.189.88.152 80]
 
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/m/mesa/mesa-va-drivers_21.0.3-0ubuntu0.3~20.04.2_amd6...  404  Not Found [IP: 91.189.88.152 80]
 
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/mesa-vdpau-drivers_21.0.3-0ubuntu0.3~20.04.2_amd64...  404  Not Found [IP: 91.189.88.152 80]
 
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

1 ACCEPTED SOLUTION

Accepted Solutions

Prabakar
Esteemed Contributor III

Cause: The VMs are pointing to the cached old mirror which is not up-to-date. Hence there is a problem with downloading the package and it's failing. 

Workaround: Use the below init script to install the package "ffmpeg". To revert to the original list, just overwrite the .bak file.

This init script works for all possible library installation failures that are caused by the mirror sync issues.

dbutils.fs.put("/databricks/ffmpeg_install.sh", """ 
 
#! /bin/bash
# do a backup of the source 
cp /etc/apt/sources.list{,.bak}
 
# remove the cached mirrors
r="deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted"
add-apt-repository --remove "${r}"
r="deb http://archive.ubuntu.com/ubuntu/ focal-updates universe"
add-apt-repository --remove "${r}"
r="deb http://security.ubuntu.com/ubuntu/ focal-security main restricted"
add-apt-repository --remove "${r}"
 
# update apt & install the package
apt-get update
apt-get install -y ffmpeg
 
""", True)

 Or a simple cleaner method to do this.

rm -f  /var/cache/apt /var/lib/apt/lists
apt-get update 
apt-get install -y ffmpeg

View solution in original post

2 REPLIES 2

Prabakar
Esteemed Contributor III

Cause: The VMs are pointing to the cached old mirror which is not up-to-date. Hence there is a problem with downloading the package and it's failing. 

Workaround: Use the below init script to install the package "ffmpeg". To revert to the original list, just overwrite the .bak file.

This init script works for all possible library installation failures that are caused by the mirror sync issues.

dbutils.fs.put("/databricks/ffmpeg_install.sh", """ 
 
#! /bin/bash
# do a backup of the source 
cp /etc/apt/sources.list{,.bak}
 
# remove the cached mirrors
r="deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted"
add-apt-repository --remove "${r}"
r="deb http://archive.ubuntu.com/ubuntu/ focal-updates universe"
add-apt-repository --remove "${r}"
r="deb http://security.ubuntu.com/ubuntu/ focal-security main restricted"
add-apt-repository --remove "${r}"
 
# update apt & install the package
apt-get update
apt-get install -y ffmpeg
 
""", True)

 Or a simple cleaner method to do this.

rm -f  /var/cache/apt /var/lib/apt/lists
apt-get update 
apt-get install -y ffmpeg

Saman
New Contributor II

remove commandinit scriptI have the same issue. Unable to install ffmpeg.

I tried running the given init script, it outputs "504 bytes wrote", and ffmpeg doesn't gets installed.

I tried the "rm" command, it says "Is a directory"

Unable to install ffmpeg at this point.

Kindly help.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group