10-25-2021 09:03 AM
While trying to install ffmpeg package using an init script on Databricks cluster, it fails with the below error.
Init script:
#! /bin/bash
set -e
sudo apt-get update
sudo apt-get -y install ffmpeg
Error message:
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libc/libcaca/libcaca0_0.99.beta19-2.1ubuntu1.20.04.1_amd... 404 Not Found [IP: 91.189.88.152 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/m/mesa/mesa-va-drivers_21.0.3-0ubuntu0.3~20.04.2_amd6... 404 Not Found [IP: 91.189.88.152 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/mesa-vdpau-drivers_21.0.3-0ubuntu0.3~20.04.2_amd64... 404 Not Found [IP: 91.189.88.152 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
10-25-2021 09:08 AM
Cause: The VMs are pointing to the cached old mirror which is not up-to-date. Hence there is a problem with downloading the package and it's failing.
Workaround: Use the below init script to install the package "ffmpeg". To revert to the original list, just overwrite the .bak file.
This init script works for all possible library installation failures that are caused by the mirror sync issues.
dbutils.fs.put("/databricks/ffmpeg_install.sh", """
#! /bin/bash
# do a backup of the source
cp /etc/apt/sources.list{,.bak}
# remove the cached mirrors
r="deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted"
add-apt-repository --remove "${r}"
r="deb http://archive.ubuntu.com/ubuntu/ focal-updates universe"
add-apt-repository --remove "${r}"
r="deb http://security.ubuntu.com/ubuntu/ focal-security main restricted"
add-apt-repository --remove "${r}"
# update apt & install the package
apt-get update
apt-get install -y ffmpeg
""", True)
Or a simple cleaner method to do this.
rm -f /var/cache/apt /var/lib/apt/lists
apt-get update
apt-get install -y ffmpeg
10-25-2021 09:08 AM
Cause: The VMs are pointing to the cached old mirror which is not up-to-date. Hence there is a problem with downloading the package and it's failing.
Workaround: Use the below init script to install the package "ffmpeg". To revert to the original list, just overwrite the .bak file.
This init script works for all possible library installation failures that are caused by the mirror sync issues.
dbutils.fs.put("/databricks/ffmpeg_install.sh", """
#! /bin/bash
# do a backup of the source
cp /etc/apt/sources.list{,.bak}
# remove the cached mirrors
r="deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted"
add-apt-repository --remove "${r}"
r="deb http://archive.ubuntu.com/ubuntu/ focal-updates universe"
add-apt-repository --remove "${r}"
r="deb http://security.ubuntu.com/ubuntu/ focal-security main restricted"
add-apt-repository --remove "${r}"
# update apt & install the package
apt-get update
apt-get install -y ffmpeg
""", True)
Or a simple cleaner method to do this.
rm -f /var/cache/apt /var/lib/apt/lists
apt-get update
apt-get install -y ffmpeg
07-05-2022 03:28 AM
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.