topic Library installation fails with mirror sync issue in Data Engineering

Library installation fails with mirror sync issue

Prabakar — Mon, 25 Oct 2021 16:03:24 GMT

While trying to install ffmpeg package using an init script on Databricks cluster, it fails with the below error.

Init script:

#! /bin/bash
set -e
sudo apt-get update
sudo apt-get -y install ffmpeg

Error message:

E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libc/libcaca/libcaca0_0.99.beta19-2.1ubuntu1.20.04.1_amd64.deb  404  Not Found [IP: 91.189.88.152 80]
 
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/m/mesa/mesa-va-drivers_21.0.3-0ubuntu0.3~20.04.2_amd64.deb  404  Not Found [IP: 91.189.88.152 80]
 
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/mesa-vdpau-drivers_21.0.3-0ubuntu0.3~20.04.2_amd64.deb  404  Not Found [IP: 91.189.88.152 80]
 
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Re: Library installation fails with mirror sync issue

Prabakar — Mon, 25 Oct 2021 16:08:38 GMT

Cause: The VMs are pointing to the cached old mirror which is not up-to-date. Hence there is a problem with downloading the package and it's failing.

Workaround: Use the below init script to install the package "ffmpeg". To revert to the original list, just overwrite the .bak file.

This init script works for all possible library installation failures that are caused by the mirror sync issues.

dbutils.fs.put("/databricks/ffmpeg_install.sh", """ 
 
#! /bin/bash
# do a backup of the source 
cp /etc/apt/sources.list{,.bak}
 
# remove the cached mirrors
r="deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted"
add-apt-repository --remove "${r}"
r="deb http://archive.ubuntu.com/ubuntu/ focal-updates universe"
add-apt-repository --remove "${r}"
r="deb http://security.ubuntu.com/ubuntu/ focal-security main restricted"
add-apt-repository --remove "${r}"
 
# update apt & install the package
apt-get update
apt-get install -y ffmpeg
 
""", True)

Or a simple cleaner method to do this.

rm -f  /var/cache/apt /var/lib/apt/lists
apt-get update 
apt-get install -y ffmpeg

Re: Library installation fails with mirror sync issue

Saman — Tue, 05 Jul 2022 10:28:59 GMT

I have the same issue. Unable to install ffmpeg.

I tried running the given init script, it outputs "504 bytes wrote", and ffmpeg doesn't gets installed.

I tried the "rm" command, it says "Is a directory"

Unable to install ffmpeg at this point.

Kindly help.