topic Re: Adding extra libraries to databricks (rosbag) in Data Engineering

Adding extra libraries to databricks (rosbag)

PiotrU — Wed, 05 Jun 2024 14:28:50 GMT

Hello

I have interesting challenge, I am required to install few libraries which are part of rosbag packages, for allowing some data deserialization tasks.

While creating cluster I do use init_script that install this software using apt

sudo apt update && sudo apt install -y curl gnupg2 lsb-release sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | sudo apt-key add - sudo sh -c 'echo "deb [arch=amd64] http://packages.ros.org/ros2/ubuntu $(lsb_release -cs) main" > /etc/apt/sources.list.d/ros2-latest.list' sudo apt update sudo apt install -y ros-humble-ros-base ros-humble-rclpy ros-humble-std-msgs python3-argcomplete /databricks/python/bin/pip install mcap echo "source /opt/ros/humble/setup.bash" >> ~/.bashrc source /opt/ros/humble/setup.bash PYTHON_VERSION=$(/databricks/python3/bin/python3 --version | cut -d' ' -f2 | cut -d'.' -f1-2) echo "export PYTHONPATH=\$PYTHONPATH:/opt/ros/humble/lib/python${PYTHON_VERSION}/site-packages" >> ~/.bashrc export PYTHONPATH=$PYTHONPATH:/opt/ros/humble/lib/python${PYTHON_VERSION}/site-packages

That part is working fine, the trouble is starting when I try to use it (whether is it master or worker node), I have selected example library rclpy - whenever I do execute it via "sh" and start with sourcing variables - it is working fine

However - If I would like to do same thing - natively using notebook - it doesn't find libraries

I have a script which read the variables generated by /opt/ros/humble/setup.bash -

import subprocess import os def source_ros_setup(): # Source the ROS 2 setup script and capture the environment variables command = ['bash', '-c', 'source /opt/ros/humble/setup.bash && env'] proc = subprocess.Popen(command, stdout=subprocess.PIPE, universal_newlines=True) env_vars = {} for line in proc.stdout: key, _, value = line.partition("=") env_vars[key] = value.strip() proc.communicate() # Update os.environ with the new environment variables os.environ.update(env_vars) # Source the ROS 2 setup script source_ros_setup() # Verify environment variables (Optional) print("PYTHONPATH:", os.environ.get('PYTHONPATH')) print("PATH:", os.environ.get('PATH')) print("LD_LIBRARY_PATH:", os.environ.get('LD_LIBRARY_PATH')) # Now run your main code try: import rclpy print("rclpy is installed and accessible") # Your main ROS 2 related code goes here except ModuleNotFoundError: print("rclpy is not found, ensure the correct PYTHONPATH is set")

but it also doesn't work -

"rclpy is not found, ensure the correct PYTHONPATH is set"

Any ideas?

Re: Adding extra libraries to databricks (rosbag)

shan_chandra — Wed, 05 Jun 2024 17:46:27 GMT

@PiotrU - Can you please check if any error in the driver logs regarding this library installation? Do you have the required access to install as a sudo user? It may require password (when i tried locally )

sudo apt install -y ros-humble-ros-base ros-humble-rclpy ros-humble-std-msgs python3-argcomplete

Re: Adding extra libraries to databricks (rosbag)

PiotrU — Wed, 05 Jun 2024 18:13:08 GMT

There are no issues while I do install packages

Re: Adding extra libraries to databricks (rosbag)

PiotrU — Thu, 06 Jun 2024 08:44:15 GMT

Issue mostly solved:

import sys import os library_path = "/opt/ros/humble/local/lib/python3.10/dist-packages" if library_path not in sys.path: sys.path.append(library_path)

That solves issue with databricks not founding libraries, still - it gives some challenges while loading library - but they are not related to paths.

Re: Adding extra libraries to databricks (rosbag)

amandaK — Sat, 17 Aug 2024 02:02:45 GMT

@PiotrU did adding the path to sys.path resolve all of your ModuleNotFoundErrors? i'm trying to do something similar and adding the path to the sys.path resolved ModuleNotFoundError for rclpy, but i continue to see others related to ros

Re: Adding extra libraries to databricks (rosbag)

amandaK — Mon, 19 Aug 2024 07:20:34 GMT

After working on it a bit, I was able to get rid of all the ModuleNotFoundErrors, but can't seem to figure out how to resolve this issue

ImportError: librcl_action.so: cannot open shared object file: No such file or directory

Did you happen to run into this as well?

@amandaK wrote:
@PiotrU did adding the path to sys.path resolve all of your ModuleNotFoundErrors? i'm trying to do something similar and adding the path to the sys.path resolved ModuleNotFoundError for rclpy, but i continue to see others related to ros

Re: Adding extra libraries to databricks (rosbag)

PiotrU — Wed, 04 Sep 2024 10:39:47 GMT

nope, also - end of the day, I've totally dropped usage of ros env on databricks