Hi @gehbiszumeis
This is a common challenge when working with Databricks jobs and git integration, especially when you need additional libraries that aren't directly supported by Databricks.
You're right that the working directory path is dynamically generated for each job run. Let me help you create a solution to get this path during runtime and copy your library.
There are a few approaches you can take:
1. Using Python's os and inspect modules
The simplest approach is to use Python's built-in modules to determine the directory of your running script:
import os
import inspect
# Get the current script's directory
current_file = inspect.getfile(inspect.currentframe())
current_dir = os.path.dirname(os.path.abspath(current_file))
print(f"Current script directory: {current_dir}")
# Now you can copy your library to this location
2. Using __file__ variable
Even simpler, you can use the __file__ variable which contains the path to the current script:
import os
# Get the current script's directory
current_dir = os.path.dirname(os.path.abspath(__file__))
print(f"Current script directory: {current_dir}")
3. Creating an init script
If you want to set up an initialization script that runs before your main script, you can create a file that:
-- Determines the script location
-- Copies the library from your workspace to that location
-- Sets up any necessary environment variables
Here's an example of what such an init script might look like:
# init_script.py
import os
import shutil
import sys
def setup_environment():
# Get the current script directory
current_dir = os.path.dirname(os.path.abspath(__file__))
print(f"Current directory: {current_dir}")
# Source library location in workspace
library_source = "/Workspace/path/to/your/library"
# Destination in the current directory
library_dest = os.path.join(current_dir, "library_name")
# Copy the library if it doesn't exist
if not os.path.exists(library_dest):
print(f"Copying library from {library_source} to {library_dest}")
shutil.copytree(library_source, library_dest)
# Add the library to Python path if needed
if library_dest not in sys.path:
sys.path.append(library_dest)
print(f"Added {library_dest} to Python path")
if __name__ == "__main__":
setup_environment()
Then in your main script, you would import and run this init script first:
# main_script.py
import init_script
# Run the setup
init_script.setup_environment()
# Now continue with your main script logic
# You can now import from your library
from library_name import some_module
LR
