topic Re: How to import a function to another notebook using Repos without %run? in Data Engineering

How to import a function to another notebook using Repos without %run?

maranBH — Tue, 19 Oct 2021 20:41:18 GMT

Hi all,

I was reading the Repos documentation: https://docs.databricks.com/repos.html#migrate-from-run-commands

It is explained that, one advantage of Repos is no longer necessary to use %run magic command to make funcions available in one notebook to another. That is to say, we can import them with:

"from notebook_in_repos import fun"

I tested it out on Repos, but it doesn´t work. I get: "No module named notebook_in_repos"

I really want this feature. It is painfull to build an entire module just to do an import. And on the other hand, the %run magic command is not a good replace; it overwrites local variables and mess up namespaces.

How can I make this work?

Thanks.

Re: How to import a function to another notebook using Repos without %run?

User16829050420 — Wed, 20 Oct 2021 08:38:25 GMT

You can import the functions using the example mentioned in the blog. https://databricks.com/blog/2021/10/07/databricks-repos-is-now-generally-available.html

Let me know if this helps with your use case?

Re: How to import a function to another notebook using Repos without %run?

-werners- — Wed, 20 Oct 2021 09:59:22 GMT

The thing in Repos is to point to the correct path.

Repos is just like in any local filesystem.

So in your case you need a notebook_in_repos folder with your .py files containing the functions.

Re: How to import a function to another notebook using Repos without %run?

madhuchennu — Wed, 20 Oct 2021 10:46:11 GMT

My solution was to tell Python of that additional module import path by adding a snippet like this one to the notebook:

import os

import sys

module_path = os.path.abspath(os.path.join('..'))

if module_path not in sys.path:

sys.path.append(module_path)

This allows you to import the desired function from the module hierarchy:

from project1.lib.module import function

# use the function normally

function(...)

Note that it is necessary to add empty __init__.py files to project1/ and lib/ folders if you don't have them already.

If you want to know more about functions in Python go through EncodingCompiler

Re: How to import a function to another notebook using Repos without %run?

maranBH — Fri, 22 Oct 2021 14:25:54 GMT

Thank you all for your help! I tried all that was suggested; but I finally realized it was my fault in first place:

I was testing Files in Repos with a runtime < 8.4.
I was trying to import a file from a DB Notebook instead of a static .py file.

Upgrading my runtime and migrating all my code to a .py file did the trick!

Thank you all again!

Re: How to import a function to another notebook using Repos without %run?

JakubSkibicki — Wed, 06 Nov 2024 10:25:26 GMT

Due to new functionalies in Runtime 16.0 regarding autoload i came across this autoload.

Performaed a practical test. It works. However had some problems at first.

As in solution the key was that definitions are places in a file.py not a notebook.