kabir
New Contributor II

‎03-02-2025 12:44 AM

Hi Team,

I have also facing same issue 

I follow  below steps

1. load_data.py

%python
from pyspark.sql import SparkSession
# Initialize Spark Session
spark = SparkSession.builder.appName("LoadData").getOrCreate()

def load_data_to_target(df😞
"""Inserts the given DataFrame into target_test_table."""
df.write.format("delta").mode("append").saveAsTable("workspace.default.target_test_table")
print("Data successfully inserted into target_test_table")

 

2. extract_data.py 

%python
from pyspark.sql import SparkSession
from load_data import load_data_to_target

# Initialize Spark Session
spark = SparkSession.builder.appName("ExtractData").getOrCreate()

def extract_data():
"""Extracts data from test_table and returns a DataFrame."""
df = spark.sql("SELECT * FROM workspace.default.test_table")
return df

# Extract the data
df_test_table = extract_data()

# Call load_data_to_target() and pass the DataFrame
load_data_to_target(df_test_table)

print("Data successfully transferred from test_table to target_test_table")

when runing extract_data.py geting below error

ModuleNotFoundError: No module named 'load_data'
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) File <command-5830530348176606>, line 2 1 from pyspark.sql import SparkSession ----> 2 from load_data import load_data_to_target 4 # Initialize Spark Session 5 spark = SparkSession.builder.appName("ExtractData").getOrCreate() File /databricks/python_shell/lib/dbruntime/autoreload/discoverability/hook.py:81, in AutoreloadDiscoverabilityHook.pre_run_cell.<locals>.patched_import(name, *args, **kwargs) 75 if not self._should_hint and ( 76 (module := sys.modules.get(absolute_name)) is not None and 77 (fname := get_allowed_file_name_or_none(module)) is not None and 78 (mtime := os.stat(fname).st_mtime) > self.last_mtime_by_modname.get( 79 absolute_name, float("inf")) and not self._should_hint): 80 self._should_hint = True ---> 81 module = self._original_builtins_import(name, *args, **kwargs) 82 if (fname := fname or get_allowed_file_name_or_none(module)) is not None: 83 mtime = mtime or os.stat(fname).st_mtime ModuleNotFoundError: No module named 'load_data'
0 Kudos