cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to write test code in databricks

rt-slowth
Contributor

 

 

 

 

from databricks.connect import DatabricksSession
from data.dbx_conn_info import DbxConnInfo

class SparkSessionManager:

    _instance = None
    _spark = None

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super(SparkSessionManager, cls).__new__(cls)
        return cls._instance

    def get_spark_session(self):
        if not self._spark:
            self._spark = DatabricksSession.builder.remote(
                host=f"https://{DbxConnInfo.retrieve_workspace_instance_name()}",
                token=DbxConnInfo.retrieve_token(),
                cluster_id=DbxConnInfo.retrieve_cluster_id()
            ).getOrCreate()
        return self._spark

 

 

 

 

When I write and run test code for this code, it keeps telling me that the databricks package doesn't exist. I don't know why.

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @rt-slowth , 

The databricks-connect module is not included by default in Python installations, so you must ensure it is installed before your code can import it.

You can install databricks-connect using pip.

Please run the following command in your terminal or command prompt and then try rerunning your code:

pip install 
databricks-connect
Additionally, make sure that you have also installed all the necessary packages that are required to run your code, including any custom packages that are used in the DbxConnInfo class.
 
If the data.dbx_conn_info module is not found, it could indicate an error in the module itself or that it is in a different location than where your code expects it to be.

Make sure that the DbxConnInfo class is located in the correct folder, and the folder is included in the Python path. You can verify the way by running the following code in your notebook:

If the folder containing the data The module is not in the list of system paths,; you can add it using the following code:
Replace /path/to/data/module with the actual path to the folder that contains the dbx_conn_info.py file.




 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!