cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to write test code in databricks

rt-slowth
Contributor

 

 

 

 

from databricks.connect import DatabricksSession
from data.dbx_conn_info import DbxConnInfo

class SparkSessionManager:

    _instance = None
    _spark = None

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super(SparkSessionManager, cls).__new__(cls)
        return cls._instance

    def get_spark_session(self):
        if not self._spark:
            self._spark = DatabricksSession.builder.remote(
                host=f"https://{DbxConnInfo.retrieve_workspace_instance_name()}",
                token=DbxConnInfo.retrieve_token(),
                cluster_id=DbxConnInfo.retrieve_cluster_id()
            ).getOrCreate()
        return self._spark

 

 

 

 

When I write and run test code for this code, it keeps telling me that the databricks package doesn't exist. I don't know why.

1 REPLY 1

Kaniz
Community Manager
Community Manager

Hi @rt-slowth , 

The databricks-connect module is not included by default in Python installations, so you must ensure it is installed before your code can import it.

You can install databricks-connect using pip.

Please run the following command in your terminal or command prompt and then try rerunning your code:

pip install 
databricks-connect
Additionally, make sure that you have also installed all the necessary packages that are required to run your code, including any custom packages that are used in the DbxConnInfo class.
 
If the data.dbx_conn_info module is not found, it could indicate an error in the module itself or that it is in a different location than where your code expects it to be.

Make sure that the DbxConnInfo class is located in the correct folder, and the folder is included in the Python path. You can verify the way by running the following code in your notebook:

If the folder containing the data The module is not in the list of system paths,; you can add it using the following code:
Replace /path/to/data/module with the actual path to the folder that contains the dbx_conn_info.py file.




 

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.