Hi,
I want to install Databricks Lakebridge on my Databricks environment and use the analyze and transpile commands through a python script.
The usecase is that we need to create an automated pipeline that will migrate the existing scripts from snowflake to databricks at a scheduled interval. The pipeline should first - fetch all DDLs from snowflake, then it should migrate it using Lakebridge and then the migrated scripts should be executed to create tables and functions.
I can get the DDLs from snowflake and dump it in unity catalog as .sql files but I am not able to run the Lakebridge on Databricks environment, I tried using the Web Terminal for Serverless compute but lakebridge installation fails with error:
"Error: python: create venv: /usr/bin/python3.12 -m venv /home/spark-***-***-***-***-***/.databricks/labs/lakebridge/state/venv: exit status 1"
I can use "subprocess" module of Python to run the Lakebridge commands but I cannot figure out how to install Lakebridge.
Please tell me if this scenario is even possible, any help is appreciated!
Please note that I have tried using Lakebridge on my local machine but my requirement is to create a pipeline for it, please help me with this.
Thanks in advance!
