topic Re: Dbfs init script migration in Data Engineering

Dbfs init script migration

Ameshj — Wed, 01 May 2024 09:08:49 GMT

I need help with migrating from dbfs on databricks to workspace. I am new to databricks and am struggling with what is on the links provided.

My workspace.yml also has dbfs hard-coded. Included is a full deployment with great expectations.

This was done by an external vendor.

Re: Dbfs init script migration

NandiniN — Wed, 01 May 2024 09:43:31 GMT

Hi @Ameshj ,

I do not see any attachments and links in this post.

However I will add the document which I feel should help you https://docs.databricks.com/en/_extras/documents/aws-init-workspace-files.pdf

If you can specify the specific error that you are facing, we can guide you better.

Thanks!

Re: Dbfs init script migration

NandiniN — Wed, 01 May 2024 09:45:32 GMT

There's also this KB specific to init script migration - https://kb.databricks.com/clusters/migration-guidance-for-init-scripts-on-dbfs

Re: Dbfs init script migration

Ameshj — Thu, 02 May 2024 04:14:10 GMT

Hello - thank you for your reply.

I have only gone as far as creating a new folder and copied the generate init file over.

where it says copy from dbfs - I am unable to as dbfs pathway nomore exists.

Regards

Re: Dbfs init script migration

Ameshj — Thu, 02 May 2024 04:40:22 GMT

hi

attached the global init script migration - but it does not work either.

dbutils.fs.put(
"/databricks/scripts/pyodbc-install.sh","""
#!/bin/bash
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list > /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql17
# optional: for bcp and sqlcmd
sudo ACCEPT_EULA=Y apt-get install -y mssql-tools
echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
source ~/.bashrc
# optional: for unixODBC development headers
sudo apt-get install -y unixodbc-dev""",
True)

it says script exit status is non-zero

Re: Dbfs init script migration

Ameshj — Thu, 02 May 2024 04:52:08 GMT

here is another error I received when running the visualize job on databricks.

Re: Dbfs init script migration

NandiniN — Sun, 05 May 2024 17:36:44 GMT

Hi @Ameshj ,

Sorry for the delay in the response.

For the all_df screenshot - how are you creating that df? Does it contain Tablename? How is it related to init script migration?

Kindly add set -x after the first line, and enable cluster logs to DBFS and share the logs if possible.

Thanks & Regards,

Nandini

Re: Dbfs init script migration

NandiniN — Mon, 06 May 2024 13:37:47 GMT

One of the other suggestions is to use Lakehouse Federation. It is possible it may be a driver issue (we will get to know from the logs)

Re: Dbfs init script migration

Ameshj — Wed, 08 May 2024 04:31:36 GMT

hi
The issue is that DBFS is not there anymore since it reached end of life.
now when the cluster starts to run the job - it needed to fire off the init script - but it cant anymore. (cluster settings image)
So how do i get the init script to start up else where - i am stuck here?

Re: Dbfs init script migration

NandiniN — Tue, 21 May 2024 09:18:22 GMT

#!/bin/bash
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list > /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql17
# optional: for bcp and sqlcmd
sudo ACCEPT_EULA=Y apt-get install -y mssql-tools
echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
source ~/.bashrc
# optional: for unixODBC development headers
sudo apt-get install -y unixodbc-dev

Re: Dbfs init script migration

Ameshj — Wed, 07 Aug 2024 04:50:13 GMT

wow, you are a magician in this.

Thank you for your guidance and assistance. It Worked finally.

Re: Dbfs init script migration

NandiniN — Thu, 31 Oct 2024 10:04:52 GMT

Hi @Ameshj , can you please "accept as solution" what worked for you.

Re: Dbfs init script migration

NandiniN — Thu, 31 Oct 2024 10:05:10 GMT

Glad it worked and helped you.