Louis_Frolio
Databricks Employee
Databricks Employee

‎04-10-2025 07:13 AM

I just thought of one more question.  Are you still trying to run your init scripts out of DBFS?  It could be causing your issues.  Here are some notes on the end of life for DBFS init scripts, hope this helps.

 

As of March 31, 2024, the End-of-Life (EoL) for running init scripts from the Databricks File System (DBFS) has been implemented for all customers that did not request an extension. Databricks no longer supports init scripts stored on DBFS for the latest Databricks Runtime (DBR) versions, and users are required to migrate their scripts to alternative storage options.
Here are the supported alternatives for storing init scripts based on your Databricks Runtime version and setup:
  1. Unity Catalog Volumes:
    • Recommended if using Databricks Runtime 13.3 LTS or above with Unity Catalog-enabled clusters.
    • Use this for secure and governed storage when Unity Catalog is available.
  2. Workspace Files:
    • Suitable for Databricks Runtime 11.3 LTS and above without Unity Catalog.
    • Workspace files provide a Databricks-native storage option with file-level permissions.
  3. Cloud Object Storage:
    • For Databricks Runtime 10.4 LTS or below, or for more complex use cases referencing additional files.
    • Examples: Amazon S3, Azure Data Lake Storage (ADLS), or Google Cloud Storage (GCS).
Summary of Supported Runtimes for Migration and Storage: - Databricks Runtime 13.3 LTS and above (with Unity Catalog): Use Unity Catalog Volumes. - Databricks Runtime 11.3 LTS and above (without Unity Catalog): Use Workspace Files. - Databricks Runtime 10.4 LTS and below: Use external cloud object storage.
It is highly recommended that users on unsupported runtimes upgrade to a supported DBR as soon as possible to avoid potential issues. If needed, migration guidance is available in the public Databricks documentation, which details steps for transitioning scripts to these secure and supported storage formats.