โ05-15-2024 12:02 AM
We have a pipeline in Azure Devops that deploys init scripts to the workspace folder on an Azure Databricks resource using the workspace API (/api/2.0/workspace/import), we use format "AUTO" and overwrite "true" to achieve this. After being uploaded the init script will fail and the cluster will is unable to be started (reason: init script failure). This behavior will persist until someone opens the init script in the workspace (without making any changes) and then the cluster will be able to run the init script normally.
โ05-21-2024 10:43 PM
If anyone else comes across this problem, the issue was a deployment powershell script was changing LF to CRLF before upload in the init script. The solution was to upload with LF line endings in the pipeline.
โ05-15-2024 02:17 AM
The azure devops pipeline never runs the script, it only uploads it to the workspace. The script should be run by the cluster on initialization, which after opening it on the workspace file system in the browser, it does on subsequent initialization of the cluster. If the file is uploaded and not opened in the browser, the init script will fail. Sorry, didn't mean to reply at the top level before.
โ05-15-2024 02:14 AM - edited โ05-15-2024 02:19 AM
The init script only generates logs in the scenario that it runs, not when it fails.
โ05-21-2024 10:43 PM
If anyone else comes across this problem, the issue was a deployment powershell script was changing LF to CRLF before upload in the init script. The solution was to upload with LF line endings in the pipeline.
Passionate about hosting events and connecting people? Help us grow a vibrant local communityโsign up today to get started!
Sign Up Now