The below approach is working for me... I had to download the packages upfront and place it on a volume though.#!/bin/bashset -euxo pipefailecho 'debconf debconf/frontend select Noninteractive' | debconf-set-selectionssudo ACCEPT_EULA=Y dpkg -i odbci...
I am not sure if you need to further curate the data before you upload it to Virtual machine, if you not you can just mount storage on VMCreate an SMB Azure file share and connect it to a Windows VM | Microsoft LearnAzure Storage - Create File Storag...
You could use Fixtures for SparkSession something like below import pytest
from pyspark.sql import SparkSession
@pytest.fixture(scope="session") # Consider "function" for reuse within a test function
def spark_session():
spark = SparkSession.bu...
Looking at the error message, issue might be related to mounting itself or incorrect reference to the mount points..Pls share the code snippet if you can
You could explore their "SPOT_WITH_FALLBAK" feature. If you don't want your jobs to fail because of eviction but this currently is not supported with interactive clusters. Hoping that they may extend this to all compute options soonCreate a pipeline ...
