Hi @Retired_mod ,
Thanks for your response. So that's the last 5 minutes of Log4J output, when libraries are installed. Is that what you mean? It fails quite exactly after 10 minutes of starting the Log4J logs. So it might be that the cluster is not in active/running state as long as the libraries as installed and therefore it is shut down?
23/10/19 12:10:11 INFO PoolingHiveClient: Hive metastore connection pool implementation is HikariCP
23/10/19 12:10:11 INFO LocalHiveClientsPool: Create Hive Metastore client pool of size 1
23/10/19 12:10:11 INFO DriverCorral: DBFS health check ok
23/10/19 12:10:12 INFO HiveClientImpl: Warehouse location for Hive client (version 0.13.1) is dbfs:/user/hive/warehouse
23/10/19 12:10:12 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
23/10/19 12:10:12 INFO ObjectStore: ObjectStore, initialize called
23/10/19 12:10:12 INFO Persistence: Property datanucleus.fixedDatastore unknown - will be ignored
23/10/19 12:10:12 INFO Persistence: Property datanucleus.connectionPool.idleTimeout unknown - will be ignored
23/10/19 12:10:12 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
23/10/19 12:10:12 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
23/10/19 12:10:12 INFO HikariDataSource: HikariPool-1 - Started.
23/10/19 12:10:13 INFO HikariDataSource: HikariPool-2 - Started.
23/10/19 12:10:13 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
23/10/19 12:10:16 INFO ObjectStore: Initialized ObjectStore
23/10/19 12:10:16 INFO HiveMetaStore: Added admin role in metastore
23/10/19 12:10:16 INFO HiveMetaStore: Added public role in metastore
23/10/19 12:10:16 INFO HiveMetaStore: No user is added in admin role, since config is empty
23/10/19 12:10:16 INFO HiveMetaStore: 0: get_database: default
23/10/19 12:10:16 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_database: default
23/10/19 12:10:17 INFO HiveMetaStore: 0: get_database: default
23/10/19 12:10:17 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_database: default
23/10/19 12:10:17 INFO DriverCorral: Metastore health check ok
23/10/19 12:10:43 INFO SharedDriverContext: Successfully attached library dbfs:/mnt/cddm-DEV/application/MIR_task/cddm-0.x-py3-none-any.whl to Spark
23/10/19 12:10:43 INFO LibraryState: [Thread 132] Successfully attached library dbfs:/mnt/cddm-DEV/application/MIR_task/cddm-0.x-py3-none-any.whl
23/10/19 12:10:43 INFO SharedDriverContext: [Thread 132] attachLibrariesToSpark PythonPyPiPkgId(torch,Some(1.13.1),None,List())
23/10/19 12:10:43 INFO SharedDriverContext: Attaching Python lib: python-pypi;torch;;1.13.1; to clusterwide nfs path
23/10/19 12:10:43 INFO Utils: resolved command to be run: List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, torch==1.13.1, --disable-pip-version-check)
23/10/19 12:13:57 INFO SharedDriverContext: Successfully attached library python-pypi;torch;;1.13.1; to Spark
23/10/19 12:13:57 INFO LibraryState: [Thread 132] Successfully attached library python-pypi;torch;;1.13.1;
23/10/19 12:13:57 INFO SharedDriverContext: [Thread 132] attachLibrariesToSpark PythonPyPiPkgId(lightning,Some(2.0.2),None,List())
23/10/19 12:13:57 INFO SharedDriverContext: Attaching Python lib: python-pypi;lightning;;2.0.2; to clusterwide nfs path
23/10/19 12:13:57 INFO Utils: resolved command to be run: List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, lightning==2.0.2, --disable-pip-version-check)
23/10/19 12:14:56 INFO DataSourceFactory$: DataSource Jdbc URL: jdbc:mariadb://consolidated-westeuropec2-prod-metastore-3.mysql.database.azure.com:3306/organization257243788442763?useSSL=true&sslMode=VERIFY_CA&disableSslHostnameVerification=true&trustServerCertificate=false&serverSslCert=/databricks/common/mysql-ssl-ca-cert.crt
23/10/19 12:14:56 INFO HikariDataSource: metastore-monitor - Starting...
23/10/19 12:14:56 INFO HikariDataSource: metastore-monitor - Start completed.
23/10/19 12:14:56 INFO HikariDataSource: metastore-monitor - Shutdown initiated...
23/10/19 12:14:56 INFO HikariDataSource: metastore-monitor - Shutdown completed.
23/10/19 12:14:56 INFO MetastoreMonitor: Metastore healthcheck successful (connection duration = 194 milliseconds)
23/10/19 12:15:09 INFO SharedDriverContext: Successfully attached library python-pypi;lightning;;2.0.2; to Spark
23/10/19 12:15:09 INFO LibraryState: [Thread 132] Successfully attached library python-pypi;lightning;;2.0.2;
23/10/19 12:15:09 INFO SharedDriverContext: [Thread 132] attachLibrariesToSpark PythonPyPiPkgId(pytorch-lightning,Some(2.0.2),None,List())
23/10/19 12:15:09 INFO SharedDriverContext: Attaching Python lib: python-pypi;pytorch-lightning;;2.0.2; to clusterwide nfs path
23/10/19 12:15:09 INFO Utils: resolved command to be run: List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, pytorch-lightning==2.0.2, --disable-pip-version-check)
23/10/19 12:15:11 INFO DriverCorral: DBFS health check ok
23/10/19 12:15:11 INFO HiveMetaStore: 0: get_database: default
23/10/19 12:15:11 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_database: default
23/10/19 12:15:11 INFO DriverCorral: Metastore health check ok
23/10/19 12:15:15 INFO SharedDriverContext: Successfully attached library python-pypi;pytorch-lightning;;2.0.2; to Spark
23/10/19 12:15:15 INFO LibraryState: [Thread 132] Successfully attached library python-pypi;pytorch-lightning;;2.0.2;
23/10/19 12:15:15 INFO SharedDriverContext: [Thread 132] attachLibrariesToSpark PythonPyPiPkgId(sentence-transformers,Some(2.2.2),None,List())
23/10/19 12:15:15 INFO SharedDriverContext: Attaching Python lib: python-pypi;sentence-transformers;;2.2.2; to clusterwide nfs path
23/10/19 12:15:15 INFO Utils: resolved command to be run: List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, sentence-transformers==2.2.2, --disable-pip-version-check)