Spark jobs not starting for saving

[OwenDB]

I am having issues with a new workspace that has been created and having trouble diagnosing what the issue is.

It seems to be related to compute resources, spark and storage.

I am an admin and able to create compute resources. Using SQL warehouses I can upload a csv to test saving as a table in unity catalog and hive (also via SQL), when using compute this operation hangs after the csv has uploaded to dbfs. Using the compute I have created (vanilla LTS, shared and personal) I can run most things on a notebook but spark jobs where I'm trying to write a delta table look like they are starting but never progress.

We have limited in-house expertise (we're a non profit), so any ideas on where I can point my infrastructure team to look?

Thanks

Owen.

[Lakshay]

Hi @OwenDB ,

Are you facing the issue while using SQL warehouse? If yes, can you try using the interactive cluster?

Also, have you had a look at the logs and do you see any message there?

[OwenDB]

Yes this looks like it, I'm on Azure and using a VNET if that makes any difference? (VNET gets referenced here https://kb.databricks.com/cloud/azure-vnet-jobs-not-progressing).

24/01/18 09:22:13 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

[OwenDB]

@Lakshay SQL warehouse is fine, I can create tables in hive and unity, via UI & SQL. I cannot do the same when using interactive cluster, either in SQL or a notebook.

@arjun_kr @Lakshay  I think I'll need to enable logs first. I'll go away and do that.

[OwenDB]

ok looks like it might be this for us....

You must have a separate pair of host/container subnets for each workspace that you deploy. It is unsupported to share subnets across workspaces or to deploy other Azure resources on the subnets that are used by your Azure Databricks workspace.

Thanks both for pointers to get us there.