topic Spark jobs not starting for saving in Administration & Architecture

Spark jobs not starting for saving

OwenDB — Wed, 17 Jan 2024 17:12:31 GMT

I am having issues with a new workspace that has been created and having trouble diagnosing what the issue is.

It seems to be related to compute resources, spark and storage.

I am an admin and able to create compute resources. Using SQL warehouses I can upload a csv to test saving as a table in unity catalog and hive (also via SQL), when using compute this operation hangs after the csv has uploaded to dbfs. Using the compute I have created (vanilla LTS, shared and personal) I can run most things on a notebook but spark jobs where I'm trying to write a delta table look like they are starting but never progress.

We have limited in-house expertise (we're a non profit), so any ideas on where I can point my infrastructure team to look?

Thanks

Owen.

Re: Spark jobs not starting for saving

Lakshay — Wed, 17 Jan 2024 19:29:16 GMT

Hi @OwenDB ,

Are you facing the issue while using SQL warehouse? If yes, can you try using the interactive cluster?

Also, have you had a look at the logs and do you see any message there?

Re: Spark jobs not starting for saving

OwenDB — Wed, 17 Jan 2024 21:58:59 GMT

@Lakshay SQL warehouse is fine, I can create tables in hive and unity, via UI & SQL. I cannot do the same when using interactive cluster, either in SQL or a notebook.

@arjun_kr @Lakshay I think I'll need to enable logs first. I'll go away and do that.

Re: Spark jobs not starting for saving

OwenDB — Thu, 18 Jan 2024 09:35:23 GMT

Yes this looks like it, I'm on Azure and using a VNET if that makes any difference? (VNET gets referenced here https://kb.databricks.com/cloud/azure-vnet-jobs-not-progressing).

24/01/18 09:22:13 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Re: Spark jobs not starting for saving

OwenDB — Thu, 18 Jan 2024 11:17:56 GMT

ok looks like it might be this for us....

You must have a separate pair of host/container subnets for each workspace that you deploy. It is unsupported to share subnets across workspaces or to deploy other Azure resources on the subnets that are used by your Azure Databricks workspace.

Thanks both for pointers to get us there.