topic Re: Can't run SQL on my cluster in Warehousing & Analytics

Can't run SQL on my cluster

neesam — Tue, 04 Nov 2025 22:58:41 GMT

I'm relatively new to Databricks and Spark and have decided to create a Spark cluster with AWS under the free 14 day trial.

The JSON to the cluster is as follows:

{
"data_security_mode": "DATA_SECURITY_MODE_DEDICATED",
"single_user_name": "me@gmail.com",
"cluster_name": "me@gmail.com's Cluster 2025-11-04 00:20:21",
"kind": "CLASSIC_PREVIEW",
"aws_attributes": {
"zone_id": "auto",
"availability": "SPOT_WITH_FALLBACK"
},
"runtime_engine": "PHOTON",
"spark_version": "16.4.x-scala2.12",
"node_type_id": "rd-fleet.xlarge",
"autotermination_minutes": 30,
"is_single_node": false,
"autoscale": {
"min_workers": 2,
"max_workers": 8
},
"cluster_id": "MY_ID"
}

I created a table from a CSV file, which I uploaded under the workspace.

I created a notebook with which I've attached the running cluster to. I'm able to run basic Python just fine (as well as utilize Spark to create a dataframe and successfully showing the dataframe) within the notebook, getting results back almost instantaneously. However, when I try to run SQL, the request is left hanging.

For example, the following code hangs indefinitely:

%sql

SHOW TABLES

I've gone into my workspace and granted myself all permissions. I also granted myself all permissions for the schema of which the table is located under.

The metastore that is attached to my cluster is of the same region. I also granted myself all permissions for the metastore.

I'm not sure what to do next.

Re: Can't run SQL on my cluster

Chiran-Gajula — Wed, 05 Nov 2025 02:46:17 GMT

Hello Neesam,

If you have too many tables. Try this:
SHOW TABLES LIKE 'pattern'
(or)
SHOW TABLES IN <another_database>

Re: Can't run SQL on my cluster

neesam — Wed, 05 Nov 2025 02:58:49 GMT

Hi Chiran,

Nothing with SQL will run, including spark.sql. I cannot run the two queries that you provided.

Another thing to note is that I can't view any sample data within the Catalog section. I just get a skeleton loader. Even for sample tables (like the nytaxi dataset), it won't load the data.

Re: Can't run SQL on my cluster

KaushalVachhani — Thu, 06 Nov 2025 01:15:51 GMT

Hi @neesam, while the command is hanging, check the driver logs from the cluster page to see if they provide any clues about what might be causing this issue.