topic Re: Making py connector to raise an error for wrong SQL when asking to plan a query in Data Engineering

Making py connector to raise an error for wrong SQL when asking to plan a query

624398 — Wed, 22 Jun 2022 10:17:50 GMT

Hey all,

My aim is to validate a given SQL string without actually running it.

I thought I could use the `EXPLAIN` statement to do so.

So I tried using the `databricks-sql-connector` for python to explain a query, and so determine whether it's valid or not. Example python code:

```

import databricks.sql

with databricks.sql.connect(...) as connection:

with connection.cursor() as cursor:

cursor.execute("EXPLAIN SELECT BAD-QUERY AS FOO")

r = cursor.fetchall()

```

The problem with that implementation is that the driver does not throws an error, but instead retrieves me a string containing the error details.

Why it's a problem? I need to parse the string result to distinguish if the explained query was valid or not.

So I was wondering if there some kind of setting / parameter / configuration or so I can use to change the described above result.

Many thanks in Advance!

Hubert-Dudek — Thu, 23 Jun 2022 20:05:46 GMT

Please try in pyspark:

try:
   spark.sql("SELECT  BAD-QUERY AS FOO")._jdf.queryExecution().toString()
except:
   print("incorrect query")

or just:

try:
   spark.sql("SELECT  BAD-QUERY AS FOO").explain()
except:
   print("incorrect query")

624398 — Sun, 03 Jul 2022 09:07:30 GMT

Hi @Hubert Dudek, thanks for replying!

The optimal solution I was looking for was in the databricks-sql-connector.

Installing large package such pyspark for this specific feature seems too much for my project purposes.

Thank you very much!

624398 — Sun, 31 Jul 2022 08:04:22 GMT

Hi' I'm still searching for a more suited solution to my use case.

Installing pyspark seems too much (build time and extra packages I don't really need).

Is there a way to make this a feature request?