cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Bug in unity catalog registering external database that is case sensitive

AurelioGesino
New Contributor II

I successfully registered in my Unity Catalog an external Database ```dwcore``` that is hosted on SQL server.
I first added the connection in "External Data": tested the connection and it was successful.
I then added the database on top: tested the connection and it was successful.

Then on the unity catalog explorer I checked I saw that the tables can be seen under the "dbo" schema.
I can see all table listed on the tree on the left.
But then, if I try to query any table, the table is not found by Spark.

I tested with another postgres database and everything works like a charm.
I think the issue is this: my SQL server table names are case sensitive.

So for example I have a table "AllAlarms" that in the external catalog gets registered all lowercase.
It becomes "allalarms".
So when I try to query the table in the unity catalog the real SQL table does not get resolved, because I'm querying with a lowercase name when it should have been registered registering its real case.
I tried as well to set Spark to be 

spark.sql.caseSensitive', True

but that does not help, because the error occurs when the table is registered in the catalog.

So I think this is a bug with Databricks.



1 REPLY 1

Hi Kaniz, thank you for your feedback.
Databases that I have to link cannot be changed, other teams and products rely on those databases as well, this change would be disruptive.

I think this is a limitation in the unity catalog behaviour, as it should take in account that external databases could have a case sensitive collation, either:

- keeping the name of the tables as they are, leaving up to the user to deal with case issues for external tables.
- remapping that internally and hide it to the user
- providing some API where to hook on in the database registration, or perform it programmatically.

So I'll open up a ticket support for it.

Thank you

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group