I am new to Databricks and currently trying to setup Unity Catalog on AWS at Enterprise level. I would like to know if Unity Catalog is enterprise ready and if there are any pointers that could help me set is up on AWS at enterprise level.Appreciate ...
Hi @Yashpal Shah​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
I am migrating my Data Lake to use Unity Catalog. However, this comes with changes to the clusters. I have tried a few options, but it seems rather complex than it should be. I need to create a cluster used by ADF that is Unity Enabled that can insta...
I exclusively use job clusters. They are cheaper.Especially when you create a pool with spot instances. I'd go for that because that is what it's made for: batch jobs/
Hi,I'm working with unity catalog for the last week. I'm refering to delta table by path, as follwing: path='s3://<my_bucket_name>/silver/data/<table_name>DeltaTable.forPath(spark, path)I get an exception that "is not a Delta table"using the table na...
I want to import data using the autoloader from a S3 bucket into a table which is managed inside a Unity Catalog.Right now, I run the code on an interactive cluster inside a notebook. In the future the code should run in a job cluster. The error I ge...
Hello,We're trying to use DBR 10.4 LTS in Shared Mode. But our python code is not working in that cluster.We're running into following messageYour administrator has only allowed sql commands on this cluster. This execution contained at least one disa...
I am trying to execute a UDF written in Python and receive the following error message:Error message: [UC_COMMAND_NOT_SUPPORTED] UDF/UDAF functions are not supported in Unity Catalog.;The limitation is not mentioned in the docs( https://docs.databric...
Indeed it is not mentioned in the limitations.This blog mentions python UDFs in UC, but in databricks SQL, not the classic DE workspace.So it seems that it is not (yet) supported, as the docs only mention SQL functions.
Hi,​Is there a know limitation in Unity Catalog workspaces ​with reading multiline json files from an S3 external location using pyspark? The file works perfectly fine if it's on dbfs, and reading json files in general works from that s3 location if ...
Hi,We are trying to adopt the Unity Catalog and are testing out creating an External table with some data from the samples catalog that comes along with creating a meta store. The command we are using is:CREATE TABLE IF NOT EXISTS my_catalog.quicksta...
In Unity catalog you have to pre-define the EXTERNAL LOCATION (and a STORAGE CREDENTIAL) before you use it this way. Please refer below:https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials.html#manage-ex...
Hey there @Carlos Queiroz​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please te...
Hello,Before Unity catalog, we were able to rename spark files to a single file (ex: accounts.csv instead of part001) using mounts with pandas or dbutils.fs. How can we achieve the same today without mounts? Is it possible to use external locations w...
It is our understanding that enabling unity catalog means that some metadata (for example usernames) will be sent to the USA for processing/storage. This is unfortunately a deal-breaker for us, and we need the data to reside solely in Europe for comp...
Usernames have always been stored centrally in the US. That's similar to most global services for the IAM Layer. The only thing additional that gets stored in the US as part of UC is the metastore names. you can check this link ( https://azure.micros...
I am trying to setup a Databricks SQL endpoint to connect to an external hive metastore. I have replicated my metastore spark config in the SQL endpoint, with the addition of adding the below to the configuration:spark.sql.hive.metastore.jars maven I...
Hi @SETH JENSEN​ , Just checking if you were able to solve this issue? If no, Can you try to modify the connection string and make sslMode=Require and see if it works?
Hi Everyone,This may sound bit odd as I haven't used Databricks much.I have a use case where I want to union data from two separate Databricks workspaces (under the same account) into a single SQL view. (Basically 2-2 tables from each workspaces and ...
Hey there @Bindi P​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? It would be really helpful for the other members too.Cheers!
I'm unable to run any command that queries data from the unity catalog within a function that executes in the event of an ipywidget button click. Code block below. I cannot do queries such as spark.sql(f"SHOW SCHEMAS;") or spark.sql(f"select * from d...
can you try to println out? val databricksApiTokenKey = CredentialContext.INHERITED_PROPERTY_DATABRICKS_API_TOKEN val databricksApiCredentialOpt = CredentialContext.getCredential(databricksApiTokenKey) val rawUrlProp = spark.sparkContext.get...
