cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
noimeta
by • Contributor III
  • 4590 Views
  • 9 replies
  • 4 kudos

Resolved! Databricks SQL: catalog of each query

Currently, we are migrating from hive metastore to UC. We have several dashboards and a huge number of queries whose catalogs have been set to hive_metastore and using <db>.<table> access pattern.I'm just wondering if there's a way to switch catalogs...

Data Engineering
Reply
  • 4590 Views
  • 9 replies
  • 4 kudos
View Replies
Latest Reply
h_h_ak
Contributor
  • 4 kudos

May you can also have a look here, if you need hot fix  https://github.com/databrickslabs/ucx 

  • 4 kudos
8 More Replies
Mr_K
by • New Contributor
  • 9324 Views
  • 2 replies
  • 2 kudos

AnalysisException: [UC_COMMAND_NOT_SUPPORTED] Spark higher-order functions are not supported in Unity Catalog.;

Hello,forecast_date = '2017-12-01' spark.conf.set('spark.sql.shuffle.partitions', 500 ) # generate forecast for this data forecasts = ( history .where(history.date < forecast_date) # limit training data to prior to our forecast date .groupBy...

Data Engineering
Reply
  • 9324 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Tharun-Kumar
Databricks Employee
  • 2 kudos

@Mr_K ApplyInPandas is a higher order function in Python. As of now, we do not support higher order functions in Unity Catalog. We do support direct calls made to python UDFs. Here is an example of how to reference UDFs in UC - https://docs.databrick...

  • 2 kudos
1 More Replies
ah0896
by • New Contributor III
  • 12591 Views
  • 17 replies
  • 10 kudos

Using init scripts on UC enabled shared access mode clusters

I know that UC enabled shared access mode clusters do not allow init script usage and I have tried multiple workarounds to use the required init script in the cluster(pyodbc-install.sh, in my case) including installing the pyodbc package as a workspa...

Data Engineering
Reply
  • 12591 Views
  • 17 replies
  • 10 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 10 kudos

@Anonymous @Retired_mod can anyone form databricks confirm on above issue please, there seems to be bit conflict on using custom scripts support  on shared access mode cluster with unity catalog enabled please

  • 10 kudos
16 More Replies
NOOR_BASHASHAIK
by • Contributor
  • 1402 Views
  • 1 replies
  • 1 kudos

Unity Catalog - addition of account groups/users to workspaces

Hi allwe have set-up metastore, and were doing certain activities as part of MVP.we realized in a particular databricks workspace that was enabled with UC, in admin settings > "Add Groups" section, user groups from other platforms/projects which leve...

Data Engineering
Reply
  • 1402 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Ismail1
New Contributor III
  • 1 kudos

Good question, didn't think of it that way, from my understanding UC uses users pushed from the account console and not from workspaces, One way to restrict would be to restrict other workspaces from using said catalog and also control ACLs with the ...

  • 1 kudos
Anonymous
by • Not applicable
  • 1794 Views
  • 2 replies
  • 0 kudos

is it possible to attach a workspace in one AWS account to an existing UC metastore created in another AWS account (same region)?

Data Engineering
Reply
  • 1794 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi Debayan,Thank you for answering my question.We were able to successfully attach the 2nd workspace (on a different AWS account) to the existing UC metastore on another AWS account.Unfortunately, we couldn't figure out what we did differently. It se...

  • 0 kudos
1 More Replies
Pat
by • Honored Contributor III
  • 2374 Views
  • 3 replies
  • 19 kudos

Resolved! UC - Service Principal/Terraform

Hi,do you know if there is a way to create Unity Catalog metastore using Service Principal?Here I can see that for creating account-level resources we need to provide a user and password (https://registry.terraform.io/providers/databricks/databricks/...

Data Engineering
Reply
  • 2374 Views
  • 3 replies
  • 19 kudos
View Replies
Latest Reply
Pat
Honored Contributor III
  • 19 kudos

This is supported right now in the Azure, but not yet in AWS, but there is plan for AWS support as well.

  • 19 kudos
2 More Replies
Labels