cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

noimeta
by Contributor II
  • 4224 Views
  • 7 replies
  • 4 kudos

Resolved! Databricks SQL: catalog of each query

Currently, we are migrating from hive metastore to UC. We have several dashboards and a huge number of queries whose catalogs have been set to hive_metastore and using <db>.<table> access pattern.I'm just wondering if there's a way to switch catalogs...

  • 4224 Views
  • 7 replies
  • 4 kudos
Latest Reply
abdulrahim
New Contributor II
  • 4 kudos

Absolutely accurate, in order to grow your business you need to create an image of your brand such that it is the first thing coming to customers mind when they think about a certain product or service that’s where social media marketing agencies com...

  • 4 kudos
6 More Replies
Mr_K
by New Contributor
  • 9047 Views
  • 2 replies
  • 2 kudos

AnalysisException: [UC_COMMAND_NOT_SUPPORTED] Spark higher-order functions are not supported in Unity Catalog.;

Hello,forecast_date = '2017-12-01' spark.conf.set('spark.sql.shuffle.partitions', 500 ) # generate forecast for this data forecasts = ( history .where(history.date < forecast_date) # limit training data to prior to our forecast date .groupBy...

  • 9047 Views
  • 2 replies
  • 2 kudos
Latest Reply
Tharun-Kumar
Honored Contributor II
  • 2 kudos

@Mr_K ApplyInPandas is a higher order function in Python. As of now, we do not support higher order functions in Unity Catalog. We do support direct calls made to python UDFs. Here is an example of how to reference UDFs in UC - https://docs.databrick...

  • 2 kudos
1 More Replies
ah0896
by New Contributor III
  • 11501 Views
  • 17 replies
  • 10 kudos

Using init scripts on UC enabled shared access mode clusters

I know that UC enabled shared access mode clusters do not allow init script usage and I have tried multiple workarounds to use the required init script in the cluster(pyodbc-install.sh, in my case) including installing the pyodbc package as a workspa...

  • 11501 Views
  • 17 replies
  • 10 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 10 kudos

@Anonymous @Retired_mod can anyone form databricks confirm on above issue please, there seems to be bit conflict on using custom scripts support  on shared access mode cluster with unity catalog enabled please

  • 10 kudos
16 More Replies
NOOR_BASHASHAIK
by Contributor
  • 1326 Views
  • 1 replies
  • 1 kudos

Unity Catalog - addition of account groups/users to workspaces

Hi allwe have set-up metastore, and were doing certain activities as part of MVP.we realized in a particular databricks workspace that was enabled with UC, in admin settings > "Add Groups" section, user groups from other platforms/projects which leve...

  • 1326 Views
  • 1 replies
  • 1 kudos
Latest Reply
Ismail1
New Contributor III
  • 1 kudos

Good question, didn't think of it that way, from my understanding UC uses users pushed from the account console and not from workspaces, One way to restrict would be to restrict other workspaces from using said catalog and also control ACLs with the ...

  • 1 kudos
Anonymous
by Not applicable
  • 1705 Views
  • 2 replies
  • 0 kudos
  • 1705 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi Debayan,Thank you for answering my question.We were able to successfully attach the 2nd workspace (on a different AWS account) to the existing UC metastore on another AWS account.Unfortunately, we couldn't figure out what we did differently. It se...

  • 0 kudos
1 More Replies
Pat
by Honored Contributor III
  • 2162 Views
  • 3 replies
  • 19 kudos

Resolved! UC - Service Principal/Terraform

Hi,do you know if there is a way to create Unity Catalog metastore using Service Principal?Here I can see that for creating account-level resources we need to provide a user and password (https://registry.terraform.io/providers/databricks/databricks/...

  • 2162 Views
  • 3 replies
  • 19 kudos
Latest Reply
Pat
Honored Contributor III
  • 19 kudos

This is supported right now in the Azure, but not yet in AWS, but there is plan for AWS support as well.

  • 19 kudos
2 More Replies
Labels