cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Governance

Forum Posts

rammy
by Contributor III
  • 4501 Views
  • 5 replies
  • 18 kudos

How would i upload file stream object to S3 bucket using pyspark?

I could able to save data using pyspark into S3 but not sure on how to save a file stream object into S3 bucket using pyspark. I could achieve this with help of python but when Unity catalog was enabled on Databrciks it always ends up with an access ...

  • 4501 Views
  • 5 replies
  • 18 kudos
Latest Reply
rammy
Contributor III
  • 18 kudos

I got to know that there is a change required at Unity-catalog to make it work with Python and got a recommendation to use pyspark to store file into S3.I do not see much information about storing a file stream object in an S3 bucket anywhere. Can an...

  • 18 kudos
4 More Replies
sandhyasr
by New Contributor III
  • 1594 Views
  • 4 replies
  • 20 kudos

Consume Delta table data using .net/python API's to present it on custom web application

How can we connect to unity catalog tables to create REST API's using Python/.NET?

  • 1594 Views
  • 4 replies
  • 20 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 20 kudos

Hi @Sandhya SR​, We haven’t heard from you since the last response from @Jensen Ackles​ and I was checking to see if their suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.Al...

  • 20 kudos
3 More Replies
quakenbush
by Contributor
  • 4611 Views
  • 4 replies
  • 5 kudos

Creating a persistent view that references both Unity Catalog and Hive Metastore objects is not supported in Unity Catalog

I tried to create a view in hive_metastore.default which would access a table from a different catalog. Is there any chance to do so?eg.create view myTest as select * from someCatalog.someSchema.someTable

  • 4611 Views
  • 4 replies
  • 5 kudos
Latest Reply
Senthil1
Contributor
  • 5 kudos

I will suggest the below links will help you, For exposing your Hive_metastore multiple catalogs, 1) Create 3 catalogs for each environment under single metastore by using Unity Catalog2) Now expose the Unity Catalog using DELTA SHARING to your BI ap...

  • 5 kudos
3 More Replies
nihar07
by New Contributor II
  • 1141 Views
  • 0 replies
  • 2 kudos

AWS VPC service endpoint for internal RDS Hive metastore

We are setting up Databricks on our AWS account using customer managed VPC and enabling private link. I could see the private endpoints for frontend and backend with which we could spin up our EC2 instances. But the problem we are facing is to connec...

  • 1141 Views
  • 0 replies
  • 2 kudos
flixlatino
by New Contributor
  • 1614 Views
  • 0 replies
  • 0 kudos

FgKtzOnWAAICHO4

Este triángulo amoroso se puso complicado... ¡Faltan solo días para el estreno de la segunda temporada de "Wildrose" en flixlatino!

  • 1614 Views
  • 0 replies
  • 0 kudos
deficiant_codge
by Contributor II
  • 2236 Views
  • 3 replies
  • 5 kudos

AWS Glue Catalog

Can we use existing AWS Glue catalog with Unity Catalog?

  • 2236 Views
  • 3 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Rahul Mishra​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 5 kudos
2 More Replies
ossinova
by Contributor II
  • 1081 Views
  • 2 replies
  • 3 kudos

When adding new columns with overwriteSchema = True it does not add it to UC

I have discovered some strange behavior when adding new columns to a table associated with Unity Catalog. I overwrote the data specifying overwriteSchema = True and passing in the abfss path as it is an external table. When quering against it, it is ...

Screenshot 2022-10-14 120710
  • 1081 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Oscar Dyremyhr​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 3 kudos
1 More Replies
LukaszJ
by Contributor III
  • 11191 Views
  • 13 replies
  • 4 kudos

Problems with Unity Catalog

Hello.I am using Azure Databricks since a few months and I want to add a Unity Catalog.Unfortunately, I have some problems:1. I am not able to create a Unity Catalog Cluster with settings:- Policy: Unrestricted- Multi Node with Single user access mod...

image
  • 11191 Views
  • 13 replies
  • 4 kudos
Latest Reply
Addi1
New Contributor II
  • 4 kudos

I'm facing the same issues as other customers listed above. I do not have the "Create Metastore" button available.

  • 4 kudos
12 More Replies
Tram
by New Contributor III
  • 1511 Views
  • 3 replies
  • 7 kudos

Unity Catalog & Delta Lake

In setting up Databricks accounnt, we need to provide storage (s3 buckets) to store the final cleaned data or aggregated data. If the Unity Catalog is enabled at the beginning of account set up, we'd need to to provide S3 buckets again to store the m...

  • 1511 Views
  • 3 replies
  • 7 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 7 kudos

Hi @Tram Nguyen​, We haven’t heard from you since the last response from @Pat Sienkiewicz​ and @karthik p​​, and I was checking back to see if their suggestions helped you.Or else, If you have any solution, please share it with the community, as it c...

  • 7 kudos
2 More Replies
venkad
by Contributor
  • 1683 Views
  • 2 replies
  • 7 kudos

Resolved! Unity Catalog support for DLT

Hi,Does the Unity catalog support the Delta Live Tables? If yes, please help me with the documentation link. If not, any update on the feature readiness timeline would help. Thank you!

  • 1683 Views
  • 2 replies
  • 7 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 7 kudos

Hi @Venkadeshwaran K​, We haven’t heard from you since the last response from @Vivian Wilfred​​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful...

  • 7 kudos
1 More Replies
Shafi
by New Contributor III
  • 3922 Views
  • 4 replies
  • 6 kudos

Resolved! Cannot create a table having a column whose name contains commas in Hive metastore.

Hi Tried to create a delta table from spark data frame using below command:destination_path = "/dbfs/mnt/kidneycaredevstore/delta/df_corr_feats_spark_4"df_corr_feats_spark.write.format("delta").option("delta.columnMapping.mode", "name").option("path"...

  • 3922 Views
  • 4 replies
  • 6 kudos
Latest Reply
Pat
Honored Contributor III
  • 6 kudos

Hi @Shafiul Alam​ ,who gave those names to columns? you can rename you columns, replace spaces / special characters, for example:%python import re list_of_columns = df_corr_feats_spark.colums renamed_list_of_columns = [ re.sub(r'[^0-9a-zA-Z]+', "_", ...

  • 6 kudos
3 More Replies
J_M_W
by Contributor
  • 3021 Views
  • 3 replies
  • 6 kudos

Resolved! Upgrade tables to Unity Catalog error (Azure)

Hi all, I have set up a Unity Catalog (UC), and I am attempting to "upgrade my Delta tables" in the hive_metastore to my new catalog.I have followed the instructions here: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catal...

image
  • 3021 Views
  • 3 replies
  • 6 kudos
Latest Reply
J_M_W
Contributor
  • 6 kudos

FYI - this is now magically resolved. I'm guessing it was a temporary bug or something.​

  • 6 kudos
2 More Replies
Anonymous
by Not applicable
  • 1604 Views
  • 2 replies
  • 2 kudos
  • 1604 Views
  • 2 replies
  • 2 kudos
Latest Reply
gaurav_gupta
New Contributor II
  • 2 kudos

@Brinda Gupta​ I found this link: Unity Catalog - DatabricksHere it is mentioned you can use "ALTER TABLE <TABLENAME> ALTER COLUMN <COLUMNNAME> SET MASK <MASKNAME>I tried this but it says there is no keyword like MASK.Please help me to understand thi...

  • 2 kudos
1 More Replies
jcasanella
by New Contributor III
  • 1507 Views
  • 3 replies
  • 7 kudos

Resolved! Terraform - Create SVC Principal under account and assign some objects to the SVC Principal

Hi, We have some problems to create some resources using terraform, after the Unity catalog migration. We have created a group and SVC Principal under account as in the doc using terraform. (AWS infrastructure) (https://registry.terraform.io/provider...

  • 1507 Views
  • 3 replies
  • 7 kudos
Latest Reply
Pat
Honored Contributor III
  • 7 kudos

Hi @Jordi Casanella​ ,I have been working with terraform for databricks lately and I would say that I had to switch my approach couple of times due to issues like you have right now (account vs workspace level API).I assume that with this part you di...

  • 7 kudos
2 More Replies
Pat
by Honored Contributor III
  • 3404 Views
  • 1 replies
  • 14 kudos

Resolved! SHOW PARTITIONS not supported

Error in SQL statement: AnalysisException: SHOW PARTITIONS is not supported in Unity Catalog;What happened with the SHOW PARTITIONS command?Is this information available anywhere in the documentation?If I go to Databricks SQL documentation there is n...

  • 3404 Views
  • 1 replies
  • 14 kudos
Latest Reply
Pat
Honored Contributor III
  • 14 kudos

I got answer from the support that there is no 'SHOW PARTITIONS' in Unity Catalog as "UC doesn't manage the partitions for tables".They suggested for work around `LIST` command, this is what I was able to came up with.The problem I see here, is for t...

  • 14 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels