cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

acsmaggart
by New Contributor III
  • 2874 Views
  • 6 replies
  • 2 kudos

`collect()`ing Large Datasets in R

Background: I'm working on a pilot project to assess the pros and cons of using DataBricks to train models using R. I am using a dataset that occupies about 5.7GB of memory when loaded into a pandas dataframe. The data are stored in a delta table in ...

collecting the data using pyspark collecting the data using R
  • 2874 Views
  • 6 replies
  • 2 kudos
Latest Reply
Annapurna_Hiriy
New Contributor III
  • 2 kudos

@acsmaggart Please try using collect_larger() to collect the larger dataset. This should work. Please refer to the following document for more info on the library.https://medium.com/@NotZacDavies/collecting-large-results-with-sparklyr-8256a0370ec6

  • 2 kudos
5 More Replies
oteng
by New Contributor III
  • 1135 Views
  • 3 replies
  • 3 kudos

DLT with unity catalog and ML

We are currently using DLT with unity catalog. DLT tables are created as materialized views in a schema inside a catalog. When we try to access these materialized view using a ML runtime (ex. 13.0 ML) cluster, it says, that we must use Single User se...

  • 1135 Views
  • 3 replies
  • 3 kudos
Latest Reply
pg5
New Contributor II
  • 3 kudos

I recently hit the same issue.Seems like this is a limitation of DLT with Unity Catalog.Did you find a workaround @oteng? Otherwise I will try copying the materialized views to a table before doing the ML work.

  • 3 kudos
2 More Replies
Sandhya1
by New Contributor
  • 1130 Views
  • 2 replies
  • 0 kudos

Attribute based access control in Unity catalog

Can I start using Attribute based access control ? Is it available now?

  • 1130 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Sandhya Reddy Veera​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best a...

  • 0 kudos
1 More Replies
animadurkar
by New Contributor III
  • 3183 Views
  • 2 replies
  • 2 kudos

Resolved! Invalid catalog and schema for table name error when creating a Feature Store

I've created a dataset that I want to create a feature store table for. I created a database in my dev unity catalog to store the feature tables, but each time I try to create the table I get this error:ValueError: Invalid catalog 'dev' or schema 'fe...

  • 3183 Views
  • 2 replies
  • 2 kudos
Latest Reply
Oogh5zai1
New Contributor II
  • 2 kudos

Hello,The error message suggests that there might be an issue with the catalog or schema name you are using when trying to create the feature store table. Here are a few tips to help you troubleshoot this problem:Verify the catalog and schema names: ...

  • 2 kudos
1 More Replies
Ismail1
by New Contributor III
  • 975 Views
  • 2 replies
  • 0 kudos

Can an HMS-managed table be upgraded to Unity Catalog?

As the question states, I am not getting the option to upgrade managed tables on UC. Is that possible, I can't find anything on the documentation?

  • 975 Views
  • 2 replies
  • 0 kudos
Latest Reply
Ismail1
New Contributor III
  • 0 kudos

In case anyone else ever faced the same issue

  • 0 kudos
1 More Replies
Spencer_Kent
by New Contributor III
  • 1316 Views
  • 2 replies
  • 1 kudos

Resolved! Lacking support for column-level select grants or attribute-based access control

In the Unity Catalog launch and its accompanying blog post, one of the primary selling points was a set of granular access control features that would at least partially eliminate the need to create a multitude of separate table views and the attenda...

  • 1316 Views
  • 2 replies
  • 1 kudos
Latest Reply
Spencer_Kent
New Contributor III
  • 1 kudos

Simply amazing that 2 years on from the initial announcement, this feature is not available. You released Unity Catalog missing one of it's most-hyped features.

  • 1 kudos
1 More Replies
venkad
by Contributor
  • 6327 Views
  • 1 replies
  • 1 kudos

Unity Catalog Pricing

Hi All, I would like to understand the pricing model of the Unity Catalog. Earlier I remember there was some mention of the data lineage and a few other features that will have a cost associated with it. If that's true, what other features cost us? W...

  • 6327 Views
  • 1 replies
  • 1 kudos
Latest Reply
LandanG
Honored Contributor
  • 1 kudos

Hi @Venkadeshwaran K​ ,All Unity Catalog features are provided at no charge to customers, provided they are using a Premium or Enterprise SKU.

  • 1 kudos
Ela
by New Contributor III
  • 5659 Views
  • 8 replies
  • 0 kudos

Resolved! Inheritance model in Unity Catalog is not working as per documentation.

As per the documentation "Securable objects in Unity Catalog are hierarchical and privileges are inherited downward. The highest level object that privileges are inherited from is the catalog". Executed following statement "GRANT SELECT ON CATALOG uc...

  • 5659 Views
  • 8 replies
  • 0 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, Both permissions have to be granted.

  • 0 kudos
7 More Replies
parthsalvi
by Contributor
  • 1998 Views
  • 1 replies
  • 2 kudos

py4j.security.Py4JSecurityException: Constructor public com.databricks.backend.daemon.dbutils.FSUtilsParallel(org.apache.spark.SparkContext) is not whitelisted. - DBR 10.4 LTS Shared mode

We're trying to update permission in Unity Catalog using spark sql statetment spark.sql("ALTER TABLE %s.%s OWNER TO `%s`;" % (raw_schema, table_name, 'Metastore_Admins'))But we're running into following error We're facing issues like these in DBR 10....

image.png
  • 1998 Views
  • 1 replies
  • 2 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 2 kudos

I have seen this beforeso there is one file in the DBFS where we have to write the whitelisted file , so you have to find out where is your whitelisted file location after that you have to add that "com.databricks.....parallel" line in that file and ...

  • 2 kudos
Sujitha
by Community Manager
  • 599 Views
  • 2 replies
  • 6 kudos

Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week. Databricks platform release notes December ...

Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notes December 5 - 16, 2022Databricks JDBC driver 2.6.32Version 2.6.32 of the Databricks JDBC driver (download and M...

  • 599 Views
  • 2 replies
  • 6 kudos
Latest Reply
Harun
Honored Contributor
  • 6 kudos

Thanks for sharing @Sujitha Ramamoorthy​ 

  • 6 kudos
1 More Replies
Ashley1
by Contributor
  • 4378 Views
  • 5 replies
  • 4 kudos

Unity Catalog - existing dbfs mounts and feature store

Hi All, We're currently considering turning on Unity Catalog but before we flick the switch I'm hoping I can get a bit more confidence of what will happen with our existing dbfs mounts and feature store. The bit that makes me nervous is the crede...

  • 4378 Views
  • 5 replies
  • 4 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 4 kudos

@Ashley Betts​ can you please check below article, as far as i know we can use external mount points by configuring storage credentials in unity catalog . default method is managed tables, but we can point external tables also. 1. you can upgrade exi...

  • 4 kudos
4 More Replies
User16345769212
by New Contributor III
  • 420 Views
  • 0 replies
  • 2 kudos

Unity Catalog Webinar: Join us to learn what's new, and what’s coming in Unity Catalog Governance for Data and AI is complex. Databricks Unity Cat...

Unity Catalog Webinar: Join us to learn what's new, and what’s coming in Unity CatalogGovernance for Data and AI is complex. Databricks Unity Catalog provides a unified governance solution for all data and AI assets on any cloud, empowering data team...

How to simplify data and AI governance with Unity Catalog
  • 420 Views
  • 0 replies
  • 2 kudos
Labels