Background: I'm working on a pilot project to assess the pros and cons of using DataBricks to train models using R. I am using a dataset that occupies about 5.7GB of memory when loaded into a pandas dataframe. The data are stored in a delta table in ...
@acsmaggart Please try using collect_larger() to collect the larger dataset. This should work. Please refer to the following document for more info on the library.https://medium.com/@NotZacDavies/collecting-large-results-with-sparklyr-8256a0370ec6
We are currently using DLT with unity catalog. DLT tables are created as materialized views in a schema inside a catalog. When we try to access these materialized view using a ML runtime (ex. 13.0 ML) cluster, it says, that we must use Single User se...
I recently hit the same issue.Seems like this is a limitation of DLT with Unity Catalog.Did you find a workaround @oteng? Otherwise I will try copying the materialized views to a table before doing the ML work.
Hi @Sandhya Reddy Veera​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best a...
I've created a dataset that I want to create a feature store table for. I created a database in my dev unity catalog to store the feature tables, but each time I try to create the table I get this error:ValueError: Invalid catalog 'dev' or schema 'fe...
Hello,The error message suggests that there might be an issue with the catalog or schema name you are using when trying to create the feature store table. Here are a few tips to help you troubleshoot this problem:Verify the catalog and schema names: ...
In the Unity Catalog launch and its accompanying blog post, one of the primary selling points was a set of granular access control features that would at least partially eliminate the need to create a multitude of separate table views and the attenda...
Simply amazing that 2 years on from the initial announcement, this feature is not available. You released Unity Catalog missing one of it's most-hyped features.
Hi All, I would like to understand the pricing model of the Unity Catalog. Earlier I remember there was some mention of the data lineage and a few other features that will have a cost associated with it. If that's true, what other features cost us? W...
As per the documentation "Securable objects in Unity Catalog are hierarchical and privileges are inherited downward. The highest level object that privileges are inherited from is the catalog". Executed following statement "GRANT SELECT ON CATALOG uc...
We're trying to update permission in Unity Catalog using spark sql statetment spark.sql("ALTER TABLE %s.%s OWNER TO `%s`;" % (raw_schema, table_name, 'Metastore_Admins'))But we're running into following error We're facing issues like these in DBR 10....
I have seen this beforeso there is one file in the DBFS where we have to write the whitelisted file , so you have to find out where is your whitelisted file location after that you have to add that "com.databricks.....parallel" line in that file and ...
Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notes December 5 - 16, 2022Databricks JDBC driver 2.6.32Version 2.6.32 of the Databricks JDBC driver (download and M...
Hi All, We're currently considering turning on Unity Catalog but before we flick the switch I'm hoping I can get a bit more confidence of what will happen with our existing dbfs mounts and feature store. The bit that makes me nervous is the crede...
@Ashley Betts​ can you please check below article, as far as i know we can use external mount points by configuring storage credentials in unity catalog . default method is managed tables, but we can point external tables also. 1. you can upgrade exi...
Unity Catalog Webinar: Join us to learn what's new, and what’s coming in Unity CatalogGovernance for Data and AI is complex. Databricks Unity Catalog provides a unified governance solution for all data and AI assets on any cloud, empowering data team...
