cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unity Catalog view access in Azure Storage account

prasha123
New Contributor II

Hi,

I have my unity catalog in Azure Storage account and I can able to access table objects but I couldn't find my views that were created on top of those table. 

1. I can can access Delta tables & related views via Databricks SQL and also find the tableId for each table created in the catalog, back to the storage container

2. But I couldn't find the views that were created in the same catalog. The corresponding object storage for view is not visible in the storage account container.

Can you please help to solve this, a critical deployment is going on.

3 REPLIES 3

Walter_C
Honored Contributor

In Databricks, views are read-only objects composed from one or more tables and views in a metastore. They reside in the third layer of Unity Catalog’s three-level namespace. A view can be created from tables and other views in multiple schemas and catalogs. However, views might have different execution semantics if they’re backed by data sources other than Delta tables. Databricks recommends that you always define views by referencing data sources using a table or view name.

When it comes to Azure Storage, views are not stored as separate objects in the storage account. Instead, they are metadata constructs within Databricks that reference the underlying tables. This is why you can't see them in the Azure Storage account container. They exist within the Databricks environment and are used to provide a specific view or representation of the underlying data.

prasha123
New Contributor II

Thank you for the explanation. So in my case I am trying to create a dataflow in azure data factory where I need those views. 
is there any way to get those views in ADF dataflow or I have to recreate in flow?

artsheiko
Honored Contributor

Hi,

Couple of options are possible : 

  1. Use Databricks to do the complex SQL queries (joins, unions, etc) and write to a staging Delta Table. Then use DataFlow to read from that staged table. Orchestrate all of this using ADF or even Databricks Workflows. 
  2. Use a Copy Activity in ADF to connect to Databricks over a view that has the complex SQL queries (joins, unions, etc) and write to the staged Delta Table that Dataflow can read from. Orchestrate both with ADF pipeline.
  3. Use the Rest API inline dataset in ADF DataFlow to connect to the Databricks SQL execution API. This is fairly complex but may be a workaround : https://www.databricks.com/blog/2023/03/07/databricks-sql-statement-execution-api-announcing-public-...

It could be useful to take a closer look at Databricks workflows : https://docs.databricks.com/en/workflows/index.html. Maybe it will be powerful enough to respond to your needs.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group