cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Reading a materialised view locally or using databricks api

ashraf1395
Contributor III

Hi there, 

This was my previous approach 
- I had a databricks notebook with a streaming table bronze level reading data from volumes which created a 2 downstream tables.

- 1st A a materialised view gold level, another a table for storing ingestion_metadata like most_recent_timestamp of events.

- ingestion metadata table was shared using open delta sharing

Then I would run an ingestion script from aws ecs / locally - I read the ingestion_metadata with delta sharing and then find the most_recent_timestamp log and fetch further data ahead from that timestamp and then other stuffs.This is was working good only issue - I needed to run the databricks notebook manually.

 

So I shifted to dlt pipeline.

Things are same but I cannot create a normal table either it will be a streaming table / materialised view / view 

and mv and views cannot be delta shared.So then I tried to access that ingestion_metadata materialised table created in dlt pipeline using databricks API but I cannot read the actual data inside it.

How to do this or any other way I should solve this case

1 ACCEPTED SOLUTION

Accepted Solutions

ashraf1395
Contributor III

I used this approach 
- Querying the materialised view using databricks serverless SQL endpoint by connecting it with SQL connect. 
Its working right now. If I face any issues, I will write it into a normal table and delta share it.

Thanks for your reply @Retired_mod as always very helpful.

View solution in original post

1 REPLY 1

ashraf1395
Contributor III

I used this approach 
- Querying the materialised view using databricks serverless SQL endpoint by connecting it with SQL connect. 
Its working right now. If I face any issues, I will write it into a normal table and delta share it.

Thanks for your reply @Retired_mod as always very helpful.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group