- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2024 01:21 AM - edited 07-17-2024 01:26 AM
Hi @MyTrh ,
Ok, I think I created similiar use case to yours. I have streaming table with column structure based on your example
CREATE OR REFRESH STREAMING TABLE clicks_raw AS SELECT *, current_timestamp() as load_time FROM cloud_files('/Volumes/dev/demo_db/landing_zone/clicks_data', "csv", map("cloudFiles.inferColumnTypes", "true"))
Now, I have another streaming table which is consuming the first one. Here I perform grouping to achieve distinct combination of user_id and department_id and I'm aggregating count for each group.
So as you can see from sceen below, take for example user_id = user_001 and departament_id = dept_01.
We can see that in clicks_raw there are two rows for this combination:
- one with 20 clicks
- another one with 15 clicks
So the final result in our target table for this user and this department should be 35.
CREATE OR REFRESH STREAMING TABLE clicks_aggregated AS SELECT user_id, department_id, SUM(clicks) as clicks FROM STREAM(LIVE.clicks_raw) GROUP BY user_id, department_id
As you can see from screen below I achieved incremental refresh.