We are having Databricks tables and we have an API which query that data, the query will be dynamic and API would allow user to query anything.
However users can query lot of data and consume lot of DBU but generic rate limiting wont help, as any single query can consume lot of DBUs so We want to rate limit user based on cost/usage. and If user of API has consumed certain DBUs in a minute we want to block them, so how can we calculate the DBU used for any SQL query, We looked at system tables but those cant help it seems, Can you help?
Note: We dont have separate accounts, its all shared in one account.
