Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
Is there any way to find a cost of one single query? i can't find any table in system catalog to connect a query with usage to find the exact cost of one query run in sql warehouse?
The published system tables do not expose the exact cost of an individual SQL statement. system.billing.usage records warehouse-attributed usage for each record's time interval, but has no statement or session ID. system.query.history records stateme...
I’ve seen two common approaches in data architectures:Landing → Bronze: Raw files are first dumped into a landing layer, archived, and then loaded into Delta tables as Bronze.Raw files = Bronze: The raw file dump itself is treated as the Bronze laye...
The choice mainly depends on replay, audit, and governance needs. If you need a reliable source of truth for reprocessing, keep raw files separate from Bronze. Otherwise, treating raw files as Bronze can reduce storage, cost, and complexity. Make the...
Hi, There is Get Table api in open sharing. and I faced error while using it. test_iceberg_table is foreign iceberg table.curl -s -X GET \ 'https://frankfurt.cloud.databricks.com/api/2.0/delta-sharing/metastores/617a9420-4294-453a-8a56-ea87eb418...
Hi @Ji-Seung,The ENDPOINT_NOT_FOUND error makes sense — the endpoint you're trying to call does not exist in the Delta Sharing open protocol. There is no standalone "Get Table" (singular) endpoint at that path.The Delta Sharing Protocol Endpoints for...
I have created Python modules containing some Python functions and I would like to import them from a notebook contained in the Workspace. For example, I have a "etl" directory, containing a "snapshot.py" file with some Python functions, and an empty...
Hello,I was facing the same issues as the OP and was unable to resolve them using the suggested solutions.In my case, the root cause turned out to be the cluster configuration rather than pytest or Databricks imports.I verified this by switching to a...
Hi Everyone, This is a follow up to a previous question I posted. We have a need for a number of apps that have a spreadsheet-like interface and read/write to our lakehouse. A common spec that comes up in these discussions is the need for managers to...
Thank you both for those very quick and excellent responses. The hovering thing is just an example of the kind of question I get, probably not an exact spec. I would be happy if the audit trail can be displayed somehow in the app.
Hi everyone,I'm working on an AI/BI dashboard and I accidentally initialized a Relationship Graph (semantic model) by clicking "Get started with Relationships" in the Data tab. The model has no relationships and no measures -- it just auto-created en...
Hi brickuser,You have hit a UI limitation with the Dashboard Relationships (Semantic Model) feature in Public Preview. Once you click Get started with Relationships, Databricks automatically initializes a semantic model from your existing datasets. W...
Setup: Lakeflow Connect Netsuite pipeline (UC Connection, deployed via UI), Azure Databricks. Target table is netsuite.transactionline, currently about 13.8M rows, 395 columns, CDC ingestion type (I know I know but there aren't that many rows changin...
Hey there, we offer a best-in-class ELT tool that can connect NetSuite to Azure Databricks out of the box.It supports incremental loading.Might you be interested in this tool?https://studio.precog.cloud
Hey, currently we have an AI/BI Dashboard pivot table with a row hierarchy such as: outlet->date->dayIn the dashboard, users can expand/collapse the hierarchy and may be viewing only the top-level outlet aggregation.However, when downloading the pivo...
Hi Anmolhhns, AIBI generally exports the underlying dataset tied to that visual across all defined hierarchy fields ignoring whether row or column groups are currently collapsed in it. You can follow belowParameter-Driven AggregationYou can shift the...
Hi everyone,Our data team frequently receives data extraction requests from business teams. Most requests are relatively simple and are handled by writing SQL queries in Databricks.We are looking for a good way to manage these SQL queries as the numb...
The catalog table generally adds maintenance overhead in manual cases as it can drift out of sync with actual queries if managed manually. Databricks capabilities might be enough as you can use Workspace search (searches across query names and conten...
I created an iceberg table within databricks and shared it via delta sharing with tokenThen I use curl command to access it with iceberg endpoint like this:curl -X GET \-H "Authorization: Bearer ***"\-H "Accept: application/json" \"https://ohio.cloud...
Thank you @rdokala , @aleksandra_ch BTW, can I share managed uniformed delta table to external Iceberg clients? I tried this but get the same error, my friend tried, seems fine. I don't know what's the difference.Best regards,Li
We have noticed that users can schedule SQL queries, but currently we haven't found a way to find these scheduled queries (this does not show up in the jobs workplane). Therefore, we don't know that people scheduled this. The only way is to look at t...
We ran into a similar governance concern and looked at it from both a prevention and a monitoring perspective.From what we've seen, scheduled SQL queries are different from the traditional Jobs that users create in the Jobs UI, which makes them harde...
I have two Databricks Apps that show RUNNING status and clean deployment logs, but both return 502 Bad Gateway when accessing their public URLs. Error: Both apps return "502 Bad Gateway" when accessed via their public URLs.Troubleshooting Attempted:-...
Hi Julie,Host or port binding mismatch between your Streamlit application and the reverse proxy is likely. Even though the app container starts up and reports RUNNING status, the platform expects the internal server process to listen on 0.0.0.0 and b...
I'm working on Databricks AI/BI dashboard containing a native visualization; to achieve custom styling such as progress bars, the dataset SQL Generates HTML strings using expressions similar to:CONCAT( '<div style="...">', CAST(value AS STRING), '</d...
Adding one option on top of the two-column approach above: depending on how attached you are to the exact progress-bar look, you may not need hand-rolled HTML at all anymore.Table visualizations in AI/BI dashboards picked up richer native formatting ...
Hi,I'm experiencing an issue with Visual Data Prep where some source tables appear to contain fewer rows than expected.When I query the same tables directly using SQL, I can see all records. However, when loading certain tables into Visual Data Prep ...
Hi dax60,What you are seeing comes down to the difference between processing limits and UI rendering limits in Visual Data Prep. When you set Rows scanned to Max you are instructing the engine to process all records from the source table for your int...
In databricks documentation it says that there's a display name when creating dashboard variables but I'm not seeing it in the dashboard settings. Does anyone know where to find them?
Hi @amekojc, We have fixed the documentation to reflect the current behaviour. Please watch the space for the specific feature you were looking for.
