- 1661 Views
- 5 replies
- 9 kudos
Resolved! How to get read/write bytes per table using Databricks system tables?
I’m working on a data usage use case and want to understand the right way to get read bytes and written bytes per table in Databricks, especially for Unity Catalog tables.What I wantFor each table, something like:DateTable name (catalog.schema.table)...
- 1661 Views
- 5 replies
- 9 kudos
- 9 kudos
system.access.audit focuses on governance and admin/security events. It doesn’t capture per-table I/O metrics such as read_bytes or written_bytes.Use system.query.history for per-statement I/O metrics (read_bytes, written_bytes, read_rows, written_ro...
- 9 kudos
- 2136 Views
- 11 replies
- 3 kudos
Resolved! Question on Ingestion Pipelines
Is there a better way to select source tables than having to manually select them 1 by 1. I have 96 tables and it's a pain. The gui keeps back to the schema and i have to search through all the tables again. Is there a way to import the tables using ...
- 2136 Views
- 11 replies
- 3 kudos
- 3 kudos
So you dont see the option to edit the pipeline ?Oronce you click on edit pipeline you dont see the option to Switch to code version(YAML)Or After you Switch to code version(YAML) you can only view that yaml and cant edit it ?
- 3 kudos
- 1619 Views
- 5 replies
- 5 kudos
Resolved! How to incrementally backup system.information_schema.table_privileges (no streaming, no unique keys
I'm trying to incrementally backup system.information_schema.table_privileges but facing challenges:No streaming support: Is streaming supported: FalseNo unique columns for MERGE: All columns contain common values, no natural key combinationNo timest...
- 1619 Views
- 5 replies
- 5 kudos
- 5 kudos
information_schema is not a Delta Table, which is why you can't stream from it. They are basically views on top of the information coming straight from the control plane database. Also your query is actually going to be quite slow/expensive (you prob...
- 5 kudos
- 3423 Views
- 4 replies
- 1 kudos
Resolved! Are you able to create a iceberg table natively in Databricks?
Been trying to create a iceberg table natively in databricks with the cluster being 16.4. I also have the Iceberg JAR file for 3.5.2 Spark.Using a simple command such as:%sql CREATE OR REPLACE TABLE catalog1.default.iceberg( a INT ) USING iceberg...
- 3423 Views
- 4 replies
- 1 kudos
- 1 kudos
Databricks supports creating and working with Apache Iceberg tables natively under specific conditions. Managed Iceberg tables in Unity Catalog can be created directly using Databricks Runtime 16.4 LTS or newer. The necessary setup requires enabling ...
- 1 kudos
- 524 Views
- 2 replies
- 2 kudos
Error in Column level tags creation in views via SQL
Hi,I'm trying to run this query using SQL and using DBR 17.3 cluster. But I get a syntax error. ALTER VIEW catalog.schema.viewALTER COLUMN column_nameSET TAGS (`METADATA` = `xyz`); But below query works- SET TAG ON COLUMN catalog.schema.view.column_n...
- 524 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks for the clarification @szymon_dybczak. I have a follow-up qn, if I have attached tag to a view column and the same view is associated with a SHARE, will the recipient see the tag in the view i.e. whether view column tags associated to shares a...
- 2 kudos
- 5247 Views
- 8 replies
- 8 kudos
Resolved! Case insensitive data
For all it's positives, one of the first general issues we had with databricks was case sensitivity.We have a lot of data specific filters in our codeProblem is, we land and view data from lots of different case insensitive source systems e.g. SQL Se...
- 5247 Views
- 8 replies
- 8 kudos
- 8 kudos
It works but there's a scenario that causes an issue.If I create a schema with defaultcollation UTF8_LCASE Then create a table, it marks all the string columns as UTF8_LCASE Which is fine and works If I create the table, in the newly created UTF8_LCA...
- 8 kudos
- 10344 Views
- 6 replies
- 17 kudos
Generate Group Id for similar deduplicate values of a dataframe column.
Inupt DataFrame'''KeyName KeyCompare SourcePapasMrtemis PapasMrtemis S1PapasMrtemis Pappas, Mrtemis S1Pappas, Mrtemis PapasMrtemis S2Pappas, Mrtemis Pappas, Mrtemis S2Mich...
- 10344 Views
- 6 replies
- 17 kudos
- 17 kudos
Hey. We’ve run into similar deduplication problems before. If the name differences are pretty minor (punctuation, spacing, small typos), fuzzy string matching can usually get you most of the way there. That kind of similarity-based clustering works f...
- 17 kudos
- 4825 Views
- 2 replies
- 1 kudos
Time travel on views
Hello,At my company, we design an application to analyze data, and we can do so on top of external databases such as Databricks. Our application cache some data in-memory and to avoid synchronization issues with the data on Databricks, we rely heavil...
- 4825 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi there,Your use case for time travel on views is really interesting. I can see why being able to track historical versions of both views and their underlying tables would be crucial for an application that relies on caching and granular queries. Ri...
- 1 kudos
- 1455 Views
- 6 replies
- 1 kudos
Resolved! DELTA_FEATURES_REQUIRE_MANUAL_ENABLEMENT DLT Streaming Table as Variant
I am attempting to ingest csv files from an S3 bucket with Autoloader. Since the schema of the data is inconsistent (each csv may have different headers), I was hoping to ingest the data as Variant following this: https://docs.databricks.com/aws/en/i...
- 1455 Views
- 6 replies
- 1 kudos
- 1 kudos
I think I found the issue... I put the table_properties in the wrong place. It goes in the decorator args not the query_function args. My bad
- 1 kudos
- 2049 Views
- 2 replies
- 0 kudos
Resolved! Connecting an S3-compatible endpoint (such as MinIO) to Unity Catalog
Hi everyone, is it possible to connect an S3-compatible storage endpoint that is not AWS S3 (for example MinIO) to Databricks Unity Catalog? I already have access using Spark configurations (3a endpoint, access key, secret key, etc.), and I can read/...
- 2049 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you for your help @MoJaMa We have a unified endpoint that is S3-compatible and connects to multiple data sources. Those data sources can be in the cloud or on-prem, and the location of the data doesn’t matter to the user. The goal of the unifie...
- 0 kudos
- 1467 Views
- 0 replies
- 0 kudos
Building Resilient Data Pipelines: Databricks Autoloader with Safe Schema Evolution and Retry Logic
Introduction: The Challenge of Schema Drift in the LakehouseThe Databricks Lakehouse Architecture, powered by Delta Lake, has revolutionized how organizations handle data ingestion, particularly in the Bronze layer. Tools like Databricks Autoloader (...
- 1467 Views
- 0 replies
- 0 kudos
- 741 Views
- 1 replies
- 1 kudos
Resolved! Serverless giving inconsistent results in Oracle UCM SOAP call
Hello ,We have implemented Data pipeline to ingest data from Oracle UCM using SOAP API, This was working fine with Job and all Purpose clusters. Recently we wanted to use Serverless to take advantage of the server startup time. In this case we were n...
- 741 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @maddan80 Thank you for reaching out with your question and providing the context about your use case. Per your comments, having a 200 Status Code in Serverless is a good initial indicator that the request is reaching the Oracle UCM server. Brain...
- 1 kudos
- 11532 Views
- 5 replies
- 9 kudos
Resolved! Why my notebook I created in a Repo can be opened safe?
I've cloned a Repo during "Get Started with Data Engineering on Databricks".Then I'm trying to run another notebook from a cell with a magic %run command.But I get that the file can't be opened safe.Here my code:notebook_aname = "John" print(f"Hello ...
- 11532 Views
- 5 replies
- 9 kudos
- 9 kudos
+1 to all the above comments. Having the %run command along with other commands will confuse the REPL execution. So, have the %run notebook_b3 command alone in a new cell, maybe as the first cell, is notebook_a, which will resolve the issue, and your...
- 9 kudos
- 1249 Views
- 5 replies
- 1 kudos
Resolved! scheduling in dlt pipeline
Hi Team, when i creaing DLT pipelines i am not able add scheduls via assed bundle,its allowing me from UI onlyis there any other option make dynamic way to create schdules like using SDK | API | CLI
- 1249 Views
- 5 replies
- 1 kudos
- 1 kudos
If you still want to use sdk here is what you can do . update the job using sdkfrom databricks.sdk import WorkspaceClient w = WorkspaceClient() def update_job_schedule(job_id: int): return w.jobs.update( job_id=job_id, new_setti...
- 1 kudos
- 1935 Views
- 1 replies
- 1 kudos
Resolved! Databricks notebook Issue
Hello Databricks Community,I’m facing multiple issues while working in Azure Databricks notebooks, and I’d appreciate guidance or troubleshooting suggestions.Issue 1: Failed to reconnectWhile running a notebook, I frequently see a “Failed to reconnec...
- 1935 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, there a few things that could cause these types of problems. 1. Azure service availablity (when these happen check the Azure service availability to make sure there are no outages) 2. Local network connection problems (verify all your other inter...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
