Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
This is my dlt pipeline event_log - why is it not in readable foramt how can i correct it.This is my pipeline code : import logging
logger = logging.getLogger(__name__)
logger.info("Error")
raise "Error is error"
Hi @ashraf1395 ,I'm working with Delta Live Tables (DLT) and the event_log table. I would like to know if it is possible to access the event handler that DLT uses to write custom logs and send them to this table when events are published.If this is n...
Hi all,I would like to publish the event_log of my DLT Pipeline to a specific schema in Unity Catalog.Following this article (https://docs.databricks.com/gcp/en/dlt/observability#query-the-event-log) this can be done by writing this into the DLTs set...
Hi @susanne , indeed , i tried to create it using dabs as well. This feature is not available using dabs I guess, maybe they will add it once event_logs moves to ga from public preview.databricks API will be a good alternative but if you try it using...
I'm encountering a puzzling schema merge issue with my Delta Live Table. My setup involves several master tables on Databricks, and due to a schema change in the source database, one of my Delta Live Tables has a column (e.g., "reference_score") that...
Hey Hung, this is a pretty common issue when working with Delta Live Tables (DLT) and schema evolution. When the data type of a column changes—like in your case from Integer to Float—Delta sees that as an incompatible schema change, and by default, D...
select regexp_replace('asdfhsdf&&1&&asdfasdf&&2&&asdf','&&[0-100]&&','') output here it replaces first pattern but not the 2nd patterno/poutputasdfhsdfasdfasdf&&2&&asdf even if we use position its not working something like both wont work select ...
>> LogManager.getLogger() seems is not working in DLT notebook DLTError: [PY4J_BLOCKED_API] You are using a Python API that is not supported in the current environment. Please check Databricks documentation for alternatives. An error occurred while c...
Is it possible to create external volume using databricks asset bundle? I have this code from databricks.yml file which is working perfectly fine for manged volume: resources: volumes: bronze_checkpoints_volume: catalog_name: ...
Hi, One of our clients is asking to switch from our API feed to have weather data delivered automatically to their Cloud Storage. What steps do I need to take from my end? Do I need to join Databricks to do so? Thanks. Tom
Hey @TomHauf, while it may not be essential in your case, you should at least consider using Databricks to facilitate loading data into your customers cloud storage. Databricks gives you a few options to make sharing with third parties simple as per ...
Hi.I have a workflow in which I write few rows into a Delta Table with auto-generated IDs. Then, I need to retrieve them back just after they're written into the table to collect those generated IDs, so I read the table and I use two columns (one is ...
When I try to delete a table, I'm getting this error:[ErrorClass=INVALID_STATE] TABLE catalog.schema.table_name cannot be deleted because it is being shared via Delta Sharing.I have checked on the internet about it, but could not find any info about ...
Hi @IGRACH ,You are facing this issue bcz I guess the table you want to delete is being shared by delta sharing. You can go to the shared object by following this dochttps://docs.databricks.com/aws/en/delta-sharing/create-share#update-sharesandThen, ...
Hello community,I have implemented a DLT pipeline.In the "Destination" setting of the pipeline I have specified a unity catalog with target schema of type external referring to an S3 destination.My DLT pipeline works well. Yet, I noticed that all str...
This won't work.best approach is create dlt sink to write to delta external table. This pipeline should only be 1 step. Read table and append flow using data sink. It works fine.
Hello,I am currently writing a Delta Lake table from Databricks to Unity Catalog using PySpark 3.5.0 (15.4 LTS Databricks runtime). We want the EXTERNAL Delta Lake tables to be readable from both UC and Dremio. Our Dremio build version is 25.0.6.The ...
Hello,I'm interested in the "Tags" feature of columns/schemas/tables of the UnityCatalog (described here: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/tags)I've been able to play with them by hand and would now lik...
Is it possible to use Liquid Clustering on Delta Live Tables? If it is available what is the Python syntax for adding liquid clustering to a Delta Live Table / Materialised view please?
@Dlt.table(name=table_name,comment="just_testing",table_properties={"quality": "gold","mergeSchema": "true"},cluster_by=["test_id", "find_date"] # Optimizes for queries filtering on these columns)def testing_table():return create_testing_table(df_fin...
Is there a centralized location in Unity Catalog that retains the table history, specifically the last timestamp, for managed delta tables?DESCRIBE HISTORY will provide it for a specific table, but I would like to get it for a number of tables.inform...
Hi Eric_Kieft,How are you doing today?, As per my understanding, yeah, Unity Catalog doesn’t currently provide a direct system table that tracks all table modifications (including inserts/updates) across multiple managed Delta tables. DESCRIBE HISTOR...
