Databricks Community

MetaRossiVinli · 04-27-2023

I am pulling data from Google BigQuery and writing it to a bronze table on an interval. I do this in a separate continuous job because DLT did not like the BigQuery connector calling collect on a dataframe inside of DLT.In Python, I would like to rea...

MetaRossiVinli · 04-03-2023

Last week, I started running a DLT pipeline with expectations that dropped rows on streaming live tables. In the side bar for a table, I saw a nice circular chart with Written/Dropped rows and Failed records stats.Today, I ran a similar DLT pipeline ...

MetaRossiVinli · 03-31-2023

Is the Secrets API 2.0 not applied to Delta Live Tables configurations? I understand that the Secrets API 2.0 is in public preview and this use case may not be supported, yet. I tried the following and both do not work for the stated reasons.In a DLT...

MetaRossiVinli · 03-30-2023

I am setting up a new workspace that will use the Unity Catalog. I want all data stored in the Unity Catalog in the following catalogs: dev, staging, prod. I want to prevent users from accidentally reading and writing data elsewhere.For the above sit...

MetaRossiVinli · 03-27-2023

I want to import a Python function stored in the following file path:`<repo>/lib/lib_helpers.py`I want to import the function from any file in my repo. For instance from these:`<repo>/notebooks/etl/bronze/dlt_bronze_elt``<repo>/workers/job_worker`It ...

MetaRossiVinli · 04-28-2023

The below code is a solution. I was missing that I could read from a table with `spark.readStream.format("delta").table("...")`. Simple. Just missed it. This is different than `dlt.read_stream()` which appears in the examples a lot.This is referenced...

MetaRossiVinli · 04-04-2023

Thanks again. I am in SQL. So, I understand that there is no need to call validate_expectations(). I will troubleshoot with those tips and report back.

MetaRossiVinli · 04-03-2023

I will continue using the dbutils.secrets workaround. Probably by making it a tiny function so that I can import it to multiple jobs. Solved for now.If it is not already on the roadmap, I suggest adding the ability to read from the secrets API in the...

MetaRossiVinli · 03-31-2023

Short answers that I derived from the above:hive_metastore - This cannot be deleted or hidden, but the default catalog can be changed with the above instructions.main - There is a new feature that can unbind catalogs from workspaces. This would remov...

MetaRossiVinli · 03-27-2023

Ok, I figured it out. If you just make it a Python module by adding an empty `__init__.py`, Databricks will load it on start. Then, you can just import it.

Databricks Community

User Stats

User Activity

In Python, Streaming read by DLT from Hive Table

DLT data quality UI was present last week. Now absent. Did I change a setting?

Can you use the Secrets API 2.0 in a Delta Live Tables configuration?

In a new workspace without any data using the Unity Catalog, can I hide/delete the hive_metastore, main, samples, and system catalogs?

Find root path to Repo for .py file import

Re: In Python, Streaming read by DLT from Hive Table

Re: DLT data quality UI was present last week. Now absent. Did I change a setting?

Re: Can you use the Secrets API 2.0 in a Delta Live Tables configuration?

Re: In a new workspace without any data using the Unity Catalog, can I hide/delete the hive_metastore, main, samples, and system catalogs?

Re: Find root path to Repo for .py file import