- 511 Views
- 6 replies
- 1 kudos
Structured streaming in Databricks using delta table
Hi everyone, I’m new to Databricks and exploring its features. I’m trying to implement Change Data Capture (CDC) from the bronze layer to the silver layer using streaming. Could anyone share sample code or reference materials for implementing CDC wit...
- 511 Views
- 6 replies
- 1 kudos
- 1 kudos
You can also look at https://www.databricks.com/resources/demos#tutorials
- 1 kudos
- 247 Views
- 2 replies
- 0 kudos
DLT pipeline observability questions (and maybe suggestions)
All my questions is around this code block@Dlt.append_flow(target=”target_table”): def flow_01(): df = spark.readStream.table(“table_01”) @dlt.append_flow(target=”target_table”): def flow_02(): df = spark.readStream.table(“table_02”)The first qu...
- 247 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @guangyi , I am getting back to you with some insights Regarding your first question about checkpointing You can manually check the checkpointing location of your stream table. The checkpoints of your Delta Live Tables are under Storage locatio...
- 0 kudos
- 395 Views
- 1 replies
- 0 kudos
Databricks Asset Bundle - variables for job trigger
I'm using the Databricks CLI to deploy an asset bundle for a job. I'm attempting to setup the configuration such that the "dev" target does not have a trigger on it, and the "prod" target does. Essentially the dev job is not scheduled to run and th...
- 395 Views
- 1 replies
- 0 kudos
- 0 kudos
in your dev target, you can add mode to pause all trigger.targets: dev: mode: development DAB also have new update,you can also use preset to handle different target setting.targets: dev: presets: name_prefix: "testing_" # pre...
- 0 kudos
- 20208 Views
- 10 replies
- 6 kudos
We've encountered an error logging you in.
I'm trying to login for past two days and i'm still facing this error: "We've encountered an error logging you in." I've tried to reset the password multiple times and nothing happened. My friend is also not able to login. I request you to resolve t...
- 20208 Views
- 10 replies
- 6 kudos
- 6 kudos
Hey, I am not able to login to my work databrick's account. I am getting the above errors.
- 6 kudos
- 204 Views
- 1 replies
- 0 kudos
Resolved! Using sql inside Notebook using question marks
Hi all,I have a very quick question that I hope someone can help with.I want to execute a very simple sql statement like %sql select * from json.`/Volumes/adfmeta/Objects.json` where ObjectName like '%SGm$RITWebsader$911a%'However, the sql does not ...
- 204 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @JensV , Question marks are used as parameter placeholders, so could you please try to escape the question mark using Backslashes?select * from json.`/Volumes/adfmeta/Objects.json` where ObjectName like '%SGm\\$RITWebsader\\$911a%'Alternatively, w...
- 0 kudos
- 1588 Views
- 1 replies
- 0 kudos
use_cached_result
i am trying to execute same query on 3 different platforms - dbeaver, python notebook and sql workflow.I was expecting after first execution of the query irrespective of the platform, subsequent execution of same query should NOT re-compute. However ...
- 1588 Views
- 1 replies
- 0 kudos
- 0 kudos
I don't think its possible unless the results written into a table and its being used in the queries across the client. Pls refer to this https://docs.databricks.com/en/sql/user/queries/query-caching.html
- 0 kudos
- 3664 Views
- 1 replies
- 0 kudos
JSON string object with nested Array and Struct column to dataframe in pyspark
I am trying to convert JSON string stored in variable into spark dataframe without specifying schema, because I have a big number of different tables, so it has to be dynamically. I managed to do it with sc.parallelize, but since we are moving to Uni...
- 3664 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi filipjankovic, SparkContext sc is a Spark 1.0 API and is deprecated on Standard and Serverless compute. However, your input data is a list of dictionaries, which are supported with spark.createDataFrame. This should give you identical output witho...
- 0 kudos
- 17866 Views
- 12 replies
- 12 kudos
Resolved! dbutils or other magic way to get notebook name or cell title inside notebook cell
Not sure it exists but maybe there is some trick to get directly from python code:NotebookNameCellTitlejust working on some logger script shared between notebooks and it could make my life a bit easier
- 17866 Views
- 12 replies
- 12 kudos
- 12 kudos
I got the solution to work in terms of printing the notebook that I was running; however, what if you have notebook A that calls a function that prints the notebook name, and you run notebook B that %runs notebook A? I get the notebook B's name when...
- 12 kudos
- 737 Views
- 2 replies
- 3 kudos
Using Managed Identity Authentication in Unity Catalog using pyodbc
Hello,I am having trouble using Managed Identity Authentication in Unity Catalog using pyodbc in Azure Databricks. The same code works on a "Legacy Shared Compute". The code snippet is below: import pyodbc jdbc_url = ( "DRIVER={ODBC 17 DRIVER PATH...
- 737 Views
- 2 replies
- 3 kudos
- 3 kudos
Thank you very much!I have spent an enormous amount of hours fighting with this and in the end it was the type of cluster... I hope that this problem will be solved in the future, because affects the developments when you use databricks-connect and s...
- 3 kudos
- 534 Views
- 1 replies
- 0 kudos
Path based access not supported for tables with row filters?
Hello, I have encountered an issue recently and was not able to find a solution yet. I have a job on databricks that creates a table using dbt (dbt-databricks>=1.0.0,<2.0.0). I am setting the location_root configuration so that this table is externa...
- 534 Views
- 1 replies
- 0 kudos
- 0 kudos
To recreate the issue:PS. Good to know: using dbt to create materialized tables is equivalent to running "create or replace table table_name"The following code will create an external table with row security:create or replace table table_name using d...
- 0 kudos
- 318 Views
- 3 replies
- 0 kudos
DLT apply_changes not accepting upsert
I have a DLT pipeline that has bronze -> silver -> gold -> platinum. I need to include a table that is joined to the gold layer for platinum that allows upserts in the DLT pipeline. This table is managed externally via Databricks API. Anytime a chang...
- 318 Views
- 3 replies
- 0 kudos
- 0 kudos
You obtain this error: "Detected a data update in the source table at version 1. This is currently not supported...."becuse DLT is based on Structured Streaming.... and for Structured Streaming any changes (deletes, updates) in the source table are n...
- 0 kudos
- 5633 Views
- 8 replies
- 9 kudos
Resolved! Pandas.spark.checkpoint() doesn't broke lineage
Hi, I'm doing some something simple on Databricks notebook:spark.sparkContext.setCheckpointDir("/tmp/") import pyspark.pandas as ps sql=("""select field1, field2 From table Where date>='2021-01.01""") df = ps.sql(sql) df.spark.checkpoint()That...
- 5633 Views
- 8 replies
- 9 kudos
- 9 kudos
checkpoint() returns a checkpointed DataFrame, so you need to assign it to a new variable:checkpointedDF = df.checkpoint()
- 9 kudos
- 237 Views
- 1 replies
- 1 kudos
Resolved! Oracle DB connection works in single-user compute but not shared compute
I can connect to an onprem oracle DB using my single-user compute but when i switch over to a shared compute, I get invalid username/password error. I can connect to my onprem singlestore db using either compute so not sure why oracle would be diffe...
- 237 Views
- 1 replies
- 1 kudos
- 1 kudos
Based on internal research I found that shared access mode does not currently support Oracle JDBC connector only in Single/Assigned access mode. There is a feature request to include Oracle connector as part of the Lakehouse Federation. Once it is in...
- 1 kudos
- 15179 Views
- 4 replies
- 0 kudos
Fastest way to write a Spark Dataframe to a delta table
I read a huge array with several columns into memory, then I convert it into a spark dataframe, when I want to write to a delta table it using the following command it takes forever (I have a driver with large memory and 32 workers) : df_exp.write.m...
- 15179 Views
- 4 replies
- 0 kudos
- 0 kudos
The answers here are not correct.TLDR: _After_ the Spark DF is materialized, saveAsTable takes ages. 35seconds for 1million rows.saveAsTable() is SLOW - terribly so. Why? Would be nice to get an answer. The workaround is to avoid spark for delta - no...
- 0 kudos
- 1162 Views
- 12 replies
- 8 kudos
Resolved! Reading a csv file
while try to read a csv file using data frame , read csv using a file format , but fail in case of formatting and column error while loading how the data in databricks ,the code i used fordf = spark.read.format("csv") \ .option("header", "true") ...
- 1162 Views
- 12 replies
- 8 kudos
- 8 kudos
You can try add multiline option: df = ( spark.read.format("csv") .option("header", "true") .option("quote", '"') .option("delimiter", ",") .option("nullValue", "") .option("emptyValue", "NULL") .option("multiline", True) .schema(schem...
- 8 kudos