cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

JissMathew
by Contributor
  • 511 Views
  • 6 replies
  • 1 kudos

Structured streaming in Databricks using delta table

Hi everyone, I’m new to Databricks and exploring its features. I’m trying to implement Change Data Capture (CDC) from the bronze layer to the silver layer using streaming. Could anyone share sample code or reference materials for implementing CDC wit...

  • 511 Views
  • 6 replies
  • 1 kudos
Latest Reply
Mike_Szklarczyk
New Contributor III
  • 1 kudos

You can also look at https://www.databricks.com/resources/demos#tutorials 

  • 1 kudos
5 More Replies
guangyi
by Contributor III
  • 247 Views
  • 2 replies
  • 0 kudos

DLT pipeline observability questions (and maybe suggestions)

All my questions is around this code block@Dlt.append_flow(target=”target_table”): def flow_01(): df = spark.readStream.table(“table_01”) @dlt.append_flow(target=”target_table”): def flow_02(): df = spark.readStream.table(“table_02”)The first qu...

  • 247 Views
  • 2 replies
  • 0 kudos
Latest Reply
Nam_Nguyen
Databricks Employee
  • 0 kudos

Hello @guangyi , I am getting back to you with some insights Regarding your first question about checkpointing You can manually check the checkpointing location of your stream table. The checkpoints of your Delta Live Tables are under Storage locatio...

  • 0 kudos
1 More Replies
madams
by Contributor
  • 395 Views
  • 1 replies
  • 0 kudos

Databricks Asset Bundle - variables for job trigger

I'm using the Databricks CLI to deploy an asset bundle for a job.  I'm attempting to setup the configuration such that the "dev" target does not have a trigger on it, and the "prod" target does.  Essentially the dev job is not scheduled to run and th...

  • 395 Views
  • 1 replies
  • 0 kudos
Latest Reply
SigaEd
New Contributor II
  • 0 kudos

in your dev target, you can add mode to pause all trigger.targets: dev: mode: development DAB also have new update,you can also use preset  to handle different target setting.targets: dev: presets: name_prefix: "testing_" # pre...

  • 0 kudos
AWS1567
by New Contributor III
  • 20208 Views
  • 10 replies
  • 6 kudos

We've encountered an error logging you in.

I'm trying to login for past two days and i'm still facing this error: "We've encountered an error logging you in." I've tried to reset the password multiple times and nothing happened. My friend is also not able to login. I request you to resolve t...

Databricks_login_issue
  • 20208 Views
  • 10 replies
  • 6 kudos
Latest Reply
rmutili
New Contributor II
  • 6 kudos

 Hey, I am not able to login to my work databrick's account. I am getting the above errors.

  • 6 kudos
9 More Replies
JensV
by New Contributor II
  • 204 Views
  • 1 replies
  • 0 kudos

Resolved! Using sql inside Notebook using question marks

Hi all,I have a very quick question that I hope someone can help with.I want to execute a very simple sql statement like %sql select * from json.`/Volumes/adfmeta/Objects.json` where ObjectName like '%SGm$RITWebsader$911a%'However, the sql does not ...

  • 204 Views
  • 1 replies
  • 0 kudos
Latest Reply
JAHNAVI
Databricks Employee
  • 0 kudos

Hi @JensV , Question marks are used as parameter placeholders, so could you please try to escape the question mark using Backslashes?select * from json.`/Volumes/adfmeta/Objects.json` where ObjectName like '%SGm\\$RITWebsader\\$911a%'Alternatively, w...

  • 0 kudos
angel_ba
by New Contributor II
  • 1588 Views
  • 1 replies
  • 0 kudos

use_cached_result

i am trying to execute same query on 3 different platforms - dbeaver, python notebook and sql workflow.I was expecting after first execution of the query irrespective of the platform, subsequent execution of same query should NOT re-compute. However ...

  • 1588 Views
  • 1 replies
  • 0 kudos
Latest Reply
SenthilRT
New Contributor III
  • 0 kudos

I don't think its possible unless the results written into a  table and its being used in the queries across the client. Pls refer to this https://docs.databricks.com/en/sql/user/queries/query-caching.html

  • 0 kudos
filipjankovic
by New Contributor
  • 3664 Views
  • 1 replies
  • 0 kudos

JSON string object with nested Array and Struct column to dataframe in pyspark

I am trying to convert JSON string stored in variable into spark dataframe without specifying schema, because I have a big number of different tables, so it has to be dynamically. I managed to do it with sc.parallelize, but since we are moving to Uni...

  • 3664 Views
  • 1 replies
  • 0 kudos
Latest Reply
cgrant
Databricks Employee
  • 0 kudos

Hi filipjankovic, SparkContext sc is a Spark 1.0 API and is deprecated on Standard and Serverless compute. However, your input data is a list of dictionaries, which are supported with spark.createDataFrame. This should give you identical output witho...

  • 0 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 17866 Views
  • 12 replies
  • 12 kudos

Resolved! dbutils or other magic way to get notebook name or cell title inside notebook cell

Not sure it exists but maybe there is some trick to get directly from python code:NotebookNameCellTitlejust working on some logger script shared between notebooks and it could make my life a bit easier

  • 17866 Views
  • 12 replies
  • 12 kudos
Latest Reply
rtullis
New Contributor II
  • 12 kudos

I got the solution to work in terms of printing the notebook that I was running; however, what if you have notebook A that calls a function that prints the notebook name, and you run notebook B that %runs notebook A?  I get the notebook B's name when...

  • 12 kudos
11 More Replies
rushi29
by New Contributor III
  • 737 Views
  • 2 replies
  • 3 kudos

Using Managed Identity Authentication in Unity Catalog using pyodbc

Hello,I am having trouble using Managed Identity Authentication in Unity Catalog using pyodbc in Azure Databricks. The same code works on a "Legacy Shared Compute". The code snippet is below: import pyodbc jdbc_url = (    "DRIVER={ODBC 17 DRIVER PATH...

  • 737 Views
  • 2 replies
  • 3 kudos
Latest Reply
mbenavent
New Contributor II
  • 3 kudos

Thank you very much!I have spent an enormous amount of hours fighting with this and in the end it was the type of cluster... I hope that this problem will be solved in the future, because affects the developments when you use databricks-connect and s...

  • 3 kudos
1 More Replies
Nes_Hdr
by New Contributor III
  • 534 Views
  • 1 replies
  • 0 kudos

Path based access not supported for tables with row filters?

Hello,  I have encountered an issue recently and was not able to find a solution yet. I have a job on databricks that creates a table using dbt (dbt-databricks>=1.0.0,<2.0.0). I am setting the location_root configuration so that this table is externa...

Data Engineering
dbt
row_filter
  • 534 Views
  • 1 replies
  • 0 kudos
Latest Reply
Nes_Hdr
New Contributor III
  • 0 kudos

To recreate the issue:PS. Good to know: using dbt to create materialized tables is equivalent to running "create or replace table table_name"The following code will create an external table with row security:create or replace table table_name using d...

  • 0 kudos
rimaissa
by New Contributor III
  • 318 Views
  • 3 replies
  • 0 kudos

DLT apply_changes not accepting upsert

I have a DLT pipeline that has bronze -> silver -> gold -> platinum. I need to include a table that is joined to the gold layer for platinum that allows upserts in the DLT pipeline. This table is managed externally via Databricks API. Anytime a chang...

  • 318 Views
  • 3 replies
  • 0 kudos
Latest Reply
Mike_Szklarczyk
New Contributor III
  • 0 kudos

You obtain this error: "Detected a data update in the source table at version 1. This is currently not supported...."becuse DLT is based on Structured Streaming.... and for Structured Streaming any changes (deletes, updates) in the source table are n...

  • 0 kudos
2 More Replies
alejandrofm
by Valued Contributor
  • 5633 Views
  • 8 replies
  • 9 kudos

Resolved! Pandas.spark.checkpoint() doesn't broke lineage

Hi, I'm doing some something simple on Databricks notebook:spark.sparkContext.setCheckpointDir("/tmp/")   import pyspark.pandas as ps   sql=("""select field1, field2 From table Where date>='2021-01.01""")   df = ps.sql(sql) df.spark.checkpoint()That...

  • 5633 Views
  • 8 replies
  • 9 kudos
Latest Reply
annafina
New Contributor II
  • 9 kudos

checkpoint() returns a checkpointed DataFrame, so you need to assign it to a new variable:checkpointedDF = df.checkpoint()

  • 9 kudos
7 More Replies
OldManCoder
by New Contributor II
  • 237 Views
  • 1 replies
  • 1 kudos

Resolved! Oracle DB connection works in single-user compute but not shared compute

I can connect to an onprem oracle DB using my single-user compute but when i switch over to a shared compute, I get invalid username/password error.  I can connect to my onprem singlestore db using either compute so not sure why oracle would be diffe...

  • 237 Views
  • 1 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

Based on internal research I found that shared access mode does not currently support Oracle JDBC connector only in Single/Assigned access mode. There is a feature request to include Oracle connector as part of the Lakehouse Federation. Once it is in...

  • 1 kudos
nakaxa
by New Contributor
  • 15179 Views
  • 4 replies
  • 0 kudos

Fastest way to write a Spark Dataframe to a delta table

I read a huge array with several columns into memory, then I convert it into a spark dataframe,  when I want to write to a delta table it using the following command it takes forever (I have a driver with large memory and 32 workers) : df_exp.write.m...

  • 15179 Views
  • 4 replies
  • 0 kudos
Latest Reply
Reiska
New Contributor II
  • 0 kudos

The answers here are not correct.TLDR: _After_ the Spark DF is materialized, saveAsTable takes ages. 35seconds for 1million rows.saveAsTable() is SLOW - terribly so. Why? Would be nice to get an answer. The workaround is to avoid spark for delta - no...

  • 0 kudos
3 More Replies
JissMathew
by Contributor
  • 1162 Views
  • 12 replies
  • 8 kudos

Resolved! Reading a csv file

while try to read a csv file using data frame , read csv using a  file format , but fail in case of formatting and column error while loading how the data in databricks ,the code i used fordf = spark.read.format("csv") \    .option("header", "true") ...

Screenshot 2024-11-14 172800.png Screenshot 2024-11-14 172727.png
  • 1162 Views
  • 12 replies
  • 8 kudos
Latest Reply
Mike_Szklarczyk
New Contributor III
  • 8 kudos

You can try add multiline option: df = ( spark.read.format("csv") .option("header", "true") .option("quote", '"') .option("delimiter", ",") .option("nullValue", "") .option("emptyValue", "NULL") .option("multiline", True) .schema(schem...

  • 8 kudos
11 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels