cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Hubert-Dudek
by • Esteemed Contributor III
  • 1357 Views
  • 1 replies
  • 1 kudos

Structured Streaming Aggregation

Utilizing structured streaming to read the change data feed from your Delta table empowers you to execute incremental streaming aggregations, such as counting and summing.

structured2.png
Data Engineering
Reply
  • 1357 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

Thank you for sharing @Hubert-Dudek !!!

  • 1 kudos
TimReddick
by • Contributor
  • 7271 Views
  • 6 replies
  • 2 kudos

Using run_job_task in Databricks Asset Bundles

Do Databrick Asset Bundles support run_job_task tasks?I've made various attempts to add a run_job_task with a specified job_id. See my the code_snippet below. I tried substituting the job_id using ${...} syntax, as well as three other ways which I've...

Data Engineering
Databrick Asset Bundles
run_job_task
Reply
  • 7271 Views
  • 6 replies
  • 2 kudos
View Replies
Latest Reply
kyle_r
New Contributor II
  • 2 kudos

Ah, I see it is a known bug in the Databricks CLI: Asset bundle run_job_task fails · Issue #812 · databricks/cli (github.com). Anyone facing this issue should comment on and keep an eye on that ticket for resolution. 

  • 2 kudos
5 More Replies
Sanjay96m
by • New Contributor
  • 1196 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks Certification exam Suspended. Need Assistance

I was taking online exam for Databricks Certified Data Analyst Associate on 06-Oct-2023 1:45PM. In between, they paused it and wanted to survey my whole room which they did, told me to clear the table of water bottle and laptop charger and then asked...

Data Engineering
Reply
  • 1196 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Cert-Team
Esteemed Contributor
  • 0 kudos

@Sanjay96m Thank you for your patience, the support team is working through support tickets and will reach out to you shortlly.

  • 0 kudos
Hubert-Dudek
by • Esteemed Contributor III
  • 1098 Views
  • 1 replies
  • 1 kudos

Foreign catalogs

With the introduction of the Unity Catalog in databricks, many of us have become familiar with creating catalogs. However, did you know that the Unity Catalog also allows you to create foreign catalogs? You can register databases from the following s...

db.png
Data Engineering
Reply
  • 1098 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

Thank you for sharing @Hubert-Dudek !!!

  • 1 kudos
117074
by • New Contributor III
  • 10637 Views
  • 1 replies
  • 1 kudos

[INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER]

Hi all,I'm trying to join 2 views in SQL editor for some analysis. I get the following error:[INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] You may get a different result due to the upgrading to Spark >= 3.0: Fail to parse '22/12/...

Data Engineering
Reply
  • 10637 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
117074
New Contributor III
  • 1 kudos

Hi Kaniz, I found the equivalent SQL code for this but it didn't seem to store the operation past the execution. I.e I would run the code to configure settings, then run the troublesome code afterwards and still get the same result. The problem has b...

  • 1 kudos
EDDatabricks
by • Contributor
  • 1359 Views
  • 0 replies
  • 1 kudos

Multiple DLT pipelines same target table

Is it possible to have multiple DLT pipelines write data concurrently and in append mode to the same Delta table? Because of different data sources, with different data volumes and required processing, we would like to have different pipelines stream...

Data Engineering
Delta tables
DLT pipeline
Reply
  • 1359 Views
  • 0 replies
  • 1 kudos
pavlos_skev
by • New Contributor III
  • 975 Views
  • 1 replies
  • 0 kudos

Potential Unity Catalog Bug: Invalid configuration value detected for fs.azure.account.keyInvalid

Hello,We are migrating to Unity Catalog (UC), and for very few of our tables, we get the below error when trying to write or even display them. We are using UC enabled clusters, usually with runtime versions 12.2 LTS. The below error, when it happens...

Data Engineering
Reply
  • 975 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16539034020
Contributor II
  • 0 kudos

Hello,  Thanks for contacting Databricks Support.  The error message indicates a problem with the configuration key fs.azure.account.key. This configuration key is used to provide the access key for the Azure Data Lake Storage account. Not sure if th...

  • 0 kudos
DBUser2
by • New Contributor III
  • 1309 Views
  • 1 replies
  • 0 kudos

Databricks sql using odbc issue

Hi,I'm connecting to a Databricks instance on Azure from a Windows Application using Simba ODBC driver, and when running SQL statements on delta tables, like INSERT, UPDATE, DELETE commands using Execute, the result doesn't indicate the no. of rows a...

Data Engineering
Reply
  • 1309 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi,I'm connecting to a Databricks instance on Azure from a Windows Application using Simba ODBC driver, and when running SQL statements on delta tables, like INSERT, UPDATE, DELETE commands using Execute, the result doesn't indicate the no. of rows a...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
marcuskw
by • Contributor II
  • 3198 Views
  • 1 replies
  • 1 kudos

Resolved! whenNotMatchedBySourceUpdate ConcurrentAppendException Partition

ConcurrentAppendException requires a good partitioning strategy, here my logic works without fault for "whenMatchedUpdate" and "whenNotMatchedInsert" logic. When using "whenNotMatchedBySourceUpdate" however it seems that the condition doesn't isolate...

Data Engineering
Reply
  • 3198 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
ConcurrentAppendException requires a good partitioning strategy, here my logic works without fault for "whenMatchedUpdate" and "whenNotMatchedInsert" logic. When using "whenNotMatchedBySourceUpdate" however it seems that the condition doesn't isolate...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
lawrence009
by • Contributor
  • 2329 Views
  • 3 replies
  • 1 kudos

Troubleshooting Spill

I am trying to troubleshoot why spill occurred during DeltaOptimizeWrite. I am running a 64-core cluster with 256 GB RAM, which I expect to be handle this amount data (see attached DAG).

IMG_1085.jpeg
Data Engineering
Reply
  • 2329 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

You can resolver the Spill to memory by increasing the shuffle partitions, but 16 GB of spill memory should not create a major impact of your job execution. Could you share more details on the actual source code that you are running?

  • 1 kudos
2 More Replies
Gilg
by • Contributor II
  • 2309 Views
  • 2 replies
  • 0 kudos

APPLY_CHANGES with json data

Hi Team,I am building a DLT pipeline and planning to use APPLY_CHANGES from Bronze to Silver. In the bronze table, a column has a json value. This value contains questions and answers as key, value pair and can change depending on list of questions h...

Data Engineering
Reply
  • 2309 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi Team,I am building a DLT pipeline and planning to use APPLY_CHANGES from Bronze to Silver. In the bronze table, a column has a json value. This value contains questions and answers as key, value pair and can change depending on list of questions h...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
1 More Replies
FabriceDeseyn
by • Contributor
  • 1160 Views
  • 0 replies
  • 0 kudos

merge breaking persistance of dataframe

Hi allIn the minimal example below you can see that executing a merge statement trigger recomputation of a persisted dataframe. How does this happen?   from delta.tables import DeltaTable table_name = "hive_metastore.default.test_table" # initializ...

FabriceDeseyn_1-1694011507567.png
Data Engineering
Reply
  • 1160 Views
  • 0 replies
  • 0 kudos
scvbelle
by • New Contributor III
  • 3233 Views
  • 3 replies
  • 3 kudos

Resolved! DLT failure: ABFS does not allow files or directories to end with a dot

In my DLT pipeline outlined below which generically cleans identifier tables, after successfully creating initial streaming tables from the append-only sources, fails when trying to create the second cleaned tables witht the following:It'**bleep** cl...

Data Engineering
abfss
azure
dlt
engineering
Reply
  • 3233 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Priyanka_Biswas
Valued Contributor
  • 3 kudos

Hi @scvbelle The error message you're seeing is caused by an IllegalArgumentException error due to the restriction in Azure Blob File System (ABFS) that does not allow files or directories to end with a dot. This error is thrown by the trailingPeriod...

  • 3 kudos
2 More Replies
Labels