cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

EDDatabricks
by Contributor
  • 2293 Views
  • 2 replies
  • 0 kudos

Concurrency issue with append only writed

Dear all,We have a pyspark streaming job (DBR: 14.3) that continuously writes new data on a Delta Table (TableA).On this table, there is a pyspark batch job (DBR: 14.3) that operates every 15 minuted and in some cases it may delete some records from ...

Data Engineering
Concurrency
DBR 14.3
delta
MERGE
  • 2293 Views
  • 2 replies
  • 0 kudos
Latest Reply
Dilisha
New Contributor II
  • 0 kudos

Hi @EDDatabricks  - were you able to find the fix for this? I am also facing a similar issue. Added more details here  - Getting concurrent Append exception after upgradin... - Databricks Community - 76521

  • 0 kudos
1 More Replies
FhSpZ
by New Contributor II
  • 557 Views
  • 1 replies
  • 0 kudos

Error AgnosticEncoder.isStruct() in Intellij using Scala locally.

I've been trying to execute a connect to Azure Databricks from Intellij using Scala locally, but I've got this error below: Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.encoders.AgnosticEncoder.isStruct()Zat o...

  • 557 Views
  • 1 replies
  • 0 kudos
Latest Reply
FhSpZ
New Contributor II
  • 0 kudos

Hi @Retired_mod,I ensured that I was using the correct Spark version that matched the version of my databricks runtime, which was the same. But I tried use the Spark version 3.5.1 locally in the .sbt dependencies, then this worked, kind strange.Anywa...

  • 0 kudos
avrm91
by Contributor
  • 1937 Views
  • 4 replies
  • 1 kudos

How to load xlsx Files to Delta Live Tables (DLT)?

I want to load a .xlsx file to DLT but struggling as it is not available with Autoloader.With the Assistant I tried to load the .xlsx first to a data frame and then send it to DLT.  import dlt from pyspark.sql import SparkSession # Load xlsx file in...

  • 1937 Views
  • 4 replies
  • 1 kudos
Latest Reply
avrm91
Contributor
  • 1 kudos

Added a feature request into Azure Community PortalXLSX - DLT Autoloader · Community (azure.com)

  • 1 kudos
3 More Replies
avrm91
by Contributor
  • 874 Views
  • 2 replies
  • 2 kudos

Resolved! XBRL File Format

I was searching for some XBRL documentation with Databricks as it is a business reporting standard format.Especially for DLT and Autoloader. Is there anything in the development pipeline? 

  • 874 Views
  • 2 replies
  • 2 kudos
Latest Reply
avrm91
Contributor
  • 2 kudos

I added the XBRL to the Azure communityXBRL · Community (azure.com)

  • 2 kudos
1 More Replies
FrankTa
by New Contributor II
  • 1597 Views
  • 2 replies
  • 2 kudos

Resolved! Unstable workflow runs lately

Hi!We are using Databricks on Azure on production since about 3 months. A big part of what we use Databricks for is processing data using a workflow with various Python notebooks. We run the workflow on a 'Pools' cluster and on a 'All-purpose compute...

  • 1597 Views
  • 2 replies
  • 2 kudos
Latest Reply
FrankTa
New Contributor II
  • 2 kudos

Hi holly,Thanks for your reply, good to hear that the 403 errors are on the radar and due to be fixed. I will reach out to support in case of further issues.

  • 2 kudos
1 More Replies
PaulMarsh
by New Contributor II
  • 758 Views
  • 1 replies
  • 0 kudos

Python databricks sql.connect takes 10minutes exact to connect to Serverless SQL Datawarehouse

Have tried this with both the SQL Warehouse up and running and turned off. The connection eventually is made and works without issue, but the initial connect() takes 10minutes to establish. No other connectivity issues between the box and the Warehou...

  • 758 Views
  • 1 replies
  • 0 kudos
Latest Reply
PaulMarsh
New Contributor II
  • 0 kudos

@Retired_mod thank you for the reply, it seems the issue came down to the script was running with Python 3.12 and I should have been using 3.11.9 

  • 0 kudos
lprevost
by Contributor
  • 2473 Views
  • 7 replies
  • 2 kudos

Resolved! Incremental Loads from a Catalog/DLT

The databricks guide outlines several use cases for loading data via delta live tables:https://docs.databricks.com/en/delta-live-tables/load.html This includes autoloader from cloudfiles, kafka messages, small static datasets, etc.But, one use case i...

  • 2473 Views
  • 7 replies
  • 2 kudos
Latest Reply
lucasrocha
Databricks Employee
  • 2 kudos

Hello @lprevost , I hope this message finds you well. In order to install this library, you can follow any of the steps below:- First step - search for libraries and versions: Go the cluster;Libraries;Install new;Maven;Search Packages;Change from Spa...

  • 2 kudos
6 More Replies
ameya
by New Contributor
  • 5099 Views
  • 1 replies
  • 0 kudos

Resolved! Adding new columns to a Delta Live table in a CDC process

Hi I am new to databricks and still learning.I am trying to do a CDC on a table.  APPLY CHANGES INTO LIVE.table1 FROM schema2.table2 KEYS (Id) SEQUENCE BY orderByColumn COLUMNS * EXCEPT (col1, col2) STORED AS SCD TYPE 1 ;  table1 is in schema1 and ...

  • 5099 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Databricks Employee
  • 0 kudos

Hi @ameya , Scenario 1: Enabling Delta schema evolution in your table or at DLT pipeline level should suffice for the scenario of new fields being added to the schema.  Scenario 2: The INSERT statement doesn't support schema evolution as described in...

  • 0 kudos
NarenderKumar
by New Contributor III
  • 975 Views
  • 2 replies
  • 2 kudos

Resolved! Can we parameterize the compute in job cluster

I have created a workflow job in databricks with job parameters.I want to run the job same with different workloads and data volume.So I want the compute cluster to be parametrized so that I can pass the compute requirements(driver, executor size and...

  • 975 Views
  • 2 replies
  • 2 kudos
Latest Reply
brockb
Databricks Employee
  • 2 kudos

Hi @NarenderKumar , Have you considered leveraging autoscaling for the existing cluster?If this does not meet your needs, are the differing volume/workloads known in advance? If so, could different compute be provisioned using Infrastructure as Code ...

  • 2 kudos
1 More Replies
yusufd
by New Contributor III
  • 5436 Views
  • 5 replies
  • 3 kudos

Pyspark serialization

Hi,I was looking for comprehensive documentation on implementing serialization in pyspark, most of the places I have seen is all about serialization with scala. Could you point out where I can get a detailed explanation on it?

  • 5436 Views
  • 5 replies
  • 3 kudos
Latest Reply
yusufd
New Contributor III
  • 3 kudos

Thank you @Retired_mod  for the prompt reply. This clears the things and also distinguishes between spark-scala and pyspark. Appreciate your explanation. Will apply this and also share any findings based on this which will help the community!

  • 3 kudos
4 More Replies
jacovangelder
by Honored Contributor
  • 1431 Views
  • 2 replies
  • 8 kudos

How do you define PyPi libraries on job level in Asset Bundles?

Hello,Reading the documentation, it does not state it is possible to define libraries on job level instead of on task level. It feels really counter-intuitive putting libraries on task level in Databricks workflows provisioned by Asset Bundles. Is th...

  • 1431 Views
  • 2 replies
  • 8 kudos
Latest Reply
jacovangelder
Honored Contributor
  • 8 kudos

Thanks @Witold ! Thought so. I decided to go with an init script where I install my dependencies rather than installing libraries. For future reference, this is what it looks like:job_clusters: - job_cluster_key: job_cluster new_cluster: ...

  • 8 kudos
1 More Replies
KendraVant
by New Contributor II
  • 15472 Views
  • 7 replies
  • 2 kudos

Resolved! How do I clear all output results in a notebook?

I'm building notebooks for tutorial sessions and I want to clear all the output results from the notebook before distributing it to the participants. This functionality exists in Juypter but I can't find it in Databricks. Any pointers?

  • 15472 Views
  • 7 replies
  • 2 kudos
Latest Reply
holly
Databricks Employee
  • 2 kudos

Yes! Run > â€ƒClear >  Clear all cell outputs Fun fact, this feature was made ~10 years ago when we realised all our customer demos looked very messy and had lots of spoilers in them!

  • 2 kudos
6 More Replies
kmaley
by New Contributor
  • 659 Views
  • 1 replies
  • 1 kudos

Concurrent append exception - Two streaming sources writing to same record on the delta table

Hi All, I have a scenario where there are 2 streaming sources Stream1( id, col1, col2) and Stream 2( id, col3, col4) and my delta table has columns (id, col1, col2, col3, col4). My requirement is to insert the record into the delta table if the corre...

  • 659 Views
  • 1 replies
  • 1 kudos
Latest Reply
Witold
Honored Contributor
  • 1 kudos

I would keep both write operations separate, i.e. they should write in own tables/partitions. In later stages (e.g. silver), you can easily merge them.

  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels