cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
andrew0117
by • Contributor
  • 2795 Views
  • 4 replies
  • 9 kudos

Resolved! How to call a few child notebooks from master notebook parallelly?

Planning using dbutils.notebook.run() to call all the child notebooks in the master notebook, but they are executed sequentially. 

Data Engineering
Reply
  • 2795 Views
  • 4 replies
  • 9 kudos
View Replies
Latest Reply
UmaMahesh1
Honored Contributor III
  • 9 kudos

Hi @andrew li​ Yes, They do run on the cluster on which the master notebook is running. Specifically, we are multithreading the Spark driver with Futures to enable parallel job submission. You can check out more on threads and futures for deeper unde...

  • 9 kudos
3 More Replies
Anonymous
by • Not applicable
  • 1352 Views
  • 0 replies
  • 0 kudos

The CDC Logs from AWS DMS not apply correctly

I have a dms task that processing the full-load and replication ongoing tasksfrom source (MSSQL) to target (AWS S3)then use delta lake to handle the CDC logsI've a notebook that would insert data into mssql continuously (with id as primary key)then d...

204293406-01bf6cc1-bb6f-42bb-9bfe-e9b1f5135ae9[1]
Data Engineering
Reply
  • 1352 Views
  • 0 replies
  • 0 kudos
apayne
by • New Contributor III
  • 6680 Views
  • 1 replies
  • 4 kudos

Databricks Jobs API not returning notebook run results?

Calling a databricks notebook using the Rest API, can confirm that it is executing the notebook, but is not accepting my parameters or returning a notebook output. Any ideas on what I am doing wrong here?My code and notebook function are below, tryin...

view view2
Data Engineering
Reply
  • 6680 Views
  • 1 replies
  • 4 kudos
View Replies
Latest Reply
apayne
New Contributor III
  • 4 kudos

Resolved this by using dbutils within the notebook being called from the API.# databricks notebook function   data = dbutils.widgets.get('data') # pulls base_parameters from API call   def add_test(i): result = i + ' COMPLETE' return result  ...

  • 4 kudos
Swapnil1998
by • New Contributor III
  • 945 Views
  • 0 replies
  • 0 kudos

How to query a MySQL Table from Databricks?

I wanted to query a MySQL Table using Databricks rather than reading the complete data using a dbtable option, which will help in incremental loads.remote_table = (spark.read .format("jdbc") .option("driver", driver) .option("url", URL) .option("quer...

Data Engineering
Reply
  • 945 Views
  • 0 replies
  • 0 kudos
Harish14
by • New Contributor III
  • 1760 Views
  • 4 replies
  • 1 kudos

Hi @Kaniz Fatma​ and @Nadia Elsayed​ , i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtain...

Hi @Kaniz Fatma​ and @Nadia Elsayed​ ,i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtained below 70% in assessment but as per the section wise results i have gained more than 70% . Can you ...

Data Engineering
Reply
  • 1760 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Nadia1
Databricks Employee
  • 1 kudos

Hello Harish - I have responded via email. Thank you

  • 1 kudos
3 More Replies
Priyanka48
by • Contributor
  • 12664 Views
  • 4 replies
  • 11 kudos

The functionality of table property delta.logRetentionDuration

We have one project requirement where we have to store only the 14 days history for delta tables. So for testing, I have set the delta.logRetentionDuration = 2 days using the below commandspark.sql("alter table delta.`[delta_file_path]` set TBLPROPER...

Data Engineering
Reply
  • 12664 Views
  • 4 replies
  • 11 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 11 kudos

Hi, by default there is a safety interval enabled. So if you set a retentionperiod lower than that interval (7 days), data in that interval will not be deleted.You have to specificall override this safety interval by setting spark.databricks.delta.r...

  • 11 kudos
3 More Replies
Bartek
by • Contributor
  • 6264 Views
  • 3 replies
  • 9 kudos

Resolved! Number of partitions in Spark UI Simulator experiment

I am learning how to optimize Spark applications with experiments from Spark UI Simulator. There is experiment #1​596 about data skew and in command 2 there is comment about how many partitions will be set as default:// Factor of 8 cores and greater ...

obraz
Data Engineering
Reply
  • 6264 Views
  • 3 replies
  • 9 kudos
View Replies
Latest Reply
UmaMahesh1
Honored Contributor III
  • 9 kudos

Hi @Bartosz Maciejewski​ Generally we arrive at the number of shuffle partitions using the following method.Input Size Data - 100 GBIdeal partition target size - 128 MBCores - 8Ideal number of partitions = (100*1028)/128 = 803.25 ~ 804To utiltize the...

  • 9 kudos
2 More Replies
640913
by • New Contributor III
  • 6128 Views
  • 2 replies
  • 1 kudos

%pip install requirements.txt - path not found

Hi everyone,I was just testing things out to come up with a reasonable way of working with version management in DB and was inspired by the commands specified here. For my team and I, it makes no sense to put the requirements file in the dbfs locatio...

Data Engineering
Reply
  • 6128 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
UmaMahesh1
Honored Contributor III
  • 1 kudos

Created a requirements.txt and pulled it into your repo folder ? Didn't get exactly this part..Maybe a screenshot should do for my understanding.If you are not storing your TEXT file in any storage space, you can't do the above stuff you are trying t...

  • 1 kudos
1 More Replies
Harish14
by • New Contributor III
  • 1517 Views
  • 3 replies
  • 4 kudos

Hi @Vidula Khanna​  i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtained below 70% in ass...

Hi @Vidula Khanna​ i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtained below 70% in assessment but as per the section wise results i have gained more than 70% . Can you please check and up...

Data Engineering
Reply
  • 1517 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Harish Raj​ Please bear with us!I already have shared this issue with the respective person in the Academy team. Thanks for your patience!

  • 4 kudos
2 More Replies
DwrCymru
by • New Contributor III
  • 3123 Views
  • 4 replies
  • 4 kudos

Data bricks issue writing to Azure synapse

Process is to pickup data from an external table in Azure Synapse and combine with other tables and write to another synapse tables. Data for external table is in Azure storage. It was fine for months ,all of sudden last week it errored out with err...

Data Engineering
Reply
  • 3123 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
UmaMahesh1
Honored Contributor III
  • 4 kudos

Also do check out this blog from this stack overflow question.https://stackoverflow.com/questions/66747544/databricks-write-back-to-azure-synapse-errorhttps://chinnychukwudozie.com/2020/11/13/write-data-from-azure-databricks-to-azure-synapse-analytic...

  • 4 kudos
3 More Replies
elgeo
by • Valued Contributor II
  • 13368 Views
  • 7 replies
  • 4 kudos

Resolved! Invalid JDBC url

Hello. I am trying to establish a connection between DBeaver and Databricks. I followed the steps in DBeaver integration with Databricks | Databricks on AWS, but I get the following error while testing the connection: Could anyone provide any insight...

jdbc_url_error
Data Engineering
Reply
  • 13368 Views
  • 7 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @ELENI GEORGOUSI​ Glad to hear! It's a request that mark an answer as best.Thanks...

  • 4 kudos
6 More Replies
mickniz
by • Contributor
  • 4458 Views
  • 3 replies
  • 6 kudos

Unable to create materialized view in Unity enabled Catalogues.

Hi Team,I was going through one of the videos of Databricks Sql Serverless and it say there is materialized view support . We can create materialized view .I tried same on my cluster of Sql Warehouse gives below error: 

image
Data Engineering
Reply
  • 4458 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
Felix
Databricks Employee
  • 6 kudos

Materialized views is in private preview right now afaik. Please talk to your account or customer success team at Databricks in order to sign up and enable it for your workspace. Thanks!

  • 6 kudos
2 More Replies
nevoezov
by • New Contributor II
  • 1597 Views
  • 0 replies
  • 2 kudos

java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2 - Delta tables dynamic partition overwrite on Databricks ACL enabled clusters

I'm working on Databricks ACL enabled clusters, and having trouble performing dynamic partition overwrite to Delta tables.I have created a test table using the following query:CREATE TABLE IF NOT EXISTS test_01 ( id STRING, name STRING, c...

Data Engineering
Reply
  • 1597 Views
  • 0 replies
  • 2 kudos
Gustavo_Az
by • Contributor
  • 4847 Views
  • 9 replies
  • 27 kudos

Resolved! When will be available the course "Data Engineering with Databricks V3" in Databricks Academy?

In the documentation of the V2 it says that it would be released 4 days ago. But searching for it in the academy only throws results for V1 and V2.

Data Engineering
Reply
  • 4847 Views
  • 9 replies
  • 27 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 27 kudos

@Gustavo Amadoz Navarro​  Updated new infoThis course will be part of the data engineer learning path once the Databricks Certified Data Engineer Associate V3 exam is released (November 19, 2022). BEFORE YOU GET STARTED: Please note that this course,...

  • 27 kudos
8 More Replies
yogu
by • Honored Contributor III
  • 1404 Views
  • 3 replies
  • 25 kudos

Can anyone plz help for mine below issues

Can anyone plz help for mine below issues

Data Engineering
Reply
  • 1404 Views
  • 3 replies
  • 25 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 25 kudos

Hi @Yogita Chavan​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks

  • 25 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels