cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

JeremyH
by New Contributor II
  • 418 Views
  • 4 replies
  • 0 kudos

CREATE WIDGETS in SQL Notebook attached to SQL Warehouse Doesn't Work.

I'm able to create and use widgets using the UI in my SQL notebooks, but they get lost quite frequently when the notebook is reset.There is documentation suggesting we can create widgets in code in SQL: https://learn.microsoft.com/en-us/azure/databri...

  • 418 Views
  • 4 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

Hi @JeremyH - can you please try adding like the below in your query and see if widgets are getting populated? {{parameter_name }}

  • 0 kudos
3 More Replies
Nathant93
by New Contributor III
  • 447 Views
  • 2 replies
  • 0 kudos

Resolved! Unzipping with Serverless Compute

HiI have started using serverless compute but have come across the limitation that I cannot use the local filesystem for temporarily storing the files and directories before moving them to where they need to be in adls.Does anyone have a way of unzip...

Data Engineering
serverless
unzip
  • 447 Views
  • 2 replies
  • 0 kudos
Latest Reply
delonb2
New Contributor III
  • 0 kudos

Do you have the ability to make a Unity Catalog volume? You could use it as temporary storage before migrating the files to adls.

  • 0 kudos
1 More Replies
wschoi
by New Contributor III
  • 3433 Views
  • 5 replies
  • 3 kudos

How to fix plots and image color rendering on Notebooks?

I am currently running dark mode for my Databricks Notebooks, and am using the "new UI" released a few days ago (May 2023) and the "New notebook editor."Currently all plots (like matplotlib) are showing wrong colors. For example, denoting:```... p...

  • 3433 Views
  • 5 replies
  • 3 kudos
Latest Reply
aleph_null
New Contributor II
  • 3 kudos

Any updated on this issue? This is a huge drawback to use the dark theme

  • 3 kudos
4 More Replies
Dave_Nithio
by Contributor
  • 6718 Views
  • 5 replies
  • 7 kudos

Resolved! Delta Live Table Pipeline with Multiple Notebooks

I have two notebooks created for my Delta Live Table pipeline. The first is a utils notebook with functions I will be reusing for other pipelines. The second contains my actual creation of the delta live tables. I added both notebooks to the pipeline...

image.png
  • 6718 Views
  • 5 replies
  • 7 kudos
Latest Reply
fecavalc08
New Contributor III
  • 7 kudos

Hi @Vivian Wilfred​ and @Dave Wilson​ we solved our reusability code with repos and pointing the code to our main code:sys.path.append(os.path.abspath('/Workspace/Repos/[your repo]/[folder with the python scripts'))from your_class import *It just wor...

  • 7 kudos
4 More Replies
QuikPl4y
by New Contributor III
  • 714 Views
  • 2 replies
  • 0 kudos

Resolved! Databricks to Oracle to Delete Rows

Hi Community!  I’m working in a Fatabricks notebook and using the Oracle JDBC Thin Client connector to query and Oracle table, merge together and select specific rows from my dataframe and write those rows to a table back in Oracle. All of this works...

  • 714 Views
  • 2 replies
  • 0 kudos
Latest Reply
delonb2
New Contributor III
  • 0 kudos

This stack overflow post ran into the same issue, would be worth trying How to delete a record from Oracle Table in Python SQLAlchemy - Stack Overflow

  • 0 kudos
1 More Replies
_TJ
by New Contributor III
  • 3617 Views
  • 4 replies
  • 2 kudos

Incremental load from source; how to handle deletes

Small introduction; I'm a BI/Data developer, mostly working in MSSQL & DataFactory, coming from SSIS. Now I'm trying Databricks, to see if it works for me and my customers. I got enthusiastic by the video; https://www.youtube.com/watch?v=PIFL7W3DmaY&...

  • 3617 Views
  • 4 replies
  • 2 kudos
Latest Reply
abajpai
New Contributor II
  • 2 kudos

@_TJ did you find a solution for sliding window

  • 2 kudos
3 More Replies
alej
by New Contributor
  • 355 Views
  • 1 replies
  • 0 kudos

Spark Scala Vs Pyspark

With the release of spark connect and used defined table functions for pyspark, I wonder, what are the remaining advantages (if any) of using scala Spark?

  • 355 Views
  • 1 replies
  • 0 kudos
Latest Reply
delonb2
New Contributor III
  • 0 kudos

The main remaining advantages of Scala are performance as there will always be some interoperation overhead when using PySpark. While I don't have any stats on me, I would assume the differences in performance are negligible at this point until very ...

  • 0 kudos
manish1987c
by New Contributor III
  • 409 Views
  • 2 replies
  • 0 kudos

Delta Live Table - Flow detected an update or delete to one or more rows in the source table

I have create a pipeline where i am ingesting the data from bronze to silver and using SCD 1, however when i am trying to create gold table as dlt it is giving me error as "Flow 'user_silver' has FAILED fatally. An error occurred because we detected ...

manish1987c_0-1718341166099.png manish1987c_1-1718341206991.png
  • 409 Views
  • 2 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@manish1987c -The Streaming does not handle input that is not an append. you can set skipChangeCommits to true 

  • 0 kudos
1 More Replies
MR07
by New Contributor II
  • 288 Views
  • 2 replies
  • 0 kudos

Databricks Managing Materialized Views in Delta Live Tables: Selective Refresh Behavior

Hi Community,I have 200 complex SQL Queries and I can't create a Streaming tables using these queries. So, I have created as Materialized Views in Delta Live Tables using these SQL queries and the DLT pipeline should be run continuously.My question i...

  • 288 Views
  • 2 replies
  • 0 kudos
Latest Reply
steyler-db
New Contributor III
  • 0 kudos

Hello team, thanks for reaching out us, it will be a pleasure to help you on this ask. That's a great catch to run through a materialized, view and regarding the question: If any record of underlying table is inserted, updated or deleted, the only re...

  • 0 kudos
1 More Replies
chevichenk
by New Contributor III
  • 414 Views
  • 3 replies
  • 3 kudos

Resolved! No userid, username, job when making modifications on tables

Hi, everyone!I'm in this situationI have some jobs that makes changes on a particular table. I use only one user to make this modifications, but then there's a process i can't identify that also makes changes on my table.The question is, there's a re...

chevichenk_1-1718308350095.png
Data Engineering
history
jobs
userid
username
  • 414 Views
  • 3 replies
  • 3 kudos
Latest Reply
chevichenk
New Contributor III
  • 3 kudos

Hi, @shan_chandra, @LuisRSanchez,Just found that there are some .jar that are executed and are writting on this table but this .jar is called through batchSo, we think this is the cause Thanks!Ingrid

  • 3 kudos
2 More Replies
Jackson1111
by New Contributor III
  • 498 Views
  • 1 replies
  • 0 kudos

Databricks job cluster logs

Hello, how can I enable Databricks to generate a separate spark log for each job run?What parameters should I use in spark configuration? 

  • 498 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@Jackson1111 - If you are talking about workflow jobs, you can try running using a job cluster to generate spark logs for a each of the workflow jobs.  But, If this is of Spark Jobs within the Spark UI, you wanted to separate out the logs. This is a ...

  • 0 kudos
thiagoawstest
by Contributor
  • 601 Views
  • 6 replies
  • 1 kudos

Resolved! Azure Devops CI/CD - AWS Databricks

Hello, there is documentation for integrating Azure Devops CI/CD pipeline with AWS Databricks.Thanks.

Data Engineering
aws devops
  • 601 Views
  • 6 replies
  • 1 kudos
Latest Reply
jacovangelder
Contributor III
  • 1 kudos

You'll need to install the AWS Toolkit in Azure DevOps, that way you can make a service connection inside your Azure DevOps project that authenticates using an AWS Access Key ID/Secret Access Key (the AWS equivalent to Azure Service Principals). Hope...

  • 1 kudos
5 More Replies
naga_databricks
by Contributor
  • 357 Views
  • 1 replies
  • 0 kudos

Overwriting same table

I have a table A that is used in a spark.sql and joins with multiple other tables to get data. this data will be overwritten to the same table A.When i tried this, i get an error consistently as below: ERROR: An error occurred while calling o382.save...

  • 357 Views
  • 1 replies
  • 0 kudos
Latest Reply
naga_databricks
Contributor
  • 0 kudos

Found this to be a transient error. Once i restarted the cluster, the overwrite was successful. 

  • 0 kudos
Babu_Krishnan
by Contributor
  • 129 Views
  • 1 replies
  • 0 kudos

Why my DLT is not working with UC?

My IAM profile is not working when accessing the SQS for file notification based ingestion?

  • 129 Views
  • 1 replies
  • 0 kudos
Latest Reply
jacovangelder
Contributor III
  • 0 kudos

I'm not sure if I fully understand the question, but what location are you monitoring? Is it a DBFS path or mount? If so, consider using a UC Volume. 

  • 0 kudos
cpayne_vax
by New Contributor III
  • 10031 Views
  • 7 replies
  • 7 kudos

Resolved! Delta Live Tables: dynamic schema

Does anyone know if there's a way to specify an alternate Unity schema in a DLT workflow using the @Dlt.table syntax? In my case, I’m looping through folders in Azure datalake storage to ingest data. I’d like those folders to get created in different...

  • 10031 Views
  • 7 replies
  • 7 kudos
Latest Reply
manish1987c
New Contributor III
  • 7 kudos

Please let me know if there is way by which we can define schema dynamically in 1 delta live table pipeline in same catalog  as below  For example:Catalog1 : schemaA : tableACatalog1 : schemaB : tableB  

  • 7 kudos
6 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels