cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Skully
by • New Contributor
  • 131 Views
  • 1 replies
  • 0 kudos

Workflow Fail safe query

I have a large SQL query that includes multiple Common Table Expressions (CTEs) and joins across various tables, totaling approximately 2,500 lines. I want to ensure that if any part of the query or a specific CTE fails—due to a missing table or colu...

Data Engineering
Reply
  • 131 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
LingeshK
Databricks Employee
  • 0 kudos

There are few options you can try. Based of the information shared, I am assuming a skeleton for you complicated query as follows: WITH cte_one AS (SELECT *FROM view_one),-- Other CTEs...-- Your main query logicSELECTFROM cte_one-- Joins and other cl...

  • 0 kudos
Krizofe
by • New Contributor II
  • 2184 Views
  • 6 replies
  • 3 kudos

Resolved! Migrating data from synapse to databricks

Hello team,I have a requirement of moving all the table from Azure Synapse (dedicated sql pool) to databricks.we have a data coming up from source to azure data lake frequently.we have Azure data factory to load data (data flow does the basic transfo...

Data Engineering
Reply
  • 2184 Views
  • 6 replies
  • 3 kudos
View Replies
Latest Reply
thelogicplus
Visitor
  • 3 kudos

Hi @Krizofe , Just gone through you deatils and thought our similar experience  with  Azure Synapse to databrick migration. We faced a similar situation and were initially hesitant, One of the my colleague recommanded to use Travinto Technologies acc...

  • 3 kudos
5 More Replies
Vetrivel
by • New Contributor III
  • 412 Views
  • 2 replies
  • 1 kudos

Resolved! SSIS packages migration to Databricks Workflows

We are doing POC to migrate SSIS packages to Databricks workflows as part of our effort to build the analytics layer, including dimension and fact tables. How can we accelerate or automate the SSIS package migration to Databricks environment?

Data Engineering
Reply
  • 412 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
thelogicplus
Visitor
  • 1 kudos

Hi  Vetrivel,There are many company  which have accelator , who can help you to migrate ssis to databricks, check with travinto.com.   we are using their accelator with services from travinto and migrated  200+ till today 24-Nov-2024. These guys are ...

  • 1 kudos
1 More Replies
EDDatabricks
by • Contributor
  • 1579 Views
  • 1 replies
  • 1 kudos

Multiple DLT pipelines same target table

Is it possible to have multiple DLT pipelines write data concurrently and in append mode to the same Delta table? Because of different data sources, with different data volumes and required processing, we would like to have different pipelines stream...

Data Engineering
Delta tables
DLT pipeline
Reply
  • 1579 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
claudiayuan
Visitor
  • 1 kudos

hello! did you get the answer?

  • 1 kudos
Jefke
by • New Contributor II
  • 69 Views
  • 4 replies
  • 0 kudos

Cloud_files function

Hi I'm fairly new to to Databricks and in some examples, blogs,... I see the cloud_files() function being used. But I'm always unable to find any documentation on it? Is there any reason for this? And what is the exact use case for the function? Most...

Data Engineering
Reply
  • 69 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
JakubSkibicki
New Contributor II
  • 0 kudos

Please see if it helps https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/cloud_files_state

  • 0 kudos
3 More Replies
somedeveloper
by • New Contributor
  • 73 Views
  • 3 replies
  • 0 kudos

Databricks Setting Dynamic Local Configuration Properties

It seems that Databricks is somehow setting the properties of local spark configurations for each notebook. Can someone point me to exactly how and where this is being done? I would like to set the scheduler to utilize a certain pool by default, but ...

Data Engineering
Reply
  • 73 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
BigRoux
Databricks Employee
  • 0 kudos

You will need to leverage cluster-level Spark configurations or global init scripts.  This will allow you to set "spark.scheduler.poo" property automatically for all workloads on the cluster. You can try navigationg to "Compute", select the cluster y...

  • 0 kudos
2 More Replies
oakhill
by • New Contributor III
  • 109 Views
  • 8 replies
  • 1 kudos

Is Delta Live Tables not supported anymore? How do I use it in Python?

Hi!Any time I try to import "dlt" in a notebook session to develop Pipelines, I get an error message saying DLT is not supported on Spark Connect clusters. These are very generic clusters, I've tried runtime 14, 15 and the latest 16, using shared clu...

Data Engineering
Reply
  • 109 Views
  • 8 replies
  • 1 kudos
View Replies
Latest Reply
BigRoux
Databricks Employee
  • 1 kudos

Oakhill, we do provide free onboard training. You might be interested in the "Get Started with Data Engineering on Databricks" session.  You can register here: https://www.databricks.com/training/catalog.  When you are searching the catalog of traini...

  • 1 kudos
7 More Replies
temucodes2024
by • New Contributor
  • 61 Views
  • 1 replies
  • 0 kudos

Temu code for exiting users and app installs

Temu code for exiting users and app installsWhat is the Temu code for existing users and app installs?The Temu code for existing users and app installs is acu570611 code in 2024.

Data Engineering
Reply
  • 61 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
temucoupons2024
New Contributor
  • 0 kudos

Temu: The Future of Online Shopping in 2024The online shopping world is evolving rapidly, and Temu has emerged as a standout platform in 2024. Known for its ease of use, affordability, and variety, it’s transforming how people shop from the comfort o...

  • 0 kudos
temucodes2024
by • New Contributor
  • 75 Views
  • 1 replies
  • 0 kudos

frg257368 Temu free gift code

What is the Temu free gift code?The Temu free gift code is frg257368 code.

Data Engineering
Reply
  • 75 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
temucoupons2024
New Contributor
  • 0 kudos

What Makes Temu Unique in 2024?While many e-commerce platforms prioritize volume over quality, Temu strikes a balance. The platform is dedicated to offering high-quality products without the premium price tag. Customer reviews and ratings help shoppe...

  • 0 kudos
Sega2
by • New Contributor III
  • 446 Views
  • 1 replies
  • 0 kudos

cannot import name 'Buffer' from 'typing_extensions' (/databricks/python/lib/python3.10/site-package

I am trying to add messages to an azure service bus from a notebook. But I get error from title. Any suggestions how to solve this?import asynciofrom azure.servicebus.aio import ServiceBusClientfrom azure.servicebus import ServiceBusMessagefrom azure...

Data Engineering
Reply
  • 446 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

@Sega2 it sounds like the error occurs because the typing_extensions library version in your Databricks environment is outdated and does not include the Buffer class, which is being imported by one of the Azure libraries. Can you first try: %pip inst...

  • 0 kudos
kalebkemp
by • New Contributor
  • 166 Views
  • 1 replies
  • 0 kudos

FileReadException error when creating materialized view reading two schemas

Hi all. I'm getting an error `com.databricks.sql.io.FileReadException` when attempting to create a materialized view which reads tables from two different schemas in the same catalog. Is this just a limitation in databricks or do I potentially have s...

Data Engineering
Reply
  • 166 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

@kalebkemp  Can you please check if this is not an access issue?: SHOW GRANTS ON SCHEMA my_catalog.my_other_schema; Also test if you can successfully run a query that access data from both schemas: SELECT * FROM my_catalog.my_schema.some_table JOIN m...

  • 0 kudos
zsh24
by • New Contributor
  • 457 Views
  • 1 replies
  • 0 kudos

Python worker exited unexpectedly (crashed)

I have a failing pipeline which results in the following failure:org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2053.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2053.0 (TID 4594) (10.171.199.129 e...

Data Engineering
Reply
  • 457 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

@zsh24 what you're seeing is a wrapper exception, the underlying and true exception, if not in the stdout log, you'll find it in the Python side. To understand what is failing in the python worker, not the executor jvm, you should analyze the code ex...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels