cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Mbinyala
by • New Contributor II
  • 22608 Views
  • 2 replies
  • 1 kudos

Connecting confluent to databricks.

Hi!!Can someone tell me how to connect the confluent cloud to Databricks? I am new to this so please elaborate on your answer.

Data Engineering
Reply
  • 22608 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
VaibB
Contributor
  • 1 kudos

You might want to watch this as well https://www.confluent.io/resources/online-talk/innovate-faster-and-easier-with-confluent-and-databricks-on-azure/?utm_medium=sem&utm_source=google&utm_campaign=ch.sem_br.nonbrand_tp.prs_tgt.dsa_mt.dsa_rgn.india_ln...

  • 1 kudos
1 More Replies
Gim
by • Contributor
  • 2649 Views
  • 1 replies
  • 3 kudos

Columns with DEFAULT missing error during INSERT

I am really confused about the DEFAULT capability of Databricks SQL. I looked at the documentation for the minimum required DBR to get the capability yet we still need to enable it as a table property? I updated my cluster's DBR from 12.2 to 13.1.Any...

Gim_0-1688465259125.png
Data Engineering
Reply
  • 2649 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
BriceBuso
Contributor II
  • 3 kudos

Hello @Gim, Got the same problem. Tried with the instruction "GENERATED ALWAYS AS (CAST(CURRENT_DATE() AS DATE))" but code is returning "Error in SQL statement: DeltaAnalysisException: current_date() cannot be used in a generated column" If you find ...

  • 3 kudos
erigaud
by • Honored Contributor
  • 3532 Views
  • 2 replies
  • 1 kudos

Incrementally load SQL Server table

I am accessing an on premise SQL Server table. The table is relatively small (10 000 rows), and I access it usingspark.read.jdbc(url=jdbcUrl, table = query)Every day there are new records in the on prem table that I would like to append in my bronze ...

Data Engineering
Reply
  • 3532 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
erigaud
Honored Contributor
  • 1 kudos

As I said, there is no unique identifier in the table that would allow me to do any sort of Join between my source table and my bronze table. 

  • 1 kudos
1 More Replies
JLL
by • New Contributor II
  • 1003 Views
  • 1 replies
  • 2 kudos

Shorten query run time

Challenges in query long run time; what are the recommended steps to improve performance 

Data Engineering
Reply
  • 1003 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
erigaud
Honored Contributor
  • 2 kudos

The question needs more precision : is it the cluster startup that takes a while ? If yes, try serverless warehousesAre there many queries running in parallel and that is where you see a slow down ? Each cluster can only run 10 queries in parallel, s...

  • 2 kudos
christo_M
by • New Contributor
  • 2271 Views
  • 4 replies
  • 0 kudos

Cost Optimization

How can I optimize the cost on our Databricks platform ? Despite some optimization actions I've taken so far it's still difficult to lower the cost. I tried different technics like Vacuum , or shutting down a cluster running after 30 mins but still d...

Data Engineering
Reply
  • 2271 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
erigaud
Honored Contributor
  • 0 kudos

Make sure you're using a cluster that is the right size for your workload. You can greatly reduce the costs by using smaller clusters.

  • 0 kudos
3 More Replies
EcuaCrisCar
by • New Contributor III
  • 1077 Views
  • 0 replies
  • 0 kudos

Sending a personalized message to email.

Greetings community, I am new to using databricks and for some time I have tried some scripts in notebook. I would like your help on a task: Carry out a personalized mailing where, First, a query of the number of records in the test table is performe...

Data Engineering
SENDEMAIL SQL
Reply
  • 1077 Views
  • 0 replies
  • 0 kudos
krucial_koala
by • New Contributor III
  • 4243 Views
  • 5 replies
  • 6 kudos

Extending DevOps Service Principal support?

As per the previous discussion:How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?The recommendation was to create a DevOps PAT for the Service Principal and upload it to Databricks using the Git Credential API. The main f...

AAD auth error
Data Engineering
Reply
  • 4243 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @James Baxter​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 6 kudos
4 More Replies
Fz1
by • New Contributor III
  • 1689 Views
  • 0 replies
  • 0 kudos

DLT + Unity Catalogue Issue accessing Dataset not defined in the pipeline

I have 2 different schemas [silver and gold] under the same Unity Catalog.We are trying to incrementally ingest data in both silver and gold layers.The silver tables were created as streaming DLT tables using dlt.create_streaming_table(....) and the ...

Data Engineering
dataset
Dataset not defined in the pipeline
dlt
schema
Unity Catalog
Reply
  • 1689 Views
  • 0 replies
  • 0 kudos
Fz1
by • New Contributor III
  • 1543 Views
  • 0 replies
  • 0 kudos

DLT with Unity Catalog pipeline not recognising tables from different schemas

I have 2 different schemas [silver and gold] under the same Unity Catalog.We are trying to incrementally ingest data in both silver and gold layers.The silver tables were created as streaming DLT tables using dlt.create_streaming_table(....) and the ...

Data Engineering
dataset
dlt
pipelines
schema
Unity Catalog
Reply
  • 1543 Views
  • 0 replies
  • 0 kudos
japan
by • New Contributor III
  • 3762 Views
  • 7 replies
  • 11 kudos

Resolved! databricks

what new anounce is most interest for you  in DAIS 2023 ？

Data Engineering
Reply
  • 3762 Views
  • 7 replies
  • 11 kudos
View Replies
Latest Reply
BriceBuso
Contributor II
  • 11 kudos

Lakehouse AI, it's bringing lots of possibilities. 

  • 11 kudos
6 More Replies
Hongbo
by • New Contributor III
  • 12583 Views
  • 2 replies
  • 4 kudos

Resolved! Delta table with Varchar column vs string column

Databricks support string data type. But I can still create delta table with varchar data type. Just wonder what is different between delta table with string and delta table with varchar:-- delta table with stringCREATE TABLE persons(first_name STRIN...

Data Engineering
Reply
  • 12583 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
erigaud
Honored Contributor
  • 4 kudos

VARCHAR allows you to specify the size of the string expected in the column. This is useful when you know your column cannot exceed a set size (ie for a name, a code etc).It is equivalent to a CHECK contraint on the size. Trying to insert a value tha...

  • 4 kudos
1 More Replies
Hongbo
by • New Contributor III
  • 2825 Views
  • 3 replies
  • 4 kudos

Resolved! ParseException: mismatched input 'PRIMARY'

I got parseException issue and I have no idea what cause the problem:ParseException: mismatched input 'PRIMARY' expecting {')', ',', 'CONSTRAINT'}(line 2, pos 44)== SQL == CREATE TABLE persons(first_name STRING NOT NULL, last_name STRING NOT NULL, ni...

Data Engineering
Reply
  • 2825 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
erigaud
Honored Contributor
  • 4 kudos

You can only declare Primary and foreign keys on fields on Unity Catalog. You also need Databricks Runtime 11.1+Otherwise only NOT NULL and CHECK constraints are supported.More info : https://docs.databricks.com/tables/constraints.html

  • 4 kudos
2 More Replies
chorongs
by • New Contributor III
  • 5053 Views
  • 3 replies
  • 6 kudos

Resolved! ON VIOLATION DROP ROW Question!

What exactly does "ON VIOLATION DROP ROW" do?

Data Engineering
Reply
  • 5053 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
pvignesh92
Honored Contributor
  • 6 kudos

Hi @chorongs , There is a feature available in Delta Live tables where you can check the quality of your dataset at the time of your load to target table and take action like dropping those invalid records that did not meet your conditions, making yo...

  • 6 kudos
2 More Replies
Chaitanya_Raju
by • Honored Contributor
  • 3485 Views
  • 4 replies
  • 4 kudos

How to create a new group in the databricks community?

Can a member create a new group in the databricks community, if yes can anyone let me know the steps, or where can I able to create a new group?

Data Engineering
Reply
  • 3485 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
OlivierAllovon
New Contributor II
  • 4 kudos

Any updates to this topic ? I'd like to create a group for my company as well.

  • 4 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels