cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

JLL
by New Contributor II
  • 748 Views
  • 1 replies
  • 2 kudos

Shorten query run time

Challenges in query long run time; what are the recommended steps to improve performance 

  • 748 Views
  • 1 replies
  • 2 kudos
Latest Reply
erigaud
Honored Contributor
  • 2 kudos

The question needs more precision : is it the cluster startup that takes a while ? If yes, try serverless warehousesAre there many queries running in parallel and that is where you see a slow down ? Each cluster can only run 10 queries in parallel, s...

  • 2 kudos
christo_M
by New Contributor
  • 1729 Views
  • 4 replies
  • 0 kudos

Cost Optimization

How can I optimize the cost on our Databricks platform ? Despite some optimization actions I've taken so far it's still difficult to lower the cost. I tried different technics like Vacuum , or shutting down a cluster running after 30 mins but still d...

  • 1729 Views
  • 4 replies
  • 0 kudos
Latest Reply
erigaud
Honored Contributor
  • 0 kudos

Make sure you're using a cluster that is the right size for your workload. You can greatly reduce the costs by using smaller clusters.

  • 0 kudos
3 More Replies
krucial_koala
by New Contributor III
  • 3325 Views
  • 5 replies
  • 6 kudos

Extending DevOps Service Principal support?

As per the previous discussion:How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?The recommendation was to create a DevOps PAT for the Service Principal and upload it to Databricks using the Git Credential API. The main f...

AAD auth error
  • 3325 Views
  • 5 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @James Baxter​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 6 kudos
4 More Replies
Fz1
by New Contributor III
  • 1298 Views
  • 0 replies
  • 0 kudos

DLT + Unity Catalogue Issue accessing Dataset not defined in the pipeline

I have 2 different schemas [silver and gold] under the same Unity Catalog.We are trying to incrementally ingest data in both silver and gold layers.The silver tables were created as streaming DLT tables using dlt.create_streaming_table(....) and the ...

Data Engineering
dataset
Dataset not defined in the pipeline
dlt
schema
Unity Catalog
  • 1298 Views
  • 0 replies
  • 0 kudos
Fz1
by New Contributor III
  • 1265 Views
  • 0 replies
  • 0 kudos

DLT with Unity Catalog pipeline not recognising tables from different schemas

I have 2 different schemas [silver and gold] under the same Unity Catalog.We are trying to incrementally ingest data in both silver and gold layers.The silver tables were created as streaming DLT tables using dlt.create_streaming_table(....) and the ...

Data Engineering
dataset
dlt
pipelines
schema
Unity Catalog
  • 1265 Views
  • 0 replies
  • 0 kudos
japan
by New Contributor III
  • 2827 Views
  • 7 replies
  • 11 kudos

Resolved! databricks

what new anounce is most interest for you  in DAIS 2023 ?

  • 2827 Views
  • 7 replies
  • 11 kudos
Latest Reply
BriceBuso
Contributor II
  • 11 kudos

Lakehouse AI, it's bringing lots of possibilities. 

  • 11 kudos
6 More Replies
Hongbo
by New Contributor III
  • 9227 Views
  • 2 replies
  • 4 kudos

Resolved! Delta table with Varchar column vs string column

Databricks support string data type. But I can still create delta table with varchar data type. Just wonder what is different between delta table with string and delta table with varchar:-- delta table with stringCREATE TABLE persons(first_name STRIN...

  • 9227 Views
  • 2 replies
  • 4 kudos
Latest Reply
erigaud
Honored Contributor
  • 4 kudos

VARCHAR allows you to specify the size of the string expected in the column. This is useful when you know your column cannot exceed a set size (ie for a name, a code etc).It is equivalent to a CHECK contraint on the size. Trying to insert a value tha...

  • 4 kudos
1 More Replies
Hongbo
by New Contributor III
  • 1944 Views
  • 3 replies
  • 4 kudos

Resolved! ParseException: mismatched input 'PRIMARY'

I got parseException issue and I have no idea what cause the problem:ParseException: mismatched input 'PRIMARY' expecting {')', ',', 'CONSTRAINT'}(line 2, pos 44)== SQL == CREATE TABLE persons(first_name STRING NOT NULL, last_name STRING NOT NULL, ni...

  • 1944 Views
  • 3 replies
  • 4 kudos
Latest Reply
erigaud
Honored Contributor
  • 4 kudos

You can only declare Primary and foreign keys on fields on Unity Catalog. You also need Databricks Runtime 11.1+Otherwise only NOT NULL and CHECK constraints are supported.More info : https://docs.databricks.com/tables/constraints.html

  • 4 kudos
2 More Replies
chorongs
by New Contributor III
  • 3504 Views
  • 3 replies
  • 6 kudos

Resolved! ON VIOLATION DROP ROW Question!

What exactly does "ON VIOLATION DROP ROW" do?

  • 3504 Views
  • 3 replies
  • 6 kudos
Latest Reply
pvignesh92
Honored Contributor
  • 6 kudos

Hi @chorongs , There is a feature available in Delta Live tables where you can check the quality of your dataset at the time of your load to target table and take action like dropping those invalid records that did not meet your conditions, making yo...

  • 6 kudos
2 More Replies
Chaitanya_Raju
by Honored Contributor
  • 2791 Views
  • 4 replies
  • 4 kudos

How to create a new group in the databricks community?

Can a member create a new group in the databricks community, if yes can anyone let me know the steps, or where can I able to create a new group?

  • 2791 Views
  • 4 replies
  • 4 kudos
Latest Reply
OlivierAllovon
New Contributor II
  • 4 kudos

Any updates to this topic ? I'd like to create a group for my company as well.

  • 4 kudos
3 More Replies
Ryan_Chynoweth
by Honored Contributor III
  • 1931 Views
  • 2 replies
  • 2 kudos

medium.com

Hi All, I recently published a streaming data comparison between Snowflake and Databricks. Hope you enjoy! Please let me know what you think! https://medium.com/@24chynoweth/data-streaming-at-scale-databricks-and-snowflake-ca65a2401649

  • 1931 Views
  • 2 replies
  • 2 kudos
Latest Reply
babyhari
New Contributor II
  • 2 kudos

Nicely done. 

  • 2 kudos
1 More Replies
THIAM_HUATTAN
by Valued Contributor
  • 512 Views
  • 0 replies
  • 0 kudos

From Visualization to Dashboard -- months order is gone

On my notebook for Visualization, it is perfect which the months are sorted correctly, as it is ordered by month_numeric. However, when I bring over to the Dashboard, its month is not sorted in the order which I need (see below). Where do I miss some...

THIAM_HUATTAN_0-1688298649642.png THIAM_HUATTAN_1-1688298929009.png
  • 512 Views
  • 0 replies
  • 0 kudos
Jesus274
by New Contributor
  • 5312 Views
  • 4 replies
  • 0 kudos

Fundamentals of the Databricks Lakehouse Platform Accreditation (V2)

Hi,After 48 hours after passing the exam, I didn´t receive any certification. My ID: E-03DK31. This is the prevous ticket Ticket URL: https://help.databricks.com/s/case/5008Y00002AAdImQAL

  • 5312 Views
  • 4 replies
  • 0 kudos
Latest Reply
JHB_bcp
New Contributor II
  • 0 kudos

I have the same issue, ID: E-03DK31Could you help me please. 

  • 0 kudos
3 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels