cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

christo_M
by New Contributor
  • 2237 Views
  • 4 replies
  • 0 kudos

Cost Optimization

How can I optimize the cost on our Databricks platform ? Despite some optimization actions I've taken so far it's still difficult to lower the cost. I tried different technics like Vacuum , or shutting down a cluster running after 30 mins but still d...

  • 2237 Views
  • 4 replies
  • 0 kudos
Latest Reply
erigaud
Honored Contributor
  • 0 kudos

Make sure you're using a cluster that is the right size for your workload. You can greatly reduce the costs by using smaller clusters.

  • 0 kudos
3 More Replies
EcuaCrisCar
by New Contributor III
  • 1047 Views
  • 0 replies
  • 0 kudos

Sending a personalized message to email.

Greetings community, I am new to using databricks and for some time I have tried some scripts in notebook. I would like your help on a task: Carry out a personalized mailing where, First, a query of the number of records in the test table is performe...

Data Engineering
SENDEMAIL SQL
  • 1047 Views
  • 0 replies
  • 0 kudos
krucial_koala
by New Contributor III
  • 4156 Views
  • 5 replies
  • 6 kudos

Extending DevOps Service Principal support?

As per the previous discussion:How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?The recommendation was to create a DevOps PAT for the Service Principal and upload it to Databricks using the Git Credential API. The main f...

AAD auth error
  • 4156 Views
  • 5 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @James Baxter​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 6 kudos
4 More Replies
Fz1
by New Contributor III
  • 1654 Views
  • 0 replies
  • 0 kudos

DLT + Unity Catalogue Issue accessing Dataset not defined in the pipeline

I have 2 different schemas [silver and gold] under the same Unity Catalog.We are trying to incrementally ingest data in both silver and gold layers.The silver tables were created as streaming DLT tables using dlt.create_streaming_table(....) and the ...

Data Engineering
dataset
Dataset not defined in the pipeline
dlt
schema
Unity Catalog
  • 1654 Views
  • 0 replies
  • 0 kudos
Fz1
by New Contributor III
  • 1519 Views
  • 0 replies
  • 0 kudos

DLT with Unity Catalog pipeline not recognising tables from different schemas

I have 2 different schemas [silver and gold] under the same Unity Catalog.We are trying to incrementally ingest data in both silver and gold layers.The silver tables were created as streaming DLT tables using dlt.create_streaming_table(....) and the ...

Data Engineering
dataset
dlt
pipelines
schema
Unity Catalog
  • 1519 Views
  • 0 replies
  • 0 kudos
japan
by New Contributor III
  • 3705 Views
  • 7 replies
  • 11 kudos

Resolved! databricks

what new anounce is most interest for you  in DAIS 2023 ?

  • 3705 Views
  • 7 replies
  • 11 kudos
Latest Reply
BriceBuso
Contributor II
  • 11 kudos

Lakehouse AI, it's bringing lots of possibilities. 

  • 11 kudos
6 More Replies
Hongbo
by New Contributor III
  • 12183 Views
  • 2 replies
  • 4 kudos

Resolved! Delta table with Varchar column vs string column

Databricks support string data type. But I can still create delta table with varchar data type. Just wonder what is different between delta table with string and delta table with varchar:-- delta table with stringCREATE TABLE persons(first_name STRIN...

  • 12183 Views
  • 2 replies
  • 4 kudos
Latest Reply
erigaud
Honored Contributor
  • 4 kudos

VARCHAR allows you to specify the size of the string expected in the column. This is useful when you know your column cannot exceed a set size (ie for a name, a code etc).It is equivalent to a CHECK contraint on the size. Trying to insert a value tha...

  • 4 kudos
1 More Replies
Hongbo
by New Contributor III
  • 2751 Views
  • 3 replies
  • 4 kudos

Resolved! ParseException: mismatched input 'PRIMARY'

I got parseException issue and I have no idea what cause the problem:ParseException: mismatched input 'PRIMARY' expecting {')', ',', 'CONSTRAINT'}(line 2, pos 44)== SQL == CREATE TABLE persons(first_name STRING NOT NULL, last_name STRING NOT NULL, ni...

  • 2751 Views
  • 3 replies
  • 4 kudos
Latest Reply
erigaud
Honored Contributor
  • 4 kudos

You can only declare Primary and foreign keys on fields on Unity Catalog. You also need Databricks Runtime 11.1+Otherwise only NOT NULL and CHECK constraints are supported.More info : https://docs.databricks.com/tables/constraints.html

  • 4 kudos
2 More Replies
chorongs
by New Contributor III
  • 4917 Views
  • 3 replies
  • 6 kudos

Resolved! ON VIOLATION DROP ROW Question!

What exactly does "ON VIOLATION DROP ROW" do?

  • 4917 Views
  • 3 replies
  • 6 kudos
Latest Reply
pvignesh92
Honored Contributor
  • 6 kudos

Hi @chorongs , There is a feature available in Delta Live tables where you can check the quality of your dataset at the time of your load to target table and take action like dropping those invalid records that did not meet your conditions, making yo...

  • 6 kudos
2 More Replies
Chaitanya_Raju
by Honored Contributor
  • 3445 Views
  • 4 replies
  • 4 kudos

How to create a new group in the databricks community?

Can a member create a new group in the databricks community, if yes can anyone let me know the steps, or where can I able to create a new group?

  • 3445 Views
  • 4 replies
  • 4 kudos
Latest Reply
OlivierAllovon
New Contributor II
  • 4 kudos

Any updates to this topic ? I'd like to create a group for my company as well.

  • 4 kudos
3 More Replies
Ryan_Chynoweth
by Esteemed Contributor
  • 2322 Views
  • 2 replies
  • 2 kudos

medium.com

Hi All, I recently published a streaming data comparison between Snowflake and Databricks. Hope you enjoy! Please let me know what you think! https://medium.com/@24chynoweth/data-streaming-at-scale-databricks-and-snowflake-ca65a2401649

  • 2322 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Nicely done. 

  • 2 kudos
1 More Replies
THIAM_HUATTAN
by Valued Contributor
  • 649 Views
  • 0 replies
  • 0 kudos

From Visualization to Dashboard -- months order is gone

On my notebook for Visualization, it is perfect which the months are sorted correctly, as it is ordered by month_numeric. However, when I bring over to the Dashboard, its month is not sorted in the order which I need (see below). Where do I miss some...

THIAM_HUATTAN_0-1688298649642.png THIAM_HUATTAN_1-1688298929009.png
  • 649 Views
  • 0 replies
  • 0 kudos
Jesus274
by New Contributor
  • 5947 Views
  • 4 replies
  • 0 kudos

Fundamentals of the Databricks Lakehouse Platform Accreditation (V2)

Hi,After 48 hours after passing the exam, I didn´t receive any certification. My ID: E-03DK31. This is the prevous ticket Ticket URL: https://help.databricks.com/s/case/5008Y00002AAdImQAL

  • 5947 Views
  • 4 replies
  • 0 kudos
Latest Reply
JHB_bcp
New Contributor II
  • 0 kudos

I have the same issue, ID: E-03DK31Could you help me please. 

  • 0 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels