cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
User16826994223
by • Honored Contributor III
  • 700 Views
  • 1 replies
  • 0 kudos

what is Delta live table? How is it different from the Normal Delta table

what is Delta's live table? How is it different from the Normal Delta table

Data Engineering
Reply
  • 700 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
aladda
Honored Contributor II
  • 0 kudos

Delta Live tables builds on normal delta table by allowing you to easily define end-to-end data pipelines by specifying the data source, the transformation logic, and destination state of the data — instead of manually stitching together siloed data ...

  • 0 kudos
Srikanth_Gupta_
by • Valued Contributor
  • 909 Views
  • 1 replies
  • 0 kudos

How does Spark SQL Catalyst optimizer work?

How does Catalyst optimizer improves the performances, what is its role?

Data Engineering
Reply
  • 909 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Srikanth_Gupta_
Valued Contributor
  • 0 kudos

Catalyst optimizer converts unresolved logical plan into executable physical plan, deep dive is available here

  • 0 kudos
User16826994223
by • Honored Contributor III
  • 1023 Views
  • 1 replies
  • 0 kudos

Unable to start cluster Error :- Defunct Resource Detected

Hi AllI am getting this error for some jobs. Can you please let me know what could be the reasonRun result unavailable: job failed with an error message -Run result unavailable: job failed with error messageUnexpected failure while waiting for the cl...

Data Engineering
Reply
  • 1023 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

This is an issue on the cloud level so try to put retries in the job as it happens not for all cluster start , it may fails once but will start after retry,Also, raise a databricks ticket , they will provide permanent solution

  • 0 kudos
jose_gonzalez
by • Moderator
  • 858 Views
  • 1 replies
  • 0 kudos

How to solve Hive connectivity issues?

I can see connectivity issues in my driver logs. How to solve this issue?

Data Engineering
Reply
  • 858 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

Can you give us some more error please, I hope you will get more error in logs, whether it is a connection issue because of JDbc URL or host name or password,something like this

  • 0 kudos
jose_gonzalez
by • Moderator
  • 5254 Views
  • 1 replies
  • 0 kudos

Resolved! How to get the size of my Delta table

I would like to know how to get the total size of my Delta table

Data Engineering
Reply
  • 5254 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

The following Kb will show a step by step example on how to get the size of a Delta table https://kb.databricks.com/sql/find-size-of-table.html

  • 0 kudos
jose_gonzalez
by • Moderator
  • 15553 Views
  • 1 replies
  • 0 kudos

Resolved! error message rpc response (of 20978566 bytes) exceeds limit of 20971520 bytes

Im getting the following error message when trying to use display()Internal error, sorry. Attach your notebook to a different cluster or restart the current cluster.com.databricks.rpc.RPCResponseTooLarge: rpc response (of 20978566 bytes) exceeds limi...

Data Engineering
Reply
  • 15553 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

It seems like the error is coming from the 20MB output limit. For more information please check this https://docs.databricks.com/jobs.html#output-size-limits

  • 0 kudos
Anonymous
by • Not applicable
  • 1550 Views
  • 1 replies
  • 0 kudos

Resolved! How long does a task have to be in the queue before the cluster autoscales?:

Data Engineering
Reply
  • 1550 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ryan_Chynoweth
Honored Contributor III
  • 0 kudos

There are two types of auto scaling in Databricks: Standard and Optimized. In both scenarios when tasks are submitted the cluster will begin scaling to execute as many of them in parallel immediately.Scaling down is different. In optimized autoscalin...

  • 0 kudos
Anonymous
by • Not applicable
  • 3951 Views
  • 1 replies
  • 0 kudos

Resolved! Can we set up alerts on cluster metrics / job failures in Databricks?

Data Engineering
Reply
  • 3951 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16019159252
New Contributor III
  • 0 kudos

Yes, you can alerts - Email alerts sent in case of job failure, success, or timeout. You can set alerts up for job start, job success, and job failure (including skipped jobs), providing multiple comma-separated email addresses for each alert type. Y...

  • 0 kudos
Anonymous
by • Not applicable
  • 646 Views
  • 0 replies
  • 0 kudos

Using multiple clouds

Are there recommendations and/or examples of leveraging AWS and Azure with Databricks? If so, is there any best practices to follow? Want to ensure we avoid expensive data transfer across clouds

Data Engineering
Reply
  • 646 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels