cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Mlaricobar
by • New Contributor
  • 486 Views
  • 0 replies
  • 0 kudos

Spark Optimization strategies

How can we get the correct size for job cluster in a job workflow pipeline based in the complexity of the process? #databricks #deltalake

Data Engineering
Reply
  • 486 Views
  • 0 replies
  • 0 kudos
Maciej
by • New Contributor II
  • 456 Views
  • 0 replies
  • 0 kudos

CDF background implementation

How Delta Lake CDF works? I seen it add additional column to data, where data was updated or deleted. So what is the purpose of change log?

Data Engineering
Reply
  • 456 Views
  • 0 replies
  • 0 kudos
tingtingchan
by • New Contributor
  • 522 Views
  • 0 replies
  • 0 kudos

Greetings for My 1st Data+AI Summit 2023!

Kudos to the amazing instructors and TAs for my first in-person Data Engineer Associate Training, and I've passed my exam! Having a fantastic time so far, can't wait for the content unfolded in the next two days!

Data Engineering
Reply
  • 522 Views
  • 0 replies
  • 0 kudos
Orianh
by • Valued Contributor II
  • 5579 Views
  • 4 replies
  • 3 kudos

function does not exist in JVM ERROR

Hello guys, I'm building a python package that return 1 row from DF at a time inside data bricks environment.To improve the performance of this package i used multiprocessing library in python, I have background process that his whole purpose is to p...

function dont exist in JVM error.
Data Engineering
Reply
  • 5579 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
dineshreddy
New Contributor III
  • 3 kudos

Using thread instead of processes solved the issue for me

  • 3 kudos
3 More Replies
Anonymous
by • Not applicable
  • 1537 Views
  • 2 replies
  • 1 kudos

Delta Tables copying

Hello, I’m trying to copy a table with all it’s versions to unity catalog, I know I can use deep cloning but I want the table with the full history, is that possible?

Data Engineering
Summit23
Reply
  • 1537 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
bikash84
New Contributor III
  • 1 kudos

To copy history, you would have to copy files along with the delta log folder and then create a delta table on that location

  • 1 kudos
1 More Replies
Kunda
by • New Contributor III
  • 893 Views
  • 1 replies
  • 0 kudos

Resolved! Welcome

Welcome!

Data Engineering
Reply
  • 893 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kunda
New Contributor III
  • 0 kudos

Welcome!

  • 0 kudos
Anonymous
by • Not applicable
  • 1056 Views
  • 2 replies
  • 0 kudos

databricks view question!

I found this phrase in the document "A view stores the text for a query type again one or more data sources or tables in the metastore."Does "view" in databricks store data in a physical location?

Data Engineering
Reply
  • 1056 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

CREATE VIEW | Databricks on AWS - Constructs a virtual table that has no physical data based on the result-set of a SQL query.

  • 0 kudos
1 More Replies
PraveenKarnam
by • New Contributor II
  • 580 Views
  • 0 replies
  • 0 kudos

Set up RBACs with hive catalog

Hello, we are not on unity catalog yet due to limitations on multi cloud implementation of UC. We still want to implement Role Based Acess Control with hive metastore. We are using DBR 11.3. Any pointers will be helpful 

Data Engineering
Reply
  • 580 Views
  • 0 replies
  • 0 kudos
Serhii
by • Contributor
  • 1776 Views
  • 3 replies
  • 1 kudos

Could not launch jobs due to node_type_id (instance) unavailability

I am running hourly job on a cluster using p3.2xlarge GPU instance, but sometimes cluster couldn't start due to instance unavailability. I wander is there is any fallback mechanism to, for example, try a different instance type if one is not availabl...

Data Engineering
Reply
  • 1776 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
abagshaw
New Contributor III
  • 1 kudos

 (AWS only) For anyone experiencing capacity related cluster launch failures on non-GPU instance types, AWS Fleet instance types are now GA and available for clusters and instance pools. They help improve chance of successful cluster launch by allowi...

  • 1 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels