cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Maciej
by • New Contributor II
  • 705 Views
  • 0 replies
  • 0 kudos

CDF background implementation

How Delta Lake CDF works? I seen it add additional column to data, where data was updated or deleted. So what is the purpose of change log?

Data Engineering
Reply
  • 705 Views
  • 0 replies
  • 0 kudos
tingtingchan
by • New Contributor
  • 714 Views
  • 0 replies
  • 0 kudos

Greetings for My 1st Data+AI Summit 2023!

Kudos to the amazing instructors and TAs for my first in-person Data Engineer Associate Training, and I've passed my exam! Having a fantastic time so far, can't wait for the content unfolded in the next two days!

Data Engineering
Reply
  • 714 Views
  • 0 replies
  • 0 kudos
Orianh
by • Valued Contributor II
  • 9787 Views
  • 4 replies
  • 3 kudos

function does not exist in JVM ERROR

Hello guys, I'm building a python package that return 1 row from DF at a time inside data bricks environment.To improve the performance of this package i used multiprocessing library in python, I have background process that his whole purpose is to p...

function dont exist in JVM error.
Data Engineering
Reply
  • 9787 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
dineshreddy
New Contributor III
  • 3 kudos

Using thread instead of processes solved the issue for me

  • 3 kudos
3 More Replies
Anonymous
by • Not applicable
  • 2742 Views
  • 2 replies
  • 1 kudos

Delta Tables copying

Hello, I’m trying to copy a table with all it’s versions to unity catalog, I know I can use deep cloning but I want the table with the full history, is that possible?

Data Engineering
Summit23
Reply
  • 2742 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
bikash84
New Contributor III
  • 1 kudos

To copy history, you would have to copy files along with the delta log folder and then create a delta table on that location

  • 1 kudos
1 More Replies
Kunda
by • New Contributor III
  • 1277 Views
  • 1 replies
  • 0 kudos

Resolved! Welcome

Welcome!

Data Engineering
Reply
  • 1277 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kunda
New Contributor III
  • 0 kudos

Welcome!

  • 0 kudos
Anonymous
by • Not applicable
  • 5207 Views
  • 2 replies
  • 0 kudos

databricks view question!

I found this phrase in the document "A view stores the text for a query type again one or more data sources or tables in the metastore."Does "view" in databricks store data in a physical location?

Data Engineering
Reply
  • 5207 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

CREATE VIEW | Databricks on AWS - Constructs a virtual table that has no physical data based on the result-set of a SQL query.

  • 0 kudos
1 More Replies
PraveenKarnam
by • New Contributor II
  • 885 Views
  • 0 replies
  • 0 kudos

Set up RBACs with hive catalog

Hello, we are not on unity catalog yet due to limitations on multi cloud implementation of UC. We still want to implement Role Based Acess Control with hive metastore. We are using DBR 11.3. Any pointers will be helpful 

Data Engineering
Reply
  • 885 Views
  • 0 replies
  • 0 kudos
Serhii
by • Contributor
  • 2686 Views
  • 3 replies
  • 1 kudos

Could not launch jobs due to node_type_id (instance) unavailability

I am running hourly job on a cluster using p3.2xlarge GPU instance, but sometimes cluster couldn't start due to instance unavailability. I wander is there is any fallback mechanism to, for example, try a different instance type if one is not availabl...

Data Engineering
Reply
  • 2686 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
abagshaw
Databricks Employee
  • 1 kudos

 (AWS only) For anyone experiencing capacity related cluster launch failures on non-GPU instance types, AWS Fleet instance types are now GA and available for clusters and instance pools. They help improve chance of successful cluster launch by allowi...

  • 1 kudos
2 More Replies
Anonymous
by • Not applicable
  • 1672 Views
  • 1 replies
  • 0 kudos

Instance type in Photon

Can Photon run on all instance/VM types?

Data Engineering
Reply
  • 1672 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
abagshaw
Databricks Employee
  • 0 kudos

No, Photon is only supported on a limited set of instance types where it's been benchmarked and tested by Databricks to have optimal performance.

  • 0 kudos
JPKC
by • New Contributor
  • 2715 Views
  • 3 replies
  • 1 kudos

Support for multiple EC2 instance types in a worker pool

As per this thread Databricks now integrates with EC2 CreateFleet API that allows customers to create Databricks pools and get EC2 instances from multiple AZs and multiple instance families & sizes. However, in the Databricks UI you can not select mo...

Data Engineering
Reply
  • 2715 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
abagshaw
Databricks Employee
  • 1 kudos

Fleet instances on Databricks is now GA and available in all AWS workspaces - you can find more details here: https://docs.databricks.com/compute/aws-fleet-instances.html

  • 1 kudos
2 More Replies
umair_hanif
by • New Contributor II
  • 3177 Views
  • 2 replies
  • 1 kudos

Ingesting more than 7 million rows into a SQL Server Table

Hi All, I hope you're super well. I need your recommendations and solution for my problem.I am using a Databricks instance DS12_v2 which has 28GB RAM and 4 cores. I am ingesting 7.2 million rows into a SQL Server table and it is taking 57 min - 1 hou...

Data Engineering
Reply
  • 3177 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

You can try to use BULK INSERT.https://learn.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?view=sql-server-ver16Also using Data Factory instead of Databricks for the copy can be helpful.

  • 1 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels