cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
01_binary
by • New Contributor III
  • 1055 Views
  • 1 replies
  • 1 kudos

Resolved! How to improve the performance of small delta tables?

How to improve the performance of small delta tables?

Data Engineering
Reply
  • 1055 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
NM
New Contributor III
  • 1 kudos

Use databricks optimize command on delta tables. It will regroup al the files and provides better performance​

  • 1 kudos
Shomari
by • New Contributor
  • 2299 Views
  • 1 replies
  • 2 kudos

Resolved! Workflow dependencies

Is it possible to make one workflow job dependent on successful completion of another job?​

Data Engineering
Reply
  • 2299 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Tony_N
New Contributor III
  • 2 kudos

I believe you can set workflow dependencies between other workflows.

  • 2 kudos
zLiu
by • New Contributor II
  • 908 Views
  • 0 replies
  • 1 kudos

Project lightspeed

It’s just a breeze for all the streaming users. What’s the best venue to learn more about it. Is there a Jira ticket that tracks all the progresses? also wonder which Spark version it will come with.

Data Engineering
Reply
  • 908 Views
  • 0 replies
  • 1 kudos
VictorP
by • New Contributor
  • 1783 Views
  • 1 replies
  • 3 kudos

Resolved! Does databricks run on GPU?

Does databricks run on GPU?

Data Engineering
Reply
  • 1783 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
ron_defreitas
Contributor
  • 3 kudos

There is support for running on GPU which will be beneficial to certain ML workloads.​Cluster​s are configured to run on CPU by default, but you can choose GPU based nodes during creation.

  • 3 kudos
Di
by • New Contributor
  • 1198 Views
  • 1 replies
  • 2 kudos

Resolved! Project Lightspeed

Is Spark Structured Streaming now comparable with Flink on streaming workloads?

Data Engineering
Reply
  • 1198 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
ron_defreitas
Contributor
  • 2 kudos

Hard to say. Project Lightspeed ​is a work in progress and has not yet been released.

  • 2 kudos
ShuImamura
by • New Contributor II
  • 2115 Views
  • 2 replies
  • 1 kudos

Resolved! How to use multi character encodings in Delta Tables?

Can we mix different encodings in Delta tables? Downstream needs different character encodings like UTF-8 and Shift JIS.

Data Engineering
Reply
  • 2115 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
ShuImamura
New Contributor II
  • 1 kudos

@Werner Stinckens​ thanks for answering!

  • 1 kudos
1 More Replies
bgarcia
by • New Contributor III
  • 997 Views
  • 1 replies
  • 0 kudos

Resolved! Delta Tables

I’m beginning my journey into Delta Tables and one thing that is still confusing me is where is the best place to save your delta tables if you need to query them later.For example I'm migrating several tables from on-prem to azure databricks into in...

Data Engineering
Reply
  • 997 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
fshimamoto
New Contributor III
  • 0 kudos

I usually recommend people to store data in a separate storage account (either mounted, or used directly), and don't use the internal storage of workspace for that tasks. Primary reason - it's easier to share this data with other workspaces, or other...

  • 0 kudos
L_
by • New Contributor II
  • 3421 Views
  • 4 replies
  • 2 kudos

How to change email on databricks community account?

I want to change the email associated with my databricks community edition account to a different email. How do I do that?

Data Engineering
Reply
  • 3421 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
User16565259302
New Contributor II
  • 2 kudos

You can add email in the account console and add them as an admin. One other solution is to close the current account and start a new community edition account with a different email.

  • 2 kudos
3 More Replies
Alex0101
by • New Contributor II
  • 4095 Views
  • 3 replies
  • 0 kudos

Resolved! Can python futures utilise all cluster nodes ?

I used python futures to call a function multiple times concurrently, however I am not sure if all nodes is utilised or how to make sure it use all cluster nodes.Can you confirm if I create a cluster with 5 works each with 8 memory cores for example....

Data Engineering
Reply
  • 4095 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Keyuri
New Contributor II
  • 0 kudos

You can create a init script and then add it during cluster start up ​

  • 0 kudos
2 More Replies
NM
by • New Contributor III
  • 1459 Views
  • 1 replies
  • 0 kudos

Resolved! How to do deterministic encryption on databricks?

How can I do deterministic encryption on databricks? And protect PI columns.

Data Engineering
Reply
  • 1459 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
01_binary
New Contributor III
  • 0 kudos

In order to deterministic encryption, we need to use aes encryption. Using AES the encrypted text will always remain the same for same input. Databricks recently implemented aes_encrypt and aes_decrypt functions and is the recommended way to perform ...

  • 0 kudos
mikesilva
by • New Contributor
  • 757 Views
  • 1 replies
  • 1 kudos

Resolved! Where can I learn more about Live tables

I keep hearing about Live tables. Where can I read more about it?​

Data Engineering
Reply
  • 757 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Fgkimball
New Contributor III
  • 1 kudos

Hello,​These are some good places to start!​Getting started docs: https://databricks.com/discover/pages/getting-started-with-delta-live-tables​​Notebook examples to walk through: https://github.com/databricks/delta-live-tables-notebooks

  • 1 kudos
Raymond_Garcia
by • Contributor II
  • 2548 Views
  • 2 replies
  • 5 kudos

Resolved! Databricks Job is slower.

Hello, I have a data bricks question. A Dataframe job that writes in an s3 bucket usually takes 8 minutes to finish, but now it takes from 8 to 9 hours to complete. Does anybody have some clues about this behavior?the data frame size is about 300 or ...

Data Engineering
Reply
  • 2548 Views
  • 2 replies
  • 5 kudos
View Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels