cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
bgarcia
by • New Contributor III
  • 937 Views
  • 1 replies
  • 0 kudos

Resolved! Delta Tables

I’m beginning my journey into Delta Tables and one thing that is still confusing me is where is the best place to save your delta tables if you need to query them later.For example I'm migrating several tables from on-prem to azure databricks into in...

Data Engineering
Reply
  • 937 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
fshimamoto
New Contributor III
  • 0 kudos

I usually recommend people to store data in a separate storage account (either mounted, or used directly), and don't use the internal storage of workspace for that tasks. Primary reason - it's easier to share this data with other workspaces, or other...

  • 0 kudos
L_
by • New Contributor II
  • 3173 Views
  • 4 replies
  • 2 kudos

How to change email on databricks community account?

I want to change the email associated with my databricks community edition account to a different email. How do I do that?

Data Engineering
Reply
  • 3173 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
User16565259302
New Contributor II
  • 2 kudos

You can add email in the account console and add them as an admin. One other solution is to close the current account and start a new community edition account with a different email.

  • 2 kudos
3 More Replies
Alex0101
by • New Contributor II
  • 3900 Views
  • 3 replies
  • 0 kudos

Resolved! Can python futures utilise all cluster nodes ?

I used python futures to call a function multiple times concurrently, however I am not sure if all nodes is utilised or how to make sure it use all cluster nodes.Can you confirm if I create a cluster with 5 works each with 8 memory cores for example....

Data Engineering
Reply
  • 3900 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Keyuri
New Contributor II
  • 0 kudos

You can create a init script and then add it during cluster start up ​

  • 0 kudos
2 More Replies
NM
by • New Contributor III
  • 1307 Views
  • 1 replies
  • 0 kudos

Resolved! How to do deterministic encryption on databricks?

How can I do deterministic encryption on databricks? And protect PI columns.

Data Engineering
Reply
  • 1307 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
01_binary
New Contributor III
  • 0 kudos

In order to deterministic encryption, we need to use aes encryption. Using AES the encrypted text will always remain the same for same input. Databricks recently implemented aes_encrypt and aes_decrypt functions and is the recommended way to perform ...

  • 0 kudos
mikesilva
by • New Contributor
  • 653 Views
  • 1 replies
  • 1 kudos

Resolved! Where can I learn more about Live tables

I keep hearing about Live tables. Where can I read more about it?​

Data Engineering
Reply
  • 653 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Fgkimball
New Contributor III
  • 1 kudos

Hello,​These are some good places to start!​Getting started docs: https://databricks.com/discover/pages/getting-started-with-delta-live-tables​​Notebook examples to walk through: https://github.com/databricks/delta-live-tables-notebooks

  • 1 kudos
Raymond_Garcia
by • Contributor II
  • 2352 Views
  • 2 replies
  • 5 kudos

Resolved! Databricks Job is slower.

Hello, I have a data bricks question. A Dataframe job that writes in an s3 bucket usually takes 8 minutes to finish, but now it takes from 8 to 9 hours to complete. Does anybody have some clues about this behavior?the data frame size is about 300 or ...

Data Engineering
Reply
  • 2352 Views
  • 2 replies
  • 5 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 5 kudos

This widget could not be displayed.
Hello, I have a data bricks question. A Dataframe job that writes in an s3 bucket usually takes 8 minutes to finish, but now it takes from 8 to 9 hours to complete. Does anybody have some clues about this behavior?the data frame size is about 300 or ...

This widget could not be displayed.
  • 5 kudos
This widget could not be displayed.
1 More Replies
Bomberone
by • New Contributor II
  • 1295 Views
  • 1 replies
  • 2 kudos

Resolved! Autoloader checkpoint issue

Hello guys, anyone issuing problems with autoloader checkpoints on azure?​

Data Engineering
Reply
  • 1295 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Me not, but it is good to specify both the autoloader checkpoint and streaming (for write) checkpoint.And it happened to me during some experiments that I had to delete everything from the autoloader directory.

  • 2 kudos
DanielWhite
by • New Contributor II
  • 938 Views
  • 2 replies
  • 2 kudos

Brilliant idea of the concept.! I want to learn much more about DataBricks!!

Brilliant idea of the concept.! I want to learn much more about DataBricks!!

Data Engineering
Reply
  • 938 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Try https://customer-academy.databricks.com/ is excellent

  • 2 kudos
1 More Replies
Anonymous
by • Not applicable
  • 1048 Views
  • 2 replies
  • 4 kudos

Become a Community Champion! As you know, Data + AI Summit 2022 is June 27-30. During this time we want to see as many of our virtual and in-person at...

Become a Community Champion! As you know, Data + AI Summit 2022 is June 27-30. During this time we want to see as many of our virtual and in-person attendees becoming the best communtiy version of themselves: a Community Champion. Because the qualifi...

Image Image
Data Engineering
Reply
  • 1048 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

Cooler will be great for summer picnics

  • 4 kudos
1 More Replies
Leladams
by • New Contributor III
  • 9799 Views
  • 9 replies
  • 2 kudos

What is the best way to read in a ms access .accdb database into Databricks from a mounted drive?

I am currently trying to read in .accdb files from a mounted drive. Based on my research it looks like I would have to use a package like JayDeBeApi with ucanaccess drivers or pyodbc with ms access drivers.Will this work?Thanks for any help.

Data Engineering
Reply
  • 9799 Views
  • 9 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Leland Adams​ Hope you are doing well. Thank you for posting your question and giving us additional information. Do you think you were able to solve the query?We'd love to hear from you.

  • 2 kudos
8 More Replies
abd
by • Contributor
  • 5728 Views
  • 2 replies
  • 2 kudos

Resolved! How spark will handles 1TB data if it has cluster of 1GB memory ?

If my cluster memory is 1GB for example and my data is 1TB how Spark will handle it?If it is in memory computing how does it handles the data that is greater than the memory size ?

Data Engineering
Reply
  • 5728 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
abd
Contributor
  • 2 kudos

@Kaniz Fatma​ @Cedric Law Hing Ping​ 

  • 2 kudos
1 More Replies
User16826990884
by • New Contributor III
  • 1450 Views
  • 1 replies
  • 0 kudos

Disable managed tables on Azure Databricks

When a user creates a table without a path, it writes it as a managed table in the root bucket. Can this functionality be disabled so users are forced to provide a storage path and follow our organization best practices?

Data Engineering
Reply
  • 1450 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
florent
New Contributor III
  • 0 kudos

Hi,I couldn't find the option to turn it off. It might be worth banning dbfs. However, you can configure the default location using a cluster policy by adding the following configuration:"spark_conf.spark.sql.warehouse.dir": {  "type": "fixed",  "val...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels