cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
alejandrofm
by • Valued Contributor
  • 2793 Views
  • 3 replies
  • 0 kudos

Cluster not following the zone: auto configuration

Hi, I'm trying some new instance types, so to be sure I will get one I set the Availability zone to 'auto'.But this is happening, the image of the error and JSON are attached.So, the cluster is trying three times to upscale and the three times is usi...

image image image.png
Data Engineering
Reply
  • 2793 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Alejandro Martinez​ Hope everything is going great.Does @Sivaprasad C S​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
2 More Replies
elgeo
by • Valued Contributor II
  • 1692 Views
  • 1 replies
  • 2 kudos

Disable auto-complete (tab button)

Hello. How could we disable autocomplete that appears with tab button? Thank you

Data Engineering
Reply
  • 1692 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
elgeo
Valued Contributor II
  • 2 kudos

Thank you @Kaniz Fatma​ 

  • 2 kudos
mcharl02
by • New Contributor III
  • 8747 Views
  • 9 replies
  • 6 kudos

How do I restore auto-close quote & auto-close parentheses functionality?

Over the last two days, my team's databricks notebooks (using Python interpreter) have stopped automatically adding a close single quote (') with a cursor between the two. Same issue with automatically adding close parentheses.The cluster has been re...

Data Engineering
Reply
  • 8747 Views
  • 9 replies
  • 6 kudos
View Replies
Latest Reply
Sajesh
Databricks Employee
  • 6 kudos

The fix for this issue will be most likely released to all regions/workspaces by 18th Nov 21

  • 6 kudos
8 More Replies
brickster_2018
by • Databricks Employee
  • 1504 Views
  • 1 replies
  • 0 kudos

Resolved! Auto-scaling not getting kicked in

I have a spark-submit job, I do not see auto-scaling happening on the cluster at the time of executions.

Data Engineering
Reply
  • 1504 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

This is working as expected. Autoscaling is not available for spark-submit jobsRun the job as jar job instead of spark-submit

  • 0 kudos
brickster_2018
by • Databricks Employee
  • 2288 Views
  • 1 replies
  • 0 kudos

Resolved! Can auto-loader process the overwritten file

I have a directory where I get files with the same multiple times. Will Auto-loader process all the files or will it process the first and ignore the rest

Data Engineering
Reply
  • 2288 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

Autoloader has an option - "cloudFiles. allowOverwrites". This determines whether to allow input directory file changes to overwrite existing data. This option is available in Databricks Runtime 7.6 and above.

  • 0 kudos
brickster_2018
by • Databricks Employee
  • 1392 Views
  • 2 replies
  • 0 kudos

Why should I move to Auto-loader?

I have a streaming workload using the S3-SQS Connector. The streaming job is running fine within the SLA. Should I migrate my job to use the auto-loader? If Yes, what are the benefits? who should migrate and who should not?

Data Engineering
Reply
  • 1392 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

That makes sense @Anand Ladda​ ! One major improvement that will have a direct impact on the performance is the architectural difference. S3-SQS uses an internal implementation of the Delta table to store the checkpoint details about the source files...

  • 0 kudos
1 More Replies
Anonymous
by • Not applicable
  • 854 Views
  • 0 replies
  • 0 kudos

What is Auto auto-logging?

How is it different from regular autologging? When should I consider enabling Auto autologging ? How can I switch the feature on?

Data Engineering
Reply
  • 854 Views
  • 0 replies
  • 0 kudos
Labels