cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
alejandrofm
by • Valued Contributor
  • 1587 Views
  • 3 replies
  • 0 kudos

Cluster not following the zone: auto configuration

Hi, I'm trying some new instance types, so to be sure I will get one I set the Availability zone to 'auto'.But this is happening, the image of the error and JSON are attached.So, the cluster is trying three times to upscale and the three times is usi...

image image image.png
Data Engineering
Reply
  • 1587 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Alejandro Martinez​ Hope everything is going great.Does @Sivaprasad C S​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
2 More Replies
mcharl02
by • New Contributor III
  • 4146 Views
  • 9 replies
  • 6 kudos

How do I restore auto-close quote & auto-close parentheses functionality?

Over the last two days, my team's databricks notebooks (using Python interpreter) have stopped automatically adding a close single quote (') with a cursor between the two. Same issue with automatically adding close parentheses.The cluster has been re...

Data Engineering
Reply
  • 4146 Views
  • 9 replies
  • 6 kudos
View Replies
Latest Reply
Sajesh
New Contributor III
  • 6 kudos

The fix for this issue will be most likely released to all regions/workspaces by 18th Nov 21

  • 6 kudos
8 More Replies
User16869510359
by • Esteemed Contributor
  • 924 Views
  • 1 replies
  • 0 kudos

Resolved! Auto-scaling not getting kicked in

I have a spark-submit job, I do not see auto-scaling happening on the cluster at the time of executions.

Data Engineering
Reply
  • 924 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

This is working as expected. Autoscaling is not available for spark-submit jobsRun the job as jar job instead of spark-submit

  • 0 kudos
User16869510359
by • Esteemed Contributor
  • 1475 Views
  • 1 replies
  • 0 kudos

Resolved! Can auto-loader process the overwritten file

I have a directory where I get files with the same multiple times. Will Auto-loader process all the files or will it process the first and ignore the rest

Data Engineering
Reply
  • 1475 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

Autoloader has an option - "cloudFiles. allowOverwrites". This determines whether to allow input directory file changes to overwrite existing data. This option is available in Databricks Runtime 7.6 and above.

  • 0 kudos
User16869510359
by • Esteemed Contributor
  • 675 Views
  • 2 replies
  • 0 kudos

Why should I move to Auto-loader?

I have a streaming workload using the S3-SQS Connector. The streaming job is running fine within the SLA. Should I migrate my job to use the auto-loader? If Yes, what are the benefits? who should migrate and who should not?

Data Engineering
Reply
  • 675 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

That makes sense @Anand Ladda​ ! One major improvement that will have a direct impact on the performance is the architectural difference. S3-SQS uses an internal implementation of the Delta table to store the checkpoint details about the source files...

  • 0 kudos
1 More Replies
Anonymous
by • Not applicable
  • 469 Views
  • 0 replies
  • 0 kudos

What is Auto auto-logging?

How is it different from regular autologging? When should I consider enabling Auto autologging ? How can I switch the feature on?

Data Engineering
Reply
  • 469 Views
  • 0 replies
  • 0 kudos
Labels