cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

alejandrofm
by Valued Contributor
  • 1539 Views
  • 3 replies
  • 0 kudos

Cluster not following the zone: auto configuration

Hi, I'm trying some new instance types, so to be sure I will get one I set the Availability zone to 'auto'.But this is happening, the image of the error and JSON are attached.So, the cluster is trying three times to upscale and the three times is usi...

image image image.png
  • 1539 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Alejandro Martinez​ Hope everything is going great.Does @Sivaprasad C S​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
2 More Replies
mcharl02
by New Contributor III
  • 4002 Views
  • 9 replies
  • 6 kudos

How do I restore auto-close quote & auto-close parentheses functionality?

Over the last two days, my team's databricks notebooks (using Python interpreter) have stopped automatically adding a close single quote (') with a cursor between the two. Same issue with automatically adding close parentheses.The cluster has been re...

  • 4002 Views
  • 9 replies
  • 6 kudos
Latest Reply
Sajesh
New Contributor III
  • 6 kudos

The fix for this issue will be most likely released to all regions/workspaces by 18th Nov 21

  • 6 kudos
8 More Replies
User16869510359
by Esteemed Contributor
  • 897 Views
  • 1 replies
  • 0 kudos

Resolved! Auto-scaling not getting kicked in

I have a spark-submit job, I do not see auto-scaling happening on the cluster at the time of executions.

  • 897 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

This is working as expected. Autoscaling is not available for spark-submit jobsRun the job as jar job instead of spark-submit

  • 0 kudos
User16869510359
by Esteemed Contributor
  • 1438 Views
  • 1 replies
  • 0 kudos

Resolved! Can auto-loader process the overwritten file

I have a directory where I get files with the same multiple times. Will Auto-loader process all the files or will it process the first and ignore the rest

  • 1438 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

Autoloader has an option - "cloudFiles. allowOverwrites". This determines whether to allow input directory file changes to overwrite existing data. This option is available in Databricks Runtime 7.6 and above.

  • 0 kudos
User16869510359
by Esteemed Contributor
  • 657 Views
  • 2 replies
  • 0 kudos

Why should I move to Auto-loader?

I have a streaming workload using the S3-SQS Connector. The streaming job is running fine within the SLA. Should I migrate my job to use the auto-loader? If Yes, what are the benefits? who should migrate and who should not?

  • 657 Views
  • 2 replies
  • 0 kudos
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

That makes sense @Anand Ladda​ ! One major improvement that will have a direct impact on the performance is the architectural difference. S3-SQS uses an internal implementation of the Delta table to store the checkpoint details about the source files...

  • 0 kudos
1 More Replies
Anonymous
by Not applicable
  • 458 Views
  • 0 replies
  • 0 kudos

What is Auto auto-logging?

How is it different from regular autologging? When should I consider enabling Auto autologging ? How can I switch the feature on?

  • 458 Views
  • 0 replies
  • 0 kudos
Labels