cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

elementalM
by New Contributor III
  • 2648 Views
  • 4 replies
  • 4 kudos

Catch-up Structured Stream hangs on last step of write job to delta sync using toTable

I'm running databricks version 10.4 on gcp. I'm running a structured stream trying to process historical files in a delta table on gcp cloud storage. This source delta table is big but maintained with OPTIMIZE.The stream repartitions which seems to b...

image
  • 2648 Views
  • 4 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Dwight Branscombe​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 4 kudos
3 More Replies
User16826994223
by Honored Contributor III
  • 2886 Views
  • 1 replies
  • 1 kudos
  • 2886 Views
  • 1 replies
  • 1 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 1 kudos

If you have configured your Structured Streaming query to use RocksDB as the state store, you can now get better visibility into the performance of RocksDB, with detailed metrics on get/put latencies, compaction latencies, cache hits, and so on. Thes...

  • 1 kudos
Labels