by
APol
• New Contributor II
- 3088 Views
- 2 replies
- 2 kudos
Hi. I assume that it can be concurrency issue. (a Read thread from Databricks and a Write thread from another system)From the start:I read 12-16 csv files (approximately 250Mb each of them) to dataframe. df = spark.read.option("header", "False").opti...
- 3088 Views
- 2 replies
- 2 kudos
Latest Reply
Hi @Anastasiia Polianska,I agree, it looks like a concurrency issue. Very possibly this concurrency problem will be caused by an erroneous ETAG in the HTTP call to the Azure Storage API (https://azure.microsoft.com/de-de/blog/managing-concurrency-in...
1 More Replies
- 6466 Views
- 7 replies
- 11 kudos
•Power BI is a publisher that uses AD group authentication to publish result sets. Since the publisher's credentials are maintained, the same user can access the data bricks database.•Number of the users are retrieving the data from the power bi or i...
- 6466 Views
- 7 replies
- 11 kudos
Latest Reply
I believe 10 is a limit as of now. See if you can increase the concurrency limit from the source.
6 More Replies
by
Phani1
• Valued Contributor II
- 1878 Views
- 2 replies
- 5 kudos
When we implemented the concurrent updates on a table which do not have a partition column we ran into ConcurrentAppendException [ensured where the condition is different for each concurrent update statement]So do we need to go by partition approach ...
- 1878 Views
- 2 replies
- 5 kudos
Latest Reply
Please check that both streaming queries don't use the same checkpoint,Auto increment id can also make problems as it is kept in schemaSchema evolution also can make problems
1 More Replies
- 2372 Views
- 4 replies
- 0 kudos
I have an SQL query to select and update rows in a table. I do this in batches of 300 rows (select 300 , update the selected 300 , select new 300 and update the newly selected and so on..) I run this query in 2 different notebooks concurrently to spe...
- 2372 Views
- 4 replies
- 0 kudos
Latest Reply
Hi @Krishna Zanwar Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
3 More Replies
- 3487 Views
- 3 replies
- 4 kudos
What is the maximum of concurrent streaming jobs for a cluster? How can I have the right amount of concurrent streaming jobs for different cluster configuration?Should I use multiple cluster for different jobs or combine it into a big cluster to hand...
- 3487 Views
- 3 replies
- 4 kudos
Latest Reply
Hi @John William it would be better to use different clusters for each streaming jobs.
2 More Replies
- 8210 Views
- 2 replies
- 4 kudos
Hello, I am developing an application that runs multiple processes that write their results to a common delta table as blind appends. According to the docs I've read online: https://docs.databricks.com/delta/concurrency-control.html#protocolchangedex...
- 8210 Views
- 2 replies
- 4 kudos
Latest Reply
I think you are right, the mergeSchema will change the schema of the table, but if you both write to that same table with another schema, which one will it be?Can you check if both of you actually write the same schema, or remove the mergeschema?
1 More Replies
