- 3574 Views
- 5 replies
- 0 kudos
working with delta files spark structure streaming , what is the maximum default chunk size in each batch?How do identify this type of spark configuration in databricks?#[Databricks SQL] #[Spark streaming] #[Spark structured streaming] #Spark
- 3574 Views
- 5 replies
- 0 kudos
Latest Reply
doc - https://docs.databricks.com/en/structured-streaming/delta-lake.html
Also, what is the challenge while using foreachbatch?
4 More Replies
- 4099 Views
- 4 replies
- 0 kudos
Have one function to create files with partitions, in that the partitions are created based on metadata (getPartitionColumns) that we are keeping. In a table we have two columns that are mentioned as partition columns, say 'Team' and 'Speciality'. Wh...
- 4099 Views
- 4 replies
- 0 kudos
Latest Reply
Hi @Thushar R Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...
3 More Replies
- 4418 Views
- 6 replies
- 10 kudos
I would need some suggestion from DataBricks Folks. As per documentation in Schema Evaluation for Drop and Rename Data is overwritten. Does it means we loose data (because I read data is not deleted but kind of staged). Is it possible to query old da...
- 4418 Views
- 6 replies
- 10 kudos
Latest Reply
Overwritte option will overwritte your data. If you want to change column name then you can first alter the delta table as per your need then you can append new data as well. So both problems you can resolve
5 More Replies
- 3957 Views
- 5 replies
- 8 kudos
Is it possible to reapply schema in delta files? For example, we have a history with field string but from some point, we need to replace string with struct.In my case merge option and overwrite schema don't work.
- 3957 Views
- 5 replies
- 8 kudos
by
a2_ish
• New Contributor II
- 1720 Views
- 2 replies
- 2 kudos
I have tried below code to write data in a delta table and save the delta files in a sink. I tried using azure storage as sink but I get error as not enough access, I can confirm that I have enough access to azure storage, however I can run the below...
- 1720 Views
- 2 replies
- 2 kudos
Latest Reply
Hi @Ankit Kumar Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
1 More Replies
- 15101 Views
- 4 replies
- 6 kudos
Hi All,I am trying to Partition By () on Delta file in pyspark language and using command:df.write.format("delta").mode("overwrite").option("overwriteSchema","true").partitionBy("Partition Column").save("Partition file path") -- It doesnt seems to w...
- 15101 Views
- 4 replies
- 6 kudos
Latest Reply
Hey @Harsha kriplani Hope you are well. Thank you for posting in here. It is awesome that you found a solution. Would you like to mark Hubert's answer as best? It would be really helpful for the other members too.Cheers!
3 More Replies
