- 1509 Views
- 2 replies
- 0 kudos
working with delta files spark structure streaming , what is the maximum default chunk size in each batch?How do identify this type of spark configuration in databricks?#[Databricks SQL]​ #[Spark streaming]​ #[Spark structured streaming]​ #Spark​
- 1509 Views
- 2 replies
- 0 kudos
Latest Reply
Hello @KARTHICK N​ ,The default value for spark.sql.files.maxPartitionBytes is 128 MB. These defaults are in the Apache Spark documentation https://spark.apache.org/docs/latest/sql-performance-tuning.html (unless there might be some overrides).To che...
1 More Replies
- 1578 Views
- 4 replies
- 0 kudos
Have one function to create files with partitions, in that the partitions are created based on metadata (getPartitionColumns) that we are keeping. In a table we have two columns that are mentioned as partition columns, say 'Team' and 'Speciality'. Wh...
- 1578 Views
- 4 replies
- 0 kudos
Latest Reply
Hi @Thushar R​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...
3 More Replies
- 1814 Views
- 6 replies
- 10 kudos
I would need some suggestion from DataBricks Folks. As per documentation in Schema Evaluation for Drop and Rename Data is overwritten. Does it means we loose data (because I read data is not deleted but kind of staged). Is it possible to query old da...
- 1814 Views
- 6 replies
- 10 kudos
Latest Reply
Overwritte ​option will overwritte your data. If you want to change column name then you can first alter the delta table as per your need then you can append new data as well. So both problems you can resolve
5 More Replies
by
Biber
• New Contributor III
- 1581 Views
- 5 replies
- 8 kudos
Is it possible to reapply schema in delta files? For example, we have a history with field string but from some point, we need to replace string with struct.In my case merge option and overwrite schema don't work.
- 1581 Views
- 5 replies
- 8 kudos
Latest Reply
Biber
New Contributor III
Hi guys! Definitely, thank you for your support.
4 More Replies
by
a2_ish
• New Contributor II
- 729 Views
- 2 replies
- 2 kudos
I have tried below code to write data in a delta table and save the delta files in a sink. I tried using azure storage as sink but I get error as not enough access, I can confirm that I have enough access to azure storage, however I can run the below...
- 729 Views
- 2 replies
- 2 kudos
Latest Reply
Hi @Ankit Kumar​ Does @Hubert Dudek​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
1 More Replies
- 7170 Views
- 4 replies
- 6 kudos
Hi All,I am trying to Partition By () on Delta file in pyspark language and using command:df.write.format("delta").mode("overwrite").option("overwriteSchema","true").partitionBy("Partition Column").save("Partition file path") -- It doesnt seems to w...
- 7170 Views
- 4 replies
- 6 kudos
Latest Reply
Hey @Harsha kriplani​ Hope you are well. Thank you for posting in here. It is awesome that you found a solution. Would you like to mark Hubert's answer as best? It would be really helpful for the other members too.Cheers!
3 More Replies
