by
MRTN
• New Contributor III
- 4336 Views
- 4 replies
- 3 kudos
I have a set of CSV files generated by a system, where the schema has evolved over the years. Some columns have been added, and at least one column has been renamed in newer files. Is there any way to elegantly load these files into a dataframe? I ha...
- 4336 Views
- 4 replies
- 3 kudos
Latest Reply
For reference - for anybody struggling with the same issues. All online examples using auto loader are written as one block statement on the form: (spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "csv")
# The schema location di...
3 More Replies
- 1912 Views
- 4 replies
- 7 kudos
Following are the details of the requirement:1. I am using databricks notebook to read data from Kafka topic and writing into ADLS Gen2 container i.e., my landing layer.2. I am using Spark code to read data from Kafka and write into landing...
- 1912 Views
- 4 replies
- 7 kudos
Latest Reply
Hi @Swapnil Kamle​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
3 More Replies
- 626 Views
- 0 replies
- 2 kudos
We do not want to use schema inference with schema evolution in Autoloader. Instead, we want to apply our schema and use the merge option. Our schema is very complex, with multiple nested following levels. When I apply this schema to Autoloader, it r...
- 626 Views
- 0 replies
- 2 kudos
- 1911 Views
- 4 replies
- 1 kudos
Hi,Currently, I'm using structure streaming to insert/update/delete to a table. A row will be deleted if value in 'Operation' column is 'deleted'. Everything seems to work fine until there's a new column.Since I don't need 'Operation' column in the t...
- 1911 Views
- 4 replies
- 1 kudos
Latest Reply
please go through this documentation https://docs.delta.io/latest/api/python/index.html
3 More Replies
- 1172 Views
- 3 replies
- 2 kudos
What are the best practices for schema drift using Delta Live tables, in a scenario where the main source is a no sql database and we have a lot of changes in the schema?​
- 1172 Views
- 3 replies
- 2 kudos
Latest Reply
Hey there @Fernando Martin​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from...
2 More Replies
- 537 Views
- 0 replies
- 1 kudos
Hi,I have a Delta Live Tables pipeline, using Auto Loader, to ingest from JSON files. I need to do some transformations - in this case, converting timestamps. Except one of the timestamp columns does not exist in every file. This is causing the DLT p...
- 537 Views
- 0 replies
- 1 kudos
by
Gapy
• New Contributor II
- 1046 Views
- 1 replies
- 1 kudos
Dear all,will (and when) will Auto Loader also support Schema-Inference and Evolution for parquet files, at this point it is only for JSON and CSV supported if i am not mistaken?Thanks and regards,Gapy
- 1046 Views
- 1 replies
- 1 kudos
Latest Reply
@Gasper Zerak​ , This will be available in near future (DBR 10.3 or later). Unfortunately, we don't have an SLA at this moment.
- 5878 Views
- 7 replies
- 0 kudos
I have file a.csv or a.parquet while creating data frame reading we can explictly define schema with struct type. instead of write the schema in the notebook want to create schema lets say for all my csv i have one schema like csv_schema and stored ...
- 5878 Views
- 7 replies
- 0 kudos
Latest Reply
@shyampsr big thanks, was searching for the solution almost 3 hours _https://luckycanadian.com/
6 More Replies
