- 3868 Views
- 5 replies
- 7 kudos
Following are the details of the requirement:1. I am using databricks notebook to read data from Kafka topic and writing into ADLS Gen2 container i.e., my landing layer.2. I am using Spark code to read data from Kafka and write into landing...
- 3868 Views
- 5 replies
- 7 kudos
Latest Reply
just to clarify, are you reading kafka and writing into adls in json files? like for each message from kafka is 1 json file in adls ?
4 More Replies
by
MRTN
• New Contributor III
- 11973 Views
- 4 replies
- 3 kudos
I have a set of CSV files generated by a system, where the schema has evolved over the years. Some columns have been added, and at least one column has been renamed in newer files. Is there any way to elegantly load these files into a dataframe? I ha...
- 11973 Views
- 4 replies
- 3 kudos
Latest Reply
For reference - for anybody struggling with the same issues. All online examples using auto loader are written as one block statement on the form: (spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "csv")
# The schema location di...
3 More Replies
- 1310 Views
- 0 replies
- 2 kudos
We do not want to use schema inference with schema evolution in Autoloader. Instead, we want to apply our schema and use the merge option. Our schema is very complex, with multiple nested following levels. When I apply this schema to Autoloader, it r...
- 1310 Views
- 0 replies
- 2 kudos
- 4416 Views
- 4 replies
- 1 kudos
Hi,Currently, I'm using structure streaming to insert/update/delete to a table. A row will be deleted if value in 'Operation' column is 'deleted'. Everything seems to work fine until there's a new column.Since I don't need 'Operation' column in the t...
- 4416 Views
- 4 replies
- 1 kudos
Latest Reply
please go through this documentation https://docs.delta.io/latest/api/python/index.html
3 More Replies
- 2799 Views
- 3 replies
- 2 kudos
What are the best practices for schema drift using Delta Live tables, in a scenario where the main source is a no sql database and we have a lot of changes in the schema?
- 2799 Views
- 3 replies
- 2 kudos
Latest Reply
Hey there @Fernando Martin Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from...
2 More Replies
- 1054 Views
- 0 replies
- 1 kudos
Hi,I have a Delta Live Tables pipeline, using Auto Loader, to ingest from JSON files. I need to do some transformations - in this case, converting timestamps. Except one of the timestamp columns does not exist in every file. This is causing the DLT p...
- 1054 Views
- 0 replies
- 1 kudos
by
Gapy
• New Contributor II
- 1629 Views
- 1 replies
- 1 kudos
Dear all,will (and when) will Auto Loader also support Schema-Inference and Evolution for parquet files, at this point it is only for JSON and CSV supported if i am not mistaken?Thanks and regards,Gapy
- 1629 Views
- 1 replies
- 1 kudos
Latest Reply
@Gasper Zerak , This will be available in near future (DBR 10.3 or later). Unfortunately, we don't have an SLA at this moment.
- 13469 Views
- 7 replies
- 0 kudos
I have file a.csv or a.parquet while creating data frame reading we can explictly define schema with struct type. instead of write the schema in the notebook want to create schema lets say for all my csv i have one schema like csv_schema and stored ...
- 13469 Views
- 7 replies
- 0 kudos
Latest Reply
@shyampsr big thanks, was searching for the solution almost 3 hours _https://luckycanadian.com/
6 More Replies
