Databricks

N_M · 01-05-2024

HiI'm using the COPY INTO command to insert new data (in form of CSVs) into an already existing table.The SQL query takes care of the conversion of the fields to the target table schema (well, there isn't other way to do that), and schema update is n...

N_M · 12-07-2023

Dear Community,I'm using the COPY INTO command to automate the staging of files that I get in an S3 bucket into specific delta tables (with some transformation on the fly).The command works smoothly, and files are indeed inserted only once (writing i...

N_M · 10-10-2023

Hi all,Due to file size and file transfer limitation, we are receiving huge files compressed and split, in the format FILE.z01, FILE.z02,...,FILE.zipHowever, I can't find a way to unzip multipart files using databricks.I tried already some of the ...

N_M · 01-09-2024

I actually found an option that could solve the newline issue I mentioned in my previous post:setting spark.sql.csv.parser.columnPruning.enabled to false withspark.conf.set("spark.sql.csv.parser.columnPruning.enabled", False)will consider malformed r...

N_M · 01-09-2024

Hi @Kaniz thank you spending time on this :).The DROPMALFORMED option will silently ignore and drop malformed lines, which is not really an applicable option in a pipeline, at least not in all the use cases I have worked in so far.A potential solutio...

N_M · 01-08-2024

hi @Kaniz thanks for the reply. I'm aware of the possibility to perform checks and error handling to handle such situation. However, this is not optimal:Perform checks on top of a file and then writing it means reading it 2 times. So a flow with thi...

N_M · 12-01-2023

How does COPY_INTO work with table restore?I made some tests, and the restore method does NOT restore the key-store values of the target at the specific version, which means that the data that came after the chosen version cannot be inserted (unless ...

Databricks

User Stats

User Activity

ignoreCorruptFiles behavior with CSV and COPY INTO

Access historical injected data of COPY INTO command

Unzip multipart files

Re: ignoreCorruptFiles behavior with CSV and COPY INTO

Re: ignoreCorruptFiles behavior with CSV and COPY INTO

Re: ignoreCorruptFiles behavior with CSV and COPY INTO

Re: How is Idempotency ensured for COPY INTO command