Databricks

MRTN · 05-16-2023

A deltaTable.dropDuplicates(columns) would be a very nice feature, simplifying the complex procedures that are suggested online. Or am I missing any existing procedures that can be done withouth merge operations or similar?

MRTN · 04-28-2023

I am currently using two streams to monitor data in two different containers on an Azure storage account. Is there any way to configure an autoloader to read from two different locations? The schemas of the files are identical.

MRTN · 04-27-2023

Am attempting to find the commit_time for a given file for a delta table using the cloud_files_state command. However, the archive_time, commit_time, and archive_time coluns are always NULL. I am running databrics runtime 11.3 and have also verified ...

MRTN · 04-04-2023

I have a set of CSV files generated by a system, where the schema has evolved over the years. Some columns have been added, and at least one column has been renamed in newer files. Is there any way to elegantly load these files into a dataframe? I ha...

MRTN · 05-16-2023

I created a feature request in the delta table project: [Feature Request] data deduplication on existing delta table · Issue #1767 · delta-io/delta (github.com)

MRTN · 05-16-2023

This would for sure be a useful feature.

MRTN · 05-16-2023

@Suteja Kanuri Thanks for this useful answer! In the meantime - we have moved onto using File Notification mode on Azure. Can we use the same "source" key to monitor two folders in this case?

MRTN · 04-12-2023

For reference - for anybody struggling with the same issues. All online examples using auto loader are written as one block statement on the form: (spark.readStream.format("cloudFiles") .option("cloudFiles.format", "csv") # The schema location di...

MRTN · 04-05-2023

Thanks for the input. I previously experimented with the Auto Loader, but was stopped by the fact that the headers in my csv files contains spaces and illegal characters. Hence the errorAnalysisException: Found invalid character(s) among " ,;{}()\n\...

Databricks

User Stats

User Activity

Feature request delta tables : drop duplicate rows

Configure multiple source paths for auto loader

Columns archive_time, commit_time, archive_time always NULL when running cloud_files_state

Load CSV files with slightly different schemas

Re: Feature request delta tables : drop duplicate rows

Re: Roll back to previous version of an AutoLoader checkpoint file

Re: Configure multiple source paths for auto loader

Re: Load CSV files with slightly different schemas

Re: Load CSV files with slightly different schemas