Databricks Community

cosminsanda · 04-16-2024

I would like to create a regular PySpark session in an isolated environment against which I can run my Spark based tests. I don't see how that's possible with the new Databricks Connect. I'm going in circles here, is it even possible?I don't want to ...

cosminsanda · 03-25-2024

I have a source table A in Unity Catalog. This table is constantly written to and is a streaming table.I also have another table B in Unity Catalog. This is a managed table with liquid clustering.Using Auto Loader I move new data from A to B using a ...

cosminsanda · 04-18-2024

Ok, so the best solution as it stands today (for me personally at least) is this:Have pyspark ^3.4 installed with the connect extra feature.My unit tests then don't have to change at all, as they use the regular spark session created on the flyFor ru...

cosminsanda · 04-17-2024

Problem is that I don't see how you can have both spark native and Databricks Connect (Spark Connect). The guidelines suggest one or the other, which is a bit of a pickle.

cosminsanda · 04-02-2024

What if I alter my target table B in advance, so that it contains the new columns before the query starts writing to them?

cosminsanda · 03-27-2024

Inded, @-werners- , you might be right, since I don't really use the cloudfiles functionality, which is really what Auto Loader is about.In any case, even if it's just regular structured streaming, given that I have a configured checkpoint where the ...

cosminsanda · 03-26-2024

To prevent data duplication, you can perform data deduplication based on a unique identifier (e.g., a primary key). -> I don't want to do data deduplication every time I add a column.If your data has a natural key or timestamp, consider using it to i...

Databricks Community

User Stats

User Activity

Unit Testing with the new Databricks Connect in Python

Adding a new column triggers reprocessing of Auto Loader source table

Re: Unit Testing with the new Databricks Connect in Python

Re: Unit Testing with the new Databricks Connect in Python

Re: Adding a new column triggers reprocessing of Auto Loader source table

Re: Adding a new column triggers reprocessing of Auto Loader source table

Re: Adding a new column triggers reprocessing of Auto Loader source table