Databricks Community

dpc · 09-23-2024

HelloWe have a development, a test and a production environmentHow do you generally deploy DDL changes?So, alter a table in development and apply to test then productione.g.table1 has column1, column2, column3I add column4I now want to deploy this ch...

dpc · 09-23-2024

HelloI've seen posts that show how to remove duplicates, something like this:MERGE into [deltatable] as targetUSING ( select *, ROW_NUMBER() OVER (Partition By [primary keys] Order By [date] desc) as rn from [deltatable] qualify rn> 1 ) as sourceON ...

dpc · 07-25-2024

Hello I have a table that has a column defined as an identity (BIGINT GENERATED ALWAYS AS IDENTITY)I will be inserting rows into this table in parallelHow can I get the identity and use that within a pipelineParallel is relevant as there will be mult...

dpc · 07-22-2024

HelloI'm building a datawarehouse with all the usual facts and dimensionsIt will flush (truncate) and rebuild on a monthly basisUsers have the need to not only view the data now but also view it historically i.e. what it was a point in timeMy initial...

dpc · 06-17-2024

HiI have a databricks table that has a column with a string datatypeWhen loading data, certain rows are prefixed by ' > 'Does anybody know what would cause this?It seems to be when the string is above a certain number of characters (around 200)Thanks

dpc · 09-25-2024

I originally tried this:Row_number() query and use DELETE statement where rn > 1It removes all rows no matter how I piece the code togetherFound this method works well in SQL Server but not databricks

dpc · 09-25-2024

Thanks. This looks more straight forward than I thoughtI'll give it a go in the morning and see if it works

dpc · 09-25-2024

Thanks.I'll step through this solution and see if I can get it working

dpc · 07-25-2024

Thanks SlashIn this case though, the batch generation is not concurrent, it's sequential but the full batch running can be concurrent (if that makes sense)So, I could be running 5 batches in parallel (not necessarily starting at the same time) and al...

dpc · 07-25-2024

Great, thanks

Databricks Community

User Stats

User Activity

table deployment (DDL) from one catalog to another

Remove Duplicate rows in tables

Returing and reusing the identity value

Approach to monthly data snapshots

Data prefixed by ' > '

Re: Remove Duplicate rows in tables

Re: Remove Duplicate rows in tables

Re: table deployment (DDL) from one catalog to another

Re: Returing and reusing the identity value

Re: Approach to monthly data snapshots