Databricks Community

dpc · 07-08-2025

HelloHistorically, we had a number of tables that have been extracted from source and loaded to databricks using 'select *'.As a result some columns that have been loaded never get used.I'd like to tidy this and remove redundant columns. Is there a w...

dpc · 09-23-2024

HelloWe have a development, a test and a production environmentHow do you generally deploy DDL changes?So, alter a table in development and apply to test then productione.g.table1 has column1, column2, column3I add column4I now want to deploy this ch...

dpc · 09-23-2024

HelloI've seen posts that show how to remove duplicates, something like this:MERGE into [deltatable] as targetUSING ( select *, ROW_NUMBER() OVER (Partition By [primary keys] Order By [date] desc) as rn from [deltatable] qualify rn> 1 ) as sourceON ...

dpc · 07-25-2024

Hello I have a table that has a column defined as an identity (BIGINT GENERATED ALWAYS AS IDENTITY)I will be inserting rows into this table in parallelHow can I get the identity and use that within a pipelineParallel is relevant as there will be mult...

dpc · 07-22-2024

HelloI'm building a datawarehouse with all the usual facts and dimensionsIt will flush (truncate) and rebuild on a monthly basisUsers have the need to not only view the data now but also view it historically i.e. what it was a point in timeMy initial...

dpc · 07-11-2025

I have the same situation and it's pretty frustration.If somebody creates a view or table, they seem to be the only person that can alter that view.Is there a way round this?I have tables created by a person who is not around and cannot do anythinig ...

dpc · 07-08-2025

ThanksWould I have to do that 1 table at a time though?Lineage is useful but it only shows which tables used the table. It doesn't actually show the column used unless you go into the notebook. Unless I am missign something here?

dpc · 07-08-2025

Thanks. I'll take a lookThe other point here though is that it's mainly ETL notebooks that access the landed data i.e. the silver layerSo, it's really a case of establishing whether the landed column has been used in the creation of a reporting table...

dpc · 09-25-2024

I originally tried this:Row_number() query and use DELETE statement where rn > 1It removes all rows no matter how I piece the code togetherFound this method works well in SQL Server but not databricks

dpc · 09-25-2024

Thanks. This looks more straight forward than I thoughtI'll give it a go in the morning and see if it works

Databricks Community

User Stats

User Activity

Can I see whether a table column is read in databricks?

table deployment (DDL) from one catalog to another

Remove Duplicate rows in tables

Returing and reusing the identity value

Approach to monthly data snapshots

Re: View alter permission for multiple members

Re: Can I see whether a table column is read in databricks?

Re: Can I see whether a table column is read in databricks?

Re: Remove Duplicate rows in tables

Re: Remove Duplicate rows in tables