Databricks Community

SamAdams

I have a DLT pipeline that reads data in S3 into an append-only bronze layer using Autoloader. The data sink needs to be changed to a new s3 bucket in a new account, and data in the existing s3 bucket migrated to the new one.Will Autoloader still be ...

SamAdams · 10-10-2024

Hello! I have a delta table with a check constraint - it's one of many that a config-driven ETL pipeline of mine generates. When someone edits the config file and deploys the change, I'd like for the check constraint to be updated as well if it's dif...

SamAdams

Might be saved by the documentation here https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/directory-listing-mode#change-source-path-for-auto-loader

SamAdams

It's in private preview, at least at the moment.

SamAdams · 10-10-2024

Figured this out with the help of @SamDataWalk 's post https://community.databricks.com/t5/data-engineering/databricks-bug-with-show-tblproperties-redacted-azure-databricks/m-p/93546It happens because Databricks thinks certain keywords in the constra...

SamAdams · 10-10-2024

Like your example that redaction behavior seemed to pick up on the column name: a condition that included a column named "URL" was redacted, but one that included a "modifiedDateTime" was not

SamAdams · 10-10-2024

@SamDataWalk thanks for posting the solution. I ran into the same problem while writing some logic that updates table constraints across many tables from a config. They were coming back as redacted - this allowed me to actually read them and update a...

Databricks Community

User Stats

User Activity

Migrating source directory in an existing DLT Pipeline with Autoloader

Redacted check constraint condition in Delta Table

Re: Migrating source directory in an existing DLT Pipeline with Autoloader

Re: Can we execute a single task in isolation from a multi task Databricks job

Re: Redacted check constraint condition in Delta Table

Re: Databricks bug with show tblproperties - redacted - Azure databricks

Re: Databricks bug with show tblproperties - redacted - Azure databricks