Databricks Community

Hello,IMHO, having a HTTP related task in a Spark cluster is an anti-pattern. This kind of code executes at the Driver, it will be synchronous and adds overhead. This is one of the reasons, DLT (or SDP - Spark Declarative Pipeline) does not have REST...

@szymon_dybczak  - thank you for your response I try:  @dlt.table( name="deserialized", comment="Raw messages from Kafka topic as JSON", table_properties={ "pipelines.autoOptimize.managed": "true", "pipelines.autoCompact.m...

Hi @Swathik ,If we're talking about where to store ETL metadata, in my opinion it's mostly a matter of preference. In my case, I prefer storing my config in YAML files, but I’ve also worked on projects where the config was stored in Delta tables.For ...

Hello @DatabricksUser5 , You can’t override committed offsets in-place for a running DLT Kafka/Event Hubs stream. If a pipeline already has a checkpoint created, startingOffsets is ignored. To replay data, you must reset the streaming checkpoints or ...

One way to get version control might be to use the Terraform resource entity_tag_assignment. I am not sure if it supports governed_tags, but I'll experiment in the coming weeks.This separates the version control on where the tags are defined and wher...

I have the same error if I query SELECT 1 from remote_query(). From documentation:> To use the remote_query function, you first need to create a Unity Catalog connectionSo, not sure why it rebels against creators

yes, I would take that as a pinch of salt

Databricks Job/Task parameter interface does not provide a built-in UI feature to restrict the possible values entered by user. Yoou can add a runtime validation code inside the notebook to allow/fail based on the values entered in.

As the User IDs and Group IDs share the same namespace in Databricks you cannot create a Group with the same email address that is already registered as a User in your Databricks account.You better rename the group.

Classic 360 Custoemr View case and Databricks is certainly the right platform to do so.Strcutred Data - Stores in the Delta TablesEmail and PDFs - stored in Volumes, however, metadata as path to the volumes stored in delta table against customer-idIn...

Hi Suheb, when using delta tables with databricks, whenever you use proper liquid clustering indexes or partitions, you should get a good performance in comparison to relational engines to deal with big data volumes.However, you can also separate tab...

Databricks can install Maven libraries by coordinate and lets you point at a custom repository URL. However, passing credentials for authenticated private Maven repositories directly through the Libraries UI/Jobs is not natively supported today and r...