Databricks Community

There are some approaches you can test: Delta Caching: Enable Delta caching to reduce the number of metadata queries. This can be done by setting the spark.databricks.io.cache.enabled configuration to true.Optimize Command: Use the OPTIMIZE command t...

  The RESTORE command is not supported on streaming tables, which is why you encountered the error. Instead, you can use the TIME TRAVEL feature of Delta Lake to query previous versions of the table. You can use the VERSION AS OF or TIMESTAMP AS OF c...

Enabling Photon Acceleration on your Databricks cluster reduces the available executor memory because Photon uses a different memory management strategy compared to standard Spark. Photon is designed to optimize performance by leveraging the underlyi...

Hi @NathanSundarara , regarding your current approach, here are the potential solutions and considerations- Deduplication: Implement deduplication strategies within your DLT pipeline. For example clicksDedupDf = ( spark.readStream.table("LIVE.rawCl...

https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html   numPartitions(none)The maximum number of partitions that can be used for parallelism in table reading and writing. This also determines the maximum number of concurrent JDBC connection...

Glad to hear you got unblocked.

Here are a few steps you can take to debug and resolve this issue: Check the Code for Either Usage: Look through your codebase for instances where Either is used. Ensure that you are handling both Left and Right cases properly. The error suggests th...

Looks like the primary key column ID is not being found during the write operation. Kindly verify the schema.  Use a command like below to create the table with id as the primary key. CREATE TABLE A_Table ( ID BIGINT IDENTITY(1,1) PRIMARY KEY NOT NUL...

Kindly use the latest jars from maven, and if that does not help shading common packages to avoid conflicts is something that should be our next resort.