Databricks Community

pantelis_mare · 07-25-2022

Hello team!As per the documentation, I understand that the table statistics can be fetched through the delta log (eg min, max, count) in order to not read the underlying data of a delta table.This is the case for numerical types, and timestamp is sup...

pantelis_mare · 07-22-2022

Hello community!I have a table with a column that is an array of a struct that has a very very long schema.When the table is written, all works well. Though, when I create a view based on this table and I try to access the view I get the error:rg.apa...

pantelis_mare · 05-04-2022

Hello all,I have a large number of light notebooks to run so I am taking the concurrent approach launching notebook runs with dbutils.notebook.run in parallel. The more I increase parallelism the more I see the duration of each notebook increasing.I ...

pantelis_mare · 04-27-2022

Hello everybody,I recently discovered (the hard way) that when a query plan uses cached data, the AQE does not kick-in. Result is that you loose the super cool feature of dynamic partition coalesce (no more custom shuffle readers in the DAG). Is ther...

pantelis_mare · 11-22-2021

Hello community!I would like to update a repo from within my Azure DevOps release pipeline. In the pipeline I generate a token using a AAD Service Principal as recommended, and I setup the databricks api using that token.When I pass the databricks re...

pantelis_mare · 09-13-2022

Hello @Gaurav Rupnar The following code snippet reproduces my statement.See how the query plan changes when you comment the cache() on the res dataframespark.conf.set("spark.sql.shuffle.partitions", 2000) spark.conf.set("spark.sql.autoBroadcastJoinT...

pantelis_mare · 09-13-2022

Hello Afzal, Unfortunately no  We have not been able to create a view on this table.

pantelis_mare · 07-26-2022

After communication with our CSE in Databricks, apparently there was an issue with truncation and precision loss that made them disactivate this feature..Actually I also had this issue on another project when trying to parse timestamp fields using js...

pantelis_mare · 07-26-2022

Yep.. attached it in my question.. Plus it's obvious by the number of tasks it is spawning.. it reads the whole table (more obvious in big tables with more read partitions).Can you reproduce it on your side as well or is it just us?

pantelis_mare · 07-26-2022

Yes! as the schema shows.. I also added an explicit cast to TimestampType (even though I create it as a java.sql.Timestamp anyway...Do you also get the same behavior? at least in my test case...

Databricks Community

User Stats

User Activity

Delta log statistics - timestamp type not working

Too long hive type string

Slow imports for concurrent notebooks

Spark 3 AQE and cache

Repos configuration for Azure Service Principal

Re: Spark 3 AQE and cache

Re: Too long hive type string

Re: Delta log statistics - timestamp type not working

Re: Delta log statistics - timestamp type not working

Re: Delta log statistics - timestamp type not working