Databricks

Jennifer

We have a table using timestampNtz type for timestamp, which is also a cluster key for this table using liquid clustering. I ran OPTIMIZE <table-name>, it failed with errorUnsupported datatype 'TimestampNTZType' But the failed optmization also broke ...

Jennifer

More people begin to use TimestampNTZ as cluster key.According to the thread here Unsupported datatype 'TimestampNTZType' with liquid clustering , optimization is not supported yet. We use this type as cluster key in Production already and can't opti...

Jennifer · 02-19-2024

I am currently trying to write a dataframe to s3 likedf.write.partitionBy("col1","col2").mode("overwrite").format("json").save("s3a://my_bucket/")The path becomes `s3a://my_bucket/col1=abc/col2=opq/`But I want to path to be `s3a://my_bucket/abc/opq/`...

Jennifer · 11-29-2023

Hello,I am running spark structured streaming, reading from one table table_1, do some aggregation and then write results to another table. table_1 is partitioned by ["datehour", "customerID"]My code is like this:spark.readStream.format("delta").tabl...

Jennifer · 03-07-2023

I followed the documentation here under the section "Import a file into a notebook" to import a shared python file among notebooks used by delta live table. But it sometimes can find the module, sometimes not and returns me exception No module named ...

Jennifer

Also, does it mean that even I specify a column of type TimestampNTZ in the clustering key, it is not clustered by this column?

Jennifer · 02-22-2024

The way I did at the end was to write files to dbfs first and then move them to s3 in order to have a customized path and file name. I could also avoid writing commit files to s3.

Jennifer · 02-19-2024

Thanks for the quick reply. If I use the new column for partitioning, "combined_col" will still be in the path I think.

Jennifer · 11-29-2023

Thanks:-)

Jennifer · 03-31-2023

It is still a problem when using DLT. But I put my functions in another notebook and schedule a job with multiple tasks, not using DLT anymore, then it works. I guess it might be a issue with DLT.

Databricks

User Stats

User Activity

Optimization failed for timestampNtz

Support for dataskipping for type TimestampNTZ

How do I write dataframe to s3 without partition column name on the path

Why does readStream filter go through all records?

Import python file to notebook doesn't work

Re: Support for dataskipping for type TimestampNTZ

Re: How do I write dataframe to s3 without partition column name on the path

Re: How do I write dataframe to s3 without partition column name on the path

Re: Why does readStream filter go through all records?

Re: Import python file to notebook doesn't work