cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Jennifer
New Contributor III
since ‎01-16-2023
Monday

User Stats

  • 15 Posts
  • 1 Solutions
  • 0 Kudos given
  • 0 Kudos received

User Activity

Can external tables be created backed by current cloud files without ingesting files in Databricks?

a week ago
Hi,We have huge amount of parquet files in s3 with the path pattern <bucket>/<customer>/yyyy/mm/dd/hh/.*.parquet.The question is can I create a external table in Unity Catalog from this external location without actually ingesting the files? Like wha...

Optimization failed for timestampNtz

04-15-2024
We have a table using timestampNtz type for timestamp, which is also a cluster key for this table using liquid clustering. I ran OPTIMIZE <table-name>, it failed with errorUnsupported datatype 'TimestampNTZType' But the failed optmization also broke ...

Support for dataskipping for type TimestampNTZ

04-11-2024
More people begin to use TimestampNTZ as cluster key.According to the thread here Unsupported datatype 'TimestampNTZType' with liquid clustering , optimization is not supported yet. We use this type as cluster key in Production already and can't opti...

How do I write dataframe to s3 without partition column name on the path

02-19-2024
I am currently trying to write a dataframe to s3 likedf.write.partitionBy("col1","col2").mode("overwrite").format("json").save("s3a://my_bucket/")The path becomes `s3a://my_bucket/col1=abc/col2=opq/`But I want to path to be `s3a://my_bucket/abc/opq/`...

Why does readStream filter go through all records?

11-29-2023
Hello,I am running spark structured streaming, reading from one table table_1, do some aggregation and then write results to another table. table_1 is partitioned by ["datehour", "customerID"]My code is like this:spark.readStream.format("delta").tabl...
View more