Databricks Community

Can you use override mode instead append?

A generation expression can use any SQL functions in Spark that always return the same result when given the same argument valuesSource: https://docs.delta.io/latest/delta-batch.html#use-generated-columnsIt means that it's intended to not work.You ca...

Hi @Zachary Higgins​ Thanks for the replyCurrently, we are also using Trigger.once so that we can handle the merge stream dependencies properly. But was wondering whether we can scale our pipeline to be streaming by changing the Trigger duration in t...

Just use tools like Goodsync and Gs Richcopy 360 to copy directly from blob to S3, I think you will never face problems like that ​

Databricks is a powerful platform for data engineering, machine learning, and analytics, and it is important to monitor the performance and health of your Databricks environment to ensure that it is running smoothly.Here are a few key metrics that yo...

Hi @Karthick Narendran​ I got the same error as you.I tried every branch in the locationtech repository but failed. Then luckily I found a rasterframe branch for databricks here.https://github.com/mjohns-databricks/rasterframes/tree/0.10.2-databricks...

Docs do show that it uses E8d as you wrote.SQL Warehouses are a different type of compute than All Purpose or Jobs clusters. The SQL warehouses always use Photon. All purpose and Jobs clusters are used for things such as notebooks or Delta Live Tab...

create_training_set is just a simple Select from delta tables. All feature tables are just registered delta tables. Here is an example code that I used to handle that: customer_features_df = spark.sql("SELECT * FROM recommender_system.customer_fea...