topic How can I create a new calculated field in databricks by using pyspark. in Data Engineering

How can I create a new calculated field in databricks by using pyspark.

kazinahian — Wed, 20 Sep 2023 21:21:08 GMT

Hello:

Great people. I am new to Databricks and pyspark learning. How can I create a new column called "sub_total"? Where I want to group by "category" "subcategory" and "monthly" sales value.

Appreciate your empathic solution.

Re: How can I create a new calculated field in databricks by using pyspark.

Miguel_Suarez — Thu, 23 Jan 2025 04:31:49 GMT

Hi @kazinahian,

I believe what you're looking for is the .withColumn() Dataframe method in PySpark. It will allow you to create a new column with aggregations on other columns: https://docs.databricks.com/en/pyspark/basics.html#create-columns

Best

Re: How can I create a new calculated field in databricks by using pyspark.

NandiniN — Sat, 01 Feb 2025 07:29:37 GMT

I want to group by "category" "subcategory" and "monthly" sales value.

sub_total_df = df.groupBy("category", "subcategory", "monthly").agg(sum("sales_value").alias("sub_total"))

You could always type in your query in the Databricks notebook, by clicking on the generate link in cell, which will help you with Databricks Assistant.