@Harun Raseed Basheer​ :

The issue with your code is that the groupBy operation returns a GroupedData object, which does not have a get_group method. Instead, you can use the filter method to filter the bronze_df DataFrame for each entity name and write the resulting DataFrames to separate Silver tables.

Here's an example of how you can modify your code to achieve this:

from pyspark.sql.functions import col
 
# Group the bronze DataFrame by entity_name
grouped_df = bronze_df.groupBy("entity_name")
 
# Extract the unique entity names
entity_names = [row.entity_name for row in grouped_df.agg({"entity_name": "first"}).collect()]
 
# Filter the bronze DataFrame for each entity name and write to a separate Silver table
for entity_name in entity_names:
    entity_df = bronze_df.filter(col("entity_name") == entity_name)
    entity_df.write.format("delta").mode("append").save(f"/mnt/silver/{entity_name}")