Hi Joao,

You’re right!—if we let Spark or Autoloader define the schema once and save it for validation, we’re still relying on inference initially. But the key difference is that we use inference just once, store the schema explicitly (as JSON or in a Delta table), and then enforce it going forward instead of letting Spark decide every time. This prevents unexpected type changes and keeps things under control. Storing everything as strings in Bronze works for flexibility, but it pushes complexity to Silver, making transformations trickier. Instead, I prefer defining schemas early, so I catch issues before they cause downstream problems. In my setup, I store schemas as JSON, load them dynamically, and apply them while reading data—this way, I don’t have to manually write schemas for every table. I also validate the schema before writing to Delta to catch mismatches early. Below is simple code snippet that shows how I do this.

Store Schema in JSON

{
"name": "customers",
"columns": [
{"name": "id", "type": "integer", "nullable": false},
{"name": "name", "type": "string", "nullable": false},
{"name": "email", "type": "string", "nullable": true},
{"name": "signup_date", "type": "date", "nullable": true}
]
}

Read Schema from JSON and Convert to StructType

import json
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DateType

# Define a mapping from JSON data types to Spark data types
type_mapping = {
"string": StringType(),
"integer": IntegerType(),
"date": DateType()
}

# Load the schema JSON file from storage (S3, ADLS, or DBFS)
schema_path = "s3://my-bucket/schema/customers_schema.json"
schema_json = spark.read.text(schema_path).collect()[0][0] # Read JSON as a string
schema_dict = json.loads(schema_json) # Convert JSON string to a Python dictionary

# Convert JSON schema to PySpark StructType
def json_to_spark_schema(schema_dict):
return StructType([
StructField(col["name"], type_mapping[col["type"]], col["nullable"])
for col in schema_dict["columns"]
])

schema = json_to_spark_schema(schema_dict)

Read Data with the Enforced Schema

df = spark.read.schema(schema).json("s3://my-bucket/raw/customers/")

df.show()
df.printSchema()

Validate Schema Before Writing to Delta

# Extract expected schema fields
expected_schema = set([f"{field.name}:{field.dataType}" for field in schema.fields])
incoming_schema = set([f"{field.name}:{field.dataType}" for field in df.schema.fields])

# Check for mismatches
if expected_schema != incoming_schema:
raise ValueError("Schema mismatch detected! Check your source data.")

# Write to Delta if schema is correct
df.write.format("delta").mode("append").saveAsTable("gold.customers")

Give a try and let me know for any additional queries.

Regards,

Brahma