Walter_C
Databricks Employee
Databricks Employee

Unfortunately, there isn't a built-in Spark SQL method to directly use a schema file when creating views or tables.

You could try to create a custom SQL function that reads the schema file and returns it as a string. Then use this function in your SQL commands.

  1. First, create a custom UDF (User Defined Function) in Scala or Python that reads the schema file:

    import org.apache.spark.sql.SparkSession
    import org.apache.spark.sql.functions.udf
    
    def readSchemaFile(path: String): String = {
      // Read the schema file and return its contents as a string
      scala.io.Source.fromFile(path).getLines().mkString
    }
    
    val spark = SparkSession.builder().getOrCreate()
    spark.udf.register("read_schema_file", readSchemaFile _)


  2. Then use this function in your SQL command:
    CREATE OR REPLACE VIEW entity_view AS
    SELECT *
    FROM read_files(
      'dbfs:/mnt/.../entity.*json',
      format => 'json',
      schema => read_schema_file('/path/to/schema/entity.json'),
      rescuedDataColumn => '_rescued_data'
    )

 

View solution in original post