Read metadata through JDBC driver

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Dear all,

The Spark JDBC driver (SparkJDBC42.jar) is unable to capture certain information from the below table structure:

1. table level comment

2. the TBLPROPERTIES key-value pair information

3. PARTITION BY information

However, it captures the column level comment (eg. the comment against employee_number column), all columns of employee table, their technical data types.

Please advise if I need to configure any additional properties to be ale to read/extract the information that the driver could not extract at the moment.

create table default.employee(
employee_number INT COMMENT 'Unique identifier for an employee', 
employee_name VARCHAR(50),
employee_age INT
)
PARTITIONED BY (employee_age) 
COMMENT 'this is a table level comment'
TBLPROPERTIES ('created.by.user' = 'Noor', 'created.date' = '10-08-2021');