Reading bigquery data using a query

naga_databricks — Wed, 23 Aug 2023 09:59:30 GMT

To read Bigquery data using spark.read, i'm using a query. This query executes and creates a table on the materializationDataset.


df = spark.read.format("bigquery") \
.option("query", query) \
.option("materializationProject", materializationProject) \
.option("materializationExpirationTimeInMinutes", materializationExpirationTimeInMinutes) \
.option("materializationDataset", materializationDataset) \
.load()
df.write.mode("overwrite").format("delta").saveAsTable(deltaTableName)

This documentation on databricks says, the spark.read will create a Materialized View, however i see a table. Is the documentation accurate? Or is there a way to select if spark can create a table or a materialized view?

Re: Reading bigquery data using a query

naga_databricks — Fri, 25 Aug 2023 13:30:56 GMT

Thanks Kaniz for the explanation. It sounds good to me that i'm working with tables here.

For me the only cause of confusion was the underlined writing:

topic Reading bigquery data using a query in Data Engineering

Reading bigquery data using a query

Re: Reading bigquery data using a query