Reading bigquery data using a query
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-23-2023 02:59 AM
To read Bigquery data using spark.read, i'm using a query. This query executes and creates a table on the materializationDataset.
df = spark.read.format("bigquery") \
.option("query", query) \
.option("materializationProject", materializationProject) \
.option("materializationExpirationTimeInMinutes", materializationExpirationTimeInMinutes) \
.option("materializationDataset", materializationDataset) \
.load()
df.write.mode("overwrite").format("delta").saveAsTable(deltaTableName)
This documentation on databricks says, the spark.read will create a Materialized View, however i see a table. Is the documentation accurate? Or is there a way to select if spark can create a table or a materialized view?
Labels:
- Labels:
-
Spark
1 REPLY 1
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-25-2023 06:30 AM
Thanks Kaniz for the explanation. It sounds good to me that i'm working with tables here.
For me the only cause of confusion was the underlined writing:

