topic What is a good way to ingest Google Analytics data into Databricks in Data Engineering

What is a good way to ingest Google Analytics data into Databricks

aladda — Fri, 18 Jun 2021 19:02:17 GMT

Re: What is a good way to ingest Google Analytics data into Databricks

Digan_Parikh — Tue, 22 Jun 2021 18:01:42 GMT

You can use a ETL tool that connects to GA and puts it into Databricks. You can also do it yourself - use the GA APIs via python and connect. This can be though as you need to set everything up yourself (API Oauth 2.0, different api endpoints, parse json etc).

Re: What is a good way to ingest Google Analytics data into Databricks

aladda — Wed, 23 Jun 2021 03:06:22 GMT

Thanks @Digan Parikh . Credit to Tahir Fayyaz, Found a couple of different paths depending on whether you're looking to bring in raw GA data vs aggregated GA data.

1) For Raw You can bring in data from GA Universal Analytics 360 Paid version or GA v4 Free and GA v4 360 (Paid) versions first into Big Query and then use either the Spark Bigquery Connector to bring that data in Delta Lake (on any cloud provider) or Land that data into GCS and then use AutoLoader on GCS to bring that into Delta

2) For aggregated GA data you can either use the GA Reporting API or the GA Data API and then bring that data in either via DB Scheduled Python Notebook Job or Partner ETL tools into Delta