topic Re: Data Ingestion from GCP in Data Engineering

Data Ingestion from GCP

delson — Tue, 03 Sep 2024 11:03:04 GMT

Hi,

I'm ingesting data from GCP to Databricks, and I think I've noticed a bug in that any datatables that have a numerical starting character are not ingested at all.

Has anyone else experienced this?

Please let me know if there is a way around this apart from re-naming the datatables (I have many)

Re: Data Ingestion from GCP

szymon_dybczak — Tue, 03 Sep 2024 11:14:01 GMT

Hi @delson ,

Could you give us some examples? And when you're saying GCP you mean bigquery?

Re: Data Ingestion from GCP

delson — Tue, 03 Sep 2024 11:37:59 GMT

Hi Slash, thanks for getting back to me

So for instance - I have data tables such as "20240901_demographics_data_v1" which I'm trying to move from BQ

Other data tables that don't include a date (or other numerical characters) at the front are being ingested but datatables with the date naming convention at the start are not being brought through

Re: Data Ingestion from GCP

szymon_dybczak — Tue, 03 Sep 2024 11:44:09 GMT

Hi @delson ,

Thanks for clarification. Could you try to escape problematic table name using backtick character?

`20240901_demographics_data_v1`

Re: Data Ingestion from GCP

delson — Tue, 03 Sep 2024 14:42:44 GMT

Hi Slash - Thanks for your response

Unfortunately, due to BQ's capabilities I'm not able to insert backticks to data table names without manually changing each of the hundreds of data tables I have

Currently, I'm ingesting the data in an automated fashion using a BQ connection through Databricks in the Catalog / Catalog Explorer

Nonetheless, thanks - hopefully another option is possible 🙂