Databricks Community

KristiLogos

I've used Airflow which allows us to create a DAG with dynamic tasks, for example we can have a list of items (such as table names), loop through an operator that accepts a table name and create a task for each table without having to create a new n...

KristiLogos

I’m running Databricks on Azure and trying to read a CSV file from Google Cloud Storage (GCS) bucket using Spark. However, despite configuring Spark with a Google service account key, I’m encountering the following error:Error getting access token fr...

KristiLogos

Hello, I have a federated connection to BigQuery that has GA events tables for each of our projects. I'm trying to query each daily table which contains about 400,000 each day, and load into another table, but I keep seeig this Simba JDBC exception. ...

KristiLogos · 01-06-2025

I was able to connect our Bigquery account to our Databricks catalog. However, all the keys in the nested dictionary columsn populate as 'v'. For example:{"v":[{"v":{"f":[{"v":"engagement_time_msec"},{"v":{"f":[{"v":null},{"v":"2"},{"v":null},{"v":nu...

KristiLogos · 11-07-2024

I'm trying to connect to Salesforce in databricks, I'm following this:https://learn.microsoft.com/en-us/azure/databricks/query-federation/salesforce-data-cloud#sql-1and when I run the "Create Catalog..." I see this error, how would I enable salesforc...

KristiLogos

I figured it out! I hope this helps someone else, I had to update the spark configuration in the cluster I was using, update with this:https://docs.databricks.com/en/connect/storage/gcs.html#global-configuration

KristiLogos

@kiko_roy unfortunately that didnt' work. the error is stating its trying to get the access token from metadata server, I wonder why from the metadata server?

KristiLogos

DId you find out why this was happening? I'm having the same issue

KristiLogos

@Alberto_Umana In addition to my last comment:For adjusting the spark.sql.shuffle.partitions and spark.executor.memory, I tried this but I was still seeing the same error spark = ( SparkSession.builder .appName("GA4 Bronze Table Ingestion") ...

KristiLogos

@Alberto_Umana My cluster's JDBC URL shows: 2.6.25 or laterAlso, where would I adjust the spark.sql.shuffle.partitions and spark.executor.memory? in the notebook?

Databricks Community

User Stats

User Activity

Making dynamic tasks like in Airflow, but in Databricks?

GCS Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/in

Simba JDBC Exception When Querying Tables via BigQuery Databricks Connection

Connection from BigQuery to Databricks populating dictionary keys as "v"

Connection type 'SALESFORCE' is not enabled. Please enable the connection to use it.

Re: GCS Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v

Re: Getting access token error when connecting from azure databricks to GCS bucket

Re: Getting access token error when connecting from azure databricks to GCS bucket

Re: Simba JDBC Exception When Querying Tables via BigQuery Databricks Connection

Re: Simba JDBC Exception When Querying Tables via BigQuery Databricks Connection