cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
inpefess
by • New Contributor II
  • 2234 Views
  • 4 replies
  • 3 kudos

Does Databricks need GCP VMs for a workspace with no clusters in it?

Hi! I'm using GCP. Does Databricks workspace always need two e2-highmem-2 instances running as soon as I create a workspace? I seem them in my VM list in GCP console no matter what (I can stop or remove a cluster, but these two machines are always th...

Data Engineering
Reply
  • 2234 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
abagshaw
New Contributor III
  • 3 kudos

To clarify, on Databricks on GCP will automatically delete the underlying GKE after 5 days of inactivity (no cluster launches or non-empty instance pools) in the workspace. You can contact Databricks support if you want to shorten the idle TTL for th...

  • 3 kudos
3 More Replies
boyelana
by • Contributor III
  • 2617 Views
  • 3 replies
  • 7 kudos

Resolved! How to start with Databricks in Google Cloud?

I am looking through Google Cloud Platform and I am looking to get started with Databricks on GCP. Happy if anyone can point me in the direction that can provide guidance on how to get started.Thansk

Data Engineering
Reply
  • 2617 Views
  • 3 replies
  • 7 kudos
View Replies
Latest Reply
martinez
New Contributor III
  • 7 kudos

Hey boyelana Databricks on Google Cloud Platform is definitely an interesting and powerful combination, and I'm thrilled to see that you're looking to get started with it, boyelana!To begin your journey with Databricks on GCP, there are a few steps y...

  • 7 kudos
2 More Replies
jdobken
by • New Contributor III
  • 8479 Views
  • 8 replies
  • 11 kudos

As the Databricks account manager; I cannot login: "Your user already belongs to a Databricks account"

On GCP I subscribed to Databricks in one project within the organization.Then I canceled this subscription and subscribed to Databricks in another project.When I try to login onto the newly subscribed databricks with google SSO:> There was an error s...

Screenshot 2023-06-07 at 11.32.30
Data Engineering
Reply
  • 8479 Views
  • 8 replies
  • 11 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 11 kudos

I can see the issue might be related to organizations or billing accounts. The new Databricks project I tried creating was on a different organization/billing-account than the test Databricks subscription I created a month back.I went back to the ori...

  • 11 kudos
7 More Replies
lugger1
by • New Contributor III
  • 2925 Views
  • 1 replies
  • 1 kudos

Resolved! What is the best way to use credentials for API calls from databricks notebook?

Hello, I have an Databricks account on Azure, and the goal is to compare different image tagging services from Azure, GCP, AWS via corresponding API calls, with Python notebook. I have problems with GCP vision API calls, specifically with credentials...

Data Engineering
Reply
  • 2925 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
lugger1
New Contributor III
  • 1 kudos

Ok, here is a trick: in my case, the file with GCP credentials is stored in notebook workspace storage, which is not visible to os.environ() command. So solution is to read a content of this file, and save it to the cluster storage attached to the no...

  • 1 kudos
jose_herazo
by • New Contributor III
  • 3293 Views
  • 5 replies
  • 5 kudos

Databricks doesn't stop compute resources in GCP

I started using Databricks in Google Cloud but it charges some unexpected costs. When I create a cluster I notice some compute resources being created in GCP but when I stop the cluster these resources are still up and never shut down. This issue res...

Data Engineering
Reply
  • 3293 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
antquinonez
New Contributor II
  • 5 kudos

The answer to the question about the kubernetes cluster regardless of dbx compute and dwh resources running is provided in this thread: https://community.databricks.com/s/question/0D58Y00009TbWqtSAF/auto-termination-for-clusters-jobs-and-delta-live-t...

  • 5 kudos
4 More Replies
Fred_F
by • New Contributor III
  • 7322 Views
  • 5 replies
  • 5 kudos

JDBC connection timeout on workflow cluster

Hi there,​I've a batch process configured in a workflow which fails due to a jdbc timeout on a Postgres DB.​I checked the JDBC connection configuration and it seems to work when I query a table and doing a df.show() in the process and it displays th...

Data Engineering
Reply
  • 7322 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
RKNutalapati
Valued Contributor
  • 5 kudos

HI @Fred Foucart​ ,The above code looks good to me. Can you try with below code as well.spark.read\  .format("jdbc") \  .option("url", f"jdbc:postgresql://{host}/{database}") \  .option("driver", "org.postgresql.Driver") \  .option("user", username) ...

  • 5 kudos
4 More Replies
ranged_coop
by • Valued Contributor II
  • 1717 Views
  • 2 replies
  • 3 kudos

Equivalent Machine Types between Databricks on Azure and GCP

Hi All,Hope everyone is doing well.We are currently validating Databricks on GCP and Azure.We have a python notebook that does some ETL (Copy, extract zip files and process files within the zip files)Our Cluster Config on AzureDBX Runtime - 10.4 - Dr...

Data Engineering
Reply
  • 1717 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
ranged_coop
Valued Contributor II
  • 3 kudos

hi @Tunde Abib​ , I have gone through the links while updating, but did not see any major documented slow downs mentioned in them. 

  • 3 kudos
1 More Replies
Tahseen0354
by • Valued Contributor
  • 10150 Views
  • 13 replies
  • 35 kudos

How do I compare cost between databricks gcp and azure databricks ?

I have a databricks job running in azure databricks. A similar job is also running in databricks gcp. I would like to compare the cost. If I assign a custom tag to the job cluster running in azure databricks, I can see the cost incurred by that job i...

Data Engineering
Reply
  • 10150 Views
  • 13 replies
  • 35 kudos
View Replies
Latest Reply
Own
Contributor
  • 35 kudos

In Azure, you can use Cost Management to track your expenses incurred by Databricks instance.

  • 35 kudos
12 More Replies
stupendousenzio
by • New Contributor III
  • 2252 Views
  • 4 replies
  • 7 kudos

Unable to access workspace after the trial period in databricks in Google cloud provider.

I was using the trial period in databricks for 14 days and had some important notebooks where I had made all the changes. Now I have extended the service and have subscribed for databricks in GCP. When I enter the workspace section I cannot see the w...

Data Engineering
Reply
  • 2252 Views
  • 4 replies
  • 7 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @Aditya Aranya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 7 kudos
3 More Replies
horatiug
by • New Contributor III
  • 4247 Views
  • 8 replies
  • 3 kudos

Create workspace in Databricks deployed in Google Cloud using terraform

In the documentation https://registry.terraform.io/providers/databricks/databricks/latest/docs https://docs.gcp.databricks.com/dev-tools/terraform/index.html I could not find documentation on how to provision Databricks workspaces in GCP. Only cre...

Data Engineering
Reply
  • 4247 Views
  • 8 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @horatiu guja​ Does @Debayan Mukherjee​ response answer your question?If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else, we can help you with more details.

  • 3 kudos
7 More Replies
elementalM
by • New Contributor III
  • 2861 Views
  • 5 replies
  • 0 kudos

Resolved! GCP auth time out in long running databricks job

I'm wondering if you can help me with a google auth issue related to structured streaming and long running databricks jobs in general. I will get this error after running for 8+ hours. Any tips on this? GCP auth issues for long running jobs?Caused by...

Data Engineering
Reply
  • 2861 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
elementalM
New Contributor III
  • 0 kudos

Thanks, yes this seems to be the best work around - the good ole retry on fail. Thanks for the help.

  • 0 kudos
4 More Replies
syedmuhammedmeh
by • New Contributor III
  • 2773 Views
  • 2 replies
  • 6 kudos

Resolved! Databricks Kafka Read Not connecting

I'm trying to read data from GCP kafka through azure databricks but getting below warning and notebook is simply not completing. Any suggestion please? WARN NetworkClient: Consumer groupId Bootstrap broker rack disconnectedPlease note I've properly c...

Data Engineering
Reply
  • 2773 Views
  • 2 replies
  • 6 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 6 kudos

Could you share the full error stack trace from your driver's logs? This is a Warning message, we need to take a look at the error level messages.

  • 6 kudos
1 More Replies
leon
by • New Contributor II
  • 1708 Views
  • 2 replies
  • 1 kudos

Error when creating a new workspace in GCP

Hello, We have several workspaces in GCP and want to create another one in another region. For some reason, after we enter all GKE IP ranges, we got a BAD_REQUEST error that applies that it couldn't get our oath token.We tried to login out and in aga...

detabrics-error
Data Engineering
Reply
  • 1708 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Leon Bam​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
1 More Replies
VikasSinha
by • New Contributor
  • 4803 Views
  • 2 replies
  • 0 kudos

Which is better - Azure Databricks or GCP Databricks?

Which cloud hosting environment is best to use for Databricks? My question pins down to the fact that there must be some difference between the latency, throughput, result consistency & reproducibility between different cloud hosting environments of ...

Data Engineering
Reply
  • 4803 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Vikas Sinha​ Does @Prabakar Ammeappin​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
ankit_k
by • New Contributor
  • 1445 Views
  • 3 replies
  • 0 kudos

Move a GCP Project with Databricks in it to new Organization

We are trying to move a GCP project to a Newly created Org and new billing account. We have a Databricks instance from GCP Marketplace with licensing As per the docs when we change a billing account for a Project the license on the first billing acco...

Data Engineering
Reply
  • 1445 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Ankit K​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best?  Else please let us know if you need more help. We'd love to hear from you.Cheers!

  • 0 kudos
2 More Replies
Labels