cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

merca
by Valued Contributor II
  • 5360 Views
  • 14 replies
  • 4 kudos

Resolved! ⬆ Bump IPython to 7.31.1

Any plans to bump IPython version to 7.31.1 on the DBR 9.1 LTS runtime? If no other motivation

  • 5360 Views
  • 14 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Merca Ovnerud​ , We haven't heard from you since my last response, and I was checking to see if my suggestions helped you. Also, please don't forget to click the "Select as Best" button" whenever the information provided helps resolve your questi...

  • 4 kudos
13 More Replies
Kaniz_Fatma
by Community Manager
  • 942 Views
  • 2 replies
  • 1 kudos

What are the best practices for the isolation of different environments in Databricks? I am trying to find out the best practice around Databricks env...

What are the best practices for the isolation of different environments in Databricks?I am trying to find out the best practice around Databricks environments creation like - dev, stage, and prod. Should it be:1. Single Databricks account with multip...

  • 942 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey @Abhijit Rai​ Does @Atanu Sarkar​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?Thank you!

  • 1 kudos
1 More Replies
GeorgeP
by New Contributor II
  • 1368 Views
  • 2 replies
  • 2 kudos

Errors when querying Azure DataBricks through DBeaver on macos

Configured DBeaver to work with either databricks latest driver or simba. I can connect and see databases, schemas, tables and columns. However, when a select statement is executed 30-40 seconds go by before I get the following error message: SQL...

  • 1368 Views
  • 2 replies
  • 2 kudos
Latest Reply
sage5616
Valued Contributor
  • 2 kudos

Has this issue been resolved? @aravhish solution did not help me. Any other options?I am experiencing the exact same issue with the same configuration on a Mac. Much help would be appreciated.

  • 2 kudos
1 More Replies
Ignacio33
by New Contributor II
  • 1158 Views
  • 2 replies
  • 1 kudos

"Backend services unavailable" when creating a default cluster

Hello all, I have been not using databricks CE for 4-5 months but today I've tried to create the default cluster, as always did, and I've got the error "Backend services unavailable". Is this a temporary problem or am I doing something wrong. Thanks ...

  • 1158 Views
  • 2 replies
  • 1 kudos
Latest Reply
tomasz
New Contributor III
  • 1 kudos

There's currently an outage of the community edition. Please follow this link for status:https://status.databricks.com/pages/incident/5cf02dde58a00904bda41926/62cd7677db464e053416a89c

  • 1 kudos
1 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 753 Views
  • 1 replies
  • 20 kudos

My favorite feature announced at Data+AI Summit is Connect from everywhere. I have been dreaming for a long time about sending SQL queries to SQL endp...

My favorite feature announced at Data+AI Summit is Connect from everywhere. I have been dreaming for a long time about sending SQL queries to SQL endpoint via API/SDK

connected_from_anywhere (1)
  • 753 Views
  • 1 replies
  • 20 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 20 kudos

Awesome!Keep Learning @Hubert Dudek​ !

  • 20 kudos
harish_s
by New Contributor II
  • 4488 Views
  • 5 replies
  • 5 kudos

Resolved! Hi, I get the following error when I enable model serving for spacy model via MLFLOW.

+ echo 'GUNICORN_CMD_ARGS=--timeout 63 --workers 4 'GUNICORN_CMD_ARGS=--timeout 63 --workers 4 + mlflow models serve --no-conda -m /tmp/tmp1a4ltdrk/spacymodelv1 -h unix:/tmp/3.sock -p12022/03/01 08:26:37 INFO mlflow.models.cli: Selected backend for f...

  • 4488 Views
  • 5 replies
  • 5 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 5 kudos

Hi @Harish S​ , We haven't heard from you on the last response from @Prabakar Ammeappin​ , me, and @Jose Gonzalez​ , and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can b...

  • 5 kudos
4 More Replies
Bhanu1
by New Contributor III
  • 3759 Views
  • 4 replies
  • 6 kudos

Resolved! Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace?

We have a development and a production data lake. Is it possible to have a production or development cluster access only respective mounts using init scripts?

  • 3759 Views
  • 4 replies
  • 6 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 6 kudos

Hi @Bhanu Patlolla​ â€‹, We haven’t heard from you on the last response from @Werner Stinckens​ and @Hubert Dudek​  and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be h...

  • 6 kudos
3 More Replies
CBull
by New Contributor III
  • 2121 Views
  • 4 replies
  • 2 kudos

Is there a way in Azure to compare data in one field?

Is there a way to compare a time stamp within on field/column for an individual ID? For example, if I have two records for an ID and the time stamps are within 5 min of each other....I just want to keep the latest. But, for example, if they were an h...

  • 2121 Views
  • 4 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Cory Bullard​, We haven't heard from you on the last response from @Merca Ovnerud​, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwis...

  • 2 kudos
3 More Replies
Judha2022
by New Contributor III
  • 1980 Views
  • 4 replies
  • 2 kudos
  • 1980 Views
  • 4 replies
  • 2 kudos
Latest Reply
Judha2022
New Contributor III
  • 2 kudos

Could you please let me know when it is available? It is critically important for me to get Databricks CE.Thanks again for your reply.

  • 2 kudos
3 More Replies
JohanRex
by New Contributor II
  • 4759 Views
  • 4 replies
  • 5 kudos

Resolved! IllegalArgumentException: requirement failed: Result for RPC Some(e100cace-3836-4461-8902-80b3744fcb6b) lost, please retry your request.

I'm using databricks connect to talk to a cluster on Azure. When doing a count on a dataframe I sometimes get this error message. Once I've gotten it once I don't seem to be able to get rid of it even if I restart my dev environment. ----------------...

  • 4759 Views
  • 4 replies
  • 5 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 5 kudos

Hi @Johan Rex​ , We haven't heard from you on my last response, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with ...

  • 5 kudos
3 More Replies
Rex
by New Contributor III
  • 4683 Views
  • 6 replies
  • 3 kudos

Resolved! Cannot use prepared statements with date functions

We are using PHP and the Databricks SQL ODBC driver and cannot run a query that users DATE functions with prepared statements. Sample script/docker setup here:https://github.com/rlorenzo/databricks_php/blob/odbc_prepare_error/test_connection.phpFor e...

  • 4683 Views
  • 6 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Rex Lorenzo​ , We haven't heard from you on the last response from @Bilal Aslam​ , and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others....

  • 3 kudos
5 More Replies
Mohit_m
by Valued Contributor II
  • 12984 Views
  • 1 replies
  • 1 kudos

Resolved! Job is failing with exception ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.

ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.Attempted credentials:EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.V...

  • 12984 Views
  • 1 replies
  • 1 kudos
Latest Reply
Mohit_m
Valued Contributor II
  • 1 kudos

Below docs are for reference:https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/identity/azure-identity/migration_guide.mdthere was a suggestion given to usefrom azure.common.credentials import ServicePrincipalCredentialsinstead offrom azure...

  • 1 kudos
Users-all
by New Contributor
  • 2251 Views
  • 2 replies
  • 1 kudos

Resolved! xml module not found error

ModuleNotFoundError: No module named 'com.databricks.spark.xml'I'm using Azure databricks, and I've added what I think is the correct library, Status InstalledCoordinatecom.databricks:spark-xml_2.12:0.13.0

  • 2251 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Christian Hamson​ , We haven't heard from you since my last response, and I was checking to see if my suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others. Also, please don't...

  • 1 kudos
1 More Replies
Christina_Marx
by New Contributor
  • 1465 Views
  • 2 replies
  • 1 kudos

Resolved! New LMS: What happened to the Developer Badges and Capstones which were in the old portal?

I was still busy with the Developer Foundations capstone. I received the Developer essentials badge, and was going to do the ILT and capstone for the Solutions Architect Badge in Jan 2022. Then we got the notice about the Academy migration, and I dec...

  • 1465 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Christina Marx​ , We haven't heard from you since my last response, and I was checking back to see if my suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others. Also, please do...

  • 1 kudos
1 More Replies
lsoewito
by New Contributor
  • 5077 Views
  • 3 replies
  • 3 kudos

Resolved! How to configure Databricks Connect to 'Assume Role' when accessing file from an AWS S3 bucket?

I have a Databricks cluster configured with an instance profile to assume role when accessing an AWS S3 bucket.Accessing the bucket from the notebook using the cluster works properly (the instance profile can assume role to access the bucket).However...

  • 5077 Views
  • 3 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @lsoewito​ , We haven't heard from you since my last response, and I was checking back to see if my suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others. Also, please don't fo...

  • 3 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels