Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
ProblemI'm unable to authenticate against the https://accounts.cloud.databricks.com endpoint even though I'm an account admin. I need it to assign account level groups to workspaces via the workspace assignment api (https://api-docs.databricks.com/re...
From this doc: To automate Databricks account-level functionality, you cannot use Databricks personal access tokens. Instead, you must use either OAuth tokens for Databricks account admin users or service principals. For more information, see:Use a s...
I'm trying to login for past two days and i'm still facing this error: "We've encountered an error logging you in." I've tried to reset the password multiple times and nothing happened. My friend is also not able to login. I request you to resolve t...
I had to sign up again using the same email. In my case, the error seemed to be related to my Azure account. So, when Databricks asks you to choose your cloud provider to sync with, you should select the community version instead.
Hello All,I get the org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient while trying to create a database scr...
Alright, we've implemented a workaround for this, and so far it's been working very well:First, we created a reusable notebook to wait until Hive has been initialized (see code below).We then execute this notebook using the %run command at the top of...
Hello,I am experiencing issues with importing from utils repo the schema file I created.this is the logic we use for all ingestion and all other schemas live in this repo utills/schemasI am unable to access the file I created for a new ingestion pipe...
@Debayan Mukherjee​ Hello, thank you for your response. please let me know if these are the correct commands to access the file from notebookI can see the files in the repo folderbut I just noticed this. the file I am trying to access the size is 0 b...
Hi All,I am getting below error when trying to execute the code.AnalysisException: Column Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to ...
Hello @vijaykumarbotla , I hope you're doing well.
This is probably because both DataFrames contain a column with the same name, and Spark is unable to determine which one you are referring to in the select statement.
To resolve this issue, you can u...
ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)should one create it in the work space manually via UI? and how?would it get overwritten if work space is created via terraform?I use 10.4 LTS runtime.
I am experiencing significant delay on my streaming. I am using changefeed connector. Its processing streaming batch very frequently but experiencing sudden halt and shows no active stage for longer time. I observed below exception continuously promp...
I'm trying to create an external location from notebook, and I got this kind of error[PARSE_SYNTAX_ERROR] Syntax error at or near 'LOCATION'(line 1, pos 16)
== SQL ==
CREATE EXTERNAL LOCATION IF NOT EXISTS test_location URL 's3://test-bronze/db/tes...
Hey everyone,I'm facing an issue with retrieving data from a volume or table that contains a string with a symbol, for example, 'databricks+'. Whenever I try to retrieve this data, I encounter a syntax error. Can anyone help me resolve this issue?
I am getting an error every time I try to view another branch or create a branch. Sometimes this has happened in the past, but usually seems to fix itself after about 10-30 minutes. This error has been lasting for over 12 hours, so I am now concerned...
I had the same problem and I could resolve it by creating the repo again with a trailing ".git" in the Git repository URL.For example, use thishttps://gitlab.mycompany.com/my-project/my-repo.gitnot this:https://gitlab.mycompany.com/my-project/my-repo...
I would like to know why I am getting this error when I tried to earn badges for lakehouse fundamentals. I can't access the quiz page. Can you please help on this?
I am trying to get the Bigquery data to Databricks using Notebooks. Following the steps based on this https://docs.databricks.com/external-data/bigquery.html. I believe I am making some mistake with this step and getting the below error. I tried givi...
I had the same issue, thanks for the info! Apparently it's also possible to fix it by removing all the actual notification in the interface (the bugged one is not displayed, but if you remove everything for some reason it removes the bugged one too)....
What is the problem?I am getting this error every time I run a python notebook on my Repo in Databricks.BackgroundThe notebook where I am getting the error is a notebook that creates a dataframe and the last step is to write the dataframe to a Delta ...
Hi @Sara Corral​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'
Hiyou can try: my_df = spark.read.format("csv") .option("inferSchema","true") # to get the types from your data .option("sep",",") # if your file is using "," as separator .option("header","true") # if you...
I am trying to run a simple training script using HF's transformers library and am running into the error `Distributed package doesn't have nccl built in` error.Runtime: DBR 13.0 ML - SPark 3.4.0 - Scala 2.12Driver: i3.xlarge - 4 coresNote: This is a...
Hi @anastassia_kor1,For CPU-only training, TrainingArguments has a no_cuda flag that should be set.For transformers==4.26.1 (MLR 13.0) and transformers==4.28.1 (MLR 13.1), there's an additional xpu_backend argument that needs to be set as well. Try u...
Cloudformation runs into error during 'assign Metastore' step.Following is the error from Cloudwatch:HTTP content: b'{ "error_code": "PERMISSION_DENIED", "message": "Cannot assign metastore to STANDARD tier workspace xyz" }What is the solution for th...
Hi @Abhilash Mhaisne​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
