cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Aakash_Bhandari
by New Contributor III
  • 8990 Views
  • 6 replies
  • 2 kudos

Resolved! Accessing a FastAPI endpoint using Personal Access Token (PAT)

Hello Community,I have a FastAPI endpoint on a cluster with addess 0.0.0.0:8084/predict. And I want to send a request to this endpoint from a React App which is locally hosted on my computer. I have a Personal access token for the workspace but dont ...

  • 8990 Views
  • 6 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Aakash Bhandari​ :To send a request from a React App to a FastAPI endpoint on a Databricks cluster using a Personal Access Token (PAT), you can use the requests module in Python to make HTTP requests.Here's an example of how to use requests to send ...

  • 2 kudos
5 More Replies
Anonymous
by Not applicable
  • 3427 Views
  • 1 replies
  • 1 kudos

"[PARSE_SYNTAX_ERROR] Syntax error at or near 'ROW'(line 2, pos 4)".

Alter table <TABLE_NAME> SET ROW FILTER <func_name> on (COLUMN)Got the below error while running the below code "[PARSE_SYNTAX_ERROR] Syntax error at or near 'ROW'(line 2, pos 4)". Please help on this issue. we tried this code as part of access polic...

Image
  • 3427 Views
  • 1 replies
  • 1 kudos
Latest Reply
Rajeev45
Databricks Employee
  • 1 kudos

Hello,Please can you confirm which DBR version are you using & do you use unity catalog?

  • 1 kudos
AEW
by New Contributor II
  • 22957 Views
  • 8 replies
  • 2 kudos

Resolved! Help with Parameters in Databricks SQL

I am following the documentation on query parameters (https://docs.databricks.com/sql/user/queries/query-parameters.html) and it's not working for me. I can't get a parameter inserted at the text caret and I can't get cmd + p to work. I've changed sp...

  • 22957 Views
  • 8 replies
  • 2 kudos
Latest Reply
labromb
Contributor
  • 2 kudos

I don't see the need for spaces around the parameter name... I have just been typing {{param_name}} and it appears automatically in SQL editor

  • 2 kudos
7 More Replies
arw1070
by New Contributor II
  • 3167 Views
  • 2 replies
  • 0 kudos

Downstream delta live table is unable to read data frame from upstream table

I have been trying to work on implementing delta live tables to a pre-existing workflow. Currently trying to create two tables: appointments_raw and notes_raw, where notes_raw is "downstream" of appointments_raw. Following this as a reference, I'm at...

image.png
  • 3167 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Anna Wuest​ : Could you please send me the code snippet here? Thanks.

  • 0 kudos
1 More Replies
jose_gonzalez
by Databricks Employee
  • 3652 Views
  • 2 replies
  • 3 kudos

NoSuchObjectException(message:There is no database named global_temp)

I can see the following error message in my driver logs. whats does it means? and how to solve it?ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)

  • 3652 Views
  • 2 replies
  • 3 kudos
Latest Reply
source2sea
Contributor
  • 3 kudos

should one create it in the work space manually via UI? would it get overwritten if work space is created via terraform?I use 10.4 LTS runtime.

  • 3 kudos
1 More Replies
akanksha_gupta
by New Contributor II
  • 3058 Views
  • 2 replies
  • 0 kudos

ERROR : Failure starting repl. Try detaching and re-attaching the notebook. Getting this error when running any python command in 10.4LTS cluster configured with https://github.com/mspnp/spark-monitoring to send databricks spark logs to Log Analytics.

ERROR Description :java.lang.Exception: Cannot run program &quot;/local_disk0/pythonVirtualEnvDirs/virtualEnv-5acc1ea9-d03f-4de3-b76b-203d42614000/bin/python&quot; (in directory &quot;.&quot;): error=2, No such file or directory at java.lang.ProcessB...

  • 3058 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Akanksha Gupta​ :The error message suggests that the Python executable file specified in the configuration of the Databricks cluster cannot be found or accessed. Specifically, it seems that the Python executable file at the path "/local_disk0/python...

  • 0 kudos
1 More Replies
Raghu1216
by New Contributor II
  • 2511 Views
  • 3 replies
  • 0 kudos

Issue withpassing parameters to the queries in spark sql temporary function

I have created a function like belowcreate function test(location STRING, designation STRING, name STRING)RETURNS TABLE (cnt INT)RETURN(SELECT CASE WHEN location = 'INDIA' THEN (SELECT COUNT(*) FROM tbl_customers where job_role = design...

  • 2511 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Raghu Dandu​ :The error message suggests that the column "designation" does not exist in the table "tbl_customers". There could be several reasons for this error, such as a typo in the column name, a missing or deleted column, or a difference in the...

  • 0 kudos
2 More Replies
Anonymous
by Not applicable
  • 16648 Views
  • 2 replies
  • 7 kudos

&#xd83d;&#xdce3; &#xd83d;&#xdce3; Weekly Raffle to Win Ticket to Data + AI Summit 2023 &#xd83d;&#xdce3; &#xd83d;&#xdce3;  NO PURCHASE NECESSARY TO ENTER OR WIN. A PURCHASE OF ANY...

Weekly Raffle to Win Ticket to Data + AI Summit 2023 NO PURCHASE NECESSARY TO ENTER OR WIN. A PURCHASE OF ANY KIND WILL NOT INCREASE YOUR CHANCES OF WINNING. VOID WHERE PROHIBITED.We are giving away one ticket to Data + AI Summit 2023 every week ...

  • 16648 Views
  • 2 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

@Fjoraldo Mamutaj​ , @Aviral Bhardwaj​ and @Shubham Soni​ â€‹ : we have emailed you about some clarifications on your week 1 participation . Request you to reply to my email

  • 7 kudos
1 More Replies
Anonymous
by Not applicable
  • 1815 Views
  • 2 replies
  • 0 kudos

Tuesday get to know each other series: Hey everyone! I&#39;m curious to know what tech books you&#39;ve been reading lately. As a community of tech en...

Tuesday get to know each other series:Hey everyone! I'm curious to know what tech books you've been reading lately. As a community of tech enthusiasts, I'm sure we all have some great recommendations to share with each other.To get the ball rolling, ...

  • 1815 Views
  • 2 replies
  • 0 kudos
Latest Reply
Serlal
Databricks Employee
  • 0 kudos

If you are interested in distributed machine learning I would suggest Scaling Machine Learning with Spark by Adi Pola k . It is just out so it has all the new goodies including the latest in ML Flow. I am half way though it already but it looks like ...

  • 0 kudos
1 More Replies
User16618471166
by Databricks Employee
  • 5728 Views
  • 3 replies
  • 1 kudos

When I aggregate over more data, I get the below error message. I&#39;ve tried multiple ways of diagnosis like going back to a version I know it was w...

When I aggregate over more data, I get the below error message. I've tried multiple ways of diagnosis like going back to a version I know it was working fine (but still got the same error below). Please advise as this is a critical report where the b...

  • 5728 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Jeff Wu​ :The error message suggests that there is a syntax error in a SQL statement, specifically near the end of the input. Without the full SQL statement or additional information, it's difficult to pinpoint the exact cause of the error. However,...

  • 1 kudos
2 More Replies
andrew0117
by Contributor
  • 4719 Views
  • 4 replies
  • 2 kudos

Resolved! master notebook cannot find the udf registered in the child notebook

The master notebook is calling a child notebook using dbutils.notebook.run("PathToChildnotebook"). The child notebook defines a user-defined function (UDF) and registers it using spark.udf.register. However, when the child notebook finishes running a...

  • 4719 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@andrew li​ :The reason why the UDF cannot be found is that when the child notebook finishes running, the Spark context that was used to define and register the UDF is destroyed. Therefore, the UDF is no longer available in the Spark context used by ...

  • 2 kudos
3 More Replies
ayesharahmat
by New Contributor II
  • 4107 Views
  • 3 replies
  • 2 kudos

AutoLoader issue - java.lang.AssertionError

The below error I am encountering . I am using microbatch for autoloader. please help to rectify this issuejava.lang.AssertionError: assertion failed: Invalid batch: path#36188,modificationTime#36189,length#36190L,content#36191,PROVIDER#36192,LOCATIO...

  • 4107 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Ayesha Rahmatali​ :The error message you provided suggests that there is an assertion failure due to invalid batch data in your AutoLoader implementation. The error specifically indicates that the schema of the incoming data is not matching with the...

  • 2 kudos
2 More Replies
Data_Engineer3
by Contributor III
  • 14091 Views
  • 4 replies
  • 5 kudos

How can i use the same spark session from onenotebook to another notebook in databricks

I want to use the same spark session which created in one notebook and need to be used in another notebook in across same environment, Example, if some of the (variable)object got initialized in the first notebook, i need to use the same object in t...

  • 14091 Views
  • 4 replies
  • 5 kudos
Latest Reply
Manoj12421
Valued Contributor II
  • 5 kudos

You can use %run and then use the location of the notebook - %run "/folder/notebookname"

  • 5 kudos
3 More Replies
yopbibo
by Contributor II
  • 5059 Views
  • 2 replies
  • 0 kudos

pip install in cluster using web UI and extra index

In an init script or a notebook, we can:pip install --index-url=<our private pypi url> --extra-index-url=https://pypi.org/simple <a module>In the cluster web UI (libraries -> install library), we can give only the url of our private repository, but n...

  • 5059 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Philippe CRAVE​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 0 kudos
1 More Replies
Sabytheseeker
by New Contributor
  • 1160 Views
  • 1 replies
  • 0 kudos

I just passed the Lakehouse Fundamentals Accreditation and I haven&#39;t received my badge yet and the certification seems to be messed up 

I just passed the Lakehouse Fundamentals Accreditation and I haven't received my badge yet and the certification seems to be messed up 

  • 1160 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Sabyasachi Samaddar​ We are going through a contract renewal with our vendor, Accredible. Once our new contract goes through you will get your badge this week.Thank you for understanding.

  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels