cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

mortenhaga
by Contributor
  • 10432 Views
  • 8 replies
  • 10 kudos

Resolved! New strange error on Runtime 12 and above: java.lang.AssertionError: assertion failed

Hi allI struggle to find out why this error message suddenly pops up after running a cell in a notebook. The notebook is trying to run a simple "INSERT INTO" command in SQL. When I only do a SELECT clause, the cell runs without error. Also, I only ge...

  • 10432 Views
  • 8 replies
  • 10 kudos
Latest Reply
entongshen__Dat
New Contributor III
  • 10 kudos

Thanks for reporting! We have identified a defect with an early version of DBR 12 related to INSERT INTO .. SELECT when certain query patterns are involved. The defect has since been fixed. Please let us know if you have any additional questions.

  • 10 kudos
7 More Replies
Chhaya
by New Contributor III
  • 3906 Views
  • 4 replies
  • 2 kudos

DLT PIPELINE RUN STATUS

Hi Everyone,Is there a way to find out DLT pipeline run status like if pipeline failed or succeeded ?I'm looking to have report which shows pipeline run info, expectation info ( I was able to get this from event log ) .

  • 3906 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Chhaya Vishwakarma​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your fe...

  • 2 kudos
3 More Replies
agagrins
by New Contributor III
  • 11338 Views
  • 17 replies
  • 0 kudos

Running `pyspark` with `databricks-connect`

Hiya,I'm trying to run `pyspark` with `databricks-connect==11.30.b0`, but am failing.The trace I see is``` File "/home/agagrins/databricks9/lib/python3.9/site-packages/py4j/java_gateway.py", line 1321, in __call__  return_value = get_return_value( Fi...

  • 11338 Views
  • 17 replies
  • 0 kudos
Latest Reply
ryojikn
New Contributor III
  • 0 kudos

How to make it work in a cluster with Unity Catalog enabled?

  • 0 kudos
16 More Replies
Mado
by Valued Contributor II
  • 1663 Views
  • 2 replies
  • 2 kudos

Can I use a cluster created in Data Science & Engineering persona to run SQL commands in the SQL persona?

Hi,I have created a single-node cluster in Data Science & Engineering persona (Standard_DS3_v2). I don't have enough vCPU to create a SQL warehouse. Is there any way I can use the cluster to run a query in SQL persona?

  • 1663 Views
  • 2 replies
  • 2 kudos
Latest Reply
Rajeev45
Databricks Employee
  • 2 kudos

Hi MadoYes, you can use cluster and run sql query in the notebook, please refer the following page for more details. https://docs.databricks.com/getting-started/quick-start.html#tutorial-query-data-with-notebookshttps://docs.databricks.com/getting-st...

  • 2 kudos
1 More Replies
JamieN
by New Contributor III
  • 24561 Views
  • 4 replies
  • 13 kudos

Resolved! Querying columns with special characters

I'm trying to do a select on a column with special characters in Databricks sql.I've tried the following%sql select ex$col from database.table limit 10;   %sql select `ex$col ` from database.table limit 10;They both don't work and will return "ex do...

  • 24561 Views
  • 4 replies
  • 13 kudos
Latest Reply
Anonymous
Not applicable
  • 13 kudos

Hi @Jamie Nathan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 13 kudos
3 More Replies
lzha174
by Contributor
  • 4153 Views
  • 4 replies
  • 16 kudos

ipywidget not able to start running another notebook

I am trying to add a button in a notebook to trigger an execution of another notebook, but it does not respond to the command. Any idea why? The run command works if I run in a separate cell

image
  • 4153 Views
  • 4 replies
  • 16 kudos
Latest Reply
mbejarano89
New Contributor III
  • 16 kudos

Did you end up finding an answer to this? I am running into the same problem

  • 16 kudos
3 More Replies
Vikad
by New Contributor II
  • 2610 Views
  • 5 replies
  • 2 kudos

Databricks certificaton voucher not recieved

Hi team,I attended the webinar on 21th feb 2023 and also took Lakehouse fundamentals badge, yet I have not received any certification voucher from databricks.regards,vikas

  • 2610 Views
  • 5 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Vikas Singh​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...

  • 2 kudos
4 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 2739 Views
  • 2 replies
  • 12 kudos

Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resource...

Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resources by triggering your Databricks jobs only when new files arrive in your cloud storage instead of mou...

ezgif-3-946af786d0
  • 2739 Views
  • 2 replies
  • 12 kudos
Latest Reply
Vartika
Databricks Employee
  • 12 kudos

Hi @Hubert Dudek​ We really appreciate you sharing this bit of information.Cheers!

  • 12 kudos
1 More Replies
Naveen_KumarMad
by New Contributor III
  • 15016 Views
  • 13 replies
  • 14 kudos

Resolved! How to find the last modified date of a notebook?

I would like to find the notebooks that are not required and not being used and then I can review and delete them. If there is a way to find last modified date of a notebook programmatically then I can get a list of notebooks, which I can review and ...

  • 15016 Views
  • 13 replies
  • 14 kudos
Latest Reply
Amit_352107
New Contributor III
  • 14 kudos

Hi @Naveen Kumar Madas​ you can go through below code block%shls -lt /dbfs/

  • 14 kudos
12 More Replies
wyzer
by Contributor II
  • 37193 Views
  • 15 replies
  • 7 kudos

Resolved! What's the equivalent of "DECLARE..." in Databricks SQL ?

Hello everyone,I'm new in Databricks SQL, and I'm comming from SQL Server.I would like to know what's the equivalent of :DECLARE @P_Name varchar(50) = 'BackOffice'It's for use it like this : CREATE DATABASE @P_NameThanks.

  • 37193 Views
  • 15 replies
  • 7 kudos
Latest Reply
Amit_352107
New Contributor III
  • 7 kudos

Hi @Salah K.​ you can go through this code block%python P_Name = 'BackOffice'spark.sql(f""" create database {P_name} """)

  • 7 kudos
14 More Replies
StephanieAlba
by Databricks Employee
  • 12359 Views
  • 4 replies
  • 2 kudos
  • 12359 Views
  • 4 replies
  • 2 kudos
Latest Reply
Debayan
Databricks Employee
  • 2 kudos

Hi, You can refer to https://docs.databricks.com/files/unzip-files.html. You can curl the file you want and then it can be unzipped as mentioned in the doc. Please let us know if this helps.Also, please tag @Debayan with your next update which will n...

  • 2 kudos
3 More Replies
RC
by Contributor
  • 1941 Views
  • 2 replies
  • 2 kudos

Not able to create a unity metastore in a specified region

Hi Team,I'm not able to create a metastore in a region (us-east-1).It tells me that "This region already contains a metastore. Only a single metastore is allowed per region"But we don't have any metastore. Earlier we had one metastore we had deleted...

  • 1941 Views
  • 2 replies
  • 2 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 2 kudos

@Rajath C​ can you please re-check if it has been properly deleted and if still old one has been tied to any of workspaces. also try to delete that storage if no data exists and retry

  • 2 kudos
1 More Replies
repcak
by New Contributor III
  • 3153 Views
  • 1 replies
  • 2 kudos

Init Scripts with mounted azure data lake storage gen2

I'm trying to access init script which is stored on mounted azure data lake storage gen2 to dbfsI mounted storage to dbfs:/mnt/storage/container/script.shand when i try to access it i got an error:Cluster scoped init script dbfs:/mnt/storage/containe...

  • 3153 Views
  • 1 replies
  • 2 kudos
Latest Reply
User16752239289
Databricks Employee
  • 2 kudos

I do not think the init script saved under mount point work and we do not suggest that. If you specify abfss , then the cluster need to be configured so that the cluster can authenticate and access the adls gen2 folder. Otherwise, the cluster will no...

  • 2 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels