- 1507 Views
- 2 replies
- 3 kudos
DLT pipeline slow streaming (root cause needs to be identified)
Dear support,we have the following situation where a set of DLT pipelines are streaming with very low rate incoming data and we need to find the root cause of this delay.In order to provide more insight about the setup of the DLT pipelines and some m...
- 1507 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @EDDatabricks EDDatabricks​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that ...
- 3 kudos
- 3267 Views
- 2 replies
- 2 kudos
Issue with reading exported tables stored in parquet
Hi All, I am exported all tables from postgres snapshot into S3 in parquet format. I am trying to read the table using databricks and i am unable to do so. I get the following error: "Unable to infer schema for Parquet. It must be specified manually....
- 3267 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @shiva charan velichala​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that bes...
- 2 kudos
- 7490 Views
- 8 replies
- 10 kudos
Resolved! New strange error on Runtime 12 and above: java.lang.AssertionError: assertion failed
Hi allI struggle to find out why this error message suddenly pops up after running a cell in a notebook. The notebook is trying to run a simple "INSERT INTO" command in SQL. When I only do a SELECT clause, the cell runs without error. Also, I only ge...
- 7490 Views
- 8 replies
- 10 kudos
- 10 kudos
Thanks for reporting! We have identified a defect with an early version of DBR 12 related to INSERT INTO .. SELECT when certain query patterns are involved. The defect has since been fixed. Please let us know if you have any additional questions.
- 10 kudos
- 2464 Views
- 4 replies
- 2 kudos
DLT PIPELINE RUN STATUS
Hi Everyone,Is there a way to find out DLT pipeline run status like if pipeline failed or succeeded ?I'm looking to have report which shows pipeline run info, expectation info ( I was able to get this from event log ) .
- 2464 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Chhaya Vishwakarma​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your fe...
- 2 kudos
- 7626 Views
- 17 replies
- 0 kudos
Running `pyspark` with `databricks-connect`
Hiya,I'm trying to run `pyspark` with `databricks-connect==11.30.b0`, but am failing.The trace I see is``` File "/home/agagrins/databricks9/lib/python3.9/site-packages/py4j/java_gateway.py", line 1321, in __call__ return_value = get_return_value( Fi...
- 7626 Views
- 17 replies
- 0 kudos
- 1085 Views
- 2 replies
- 2 kudos
Can I use a cluster created in Data Science & Engineering persona to run SQL commands in the SQL persona?
Hi,I have created a single-node cluster in Data Science & Engineering persona (Standard_DS3_v2). I don't have enough vCPU to create a SQL warehouse. Is there any way I can use the cluster to run a query in SQL persona?
- 1085 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi MadoYes, you can use cluster and run sql query in the notebook, please refer the following page for more details. https://docs.databricks.com/getting-started/quick-start.html#tutorial-query-data-with-notebookshttps://docs.databricks.com/getting-st...
- 2 kudos
- 3422 Views
- 2 replies
- 0 kudos
how to install custom libraries of python on Shared or multi user cluster
Hello all,wanted to know how to install custom python libraries or load the library files on multiuser or shared databrick cluster.thanks
- 3422 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello all,wanted to know how to install custom python libraries or load the library files on multiuser or shared databrick cluster.thanks
- 0 kudos
- 17837 Views
- 4 replies
- 13 kudos
Resolved! Querying columns with special characters
I'm trying to do a select on a column with special characters in Databricks sql.I've tried the following%sql select ex$col from database.table limit 10; %sql select `ex$col ` from database.table limit 10;They both don't work and will return "ex do...
- 17837 Views
- 4 replies
- 13 kudos
- 13 kudos
Hi @Jamie Nathan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 13 kudos
- 2652 Views
- 4 replies
- 16 kudos
ipywidget not able to start running another notebook
I am trying to add a button in a notebook to trigger an execution of another notebook, but it does not respond to the command. Any idea why? The run command works if I run in a separate cell
- 2652 Views
- 4 replies
- 16 kudos
- 16 kudos
Did you end up finding an answer to this? I am running into the same problem
- 16 kudos
- 1908 Views
- 5 replies
- 2 kudos
Databricks certificaton voucher not recieved
Hi team,I attended the webinar on 21th feb 2023 and also took Lakehouse fundamentals badge, yet I have not received any certification voucher from databricks.regards,vikas
- 1908 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Vikas Singh​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...
- 2 kudos
- 1791 Views
- 2 replies
- 12 kudos
Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resource...
Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resources by triggering your Databricks jobs only when new files arrive in your cloud storage instead of mou...
- 1791 Views
- 2 replies
- 12 kudos
- 12 kudos
Hi @Hubert Dudek​ We really appreciate you sharing this bit of information.Cheers!
- 12 kudos
- 9566 Views
- 13 replies
- 14 kudos
Resolved! How to find the last modified date of a notebook?
I would like to find the notebooks that are not required and not being used and then I can review and delete them. If there is a way to find last modified date of a notebook programmatically then I can get a list of notebooks, which I can review and ...
- 9566 Views
- 13 replies
- 14 kudos
- 14 kudos
Hi @Naveen Kumar Madas​ you can go through below code block%shls -lt /dbfs/
- 14 kudos
- 32841 Views
- 15 replies
- 7 kudos
Resolved! What's the equivalent of "DECLARE..." in Databricks SQL ?
Hello everyone,I'm new in Databricks SQL, and I'm comming from SQL Server.I would like to know what's the equivalent of :DECLARE @P_Name varchar(50) = 'BackOffice'It's for use it like this : CREATE DATABASE @P_NameThanks.
- 32841 Views
- 15 replies
- 7 kudos
- 7 kudos
Hi @Salah K.​ you can go through this code block%python P_Name = 'BackOffice'spark.sql(f""" create database {P_name} """)
- 7 kudos
- 9785 Views
- 4 replies
- 2 kudos
- 9785 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi, You can refer to https://docs.databricks.com/files/unzip-files.html. You can curl the file you want and then it can be unzipped as mentioned in the doc. Please let us know if this helps.Also, please tag @Debayan with your next update which will n...
- 2 kudos
- 1253 Views
- 2 replies
- 2 kudos
Not able to create a unity metastore in a specified region
Hi Team,I'm not able to create a metastore in a region (us-east-1).It tells me that "This region already contains a metastore. Only a single metastore is allowed per region"But we don't have any metastore. Earlier we had one metastore we had deleted...
- 1253 Views
- 2 replies
- 2 kudos
- 2 kudos
@Rajath C​ can you please re-check if it has been properly deleted and if still old one has been tied to any of workspaces. also try to delete that storage if no data exists and retry
- 2 kudos