- 2283 Views
- 2 replies
- 1 kudos
How install a library that is under the /Workspace/Shared/ directory using the init.sh script in a cluster?
I would like to install a library that is under the /Workspace/Shared/ directory using the init.sh script in a cluster. How to access the /Workspace/Shared/ folder in shell? This page only shows how to access manually but doesn't show how to access i...
- 2283 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Juned Mala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 1 kudos
- 6434 Views
- 2 replies
- 3 kudos
Do we really need Autoloader for batch processing.?
Hi All,It seem AutoLoader is good option for even driven data ingestion but if my job runs only once , do I still need autoloader ? I dont want to spend money to spin a cluster whole day.I know we have RunOnce option available while running a job but...
- 6434 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Deepak Bhatt​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Thanks and regards
- 3 kudos
- 11463 Views
- 3 replies
- 3 kudos
Resolved! Explicit transaction blocks
I know delta tables are supporting the ACID properties and my understanding is Merge, Insert, delete, etc. are inside a transaction by default and if any error occurred during these operations, that transaction will be roll backed. I hope this unders...
- 11463 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Thushar R​​, We can build a thriving shared knowledge and insights community. Return and mark the best answers to contribute to our ongoing pursuit of excellence.
- 3 kudos
- 5337 Views
- 2 replies
- 1 kudos
Is ABAC feature enabled?
Can anyone please share me the example of how it works in terms of access controls?
- 5337 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Naveena G​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
- 1 kudos
- 1502 Views
- 2 replies
- 3 kudos
DLT pipeline slow streaming (root cause needs to be identified)
Dear support,we have the following situation where a set of DLT pipelines are streaming with very low rate incoming data and we need to find the root cause of this delay.In order to provide more insight about the setup of the DLT pipelines and some m...
- 1502 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @EDDatabricks EDDatabricks​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that ...
- 3 kudos
- 3236 Views
- 2 replies
- 2 kudos
Issue with reading exported tables stored in parquet
Hi All, I am exported all tables from postgres snapshot into S3 in parquet format. I am trying to read the table using databricks and i am unable to do so. I get the following error: "Unable to infer schema for Parquet. It must be specified manually....
- 3236 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @shiva charan velichala​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that bes...
- 2 kudos
- 7455 Views
- 8 replies
- 10 kudos
Resolved! New strange error on Runtime 12 and above: java.lang.AssertionError: assertion failed
Hi allI struggle to find out why this error message suddenly pops up after running a cell in a notebook. The notebook is trying to run a simple "INSERT INTO" command in SQL. When I only do a SELECT clause, the cell runs without error. Also, I only ge...
- 7455 Views
- 8 replies
- 10 kudos
- 10 kudos
Thanks for reporting! We have identified a defect with an early version of DBR 12 related to INSERT INTO .. SELECT when certain query patterns are involved. The defect has since been fixed. Please let us know if you have any additional questions.
- 10 kudos
- 2458 Views
- 4 replies
- 2 kudos
DLT PIPELINE RUN STATUS
Hi Everyone,Is there a way to find out DLT pipeline run status like if pipeline failed or succeeded ?I'm looking to have report which shows pipeline run info, expectation info ( I was able to get this from event log ) .
- 2458 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Chhaya Vishwakarma​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your fe...
- 2 kudos
- 7592 Views
- 17 replies
- 0 kudos
Running `pyspark` with `databricks-connect`
Hiya,I'm trying to run `pyspark` with `databricks-connect==11.30.b0`, but am failing.The trace I see is``` File "/home/agagrins/databricks9/lib/python3.9/site-packages/py4j/java_gateway.py", line 1321, in __call__ return_value = get_return_value( Fi...
- 7592 Views
- 17 replies
- 0 kudos
- 1082 Views
- 2 replies
- 2 kudos
Can I use a cluster created in Data Science & Engineering persona to run SQL commands in the SQL persona?
Hi,I have created a single-node cluster in Data Science & Engineering persona (Standard_DS3_v2). I don't have enough vCPU to create a SQL warehouse. Is there any way I can use the cluster to run a query in SQL persona?
- 1082 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi MadoYes, you can use cluster and run sql query in the notebook, please refer the following page for more details. https://docs.databricks.com/getting-started/quick-start.html#tutorial-query-data-with-notebookshttps://docs.databricks.com/getting-st...
- 2 kudos
- 3400 Views
- 3 replies
- 0 kudos
Resolved! how to install custom libraries of python on Shared or multi user cluster
Hello all,wanted to know how to install custom python libraries or load the library files on multiuser or shared databrick cluster.thanks
- 3400 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Paresh Desai​, To install custom Python libraries or load library files on a multi-user or shared Databricks cluster, you can follow the steps below:Create a library: You can create a library by clicking on the "Libraries" tab in the left-hand pa...
- 0 kudos
- 17747 Views
- 5 replies
- 16 kudos
Resolved! Querying columns with special characters
I'm trying to do a select on a column with special characters in Databricks sql.I've tried the following%sql select ex$col from database.table limit 10; %sql select `ex$col ` from database.table limit 10;They both don't work and will return "ex do...
- 17747 Views
- 5 replies
- 16 kudos
- 16 kudos
Hi @Jamie Nathan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 16 kudos
- 2639 Views
- 4 replies
- 16 kudos
ipywidget not able to start running another notebook
I am trying to add a button in a notebook to trigger an execution of another notebook, but it does not respond to the command. Any idea why? The run command works if I run in a separate cell
- 2639 Views
- 4 replies
- 16 kudos
- 16 kudos
Did you end up finding an answer to this? I am running into the same problem
- 16 kudos
- 954 Views
- 1 replies
- 5 kudos
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and...
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and reliable platform for users to build and deploy their applications. As part of this release, the en...
- 954 Views
- 1 replies
- 5 kudos
- 5 kudos
Hi @Hubert Dudek​,We express appreciation for the informative content you've contributed to our community.Your posts have sparked engaging discussions and proven invaluable resources for our members.You've truly made a difference in our community, an...
- 5 kudos
- 932 Views
- 1 replies
- 2 kudos
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:30-5:00pm PT.We'll have Dave Ma...
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:30-5:00pm PT.We'll have Dave Mariani - CTO & Founder at AtScale, and Riley Phillips - Enterprise Solution Engineer at Matillion to shar...
- 932 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Lu Wang​, Thank you for your outstanding efforts in sharing important announcements within our community. Your proactive approach to communication has helped maintain a well-informed and cohesive work environment for our team.Your dedication to k...
- 2 kudos