- 14058 Views
- 4 replies
- 5 kudos
How can i use the same spark session from onenotebook to another notebook in databricks
I want to use the same spark session which created in one notebook and need to be used in another notebook in across same environment, Example, if some of the (variable)object got initialized in the first notebook, i need to use the same object in t...
- 14058 Views
- 4 replies
- 5 kudos
- 5 kudos
You can use %run and then use the location of the notebook - %run "/folder/notebookname"
- 5 kudos
- 5044 Views
- 2 replies
- 0 kudos
pip install in cluster using web UI and extra index
In an init script or a notebook, we can:pip install --index-url=<our private pypi url> --extra-index-url=https://pypi.org/simple <a module>In the cluster web UI (libraries -> install library), we can give only the url of our private repository, but n...
- 5044 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Philippe CRAVE​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 0 kudos
- 1153 Views
- 1 replies
- 0 kudos
I just passed the Lakehouse Fundamentals Accreditation and I haven't received my badge yet and the certification seems to be messed upÂ
I just passed the Lakehouse Fundamentals Accreditation and I haven't received my badge yet and the certification seems to be messed up
- 1153 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sabyasachi Samaddar​ We are going through a contract renewal with our vendor, Accredible. Once our new contract goes through you will get your badge this week.Thank you for understanding.
- 0 kudos
- 2888 Views
- 1 replies
- 0 kudos
Kafka consumer groups in Databricks
I was trying to find information about configuring the consumer groups for kafka stream in databricks. By doing so I want to parallelize the stream and load it into databricks tables. Does the databricks handle this internally? If we can configure th...
- 2888 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, we have a few examples on stream processing using Kafka (https://docs.databricks.com/structured-streaming/kafka.html), there is no straight public document for Kafka consumer group creation. You can refer to https://kafka.apache.org/documentation...
- 0 kudos
- 4675 Views
- 4 replies
- 2 kudos
Delta table property is not set.
I have set the delta table property at cluster level.spark.databricks.delta.retentionDurationCheck.enabled falseWhen I create a new table, retentionDurationCheck property is not shown in the table details. But when I set this with ALTER TABLE for a s...
- 4675 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Mahesh Chahare​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
- 2 kudos
- 2495 Views
- 2 replies
- 2 kudos
DataBricks Certification Exam Got Suspended. Need help in resolving the issue
Hi Team,My databricks exam got suspened on 16th April today Morning and it is still in the suspended state. I have raised a support request using the below linkhttps://help.databricks.com/s/contact-us?ReqType=training .​ but I haven’t received the ti...
- 2495 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Sachin Kumara​ We are going through a contract renewal with our vendor, Accredible. Once our new contract goes through you will get your badge this week. Thank you for understanding!
- 2 kudos
- 3696 Views
- 3 replies
- 0 kudos
Resolved! Problem with accessing element using Pandas UDF in Image Processing
Hi everyone,I was stuck at this for very long time. Not a very familiar user of using Spark for image processing. I was trying to resize images that are loaded into a Spark DF. However, it keeps throwing error that I am not able to access the element...
- 3696 Views
- 3 replies
- 0 kudos
- 0 kudos
@Yan Chong Tan​ :The error you are facing is due to the fact that you are trying to access the attribute "width" of a string object in the resize_image function. Specifically, input_dim is a string object, but you are trying to access its width attr...
- 0 kudos
- 7033 Views
- 4 replies
- 0 kudos
Resolved! partition on a csv file
When I use SQL code like "create table myTable (column1 string, column2 string) using csv options('delimiter' = ',', 'header' = 'true') location 'pathToCsv'" to create a table from a single CSV file stored in a folder within an Azure Data Lake contai...
- 7033 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @andrew li​, When you specify a path with LOCATION keyword, Spark will consider that to be an EXTERNAL table. So when you dropped the table, you underlying data if any will not be cleared. So in you case, as this is an external table, you folder s...
- 0 kudos
- 8012 Views
- 3 replies
- 3 kudos
Resolved! How to delay a new job run after job
I have a daily job run that occasionally fails with the error: The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached. After I get the notification that this job failed on schedule, I manually run ...
- 8012 Views
- 3 replies
- 3 kudos
- 3 kudos
According to this documentation, you can specify the wait time between the "start" of the first run and the retry start time.
- 3 kudos
- 6201 Views
- 0 replies
- 0 kudos
What are executeAdhocQuery and executeFastQuery operations in the Azure SQL Logs?
Hi,-Im performing some analysis using the databricks sql logs, and seeing these operation names.-I notice these events dont seem to have a duration nor query text, unlike commandSubmit operations.-Any explanation on what these operations mean exactly...
- 6201 Views
- 0 replies
- 0 kudos
- 8388 Views
- 3 replies
- 0 kudos
Error when calling SparkR from within a Python notebook
I’ve had success with R magic (R cells in a Python notebook) and running an R script from a Python notebook, up to the point of connecting R to a Spark cluster. In either case, I can’t get a `SparkSession` to initialize. 2-cell (Python) notebook exa...
- 8388 Views
- 3 replies
- 0 kudos
- 0 kudos
The answer I can give you to have this work for you is to call the R notebooks from your Python notebook. Just save each dataframe as a delta table to pass between the languages.How to call a notebook from another notebook? here is a link
- 0 kudos
- 3591 Views
- 2 replies
- 2 kudos
Using dbutils.fs.ls on URI with square brackets results in error
Square brackets in ADLS are accepted, so why can't I list the files in the folder? I have tried escaping the square brackets manually, but then the escaped values are re-escaped from %5B to %255B and %5D to %255D. I get:URISyntaxException: Illegal ...
- 3591 Views
- 2 replies
- 2 kudos
- 2 kudos
@Joshua Stafford​ :The URISyntaxException error you are encountering is likely due to the fact that square brackets are reserved characters in URIs (Uniform Resource Identifiers) and need to be properly encoded when used in a URL. In this case, it ap...
- 2 kudos
- 9139 Views
- 11 replies
- 4 kudos
Certificate and Badge not received
I have cleared my certification exam, Databricks Certified Data Engineer Associate on 07 April 2023. I haven't received any certification or badge yet.I sent an email to training databricks and they told me that this problem has been solved...Any Hel...
- 9139 Views
- 11 replies
- 4 kudos
- 4 kudos
Hello all,We are going through a contract renewal with our vendor, Accredible. Once our new contract goes through you will get your badge this week. Thank you for understanding.
- 4 kudos
- 46073 Views
- 13 replies
- 8 kudos
Resolved! Not able to generate Access Token for Service Principal using rest API
I am trying to generate a Databricks token for a service principal (SP). I have created the SP in Azure AD and have used the Databricks rest api to add it as an admin.When using the Databricks rest API "/api/2.0/token-management/on-behalf-of/tokens" ...
- 46073 Views
- 13 replies
- 8 kudos
- 8 kudos
Hi all,I believe I found a temporary fix for this -Generate an AAD token for the service principle in Azure. Follow this guide if you don't know how to -https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/aad/service-prin-aad-toke...
- 8 kudos
- 7468 Views
- 2 replies
- 0 kudos
Unexpected failure while fetching notebook - What can we do from our side?
Hello!We got some pipelines running in Azure Data Factory that call Databricks Notebooks to run data transformations. This morning at 6:21 AM (UTC) we got an error " Unexpected failure while fetching notebook" inside a notebook that calls another one...
- 7468 Views
- 2 replies
- 0 kudos
- 0 kudos
@Rita Fernandes​ :Based on the error message you provided, it seems like the issue might be related to the version mismatch between the ANTLR tool used for code generation and the current runtime version. Additionally, the error message suggests that...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 484 | |
| 349 | |
| 287 |
