- 7176 Views
- 2 replies
- 7 kudos
Where can I find API documentation to dbutils.notebook.entry_point?
dbutils.notebook.help only lists "run" and "exit" methods. I could only find references to dbutils.notebook.entry_point spread across the web but there does not seem to be an official Databricks API documentation to its complete APIs anywhere. Can so...
- 7176 Views
- 2 replies
- 7 kudos
- 7 kudos
Hi @James kuo​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 7 kudos
- 2534 Views
- 1 replies
- 6 kudos
Creating a DLT pipeline that reads from a JDBC source
Hey,I'm trying to create a DLT pipeline that reads from a JDBC source, and the code I'm using looks something like this in python:import dlt @dlt.table def table_name(): driver = 'oracle.jdbc.driver.OracleDriver' url = '...' query = 'SELECT ......
- 2534 Views
- 1 replies
- 6 kudos
- 6 kudos
Hi @Reda Bitar​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks
- 6 kudos
- 4452 Views
- 4 replies
- 6 kudos
mount s3 bucket with specific endpoint
Environment:AZURE-DatabricksLanguage: PythonI can access my s3 bucket via:boto3.client('s3', endpoint_url='https://gateway.storjshare.io', ... )and it also works via:boto3.resource('s3', endpoint_url='https://gateway.storjshare.io', ... )As a next st...
- 4452 Views
- 4 replies
- 6 kudos
- 6 kudos
Hi @Kevin Ostheimer​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 6 kudos
- 2446 Views
- 1 replies
- 4 kudos
Delta sharing in Databricks doesn't work
Databricks Delta sharing server seems to be broken. We have a table ~ 10M rows and there is no way for us to query the shared data via any methods (e.g Python/Spark or even another Databricks account that the data was shared with)Any ideas on why thi...
- 2446 Views
- 1 replies
- 4 kudos
- 4 kudos
Hi @Khoa Ho​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 4 kudos
- 2190 Views
- 3 replies
- 4 kudos
Haven't received Databricks Certificate or any form of correspondence
I passed the Databricks Certified Associate Developer for Apache Spark 3.0 - Python on 10/22/2022 with a score of 85%.My kryterion webassessor account shows a pass. However, I've not yet received any correspondence/badge from Databricks
- 2190 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Kartikeya Shukla​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 4 kudos
- 2900 Views
- 2 replies
- 6 kudos
Community Edition SQL Warehouse is not starting up, is it not free to even trail ?
When i start the sql warehosue , i get this error. Is there a way around to start up, I just want to try some features. Clusters are failing to launch. Cluster launch will be retried.Details for the latest failure: Error: Error code: PublicIPCountLim...
- 2900 Views
- 2 replies
- 6 kudos
- 6 kudos
Hi @Anbarasan Dhanushkodi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from ...
- 6 kudos
- 9989 Views
- 2 replies
- 5 kudos
Using large dataframe in-memory (data not allowed to be "at rest") results in driver crash and/or out of memory
I'm having trouble working on Databricks with data that we are not allowed to save off or persist in any way. The data comes from an API (which returns a JSON response). We have a scala package on our cluster that makes the queries (almost 6k queries...
- 9989 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi @James Held​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 5 kudos
- 2850 Views
- 2 replies
- 3 kudos
What is default location when using "writeStream"?
Hi,Assume that I want to write a table by" writeStream". Where is the default location on DBFS where the table is saved?Sample code:spark.table("TEMP_SILVER").writeStream .option("checkpointLocation", "dbfs:/user/AAA@gmail.com") ....
- 2850 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Mohammad Saber​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 3 kudos
- 1482 Views
- 2 replies
- 3 kudos
community edition workspace password reset issue
I am not able to reset my community edition workspace password. It continuously processing and password not getting rese
- 1482 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Akil Kapasi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 3 kudos
- 1539 Views
- 1 replies
- 3 kudos
How to move the "__apply changes_storage_mytablename" when creating a streaming live table?
As the title suggests, whenever I create a streaming live table it creates a __apply_changes_storage_"mytablename" section in the database on databricks. Is there a way to specify a different cloud location for these files?
- 1539 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Logan Nicol​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks
- 3 kudos
- 4219 Views
- 2 replies
- 5 kudos
Need help in a pyspark code in Databricks to calculate a new measure column.
Details of the requirement is as below:I have a table with below structure:So i have to write a code in pyspark to calculate a new column.Logic for new column is Sum of Magnitude for different Categories divided by the total Magnitude.And it should b...
- 4219 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi @Faizan Arefin​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 5 kudos
- 5830 Views
- 3 replies
- 4 kudos
Create new job api error "MALFORMED_REQUEST"
hi,i'm trying to test create a new job api (v 2.1) with python, but i got error:{ 'error_code': 'MALFORMED_REQUEST', 'message': 'Invalid JSON given in the body of the request - expected a map'}How do i validate json body before posting ?this is my js...
- 5830 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @tum m​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 4 kudos
- 13077 Views
- 5 replies
- 10 kudos
Is ZORDER required after table overwrite?
Hi,After appending new values to a delta table, I need to delete duplicate rows.After deleting duplicate rows using PySpark, I overwrite the table (keeping the schema).My question is, do I have to do ZORDER again?Another question, is there another wa...
- 13077 Views
- 5 replies
- 10 kudos
- 10 kudos
Hii @Nurettin Ersoz​ try to use incremental load of data so it will avoid duplicate and you can use full load once if you have updation in your data
- 10 kudos
- 6998 Views
- 7 replies
- 23 kudos
Resolved! Is there syllabus change in self paced Data Engineering with Databrick course video?
Is there syllabus change in self paced Data Engineering with Databrick course video?Last week i started that video lecture, but today i found that everything is change.https://partner-academy.databricks.com/learn/course/62/data-engineering-with-datab...
- 6998 Views
- 7 replies
- 23 kudos
- 23 kudos
Hi @Milind Singh​ yes there is keep on updation of syllabus so it is required to be updated on self paced course
- 23 kudos
- 5718 Views
- 3 replies
- 4 kudos
How to identify or determine how many jobs will be performed if I submit code
I’m not able to find a source where it explains how to determine how many job a written piece of pyspark code will trigger. Can you please help me here. About stages I know that the number of shuffles equals to the number of stages.
- 5718 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @sagar Varma​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 4 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
