- 10795 Views
- 1 replies
- 4 kudos
Databricks Jobs API not returning notebook run results?
Calling a databricks notebook using the Rest API, can confirm that it is executing the notebook, but is not accepting my parameters or returning a notebook output. Any ideas on what I am doing wrong here?My code and notebook function are below, tryin...
- 10795 Views
- 1 replies
- 4 kudos
- 4 kudos
Resolved this by using dbutils within the notebook being called from the API.# databricks notebook function data = dbutils.widgets.get('data') # pulls base_parameters from API call def add_test(i): result = i + ' COMPLETE' return result ...
- 4 kudos
- 1789 Views
- 0 replies
- 0 kudos
How to query a MySQL Table from Databricks?
I wanted to query a MySQL Table using Databricks rather than reading the complete data using a dbtable option, which will help in incremental loads.remote_table = (spark.read .format("jdbc") .option("driver", driver) .option("url", URL) .option("quer...
- 1789 Views
- 0 replies
- 0 kudos
- 3787 Views
- 4 replies
- 1 kudos
Hi @Kaniz Fatma​ and @Nadia Elsayed​ , i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtain...
Hi @Kaniz Fatma​ and @Nadia Elsayed​ ,i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtained below 70% in assessment but as per the section wise results i have gained more than 70% . Can you ...
- 3787 Views
- 4 replies
- 1 kudos
- 23765 Views
- 4 replies
- 11 kudos
The functionality of table property delta.logRetentionDuration
We have one project requirement where we have to store only the 14 days history for delta tables. So for testing, I have set the delta.logRetentionDuration = 2 days using the below commandspark.sql("alter table delta.`[delta_file_path]` set TBLPROPER...
- 23765 Views
- 4 replies
- 11 kudos
- 11 kudos
Hi, by default there is a safety interval enabled. So if you set a retentionperiod lower than that interval (7 days), data in that interval will not be deleted.You have to specificall override this safety interval by setting spark.databricks.delta.r...
- 11 kudos
- 13688 Views
- 3 replies
- 10 kudos
Resolved! Number of partitions in Spark UI Simulator experiment
I am learning how to optimize Spark applications with experiments from Spark UI Simulator. There is experiment #1​596 about data skew and in command 2 there is comment about how many partitions will be set as default:// Factor of 8 cores and greater ...
- 13688 Views
- 3 replies
- 10 kudos
- 10 kudos
Hi @Bartosz Maciejewski​ Generally we arrive at the number of shuffle partitions using the following method.Input Size Data - 100 GBIdeal partition target size - 128 MBCores - 8Ideal number of partitions = (100*1028)/128 = 803.25 ~ 804To utiltize the...
- 10 kudos
- 2956 Views
- 3 replies
- 4 kudos
Hi @Vidula Khanna​  i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtained below 70% in ass...
Hi @Vidula Khanna​ i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtained below 70% in assessment but as per the section wise results i have gained more than 70% . Can you please check and up...
- 2956 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Harish Raj​ Please bear with us!I already have shared this issue with the respective person in the Academy team. Thanks for your patience!
- 4 kudos
- 6669 Views
- 4 replies
- 4 kudos
Data bricks issue writing to Azure synapse
Process is to pickup data from an external table in Azure Synapse and combine with other tables and write to another synapse tables. Data for external table is in Azure storage. It was fine for months ,all of sudden last week it errored out with err...
- 6669 Views
- 4 replies
- 4 kudos
- 4 kudos
Also do check out this blog from this stack overflow question.https://stackoverflow.com/questions/66747544/databricks-write-back-to-azure-synapse-errorhttps://chinnychukwudozie.com/2020/11/13/write-data-from-azure-databricks-to-azure-synapse-analytic...
- 4 kudos
- 24855 Views
- 7 replies
- 4 kudos
Resolved! Invalid JDBC url
Hello. I am trying to establish a connection between DBeaver and Databricks. I followed the steps in DBeaver integration with Databricks | Databricks on AWS, but I get the following error while testing the connection: Could anyone provide any insight...
- 24855 Views
- 7 replies
- 4 kudos
- 4 kudos
Hi @ELENI GEORGOUSI​ Glad to hear! It's a request that mark an answer as best.Thanks...
- 4 kudos
- 7650 Views
- 3 replies
- 6 kudos
Unable to create materialized view in Unity enabled Catalogues.
Hi Team,I was going through one of the videos of Databricks Sql Serverless and it say there is materialized view support . We can create materialized view .I tried same on my cluster of Sql Warehouse gives below error:
- 7650 Views
- 3 replies
- 6 kudos
- 6 kudos
Materialized views is in private preview right now afaik. Please talk to your account or customer success team at Databricks in order to sign up and enable it for your workspace. Thanks!
- 6 kudos
- 8421 Views
- 9 replies
- 27 kudos
Resolved! When will be available the course "Data Engineering with Databricks V3" in Databricks Academy?
In the documentation of the V2 it says that it would be released 4 days ago. But searching for it in the academy only throws results for V1 and V2.
- 8421 Views
- 9 replies
- 27 kudos
- 27 kudos
@Gustavo Amadoz Navarro​ Updated new infoThis course will be part of the data engineer learning path once the Databricks Certified Data Engineer Associate V3 exam is released (November 19, 2022). BEFORE YOU GET STARTED: Please note that this course,...
- 27 kudos
- 2539 Views
- 3 replies
- 25 kudos
Can anyone plz help for mine below issues
Can anyone plz help for mine below issues
- 2539 Views
- 3 replies
- 25 kudos
- 25 kudos
Hi @Yogita Chavan​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks
- 25 kudos
- 8393 Views
- 4 replies
- 11 kudos
How to setup an all-purpose cluster pool for all my jobs?
Today, we start working on setting up an all-purpose cluster pool for all the jobs that we are running on databricks. We used the documentation for this but we got some issues when running our jobs.The errors in the jobs are the following: The jobs a...
- 8393 Views
- 4 replies
- 11 kudos
- 11 kudos
Hi @Siebert Looije​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 11 kudos
- 2139 Views
- 1 replies
- 4 kudos
Structured Streaming from SFTP
Hello,Is it possible to use a SFTP location to load from for structured streaming.At the moment we are going from SFTP->S3->databricks via structured streaming. I would like to cut out the S3 part.CheersChris
- 2139 Views
- 1 replies
- 4 kudos
- 4 kudos
Hi @Chris Lant​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks.
- 4 kudos
- 4227 Views
- 4 replies
- 9 kudos
How long does it take to enable SQL Serverless for Azure Databricks?
I have filled the form to enable SQL Serverless twice in our Azure Databricks workspace, but I have not received any response through email nor I have serverless enabled in our workspace.
- 4227 Views
- 4 replies
- 9 kudos
- 9 kudos
Hi @Manuel Torrez​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 9 kudos
- 10856 Views
- 1 replies
- 4 kudos
Resolved! TypeError: 'JavaPackage' object is not callable - DocumentAssembler() - Spark NLP
Hi,I am doing a grammar check using spark NLP using azure databricks. But am getting TypeError: 'JavaPackage' object is not callable - in DocumentAssembler() intialization line.document_assembler = DocumentAssembler()\ .setInputCol("text")\ .setOutpu...
- 10856 Views
- 1 replies
- 4 kudos
- 4 kudos
Hi its working after adding below configs in cluster. Please check URL for more info :- https://nlp.johnsnowlabs.com/docs/en/install#databricks-support
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
