- 9528 Views
- 2 replies
- 2 kudos
Can assign a default value for job parameter from the widget?
The Databricks widget (dbutils) provides the get function for accessing the job parameters of a job.​dbutils.widgets.get('my_param')Unlike Python dict, where get returns None or an optional argument if the dict doesn't contain the parameter, the widg...
- 9528 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Mattias P​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
- 2 kudos
- 3446 Views
- 2 replies
- 0 kudos
Databricks-Connect Error occurred while running *** java.io.StreamCorruptedException: invalid type code: 00
I am using databricks-connect to access a remote cluster. Everything works as expected and I can set breakpoints and interrogate the results, same for when it trys to execute the following code:val testDF = spark.createDataFrame(spark.sparkContext .e...
- 3446 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @James Metcalf​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 0 kudos
- 8881 Views
- 2 replies
- 2 kudos
Resolved! DLT data quality UI was present last week. Now absent. Did I change a setting?
Last week, I started running a DLT pipeline with expectations that dropped rows on streaming live tables. In the side bar for a table, I saw a nice circular chart with Written/Dropped rows and Failed records stats.Today, I ran a similar DLT pipeline ...
- 8881 Views
- 2 replies
- 2 kudos
- 2 kudos
@Kevin Rossi​ :The circular chart with Written/Dropped rows and Failed records stats that you saw in the sidebar of a table in Delta Live Tables (DLT) is a built-in feature called "Data Quality Metrics" that provides a visual representation of the da...
- 2 kudos
- 2010 Views
- 1 replies
- 4 kudos
Resolved! while creating serverless warehouse we are receiving below message <workspaceId> is no longer eligible for Serverless Compute. Please reach out to your administrator
HI team,As far as limitations and pre-requisites we have met all, able to create warehouse in other workspace which are part of same account, for one of the workspaces we are seeing above issue. we don't any clear error log other than <workspaceId >i...
- 2010 Views
- 1 replies
- 4 kudos
- 4 kudos
@karthik p​ Your workspace might have blocked from serverless feature if there are some uncleared bills. If that's not the case, please file a support case to us.
- 4 kudos
- 2297 Views
- 1 replies
- 1 kudos
Is it possible to have customized task parameter variables
I found some Task parameter variables in this document, https://community.databricks.com/s/feed/0D58Y0000A7AUWhSQO. Sounds like it will be much flexible with double curly braces for Task parameter variables. However, from the documentation, seems onl...
- 2297 Views
- 1 replies
- 1 kudos
- 1 kudos
@Yanan Zhang​ :As per the documentation you shared, Databricks Task parameter variables are used to parameterize notebook tasks in a Databricks workspace. These variables are used to pass values from the parent notebook to the child notebook that is ...
- 1 kudos
- 5540 Views
- 1 replies
- 0 kudos
I am getting an exception "RuntimeException: Caught Hive MetaException attempting to get partition metadata by filter from Hive."
I have a parquet dataframe df. I first add a column using df.withColumn("version",lit(currentTimestamp)) and append it a table db.tbl with format parquet and partitioned by the "version" column. I then ran MSCK REPAIR TABLE db.tbl. I have then create...
- 5540 Views
- 1 replies
- 0 kudos
- 0 kudos
@vikashk84The exception "RuntimeException: Caught Hive MetaException attempting to get partition metadata by filter from Hive" typically occurs when there is an issue with Hive metadata related to partitioning in Databricks. Here are a few steps you ...
- 0 kudos
- 3609 Views
- 2 replies
- 0 kudos
geojson file doesn't plot points on mapbox in jupyter notebook
I am running jupyter notebook on a Databricks cluster and attempting to render a map. I tried the example from the documentation to test and it doesn't plot the points. # Load data from sample csv data_url = 'https://raw.githubusercontent.com/mapb...
- 3609 Views
- 2 replies
- 0 kudos
- 0 kudos
@Kaniz Fatma​ Do you have any updates? were you able to reproduce the error? i am not able to plot within jupyter notebook.
- 0 kudos
- 6898 Views
- 5 replies
- 2 kudos
Resolved! Databricks Lakehouse Fundamentals Training -- Voucher
I just attended the Databricks Lakehouse Fundamentals Training (28 March 2023) and I am still wondering if I should expect any email with infos regarding a survey I should do and the voucher I should get. The Webcast finished without giving any infos...
- 6898 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Alexander Mora Araya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...
- 2 kudos
- 2448 Views
- 1 replies
- 0 kudos
Acceptable pythons to use with databricks-sql-cli (dbsqlcli)?
I have recently had to rebuild my laptop. Before it crashed I was able to use the dbsqlcli and have a set of scripts which I was using to do simple extracts into text files.The rebuild shows Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 ...
- 2448 Views
- 1 replies
- 0 kudos
- 0 kudos
@John Zajic​ you need -Python 3.7 or higher. To check whether you have Python installed, run the command Install Python, if you do not have it already installed.I believe one version should be sufficient.
- 0 kudos
- 1690 Views
- 1 replies
- 8 kudos
Databricks will release a new navigation experience to enhance user efficiency and convenience. The update includes a single, unified homepage with a ...
Databricks will release a new navigation experience to enhance user efficiency and convenience. The update includes a single, unified homepage with a list of recently viewed objects, a single search box, and common Lakehouse categories at the top of ...
- 1690 Views
- 1 replies
- 8 kudos
- 8 kudos
Thank you for sharing @Hubert Dudek​
- 8 kudos
- 3060 Views
- 2 replies
- 4 kudos
Specifying cluster on running a job
Hi,Let's say that I am starting jobs with different parameters at a certain time each day in the following manner:response = requests.post( "https://%s/api/2.0/jobs/run-now" % (DOMAIN), headers={"Authorization": "Bearer %s" % TOKEN}, json={ ...
- 3060 Views
- 2 replies
- 4 kudos
- 4 kudos
@Tjadi Peeters​ You can select option Autoscaling/Enhanced Scaling in workflows which will scale based on workload
- 4 kudos
- 3272 Views
- 3 replies
- 1 kudos
In azure databricks custom alert i am able to preview but when it goes to mail data gets missing
In azure databricks custom alert i am able to preview but when it goes to mail data gets missing
- 3272 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi, what is the custom alert about? Also, could you please raise an Azure support case on the same to triage this further? Please tag @Debayan​ with your next response which will notify me. Thank you!
- 1 kudos
- 3402 Views
- 1 replies
- 2 kudos
Execute code on Application End
Hello,I want to execute a custom code onApplicationEnd. Outisde Databricks, I have used the Spark Listener onApplicationEnd without problems. But it is not working on Databricks (I tried listener onJobEnd and this one worked).I have also tried Spark ...
- 3402 Views
- 1 replies
- 2 kudos
- 2 kudos
Did you find any solution?
- 2 kudos
- 7340 Views
- 7 replies
- 4 kudos
On-Behalf of tokens disabled for Azure Environments?
While trying to setup a Power BI connection to the Azure Delta Lake we ran into several issues around Service Principals. ​1) The API listed on the learn.microsoft site (link 1 below) indicates that there is an API you can use to create SP tokens. Wh...
- 7340 Views
- 7 replies
- 4 kudos
- 4 kudos
hello,i am new here from india, here to share some thoughts with you all
- 4 kudos
- 4014 Views
- 3 replies
- 1 kudos
Databricks Repos are pushing all changes to GitHub
When I make any changes within a repo, when I go to commit and push to GitHub if I uncheck any changes that I don't want pushed, still get pushed to Github. Any help would be appreciated
- 4014 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Vidula Khanna​ Problem is now resolved thanks, didn't have to do anything different, the functionality just started working as expected.
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
