- 2438 Views
- 2 replies
- 0 kudos
How we can read data from adls gen 2 using bash (%sh) command.(without mounting)
Hi @Ananth Arunachalam/Team,Can we read file from ADLS gen 2 using shell script (%%bash or %%sh ) without doing mounting.​ Please let me know. Thank you.​
- 2438 Views
- 2 replies
- 0 kudos
- 0 kudos
@S S​ you can access data in ADLS GEn2 using multiple ways, please check below article.easy way is using storage account access key method https://learn.microsoft.com/en-us/azure/databricks/storage/azure-storage
- 0 kudos
- 4810 Views
- 2 replies
- 3 kudos
- 4810 Views
- 2 replies
- 3 kudos
- 3 kudos
The workaround is available here.Idea is to use spark.conf.set(), spark.conf.get() methods.
- 3 kudos
- 7100 Views
- 3 replies
- 0 kudos
DE 4.1 - DLT UI Walkthrough Error in Classroom Setup
Trying to follow along with the DLT videos in the academy. I get an error when running the setup script. Error trace below. It stems from running Classroom-Setup-04.1DA = DBAcademyHelper(course_config=course_config, lesson_config=...
- 7100 Views
- 3 replies
- 0 kudos
- 0 kudos
I tried with Py4J versions 0.10.9.5, .3, and .1. None of those versions worked. I also tried upgrading the runtime to 13.0 and 12.1 and saw the same issue. The 13.0 runtime upgraded Py4J to 0.10.9.7 and that didn't resolve the issue. The error stayed...
- 0 kudos
- 42365 Views
- 9 replies
- 6 kudos
Resolved! How to access the result of a %sql cell from python
I see the way to move from python to sql is to create a temp view, and then access that dataframe from sql, and in a sql cell. Now the question is, how can I have a %sql cell with a select statement in it, and assign the result of that statement to ...
- 42365 Views
- 9 replies
- 6 kudos
- 6 kudos
Results from an SQL cell are available as a Python DataFrame. The Python DataFrame name is _sqldf.To save the DataFrame, run this code in a Python cell:df = _sqldfKeep in mind that the value in _sqldf is held in memory and will be replaced with the m...
- 6 kudos
- 5400 Views
- 4 replies
- 4 kudos
Urgent - Use Python Variable in shell command in databricks notebook
I am trying to read a csv and do an activity from azure storage account using databricks shell script. I wanted to add this shell script into my big python code for other sources as well. I have created widgets for file path in python. I have created...
- 5400 Views
- 4 replies
- 4 kudos
- 4 kudos
You can mount the storage account and then can set env level variable and can do the operation that you want.
- 4 kudos
- 3958 Views
- 9 replies
- 5 kudos
It would be great if Databricks starts increasing the number of rewards, as the no of users in community ae increasing. When we want to redeem somethi...
It would be great if Databricks starts increasing the number of rewards, as the no of users in community ae increasing. When we want to redeem something the limited goodies available in community rewards portal are out of stock. So its better to incr...
- 3958 Views
- 9 replies
- 5 kudos
- 5 kudos
@Kaniz Fatma​ @Vidula Khanna​ Hi. I just see the below rewards available to redeem. Is this different based on the location?
- 5 kudos
- 7690 Views
- 2 replies
- 1 kudos
Can assign a default value for job parameter from the widget?
The Databricks widget (dbutils) provides the get function for accessing the job parameters of a job.​dbutils.widgets.get('my_param')Unlike Python dict, where get returns None or an optional argument if the dict doesn't contain the parameter, the widg...
- 7690 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Mattias P​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
- 1 kudos
- 2754 Views
- 2 replies
- 0 kudos
Databricks-Connect Error occurred while running *** java.io.StreamCorruptedException: invalid type code: 00
I am using databricks-connect to access a remote cluster. Everything works as expected and I can set breakpoints and interrogate the results, same for when it trys to execute the following code:val testDF = spark.createDataFrame(spark.sparkContext .e...
- 2754 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @James Metcalf​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 0 kudos
- 7597 Views
- 2 replies
- 2 kudos
Resolved! DLT data quality UI was present last week. Now absent. Did I change a setting?
Last week, I started running a DLT pipeline with expectations that dropped rows on streaming live tables. In the side bar for a table, I saw a nice circular chart with Written/Dropped rows and Failed records stats.Today, I ran a similar DLT pipeline ...
- 7597 Views
- 2 replies
- 2 kudos
- 2 kudos
@Kevin Rossi​ :The circular chart with Written/Dropped rows and Failed records stats that you saw in the sidebar of a table in Delta Live Tables (DLT) is a built-in feature called "Data Quality Metrics" that provides a visual representation of the da...
- 2 kudos
- 1465 Views
- 1 replies
- 4 kudos
Resolved! while creating serverless warehouse we are receiving below message <workspaceId> is no longer eligible for Serverless Compute. Please reach out to your administrator
HI team,As far as limitations and pre-requisites we have met all, able to create warehouse in other workspace which are part of same account, for one of the workspaces we are seeing above issue. we don't any clear error log other than <workspaceId >i...
- 1465 Views
- 1 replies
- 4 kudos
- 4 kudos
@karthik p​ Your workspace might have blocked from serverless feature if there are some uncleared bills. If that's not the case, please file a support case to us.
- 4 kudos
- 1832 Views
- 1 replies
- 1 kudos
Is it possible to have customized task parameter variables
I found some Task parameter variables in this document, https://community.databricks.com/s/feed/0D58Y0000A7AUWhSQO. Sounds like it will be much flexible with double curly braces for Task parameter variables. However, from the documentation, seems onl...
- 1832 Views
- 1 replies
- 1 kudos
- 1 kudos
@Yanan Zhang​ :As per the documentation you shared, Databricks Task parameter variables are used to parameterize notebook tasks in a Databricks workspace. These variables are used to pass values from the parent notebook to the child notebook that is ...
- 1 kudos
- 4148 Views
- 1 replies
- 0 kudos
I am getting an exception "RuntimeException: Caught Hive MetaException attempting to get partition metadata by filter from Hive."
I have a parquet dataframe df. I first add a column using df.withColumn("version",lit(currentTimestamp)) and append it a table db.tbl with format parquet and partitioned by the "version" column. I then ran MSCK REPAIR TABLE db.tbl. I have then create...
- 4148 Views
- 1 replies
- 0 kudos
- 0 kudos
@vikashk84The exception "RuntimeException: Caught Hive MetaException attempting to get partition metadata by filter from Hive" typically occurs when there is an issue with Hive metadata related to partitioning in Databricks. Here are a few steps you ...
- 0 kudos
- 2821 Views
- 2 replies
- 0 kudos
geojson file doesn't plot points on mapbox in jupyter notebook
I am running jupyter notebook on a Databricks cluster and attempting to render a map. I tried the example from the documentation to test and it doesn't plot the points. # Load data from sample csv data_url = 'https://raw.githubusercontent.com/mapb...
- 2821 Views
- 2 replies
- 0 kudos
- 0 kudos
@Kaniz Fatma​ Do you have any updates? were you able to reproduce the error? i am not able to plot within jupyter notebook.
- 0 kudos
- 5448 Views
- 5 replies
- 2 kudos
Resolved! Databricks Lakehouse Fundamentals Training -- Voucher
I just attended the Databricks Lakehouse Fundamentals Training (28 March 2023) and I am still wondering if I should expect any email with infos regarding a survey I should do and the voucher I should get. The Webcast finished without giving any infos...
- 5448 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Alexander Mora Araya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...
- 2 kudos
- 1659 Views
- 1 replies
- 0 kudos
Acceptable pythons to use with databricks-sql-cli (dbsqlcli)?
I have recently had to rebuild my laptop. Before it crashed I was able to use the dbsqlcli and have a set of scripts which I was using to do simple extracts into text files.The rebuild shows Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 ...
- 1659 Views
- 1 replies
- 0 kudos
- 0 kudos
@John Zajic​ you need -Python 3.7 or higher. To check whether you have Python installed, run the command Install Python, if you do not have it already installed.I believe one version should be sufficient.
- 0 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
