- 1074 Views
- 2 replies
- 1 kudos
Job optimization
How to increase the resource efficiency in databricks jobs?We see that idle cost is more than the utilization cost. Any guidelines will be helpful Please share some examples.
- 1074 Views
- 2 replies
- 1 kudos
- 1 kudos
My main improvements are:- use singlenode job clusters for small data- cluster reuse (so use the same job cluster for multiple tasks, in parallel or serial)- use autoscaling only when it is very hard to find a good fixed sizing, otherwise go for fixe...
- 1 kudos
- 1192 Views
- 1 replies
- 1 kudos
StringIndexer method fails with shared compute
Dear TeamStringIndexer method of mlflow library upon running code on No Isolation Shared access mode data bricks cluster it works but it is failing on Unity catalog enabled data bricks cluster having Shared access mode. here is the library name: from...
- 1192 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, The issue you're encountering with the StringIndexer method from the MLflow library failing on a Unity Catalog-enabled Databricks cluster with Shared access mode is likely due to the limitations associated with Shared access mode in Unity Catalog...
- 1 kudos
- 22433 Views
- 3 replies
- 3 kudos
How can I make Databricks API calls from notebook?
Access to Databricks APIs require the user to authenticate. This usually means creating a PAT (Personal Access Token) token. Conveniently, a token is readily available to you when you are using a Databricks notebook.databricksURL = dbutils.notebook....
- 22433 Views
- 3 replies
- 3 kudos
- 3 kudos
@User16752245312 You can use Databricks Secret Scope to manage sensitive data such as personal access tokens (PATs) securely. Storing your token in a secret scope ensures you don’t hard-code credentials in your notebook, making it more secure.For mo...
- 3 kudos
- 576 Views
- 1 replies
- 0 kudos
how to get access_token from REST API without a user password
using rest api /oauth2/token how do i get access_token programmatically
- 576 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you read this blogpost?https://medium.com/@wernkai95/generate-azure-databricks-workspace-personal-access-token-for-azure-service-principal-8dff72d045c6Also refer to the API docs: https://docs.databricks.com/api/azure/workspace/tokens/create
- 0 kudos
- 856 Views
- 1 replies
- 0 kudos
Large Dataset Processing: How AI-Powered PCs Measure Up Against Cloud Solutions
Hey community,I’ve been diving deep into AI-powered PCs and their growing capabilities, particularly when it comes to processing large datasets. As cloud solutions like AWS, Google Cloud, and Azure have been the go-to for scaling data-heavy tasks, I’...
- 856 Views
- 1 replies
- 0 kudos
- 0 kudos
My answer to the same question in another topic:https://community.databricks.com/t5/data-engineering/how-does-an-ai-powered-pc-handle-large-datasets-compared-to/m-p/91889#M38291
- 0 kudos
- 27910 Views
- 4 replies
- 2 kudos
Resolved! Executor heartbeat timed out
Hello, I'm trying to read a table that is located on Postgreqsl and contains 28 million rows. I have the following result:"SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in sta...
- 27910 Views
- 4 replies
- 2 kudos
- 2 kudos
Please also review the Spark UI to see the failed Spark job and Spark stage. Please check on the GC time and data spill to memory and disk. See if there is any error in the failed task in the Spark stage view. This will confirm data skew or GC/memory...
- 2 kudos
- 863 Views
- 4 replies
- 2 kudos
I want to consolidate the delta tables historical versions and write to another table in append mode
Hi Team, I have a table let's say, employee and it has 5 versions .version 0 and 1 has column A, column B, from version 2 column A has been changed to column C and column B to column D. I want to ingest version by version without manual intervention....
- 863 Views
- 4 replies
- 2 kudos
- 680 Views
- 1 replies
- 1 kudos
Upload files to DBFS
Hello, since yesterday I cannot upload files to DBFS, only the S3 option appears, I am a little desperate with this, because I have to teach some courses and the tool does not work for me, is there a way to upload the csv files? Thank you!
- 680 Views
- 1 replies
- 1 kudos
- 1 kudos
https://community.databricks.com/t5/data-engineering/suddenly-can-t-find-the-option-to-uplaod-files-into-databricks/m-p/94359/highlight/true#M38884
- 1 kudos
- 354 Views
- 1 replies
- 0 kudos
Databricks community edition browse data to create New Table not available
Hi,From today (17-10-24), I can't seems to browse data to create New Table. Is there any change? If so how to import data on the community edition to run pySpark cluster. Please let me know as I run academic classes using this tool.Thank you Sandra
- 354 Views
- 1 replies
- 0 kudos
- 0 kudos
https://community.databricks.com/t5/data-engineering/suddenly-can-t-find-the-option-to-uplaod-files-into-databricks/m-p/94359/highlight/true#M38884
- 0 kudos
- 357 Views
- 1 replies
- 0 kudos
Not able to create table using upload file option
I was trying to create table using a CSV file in community addition. I am not able to find any option to do that on the portal.
- 357 Views
- 1 replies
- 0 kudos
- 0 kudos
https://community.databricks.com/t5/data-engineering/suddenly-can-t-find-the-option-to-uplaod-files-into-databricks/m-p/94359/highlight/true#M38884
- 0 kudos
- 527 Views
- 1 replies
- 0 kudos
DBFS FILE BROWSER BUTTON
Hello, I can not see the filestore when I open the ' catalog' tab and when I want to activate the ' dbfs file browser', the option to enable it in 'settings/Advanced' does not appear. How could I do it so I can have the Filestore active in my Databri...
- 527 Views
- 1 replies
- 0 kudos
- 0 kudos
Please mount your AWS S3 bucket. https://community.databricks.com/t5/data-engineering/suddenly-can-t-find-the-option-to-uplaod-files-into-databricks/m-p/94359/highlight/true#M38884
- 0 kudos
- 1819 Views
- 1 replies
- 1 kudos
DLT x UC x Auto Loader
Now that the Directory Listing Mode of Auto Loader is officially deprecated, is there a solution for using File Notification Mode in a DLT pipeline writing to a UC-managed table? My understanding is that File Notification Mode is only available on si...
- 1819 Views
- 1 replies
- 1 kudos
- 1 kudos
I am having the same concern and am reaching out to our Solutions Architect to better understand how AutoLoader &DLT can be used. DLT and AutoLoader should go hand in hand especially when using file notification mode.
- 1 kudos
- 33622 Views
- 8 replies
- 8 kudos
Resolved! How to merge small parquet files into a single parquet file?
I have thousands of parquet files having same schema and each has 1 or more records. But reading with spark these files is very very slow. I want to know if there is any solution how to merge the files before reading them with spark? Or is there any ...
- 33622 Views
- 8 replies
- 8 kudos
- 8 kudos
We can combine all these small parquet files into single file using optimize command..Optimize delta_table_name
- 8 kudos
- 1032 Views
- 3 replies
- 7 kudos
Cant find DBFS File Browser option in community edition settings
Hi,I was able to access DBFS, but I can no longer find the DBFS File Browser option in the settings. I used it two days ago, but now it's missing.Please help.Thanks,Shashi
- 1032 Views
- 3 replies
- 7 kudos
- 7 kudos
Thanks for the update @szymon_dybczak . It's reassuring to know that others are experiencing the same issue. I'll keep an eye out for any official announcements regarding the change.
- 7 kudos
- 1449 Views
- 5 replies
- 0 kudos
Internal Errors in Databricks Queries via Go Library.
While executing a query directly from the Databricks editor for the entire table, no errors are observed. However, when querying through the Go library, we encounter an internal error.Your guidance on this issue would be greatly appreciated.
- 1449 Views
- 5 replies
- 0 kudos
- 0 kudos
@Vetrivel wrote:While executing a query directly from the Databricks editor for the entire table, no errors are observed. However, when querying through the Go library, we encounter an internal error.Your guidance on this issue would be greatly appre...
- 0 kudos
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
