- 1091 Views
- 1 replies
- 0 kudos
How to transform of date to EDM
How to transform of date to EDM
- 1091 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you please provide some more details about your question?
- 0 kudos
- 2201 Views
- 1 replies
- 0 kudos
Jenkins Alternatives for Data Pipeline Automation?
I’ve been managing data pipelines with Jenkins but would like to explore other options that might work better for data-centric projects. Has anyone tried GitLab CI or Azure DevOps for similar use cases? Looking for insights into what might offer bett...
- 2201 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use Github Actions or Azure DevOps as an alternative.I use Azure DevOps Pipelines for my projects and have had very good experiences. You can find instructions on how to do the whole thing with Github Actions for example, in the official docu...
- 0 kudos
- 4419 Views
- 4 replies
- 1 kudos
Resolved! Get job run_id of run_job_task in an orchestration job
Hi,I have a Databricks job which orchestrates the run of two jobs: job1 and job2 using run_job_task.job2 depends on job1 and would need to use the run_id of job1 as a parameter. How can this be done?I see that you can only easily access the task run ...
- 4419 Views
- 4 replies
- 1 kudos
- 1 kudos
@MihaiTache You can achieve this by utilizing a combination of dbutils.widgets.get and dbutils.jobs.taskValues.set. The approach involves extracting the run_id from Job1 and passing it as a value to Job2 using taskValues.set. This allows seamless com...
- 1 kudos
- 5283 Views
- 4 replies
- 1 kudos
Can you use the Databricks API from a notebook?
I want to test out different APIs directly from a Databricks notebook instead of using Postman or CURL. Is this possible?
- 5283 Views
- 4 replies
- 1 kudos
- 1 kudos
@Panda There is no REST API for databricks. "RE" in REST stands for Ready Everywhere. You cannot connect to the API in workspace 1, from a notebook in workspace 2. Therefor it is Not Ready Everywhere. Workspace 1 cannot resolve the hostname for Works...
- 1 kudos
- 2331 Views
- 2 replies
- 1 kudos
Job optimization
How to increase the resource efficiency in databricks jobs?We see that idle cost is more than the utilization cost. Any guidelines will be helpful Please share some examples.
- 2331 Views
- 2 replies
- 1 kudos
- 1 kudos
My main improvements are:- use singlenode job clusters for small data- cluster reuse (so use the same job cluster for multiple tasks, in parallel or serial)- use autoscaling only when it is very hard to find a good fixed sizing, otherwise go for fixe...
- 1 kudos
- 28977 Views
- 3 replies
- 4 kudos
How can I make Databricks API calls from notebook?
Access to Databricks APIs require the user to authenticate. This usually means creating a PAT (Personal Access Token) token. Conveniently, a token is readily available to you when you are using a Databricks notebook.databricksURL = dbutils.notebook....
- 28977 Views
- 3 replies
- 4 kudos
- 4 kudos
@User16752245312 You can use Databricks Secret Scope to manage sensitive data such as personal access tokens (PATs) securely. Storing your token in a secret scope ensures you don’t hard-code credentials in your notebook, making it more secure.For mo...
- 4 kudos
- 1341 Views
- 1 replies
- 0 kudos
how to get access_token from REST API without a user password
using rest api /oauth2/token how do i get access_token programmatically
- 1341 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you read this blogpost?https://medium.com/@wernkai95/generate-azure-databricks-workspace-personal-access-token-for-azure-service-principal-8dff72d045c6Also refer to the API docs: https://docs.databricks.com/api/azure/workspace/tokens/create
- 0 kudos
- 3445 Views
- 1 replies
- 0 kudos
Large Dataset Processing: How AI-Powered PCs Measure Up Against Cloud Solutions
Hey community,I’ve been diving deep into AI-powered PCs and their growing capabilities, particularly when it comes to processing large datasets. As cloud solutions like AWS, Google Cloud, and Azure have been the go-to for scaling data-heavy tasks, I’...
- 3445 Views
- 1 replies
- 0 kudos
- 0 kudos
My answer to the same question in another topic:https://community.databricks.com/t5/data-engineering/how-does-an-ai-powered-pc-handle-large-datasets-compared-to/m-p/91889#M38291
- 0 kudos
- 35502 Views
- 4 replies
- 2 kudos
Resolved! Executor heartbeat timed out
Hello, I'm trying to read a table that is located on Postgreqsl and contains 28 million rows. I have the following result:"SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in sta...
- 35502 Views
- 4 replies
- 2 kudos
- 2 kudos
Please also review the Spark UI to see the failed Spark job and Spark stage. Please check on the GC time and data spill to memory and disk. See if there is any error in the failed task in the Spark stage view. This will confirm data skew or GC/memory...
- 2 kudos
- 2317 Views
- 4 replies
- 2 kudos
I want to consolidate the delta tables historical versions and write to another table in append mode
Hi Team, I have a table let's say, employee and it has 5 versions .version 0 and 1 has column A, column B, from version 2 column A has been changed to column C and column B to column D. I want to ingest version by version without manual intervention....
- 2317 Views
- 4 replies
- 2 kudos
- 1501 Views
- 1 replies
- 1 kudos
Upload files to DBFS
Hello, since yesterday I cannot upload files to DBFS, only the S3 option appears, I am a little desperate with this, because I have to teach some courses and the tool does not work for me, is there a way to upload the csv files? Thank you!
- 1501 Views
- 1 replies
- 1 kudos
- 1 kudos
https://community.databricks.com/t5/data-engineering/suddenly-can-t-find-the-option-to-uplaod-files-into-databricks/m-p/94359/highlight/true#M38884
- 1 kudos
- 847 Views
- 1 replies
- 0 kudos
Databricks community edition browse data to create New Table not available
Hi,From today (17-10-24), I can't seems to browse data to create New Table. Is there any change? If so how to import data on the community edition to run pySpark cluster. Please let me know as I run academic classes using this tool.Thank you Sandra
- 847 Views
- 1 replies
- 0 kudos
- 0 kudos
https://community.databricks.com/t5/data-engineering/suddenly-can-t-find-the-option-to-uplaod-files-into-databricks/m-p/94359/highlight/true#M38884
- 0 kudos
- 849 Views
- 1 replies
- 0 kudos
Not able to create table using upload file option
I was trying to create table using a CSV file in community addition. I am not able to find any option to do that on the portal.
- 849 Views
- 1 replies
- 0 kudos
- 0 kudos
https://community.databricks.com/t5/data-engineering/suddenly-can-t-find-the-option-to-uplaod-files-into-databricks/m-p/94359/highlight/true#M38884
- 0 kudos
- 1251 Views
- 1 replies
- 0 kudos
DBFS FILE BROWSER BUTTON
Hello, I can not see the filestore when I open the ' catalog' tab and when I want to activate the ' dbfs file browser', the option to enable it in 'settings/Advanced' does not appear. How could I do it so I can have the Filestore active in my Databri...
- 1251 Views
- 1 replies
- 0 kudos
- 0 kudos
Please mount your AWS S3 bucket. https://community.databricks.com/t5/data-engineering/suddenly-can-t-find-the-option-to-uplaod-files-into-databricks/m-p/94359/highlight/true#M38884
- 0 kudos
- 41464 Views
- 8 replies
- 8 kudos
Resolved! How to merge small parquet files into a single parquet file?
I have thousands of parquet files having same schema and each has 1 or more records. But reading with spark these files is very very slow. I want to know if there is any solution how to merge the files before reading them with spark? Or is there any ...
- 41464 Views
- 8 replies
- 8 kudos
- 8 kudos
We can combine all these small parquet files into single file using optimize command..Optimize delta_table_name
- 8 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
