- 2192 Views
- 1 replies
- 0 kudos
Job running time too long
Hi all,I'm doing my first data jobs.I create one job that consists of 4 other jobs.Yesterday I ran the 4 jobs separately and it worked fine (about half hour)-today I ran the big job, and the 4 jobs is running for 2 hours (and still running), Why is t...
- 2192 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @galzamo how are you? You can check on the SparkUI for long running stages that might give you a clue where it's spending the most time on each task. Somethings can be the reason: 1. Increase of data and partitions on your source data 2. Cluste...
- 0 kudos
- 2157 Views
- 1 replies
- 0 kudos
Expected size of managed Storage Accounts
Dear all,we are monitoring the size of managed storage accounts associated with our deployed Azure databricks instances.We have 5 databricks instances for specific components of our platform replicated in 4 environments (DEV, TEST, PREPROD, PROD).Dur...
- 2157 Views
- 1 replies
- 0 kudos
- 8049 Views
- 3 replies
- 7 kudos
Resolved! SQL Warehouse Timeout / Prevent Long Running Queries
We have an external service connecting to a SQL Warehouse, running a query that normally lasts 30 minutes.On occasion an error occurs and it will run for 6 hours.This happens overnight and is contributing to a larger bill. Is there any way to force l...
- 8049 Views
- 3 replies
- 7 kudos
- 7 kudos
@lucasrocha @raphaelblg That is exactly what I was hoping to find. Thank you!
- 7 kudos
- 1070 Views
- 0 replies
- 0 kudos
Best Practices for Filling Table Information for Databricks Assistant
Hi everyone,I’m looking for advice on the best practices for populating table information in Databricks to ensure that non-SQL users (such as PMs and Marketing teams) can easily query the tables using Databricks Assistant.Specifically, I have a few q...
- 1070 Views
- 0 replies
- 0 kudos
- 1545 Views
- 0 replies
- 0 kudos
add active directory group permission
Hi, I'm using Databricks on AWS, I did the single sign-on integration with Azure extra ID (active directory), everything is working fine, I can add users, but when I try to add a group that was created in AD, it can't be found the group.How should I ...
- 1545 Views
- 0 replies
- 0 kudos
- 1143 Views
- 0 replies
- 0 kudos
Issues while writing into bad_records path
Hello All,I would like to get your inputs with a scenario that I see while writing into the bad_records file.I am reading a ‘Ԓ’ delimited CSV file based on a schema that I have already defined. I have enabled error handling while reading the file to ...
- 1143 Views
- 0 replies
- 0 kudos
- 830 Views
- 1 replies
- 0 kudos
- 830 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mitashi12 it's possible as long as you're on DBR 14.3+. Limitations with Databricks Connect for Python .
- 0 kudos
- 1344 Views
- 0 replies
- 0 kudos
Problem using from_avro function
Hello everyone,I need your help with a topic that has been preoccupying me for a few days."from_avro" function gives me a strange result when I pass it the json schema of a Kafka topic.=================================================================...
- 1344 Views
- 0 replies
- 0 kudos
- 1376 Views
- 2 replies
- 0 kudos
Merge operation with ouputMode update in autoloader databricks
Hi team,I am trying to do merge operation along with outputMode('update') and foreachmode byusing below code but it is not updating data could you please any help on this?output=(casting_df.writeStream.format('delta').trigger(availableNow=True).optio...
- 1376 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @db_knowledge Please try .foreachBatch(upsertToDelta) instead of creating the lambda inside it. Best, Alessandro
- 0 kudos
- 1027 Views
- 0 replies
- 0 kudos
Generate embeddings from third party API in Delta Live Tables
HiWe currently have a Delta Live Tables flow through which textual data is flowing. As a final enrichment step we would also want to generate embeddings using a third party api provider (probably Voyage.AI). They support batch embedding which would g...
- 1027 Views
- 0 replies
- 0 kudos
- 1690 Views
- 2 replies
- 0 kudos
Reprocess of old data stored in adls
Hi,We have a requirement fir a scenario to reprocess old data using data factory pipeline.Here are the detailsStorage in ADLSGEN2Landing zone(where the data will be stored in the same format as we get from source),Data will be loaded from sql server ...
- 1690 Views
- 2 replies
- 0 kudos
- 0 kudos
@Retired_mod I just posted a possible solution for the above problem and it has been rejected community moderator without any explanation. This has happened to me twice in past as well.Can you please help in this case.
- 0 kudos
- 1925 Views
- 0 replies
- 0 kudos
running git clone from databricks notebook
Hey,We have a use-case where we want to clone a git repository in Azure DevOps to a storage container (Blob storage).When I'm trying to run the "git clone" command to local storage I keep getting `Operation not supported` error.Git is installed and I...
- 1925 Views
- 0 replies
- 0 kudos
- 7926 Views
- 5 replies
- 1 kudos
Resolved! I am trying to use Databricks Autoloader with File Notification Mode
When i run my command for readstream using .option("cloudFiles.useNotifications", "true") it start reading the files from Azure blob (please note that i did not provide the configuration like subscription id , clint id , connect string and all while...
- 7926 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi,I would like to share the following docs that might be able to help you with this issue. https://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html#required-permissions-for-configuring-file-notification-for-adls-gen2-and-azure-b...
- 1 kudos
- 1967 Views
- 3 replies
- 1 kudos
Databricks Certified Associate Developer Exam Got Suspended. Require support for the same.
Request #00482566Hello Team, I encountered Pathetic experience while attempting my 1st Databricks certification. I had some network issues and lighting issues. My test was stopped in the middle and I was connected with the proctor for reviewing. As r...
- 1967 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Kaniz @Sujitha @APadmanabhan @Cert-Team @Cert-Bricks @Cert-TeamOPS I have been waiting for more than 40+ hours since I raised my ticket. Till now I dint get any response from the support team nor from anyone. Can you please escalate this issue...
- 1 kudos
- 1545 Views
- 1 replies
- 1 kudos
Since moving to dbr 14.3 with python jobs I don't see the stack trace for exceptions
or even the logs don't contain the error line I see (downloaded all logs file from the UI and checked them)How can I see the stacktrace? it's essential to debug certain issues
- 1545 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for the answer, but i fail to see what it has to do with my questions. it's not a "general python error", I run lots of jobs with python on Databricks clusters and know how to run python jobs and dependencies, I'm pointing to a specific issue ...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 791 | |
| 553 | |
| 349 | |
| 287 |
