- 1588 Views
- 2 replies
- 0 kudos
Get run details of a databricks job that provides similar data without using api '/api/2.0/jobs/runs
I have a notebook, which is attached to a task at the end of a job. This task will pull the status of all other tasks in the job and checks if they are success or failure. Depending on the result, this last task will send a slack notification (custom...
- 1588 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @mannepk85 ,You can take a look on jobs system table. Notice though, that it is in public preview now so use it with caution: https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/jobs
- 0 kudos
- 1227 Views
- 1 replies
- 0 kudos
Get the external public IP of the Job Compute cluster
We just moved our workflow from "all purpose compute cluster" to "job compute cluster". We need to find out the external public IP of the Job Compute cluster. On the all purpose compute cluster, we get the IP by attaching a notebook and run the comm...
- 1227 Views
- 1 replies
- 0 kudos
- 0 kudos
I found the following IPs from the Cluster JSON file:"driver": {"private_ip": "10.*.*.*","public_dns": "172.*.*.*","node_id": "80*****",Similar the executors configuration"executors": [{"private_ip": "10.*.*.*","public_dns": "172.*.*.*","node_id": "7...
- 0 kudos
- 920 Views
- 0 replies
- 0 kudos
Workflow scheduler cancel unreliable
Workflow paramtersWarning: 4m 30s | Timeout: 6m 50sThe jobs took 20-50 minutes to cancel.This workflow must have high reliability for our requirements. Does anyone know why the scheduler failed this morning at ~5:20 AM PT?After several failures, we'r...
- 920 Views
- 0 replies
- 0 kudos
- 2990 Views
- 7 replies
- 1 kudos
Error while installing ODBC to shared cluster
I previously used the following script to install and configure the ODBC driver on our shared cluster in Databricks, and it was functioning correctly. However, I am currently experiencing issues where the installation is not working as expected. Plea...
- 2990 Views
- 7 replies
- 1 kudos
- 1 kudos
The below approach is working for me... I had to download the packages upfront and place it on a volume though.#!/bin/bashset -euxo pipefailecho 'debconf debconf/frontend select Noninteractive' | debconf-set-selectionssudo ACCEPT_EULA=Y dpkg -i odbci...
- 1 kudos
- 968 Views
- 0 replies
- 0 kudos
No such file or directory error when accessing Azure Storage Container through Unity Catalog
Hello,I have a Databricks notebook that is attempting to unzip an archive located in Azure Storage Container. Unity Catalog is setup to provide access to the container, yet I receive the following file not found error:FileNotFoundError: [Errno 2] No...
- 968 Views
- 0 replies
- 0 kudos
- 1843 Views
- 1 replies
- 0 kudos
Databricks Asset Bundles
Hi, I'm implementing Databricks Asset bundles, my scripts are in GitHub and my /resource has all the .yml of my Databricks workflow which are pointing to the main branch git_source: git_url: https://github.com/xxxx git_provider: ...
- 1843 Views
- 1 replies
- 0 kudos
- 0 kudos
Why not use Substitutions and Custom variables that can be specified on command line using --var="<key>=<value>"?With all the features your databricks.yml would look as follows:variables: git_branch: default: maingit_source: git_url: https://git...
- 0 kudos
- 1463 Views
- 2 replies
- 1 kudos
right semi join
Hi All,I am having issue running a simple right semi join in my community databricks edition.select * from Y right semi join X on Y.y = X.a;Error : [PARSE_SYNTAX_ERROR] Syntax error at or near 'semi': extra input 'semi'. Not sure what is the issue wi...
- 1463 Views
- 2 replies
- 1 kudos
- 6762 Views
- 4 replies
- 0 kudos
ipywidgets: Uncaught RefferenceError require is not defined
Hi,When I tried to use ipywidgets, it returns the following error.I’m using Databricks with PrivateLink enabled on AWS, and Runtime version is 12.2 LTS.Is there something that I need to use ipywidgets in my environment?
- 6762 Views
- 4 replies
- 0 kudos
- 0 kudos
I am currently experiencing the same error, Azure DataBricks, Runtime version is 15.3 ML, default Notebook Editor.
- 0 kudos
- 931 Views
- 1 replies
- 0 kudos
State store configuration with applyInPandasWithState for optimal performance
Hello,We are using a stateful pipeline for data processing and analytics. For state store, we are using applyInPandasWithState function however the state needs to be persistent across node restarts etc. At this point, we are not sure how the state ca...
- 931 Views
- 1 replies
- 0 kudos
- 1077 Views
- 2 replies
- 2 kudos
Saving failed records with failed expectation name(s)
Hi all,I am using Databricks expectations to manage my data quality. But I wanted to save the failed records along side with the expectation name(s) - one or many - that the record failed. The only way I figure out is, not to use Databricks expectati...
- 1077 Views
- 2 replies
- 2 kudos
- 2 kudos
@youcanlearn Have you explore this : https://docs.databricks.com/en/delta-live-tables/expectations.html
- 2 kudos
- 810 Views
- 0 replies
- 0 kudos
use dbutils outside a notebook
Hello everyone, I want to use dbtuil function outside my notebook, so i will use it in my external jar.I have add dbutil librairies in my build.sbt file "com.databricks" %% "dbutils-api" % "0.0.6"I have import the librairie on top of my code import c...
- 810 Views
- 0 replies
- 0 kudos
- 784 Views
- 1 replies
- 0 kudos
Get Level Cluster Metrics
Im looking for a way to Optimize the consumption of the jobs in my company and the last piece of data to achieve this is the statistics of the Cluster Level Metrics called Active Tasks over time. Do we have any way to get this? Seems easy when is alr...
- 784 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @erwingm10 ,Unfortunately, currently that there is no direct endpoint in REST API to get cluster metrics. You can extract some ganglia metrics through custom scripting, but they're not so detailed like the one you looking for.Look at below links ...
- 0 kudos
- 665 Views
- 1 replies
- 0 kudos
Mosaic AI
Hi,While going through recent releases of databricks, I came to know about the Mosaic AI.And i am little bit confused what mosaic AI exactly is? what it is offering? from a data engineering point of view what benefits can i expect?Anyone please answe...
- 665 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Avinash_Narala ,I think it is more targeted to people that are creating machine learning models or data scientist.Here you can read about it and as you can see it's all related to ML models, Gen AI, RAG etc.:https://www.databricks.com/product/mac...
- 0 kudos
- 5767 Views
- 3 replies
- 0 kudos
Databricks External Data SQL Server Connection Dirty Reads
I've connected a SQL Server database as an external connection in Unity Catalog. It looks like when I write SELECT queries to that connection I end up locking my tables on the SQL Server. Is there a way to query these tables using a "with (nolock)" c...
- 5767 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks Slash - where would the "with (nolock)" command fall into the SQL Syntax...within the OPTIONS section? What would the specific command look like?
- 0 kudos
- 1801 Views
- 4 replies
- 1 kudos
What is difference between _RAW tables and _APPEND_RAW tables of Bronze-Layer of Azure Databricks
Hi Team,I would like to know difference between _RAW tables and _APPEND_RAW tables of Bronze-Layer.As both are STREAMING Tables then why we need 2 separate tables.Note: we are following Medalion Architecture. Also above tables are created via Delta L...
- 1801 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod , I saw your replies to other posts, so thought to ask you....would you like to help me on this...!!!
- 1 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
