02-27-2024 06:45 AM - edited 02-27-2024 06:46 AM
Hello all!
I've been working on integrating a Databricks Repos update API call to a DevOps Pipeline so that the Databricks local repo stays up to date with the remote staging branch (Pipeline executes whenever there's a new commit in to the staging branch). Everything seems to work fine (request returns 200), but for some reason, after the pipeline executes, the Databricks local repo has yet to perform the git pull operation.
I tried checking out to main and then staging to see if it would work, but it didn't change anything.
The weird thing is that if I run the same code using Postman or python, the repo gets updated as expected.
Here's the pipeline's code:
task: PythonScript@0
inputs:
scriptSource: 'inline'
script: |
import requests as rq
import os
access_token = os.environ["DATABRICKS_TOKEN_GENERATOR_ACCESS_TOKEN"]
url = "$(databrick-hostname)/api/2.0/repos/$(repo-id)"
headers = {'Authentication': f'Bearer {access_token}'}
data = '{"branch": "main"}'
response = rq.patch(url, headers=headers, data = data)
data = '{"branch": "staging"}'
response = rq.patch(url, headers=headers, data = data)
print(response)
displayName: 'Update Staging project'
Any clues as to where I might be messing up?
Thanks in advance!
02-29-2024 07:32 AM
@BookerE1 I found it!. There was already another thread related to this problem and someone else helped me find the solution (Problem was the pool that I was using for the pipeline)
This is the link to the other thread: https://community.databricks.com/t5/get-started-discussions/getting-html-sign-i-page-as-api-response...
Thanks for all your help and goodwill. Have a good one!
02-28-2024 12:13 AM
Hello,
I can try to give you some possible solutions for your problem, based on the web search results that I found. Here are some suggestions:
import requests as rq
import os
import time
access_token = os.environ["DATABRICKS_TOKEN_GENERATOR_ACCESS_TOKEN"]
url = "$(databrick-hostname)/api/2.0/repos/$(repo-id)"
headers = {'Authentication': f'Bearer {access_token}'}
# Add a delay of 10 seconds between each API call
time.sleep(10)
data = '{"branch": "main"}'
response = rq.patch(url, headers=headers, data = data)
print(response)
time.sleep(10)
data = '{"branch": "staging"}'
response = rq.patch(url, headers=headers, data = data)
print(response)
# Add a retry mechanism with 3 attempts and a backoff factor of 1 second
session = rq.Session()
adapter = rq.adapters.HTTPAdapter(max_retries=rq.packages.urllib3.util.retry.Retry(total=3, backoff_factor=1))
session.mount('https://', adapter)
session.mount('http://', adapter)
data = '{"branch": "main"}'
response = session.patch(url, headers=headers, data = data)
print(response)
data = '{"branch": "staging"}'
response = session.patch(url, headers=headers, data = data)
print(response)
I hope this helps you to fix your problem and update your Databricks local repo using your DevOps pipeline. If you have any other questions or requests, please let me know.
02-28-2024 11:11 AM
Hello @BookerE1 !
Thanks for your reply! Unfortunately, I tried everything and still nothing. I'm printing the text of the response that I'm getting and it seems it's trying to log in to Databricks to execute the request
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="Content-Language" content="en">
<title>Databricks - Sign In</title>
<meta name="viewport" content="width=960">
<link rel="icon" type="image/png" href="https://databricks-ui-assets.azureedge.net/favicon.ico">
<meta http-equiv="content-type" content="text/html; charset=UTF8">
<script id="__databricks_react_script"></script>
<script>window.__DATABRICKS_SAFE_FLAGS__={"databricks.infra.showErrorModalOnFetchError":true,"databricks.fe.infra.useReact18":true,"databricks.fe.infra.useReact18NewAPI":false},window.__DATABRICKS_CONFIG__={"publicPath":{"mlflow":"https://databricks-ui-assets.azureedge.net/","dbsql":"https://databricks-ui-assets.azureedge.net/","feature-store":"https://databricks-ui-assets.azureedge.net/","monolith":"https://databricks-ui-assets.azureedge.net/","jaws":"https://databricks-ui-assets.azureedge.net/"}}</script>
<link rel="icon" href="https://databricks-ui-assets.azureedge.net/favicon.ico">
<script>
function setNoCdnAndReload() {
document.cookie = `x-databricks-cdn-inaccessible=true; path=/; max-age=86400`;
const metric = 'cdnFallbackOccurred';
const browserUserAgent = navigator.userAgent;
const browserTabId = window.browserTabId;
const performanceEntry = performance.getEntriesByType('resource').filter(e => e.initiatorType === 'script').slice(-1)[0]
sessionStorage.setItem('databricks-cdn-fallback-telemetry-key', JSON.stringify({ tags: { browserUserAgent, browserTabId }, performanceEntry}));
window.location.reload();
}
</script>
<script>
// Set a manual timeout for dropped packets to CDN
function loadScriptWithTimeout(src, timeout) {
return new Promise((resolve, reject) => {
const script = document.createElement('script');
script.defer = true;
script.src=src;
script.onload = resolve;
script.onerror = reject;
document.head.appendChild(script);
setTimeout(() => {
reject(new Error('Script load timeout'));
}, timeout);
});
}
loadScriptWithTimeout('https://databricks-ui-assets.azureedge.net/static/js/login/login.21e80507.js', 10000).catch(setNoCdnAndReload);
</script>
</head>
<body class="light-mode">
<uses-legacy-bootstrap>
<div id="login-page"></div>
</uses-legacy-bootstrap>
</body>
</html>
02-28-2024 08:39 PM
@SJRDogNeedsBest wrote:Hello @BookerE1 !
Thanks for your reply! Unfortunately, I tried everything and still nothing. I'm printing the text of the response that I'm getting and it seems it's trying to log in to Databricks to execute the request
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="Content-Language" content="en">
<title>Databricks - Sign In</title>
<meta name="viewport" content="width=960">
<link rel="icon" type="image/png" href="https://databricks-ui-assets.azureedge.net/favicon.ico">
<meta http-equiv="content-type" content="text/html; charset=UTF8">
<script id="__databricks_react_script"></script>
<script>window.__DATABRICKS_SAFE_FLAGS__={"databricks.infra.showErrorModalOnFetchError":true,"databricks.fe.infra.useReact18":true,"databricks.fe.infra.useReact18NewAPI":false},window.__DATABRICKS_CONFIG__={"publicPath":{"mlflow":"https://databricks-ui-assets.azureedge.net/","dbsql":"https://databricks-ui-assets.azureedge.net/","feature-store":"https://databricks-ui-assets.azureedge.net/","monolith":"https://databricks-ui-assets.azureedge.net/","jaws":"https://databricks-ui-assets.azureedge.net/"}}</script>
<link rel="icon" href="https://databricks-ui-assets.azureedge.net/favicon.ico">
<script>
function setNoCdnAndReload() {
document.cookie = `x-databricks-cdn-inaccessible=true; path=/; max-age=86400`;
const metric = 'cdnFallbackOccurred';
const browserUserAgent = navigator.userAgent;
const browserTabId = window.browserTabId;
const performanceEntry = performance.getEntriesByType('resource').filter(e => e.initiatorType === 'script').slice(-1)[0]
sessionStorage.setItem('databricks-cdn-fallback-telemetry-key', JSON.stringify({ tags: { browserUserAgent, browserTabId }, performanceEntry}));
window.location.reload();
}
</script>
<script>
// Set a manual timeout for dropped packets to CDN
function loadScriptWithTimeout(src, timeout) {
return new Promise((resolve, reject) => {
const script = document.createElement('script');
script.defer = true;
script.src=src;
script.onload = resolve;
script.onerror = reject;
document.head.appendChild(script);
setTimeout(() => {
reject(new Error('Script load timeout'));
}, timeout);
});
}
loadScriptWithTimeout('https://databricks-ui-assets.azureedge.net/static/js/login/login.21e80507.js', 10000).catch(setNoCdnAndReload);
</script>
</head>
<body class="light-mode">
<uses-legacy-bootstrap>
<div id="login-page"></div>
</uses-legacy-bootstrap>
</body>
</html>
Hello,
I’m sorry to hear that you are having trouble logging in to Databricks.
Please Try the below step for me. I think it's helpful for you.
I hope this helps you resolve your login issue. If you have any other questions or feedback, please let me know. I’m always happy to help.
Best Regard,
BookerE1
02-29-2024 07:32 AM
@BookerE1 I found it!. There was already another thread related to this problem and someone else helped me find the solution (Problem was the pool that I was using for the pipeline)
This is the link to the other thread: https://community.databricks.com/t5/get-started-discussions/getting-html-sign-i-page-as-api-response...
Thanks for all your help and goodwill. Have a good one!
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group