I cannot curl a URL in a notebook

[jq2024]

I tried to curl the following url in a notebook

%sh

curl https://staging-api.newrelic.com/graphql -v

 But I got the following error message

{ [5 bytes data]
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
< HTTP/2 418 
< date: Tue, 27 Aug 2024 09:16:43 GMT
< content-type: text/plain
< content-length: 23
< server: cloudflare
< cf-ray: 8b9af6e55b080580-IAD
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
100    23  100    23    0     0    283      0 --:--:-- --:--:-- --:--:--   283
* Connection #0 to host staging-api.newrelic.com left intact
Unauthorized IP address

How can I fix this  Unauthorized IP address issue?

[szymon_dybczak · Accepted solution]

Hi @jq2024 ,

It looks like this API is behind some kind of firewall. You need to find IP address of your cluster and then add this IP address to whitelist.
To find your cluster ip addresses, you can use below code:

def get_external_ip(x):
    import requests
    import socket

    hostname = socket.gethostname()
    r = requests.get("https://api.ipify.org/")
    public_IP = r.content
    return(f"#{x} From {hostname} with publicIP {public_IP}.")

print('DRIVER:')
rdd1 = get_external_ip(0)
print(rdd1)
print('WORKERS:')   
rdd2 = sc.parallelize(range(1, 4)).map(get_external_ip)
datacoll2 = rdd2.collect()
for row in datacoll2:
    print(row)

[jq2024]

Thank you for the suggestion. It works