Please make sure the connection from Databricks VPC to the redshift works. You can start by looking at the security groups > inbound/outbound rules for DB default SG and the redshift SG.
@kennyg To test connectivity to redshift, please use the nc command. From the exception it looks like the spark nodes are unable to establish connectivity to redshift on port 5439%sh nc -v <redshit host> 5439
This appears to be occurring due to IMDSv2 enabled in the workspace.https://docs.databricks.com/administration-guide/cloud-configurations/aws/imdsv2.html#migrateTo fix this, can you please try adding the below spark config under SQL Admin Console > ...
Jobs API 2.1 jobs list responses will be capped at a limit of 25. With the introduction of pagination in Jobs API 2.1, and to stay in-line with providing increased stability, a limit was introduced on the amount Jobs API 2.1 jobslist responses.
Hello,Instance type can never be changed to a different one if the defined type is unavailable in the AWS AZ.Have you setup auto-AZ for this job? It will let databricks try a different availability zone within the same region if the instance_type is ...
