BAD_REQUEST: ExperimentIds cannot be empty when checking ACLs in bulk

[sarvar-anvarov]

I was going through this tutorial https://mlflow.org/docs/latest/getting-started/tracking-server-overview/index.html#method-2-start-yo..., I ran the whole script and when I try to open the experiment on the databricks website I get this error: "BAD_REQUEST: ExperimentIds cannot be empty when checking ACLs in bulk". I get the same error when I try to create a new experiment directly on the databricks website.

[Kaniz]

Hi @sarvar-anvarov, It appears that you’re encountering an issue related to ACLs (Access Control Lists) when working with Databricks and Amazon S3.

Let’s break down the problem and explore potential solutions:

1. ACLs and Object Ownership:

   • ObjectOwnership is a setting that affects how ACLs are handled in Amazon S3 buckets.
   • By default, for newly created buckets, ObjectOwnership is set to BucketOwnerEnforced. This means that ACLs are disabled, and the bucket owner automatically owns and has full control over...¹.
   • ACLs no longer affect access permissions to data in the S3 bucket; instead, the bucket uses policies exclusively to define access control.

2. The Error Message:

   • The error message you’re encountering, “BAD_REQUEST: ExperimentIds cannot be empty when checking ACLs in bulk,” suggests that there’s an issue related to ACLs during an operation involving multiple experiments.
   • It seems to be related to the interaction between Databricks and Amazon S3.

3. Possible Solutions:

   • Let’s explore some steps to resolve this issue:
     • 1. Check Existing ACLs:
       • Ensure that there are no existing ACLs set on the bucket.
       • If you want to apply the bucket owner enforced setting, you must remove access granted to other AWS ...².
     • 2. Bucket Policy:
       • Consider using a bucket policy instead of ACLs.
       • Create a bucket policy that allows public read access. For example:

         {
             "Version": "2012-10-17",
             "Statement": [
                 {
                     "Sid": "Stmt1380877761162",
                     "Effect": "Allow",
                     "Principal": {
                         "AWS": "*"
                     },
                     "Action": "s3:GetObject",
                     "Resource": "arn:aws:s3:::bucketname/*"
                 }
             ]
         }
         

       • Make sure to replace "bucketname" with your actual bucket name.
     • 3. Disable ObjectOwnership:
       • If you want to enable ACLs for a bucket, set the ObjectOwnership parameter to ObjectWriter during bucket creation.
       • Alternatively, call DeleteBucketOwnershipControls after creating the bucket ³.

4. Review AWS Changes:

   • AWS made changes to the default security configuration for new S3 buckets starting in April 2023.
   • New buckets now have S3 Block Public Access enabled by default, and ACLs are disabled.
   • If you have specific use cases for public bucket access or ACLs, consider adjusting your settings ac...².

Remember to adjust the steps based on your specific use case and ensure that your bucket configuration aligns with your requirements. If you’re still facing issues, consider reaching out for further assistance. 🚀

[stanjs]

Hi, I also encounter the same issue. I used Mlflow to log results as experiments in Databricks CE. It works perfectly fine before but I suddenly encountered this error. I am sure I didn't change any settings. And strangely I noticed that the URL is different from before when I clicked the same experiment.

[kdfkas]

Hi @Kaniz,

I'm also having the same issue in Databricks CE. 
This error is related to S3 bucket hosting CE. Is it possible to change the bucket policy on user's end?
I researched it, but I couldn't get the answer.

[Kaniz]

Hi @sarvar-anvarov, It seems you’ve encountered an issue related to ACLs (Access Control Lists) while working with MLflow and Databricks.

Let’s break down the problem and explore potential solutions.

1. Background:

   • ACLs are used to manage access permissions for Amazon S3 buckets and objects.
   • Object ownership is a setting that determines how ACLs and ownership work together.
   • BucketOwnerEnforced is an ObjectOwnership setting that disables ACLs and ensures the bucket owner au...¹².

2. Error Message:

   • The error message you encountered, “BAD_REQUEST: ExperimentIds cannot be empty when checking ACLs in bulk,” suggests that there’s an issue with ACLs when checking experiment permissions in bulk.

3. Possible Solutions:

   • 1. Object Ownership Settings:
     • Bucket owner enforced (default): This setting disables ACLs, and the bucket owner automatically owns and controls every object in the bucket. ACLs no longer affect access permissions to data in the S3 bucket. The bucket uses policies exclusively for access control ².
     • Solution: Check if your bucket has the correct ObjectOwnership setting. If it’s set to BucketOwnerEnforced, ACLs won’t work. You may need to adjust this setting.
   • 2. Existing Bucket ACLs:
     • The error message indicates that you have existing bucket ACLs. Applying Bucket owner enforced requires removing access granted to other AWS accounts or groups from...¹.
     • Solution: Review your bucket ACLs and ensure they align with your desired configuration. If you want to apply the bucket owner enforced setting, consider removing existing ACLs.
   • 3. Bucket Policy:
     • You attempted to use a bucket policy to grant public access. However, if your bucket already has ACLs, applying bucket owner enforced is not straightforward.
     • Solution: Consider removing existing ACLs and then applying the bucket owner enforced setting. Adjust your bucket policy accordingly.
   • 4. CloudFormation:
     • If you’re using CloudFormation, ensure that your template doesn’t conflict with the ObjectOwnership settings.
     • Solution: Check your CloudFormation template and verify that it aligns with the desired settings ³.

4. Next Steps:

   • Review Object Ownership:
     • Familiarize yourself with the concept of Object Ownership and how it affects ACLs.
   • Check Bucket Settings:
     • Verify the ObjectOwnership setting for your bucket.
   • Adjust ACLs or Policies:
     • Depending on your use case, either adjust ACLs or create/update bucket policies.
   • Test Again:
     • After making changes, try accessing your experiment again.

Remember to proceed carefully, especially if you’re new to AWS. If you encounter any issues, consult AWS documentation or seek assistance from experienced AWS users. Good luck! 🚀

[stanjs]

Hi did u resolve that? I encountered the same error

[sarvar-anvarov]

Hi, @stanjs , couldn't resolve that. I forgot to mention that I am also using databricks ce, so I can't change any s3 bucket configurations or etc. I'm guessing they are provided by MLflow or Databricks, but I am not sure.