When you say they can't talk to him, what do you mean? Can they see him at all? or is it that Genie returns no results to them? Have you published the agent? https://learn.microsoft.com/en-us/microsoft-copilot-studio/publication-add-bot-to-microsoft-teams
