The connect users are able to see any users in the instance and there is no way to restrict them from doing so.
All available releases
Connect chat is designed for the end users to chat with anyone in the organization and it does allow any sort of restrictions as of now. We have checked the backend code to verify if there is a way to restrict the user selection.
We have opened an enhancement request with the development team to consider this as a product enhancement in the future releases. The Enhancement Request number is FTASK27139.