Connect support conversations that are closed have the "Rejoin the session" link that users can click on to reopen the conversation.
This is by design of the application and unfortunately, there's no way to disable it.
The "Rejoin the session" link is designed to appear for conversations that were closed by the user before an agent can accept it. The corresponding chat_queue_entry record for this conversation would be marked as "Closed Abandoned". Conversations that are marked as "Closed Complete" would not have this link.