Sorry, if a person is no longer part of a group, you need to be in their Telegram contacts to add them back. Note that they can still join via the group's invite link as long as they are not in the Removed Users list.