Sorry, if a person is no longer part of a channel, you need to be in their Telegram contacts in order to add them back. Note that they could still join via the channel's invite link as long as they are not in the Removed users list.