It's slightly counter-intuitive.

You have to send your invites as the game prompts you and then choose your character. Afterwards, the invite will be sent, to which the other party can respond. Otherwise, you'll be waiting in the character screen for a while for an invite that'll never get sent.