First, ensure that your device is connected to the Internet, then try again.
If this doesn't work, you can try hosting the Room and inviting your friend to enter your Room instead.
Alternatively, extend your invitation directly to your friend through the Friend Match > Invite Players > Invite Friends screen!