Don't match up against Nagoriyuki

You can't lose the matches you don't play.
Nagoriyuki can't win against you when you simply aren't there to begin with.
You are the victor of the duel, you pressed the button and sent them away.