It's not the horse duel. I haven't done it either. And for me, it did start with the sparring practice with the knights. Though I did it last. I first did the shooting practice, then the race practice.
If it can be triggered during either race practice or shooting practice, this could be why...