No.

Originally posted by Matthew Earl:

I have 3 people in my party and see horse slot for each. I only have one horse on myself. Do I need to slot the other horses on each character to go faster?

battle crystals also go in the horse slot so no but if you dont give them one you risk them being slow on the map which is bad if you get sieged somewhere far

I guess you can give every hero an enhanced Armored Horses for the stat bonus and not the world movement.