I found it quite wierd throughout the whole game that the number of troops under your banner is just an arbitrary number that does not have any impact whatsoever...Perhaps there is some ingame mechanic that will make you fail all major battles without a certain number? But I did not encounter anything like that.