Mount & Blade: Warband

Mount & Blade: Warband

A World of Ice and Fire
Own factions troops deserting because of relation; any fix?
Doing a Blackfyre rebellion play through starting with Valyrian as my culture; and was hostile with the Westernlands from the get go so couldn't start the Varys quests till I'd taken King's Landing from them. When the Targeryn invasion started they were automatically hostile towards me. Was attacked by Aegon in the field and managed to take hostage, so I sent him to the wall for his sword and helm. Now my Targeryn faction relation is so low my own native Targeryn troops are defecting; which is about half of my force and obviously all my new recruits from anything I control. Any way to fix? My thought was to eliminate the faction by sending them all to the wall one at a time; take their castle and eliminate the faction. However I don't know if that will allow me to keep Targeryn troops again.