same with my own brother

There is a workaround for that.
Tell your son to log into your PC with their Steam account and start any game for like 1-2 minutes.
Then switch over to your own Steam account and send them the invite, it will now work.
Thats because to Steam it will show your sons last login and playtime location as the same as yours.
After he is part of your Steam Family it does not matter where he lives in the country, i have added family members living hundreds of kilometers aways like this.