From what I've read, as soon as somebody in the group joins the server, anybody can. I'm sure Valve is working on fixing this bug, so hang in there. For now, just hit esc in game and call a vote to kick a player if need be.

For when this bug gets fixed, I've made everybody an officer so that you can invite your friends to the server group as well.

It appears that it takes some time for the server to show up on your main menu the first time (20 minutes for me). After that though, its seems to be there right away just like your friends. Well, actually the "Steam Group Servers" takes a few extra seconds to pop up. In the mean time, if you want to connect right away, just type "connect 24.201.146.17:27019" in console.

Another thing you can do, it type "openserverconsole" in console, then add the server to your favorites.

On your main L4D screen, the server might show 0/4 players, but it might actually be full! If you can't join, that might be why. The openserverbrowser should show you if it's actually full or not.