There seems to be some misconception that everyone WANTS to let any and everyone in.
I personally enjoy playing the game with my friend, so there's always a private slot. If another friend wants in, they get priority.
I always leave one slot open for the public though.