Yes if you don't want that they start a party against you. Because if you see them "idle", they must see you "idle" too.
And some players, who don't know what a ping is, start a party against every players. ( good ping or not )
I don't deny : I just restart ( Start+Select+L1+L2+R1+R2 as on PSX ) when I see that the party'll should lag.