Thinking about it critically, the two top tiers are probably his worse matchups.
Vs. Aigis you're pretty much at a disadvantage whether your blocking or being hit, and since Yosuke's health isn't very high, and Aigis does great damage, you don't get to fuck up very many defensive situations before you're dead. I imagine 6-4 Aigis.
Vs. Mitsuru she's got pretty much all ranges covered. Makes it hard to get it with her j.a, and her buttons in general just having superior range. Plus, she does great damage too. God help you if you get Fatal Countered with no burst. Probably 6-4 Mitsuru.