It's hard, for sure. It doesn't mean that you can't mess around with other characters, but it'll definitely be more worth your time to go into the lab with 'em, get your shit down, and even learn match-ups that aren't very favorable.
I hate playing Sakura against Guile or Bison, but damn if I don't try my hardest.