The best solution would be to stop losing your calm and just wait it out. Assuming you either 1: aren't in range to punish them or 2: don't have the reaction time to do so, it would be best to just sit there and avoid/IB the swords. If you IB them, he won't have enough time to get close enough to pressure you (if he can, then you could have punished him throwing out the ice sword).
If you really want to be aggressive, just walk forward until you are in range to punish ice swords and then wait. If you throws one out, punish it. If he tries to approach, do what you normally do to stop his approaches. If he backs off, you walk forward again until he either moves forward, throws a sword or backs himself into a corner.