Even your average latency of 25ms is pretty high. Please talk to your storage team and have them have a good like at the filer and see what the storage latencies are like from their end.
I've certainly seen storage vMotions generate a lot of latency on our 3140 controllers. PAM cards certainly help on the filer but they're not typically the only solution to the problem.