It is the controller. Anyways, I dont care anymore about it, I just threw the thing in the garbage can. Same as Virtua Fighter 4. I tell you... this game is not recommended to play with a ds2. The ds2 are just not made for fighting games, at least for VF4. The eight way dash is bad, controls are unresponsive, and using Akira is a pain in the butt. There is just one way to do all his moves, and if you miss it by a frame, the move doesnt work.
Right now, I am just using the other ds2 I have. See, the first time I bought my ps2, I got an extra ds2 cuz I knew one of them would get broken. And indeed it did, with GT3. Then I got another ds2, but this time, it came broken. The shoulder buttons would stick together after being pressed, then the x button no longer felt tight, like it was too soft to press. Then, the right digital button wouldnt work, unless pressed too hard, then why I did that, the left analog button got crazy and wouldnt stop moving on the screen. I just threw it to the garbage.
The idea about getting an extra ds2 was to use it if a relative came over to my house and wanted to play with it, but I dont get that many visits, so it doesnt matter. Now I am stuck with just one ds2 (the one it came with the ps2). I'll just get another controller If I get visited all of a sudden or this one gets broken (very unlikely).
Anyways, I have heard a lot of good things about the Madcatz Nyko Air Flow controller.