Skinny arms or funky pattern?

Ack. I am about halfway done with one of the sleeves for my cardigan. (I decided to knit the sleeve before the second half of the front so I could have one full side done and see how it looks.) I thought it looked big, so I pinned it together and stuck my arm in - it's HUGE! Very very loose. I'm using the size small directions, and my gauge is spot-on (I'm even using the right yarn). Help!! Should I frog it? I guess I have to. But how do I prevent it from being too big again? Argh. I guess it's math time.