You missed a rule, but a lot of people miss it, especially if they're taught the basic moves by a casual player (who likely also didn't know it). It's called En Passant.
Basically, if you have a pawn on your 5th rank and your opponent moves their adjacent pawn 2 spaces as its first move in an attempt to avoid capture, you can still capture it "in passing," but only if that is your next move. Once you make a different move, En Passant is no longer available.