A defensive stat would seem appropriate when comparing how two quarterbacks have played. How good or bad an opponnent's offense has has significantly less (zero direct) impact on how our quarterbacks perform. In your example, Colorado has a pretty good offense but a terrible defense. But Colorado's good offense wasn't making it tougher for Sims.
So, no, just going off an opponent's W-L record isn't "really straightforward", it's overly simplified to the point of not really telling us anything about how good or bad a QB should have played against them. Thus, talking about how good the defenses that we've faces isn't being sarcastic, it's actually looking to see what has happened.