At the point of divergence, ask if the difference is intentional, or a mistake. "The usual roll is Dex + Brawl, and you're asking me to roll Str + Intimidate. Is that a change you're making?" This gives the GM a chance to correct the mistake without losing too much face, or emphasize that this is a house rule being enforced. If you're not happy with the answer you get, then it's worth following up after the game, possibly with text references.
Regardless of the rules you're using, even if it has some kind of crazy "GM is God" rule, the game lives or dies based on a mutual agreement of the players to continue. That is, a dissatisfied player doesn't have to stick around, and everyone intuitively gets that unless they're socially broken. Right?
So assuming you're a bunch of mature people, you should be able to challenge each other in different ways and discuss problems without it being an enormous issue.
When do you just let it slide? When the ruling feels right. When the ruling doesn't bother you, even if it feels wrong. When it doesn't matter.
When do you raise the issue? When it bothers you. When it matters. When it can't wait until after the game is over. I'd say raise the issue when it matters. I wouldn't recommend holding back on /raising/ issues that matter to you in the interest of group harmony. Really, if the group harmony can't withstand someone going, "Hey, guys, is this right?" then the group is pretty fragile. Arguing the issue to death is not what I am getting at.
How do you raise the issue? Like a mature person acting as a peer of the GM. Respectfully, confidently, clearly, briefly. Speak your peace and then give the GM space to think about it. Weight how important this issue is relative to other aspects of the game. Even if you think it's the wrong decision, can you suck it up this time, as long as it's fixed going forward?
I think you're always OK to say "Hey, shouldn't it be X instead?" No one gets offended at this unless the person is coming off like a know-it all (e.g. "Well I rolled this instead because per the book that's what it is nyah"). Maybe it's a mistake, maybe it's a specific thing to that encounter, maybe it's a house rule, maybe you're wrong, whatever. If you care, it's worth asking (note "asking," not "telling.")
Where it turns into rules lawyering is when you state it and the GM states their ruling and you continue to argue it. Take that off line till after the session. And yes, I'm old school, and what the GM says goes. They're running the world and the NPCs and the gods and the rules, and you run your character inside how their world works (assuming you're playing a trad RPG, of course something more shared-narrative works differently). Bringing it up isn't annoying, insisting/arguing is. Respect your GM; they are doing a large amount of work so that you can enjoy yourself. Run your own game if you want the rules to always work the way you think they should.
In my group (in all the groups I've ever played in, actually) you're always welcome to say "I think per the book I should be rolling X instead." But if the GM says "I am ruling it is Y," that's when you shut up and keep playing.
Here's a real play example, I was GMing D&D 3.5e (Pathfinder actually) and a PC wanted his snake pet to attack a guy on horseback. I said "No, it can't reach." He said, "Well technically there's nothing in the rules that says it can't." Which is true, the default D&D combat rules don't take 3D into account worth a darn. I said "Yes, that's true. But it can't reach. Next!" Being a good player, he didn't argue the point as I had ruled and it was clear I was putting game world realism over some arbitrary ruleset. The snake had a climb speed so he commanded it to climb a tree and drop down on the guy, which worked well and was clever.
What can help sometimes as well is asking about different approaches. If it really is a "roll Dex vs roll Str" question, you can help by justifying it in game - "Well instead of an agility trick to distract him, I want to use my massive strength to break the beam he's on and throw him off." (That's a real play example from a Savage Worlds game).
I'm a consultant IRL, and run into this kind with every client I have - So, I have a standard practice in life: whenever I encounter a situation where someone has made an important declaration that I think might be in error, I say something like:
"I think I'm confused." - I always assume that I might be wrong (even when I'm pretty sure I'm not.) I would then follow up with something like "I thought that it was vs. Dex, not Str. What am I missing?"
Then, I gracefully accept whatever answer I am given with an "OK, thanks.", even if I know it is wrong. Some folks take the self-deprecation as an invitation to review the matter under discussion, some immediately apologize and correct the error (if there is one), still others explain it (again), a few park/table the matter till later, and the smallest group just ignore me. I accept it all.
Here's an important point: Sometimes I AM confused and this approach sorts things out with the minimum of fuss. Because my fellow players see me use the same approach when honestly confused and when I'm pretty sure there's a mistake, I use that ambiguity to avoid confrontation.
Building on F. Randall's excellent answer, I'd add that where the issue isn't resolved quickly, ie. clarified in a quick exchange, that it be side boarded to be resolved at a later stage.
This, of course, requires an understanding between players and GM that everyone's input into rules determination is welcome. I offer this to my players, as I think that we are all participants in the game and if a rule is just plain stupid, confusing or could be improved to match the groups style of play, then why shouldn't it?
We then discuss these more weightier issues at the ed of the session.
For quick arbitration that keeps the action going, let the GM's word be final. But for lengthier issues that are more involved, why not involve everyone in the decision making?
The important elements are to remember that rules arguments are generally not fun for the parties witnessing them, and often are not fun for those participating in them.
In addition to the answers above, this depends on the social context. Speak up if:
Consider keeping quiet if:
A lot of people seem to go with the "I thought that it was vs. Dex, not Str. What am I missing?" line of thought. I find it helpful to include why as well, this way you are actually giving the GM information on which to base their decision, rather than posing them with a question they might not understand. For instance, something along the lines of, “Con, not Dex? Do I only get to resist the attack and not avoid it?”
This does two things, lets them save face and reminds or gives them new information about a possible additional mechanic.
There is always the chance there is something unique or rare taking place and so I would never assume I knew what the roll must be. However, no GM should be threatened by someone asking them to clarify what the player is doing. It also, isn’t the worst idea for a GM who doesn’t have a great grasp of rules to rely on someone else for base forumlas.
The worst time is during game time. Ask for clarification once, then drop it. The logistics of most play groups is that actual playtime is going to be the most limited resource you have, and it's usually not worth it to burn playtime for a rules argument.
After the game, e-mail is your friend. Wait a day and send one to the DM, preferably along the lines of "I'm still confused, page 85 makes it look like it's an X+Y roll, why do we roll Z?"
You'll either get a direct response of "We do Z because X+Y makes no sense," "I don't know, now that you mention it X+Y does look better," or the DM is going to throw a fit.
If he actually throws a fit, never play with him again because he's at best a total jerk. Otherwise, you'll get a good calm answer, in writing, away from time concerns, when everyone has time to review the appropriate materials, and those are the answers that make the difference in the long run.
There are many great answers here that focus on what to do when an instance like this comes up (i.e. how to raise a specific question during or after the game).
What I'll add is that (given this will probably come up again) it can also be helpful to have a general conversation that prepares you for the next time. Depending on my relationship with the GM, I might ask when it's just the two of us, or as a group before/after a play session.
Planning together for the next time:
"Hey, I have a question about what you like from players when you're GMing. If you make a ruling that I'm confused by or think may not match the rules as written, what's your preference?"
Some folks will have a preference for when, or how it's raised, or how to let you know to drop it, or they might let you know that they are more into house ruling based on what seems cool to them than on following rules as written, or whatever.
Depending on the GM's answer, I might ask for more details, e.g.
"Do you like folks to raise it during the group? Wait until after? Do you have a preference about how it's raised?"
This generalized conversation does a few things: