What if, instead of your class determining your hit point die and progression, your race did? Leaving your chosen class to give you a penalty, or bonus depending on what you chose? I know it's a mechanical thing, but it is something that always bugged me about any of the d&d's that your health level was determined by your profession, and not race.This is, of course, another example of the "luck vs. health" divide that runs through the hobby. If you interpret hit points as resistance to physical damage, even only a little bit, then linking hit dice to something non-physical like class becomes an annoying flaw. (And I'm not even going to get into the "class vs. profession" divide hinted at here...)
My long-time readers know that I'm firmly in the "hit points are luck" camp. But if hit points are luck, what are hit dice? They aren't just the potential for luck; they were also originally an abstract indicator of combat potential. Creatures with higher hit dice have a greater chance to hit and take longer to defeat. That's still true for monsters, but when Greyhawk changed the way hit dice worked for character classes, the meaning of "hit dice" became obscured. That's one of the reasons that I stick to the original d6-only hit dice scheme for fighters and magic-users, and why I tweaked the cleric progression for Liber Zero so that I could use a straight Target 20 approach for all attacks.
I do allow physical size to exert a minor effect on hit dice. Human-sized creatures are all 4 hit dice or less, but larger than human creatures multiply 4 hit dice by their size relative to humans to get their base hit dice. Thus, giant who are twice human size have 8 HD or more. This reflects the fact that size influences combat potential, though, and has nothing to do with the amount of physical damage a larger creature can take.