The Fantasy Trip -- Melee, Wizard, the advanced versions of both, and In the Labyrinth -- used 3d6 by default. Roll under an ability score (there were only three;) the highest possible roll is a fumble. The mechanic has two "lapses of elegance": there are little modifiers to add or subtract from your target score, and the critical success is the lowest possible roll, instead of an exact match with your ability score. Those are easily fixed. But more importantly, aside from a few modifiers, difficult or easy tasks add or drop dice; a parry or dodge increases the number of dice that the attacker rolls to 4 or 5 dice, for example. In the Labyrinth added Talents, which generally drop a die from the roll. As it so happens, one of the "classic" D&D lines, BECMI I believe, included ability checks that worked almost the same way: roll 3d6 under an ability score, or 4 dice for difficult tasks, 2 dice for easy tasks. It's this dice trick that's key to making a more elegant system.
Here's my modified version of the TFT mechanic:
- Roll 3d6 under ability, 4 dice for unfavorable situations, 2 dice for favorable situations. An Expert drops one die, a Master drops 2 dice. Add 2 dice if task is difficult even under perfect conditions or completely outside character's experience/training.
- Exact Match = critical success, All dice matching on a failed roll = fumble (all 6s = automatic fumble, regardless of skill.)
- Higher is better for opposed rolls; having a higher class level counts as a favorable situation.
If the roll is less than 2 dice, don't bother to roll.
For combat, wearing any armor at all or using a shield adds 1 die (these are cumulative.) Low AC (2 to 5) also acts as a damage cap. Use standard damage rolls. For Ascending AC, use opposed rolls: attacker rolls under Dex, defender rolls under AC, highest wins.) For ranged attacks, treat short and long ranges as favorable/unfavorable situations.