I wanted to highlight and discuss part of Mphs. Steve's comment on his experience with the Subhex Wilderness Crawl system.
"...The drop die method was better for my purposes since it puts more points of interests on a single sheet. I used two of the paths off the starting point to create a coastline. It ended up being too straight so I used it as a general guide with natural looking irregularity added by using a rule that I borrowed from ICE's Campaign Law for creating coastlines. The end result was a map of a nicely detailed small area with a lot of opportunity for adventure."
I don't know what the rule from Campaign Law was (never had the Rolemaster books,) although I'd be interested in hearing about it. But I can understand why the subhex system probably wouldn't make satisfactory coastlines.
Theoretically, you can treat a coastline as a path, rolling d12s to establish where the coast changes direction. But the problem is, the path rolls are geared toward the viewpoint of traveling characters. The length of each leg of a journey is based on travel speed modified by terrain. These things should not affect coastlines in the same way. Also, coastlines are more fractal and more "noisy". In theory, what you should do for a coastline would be to roll d12s to establish a rough coastline, then for each segment of the coast, roll more d12s to divide it into smaller segments, and then repeat a couple times, zooming in to smaller and smaller coastal changes.
That's kind of unwieldy.
I haven't fully thought it through, but a good compromise would be to roll some elevation checks -- the roll with three light dice and three dark dice, but use more dice, something like 8d6 light and 8d6 dark. Read them in a line to establish broad details of the coastline: light dice are spits of land or rocky cliffs jutting out into the water, the dark dice are coves and inlets. That's the rough outline, and the places where the first draft of the coastline changes are the defining points. For each pair of defining points, roll 5d12 to establish the deviations of that section of coastline from the straight line. Or, if you prefer, start at one point and roll 1d12 for every four squares of coastline until you reach the next defining point. That might work better for rolling a coastline on the fly when players decide to follow and map it.