Could be game related. "machine=hercules" works fine for me with Ultima 6 (apart from looking ugly).
Have you tried different output methods? Like ddraw, opengl etc.?

yhep, tried with all the diffrent output types and several games. at work the game crashes out of dosbox at best I get a black screen (or a blinking cursour and I can hear the music as if the game is running but there is nothing being displayed

it might be game related as Xelasarg said,

I've Sharkey's 3D Pool, which works with "machine=hercules",
I also have ATech's Image72 ver4.2 (not a game but a paint-like program) that also work on the same "hercules" setting.

a bit question,
I trying to find which 'color' were better for my taste, by pressing Alt+F11
then i found this:

is the behaviour intentional?
as i tried the same on vanilla 0.74, but i unable replicate that.

Test Drive 2 works fine with hercules graphics for me. It sounds like you may be using an incomplete copy that only includes the EGA/VGA executable. There should be a DUEL.EXE launcher program and TD2CGA.EXE, TD2TDY.EXE, and TD2EGA.EXE executable files. TD2CGA.EXE supports hercules when you pass it a "herc" parameter on the command line, which is what the DUEL.EXE launcher program does.

OutRun also works fine with hercules graphics, but as with TD2 you may be trying to run the EGA/VGA executable because you have an incomplete copy. There should be an OUTRUN.EXE launcher program and OUTCGA.EXE, OUTTDY.EXE, and OUTEGA.EXE executable files. However, there is one version of OutRun that has broken hercules support due to an oversight in the launcher program where it doesn't pass the "herc" parameter to the launched executable. If you have an OUTRUN.EXE file that is larger than 1024 bytes and CHEVY.PES (CGA and Hercules), BEETLE.PES (Tandy), and CORV.PES (EGA/VGA) executable files, then you have the version with broken hercules support. As a workaround you can copy CHEVY.PES to CHEVY.EXE and run the game with the command "chevy herc".

looks like I have the broken version. I'm running dosbox through the D-Fend frontend is there a chance that could be causeing the issue?

nope, nothing I try works. I tried your work around and all i get is CGA mode when I try it through dosbox, if i acually set the card to hercules I just get the music and no display. every game ive tried will not do herc mode. must be something with my pc?

I tried your work around and all i get is CGA mode

Huh? How can you get CGA mode when the machine type is set to hercules? You are setting the machine type to hercules, aren't you? Maybe the frontend you're using is not working or you're using it wrong.

When I try it through dosbox

Where else would you be trying it?

every game ive tried will not do herc mode

Most of the games that support hercules graphics DO work if you properly configure DOSBox, although there are a few with an issue that is fixed in SVN.

must be something with my pc?

If other machine types work then that seems unlikely.

i meant the front end d-fend as opposed to just going into acual dosbox. if by chooseing herculeas you mean goining under the graphics section and choiceing the "hercules graphics card" as opposed to say "svga_s3" then yes, ive tried it under the front end and dosbox itself with the same result. i've tried using all the diffrent render options as well to the same effect.

hmm, ok I got it to work with outrun going through all your steps and changing the config file....it will not run via the front end though...wierd. thanks for the help though.