your recoder is capable of handling ZMBV ?
ffmpeg and virtualdub (with codec from startmenu installed) are capable of handling it.

Regular VirtualDub doesn't seem to want to do anything, won't even see the codecs I have installed. At first I couldn't even get the ZMBV codec to work for playback even after using the start menu install. I had to hunt down an actual installer program that installed it.

So I'm using VdubMOD, which sees all my codecs and can open the DosBOX recorded AVI's fine. It just throws me that error when I try to DivX something.

I can dump it into Xvid, but that just makes the file size bigger.. And RGB of course doesn't complain, will happilly make gigabyte sized files without an issue. Even from those formats I can't get it to DivX stuff.

So that's been my experience. In case it matters I'm using XP 64-bit

You installed the codec that comes with the dosbox 0.70 release, right?

Like I said, I tried. I clicked the shortcut but it wouldn't install properly from what I can tell. So I found an installer that installed it with the version it had bundled. Only then was I even able to view playback.

edit: I looked in VdubMOD and the ZMBV codec is listed as "Zipped Motion Block Video v0.1"

edit2: I did manually copy the dll to the windows system folder(s) and ran the start menu install again, so I believe I am using the codec supplied with DOSBox at this point..

You can encode your DOSBox videos firstly through Nero into .mpg file and then to any format you want e.g. to the .avi file trough AllToAVI. When you encode DOSBox videos into .mpg through Nero you get file 4 times bigger than the original, but by converting the .mpg file back to .avi through AllToAVI, the file size reduces 4 times, often getting even a little bit smaller than the original DOSBox file without quality loss. In fact the bigger the .mpg file to be converted, the smaller the .avi file that comes out of it through AllToAVI., but the original DOSBox videos get 4 times more compressed than the .avi files encoded through AllToAVI, and the quality of the DOSBox video files converted directly through AllToAVI is bad, therefore DOSBox video at first must be proceeded through Nero. Let me know if you want more details.