Scalers go up to 3x, or 4x in the Dosbox-X fork, but on a high resolution screen that's still pretty pitiful for 320x200 modes. With scaler set to none, you get automatic bicubic interpolation up to your set windowed or fullscreen resolution. Unfortunately, it seems like it's impossible to combine them, even though that's a pretty important use case (because bicubic on its own sucks, but in combination with a good-looking scaler can be reasonable).

If I want to play windowed, the only options are postage stamp or ugly. For fullscreen, that means I have to switch resolutions or have a postage stamp in the middle of the screen -- switching resolutions is abhorrent because it completely rearranges my multi-monitor desktop. (I also consider it a bug that fullscreenresolution=desktop still switches resolutions when you use a scaler; you have to explicitly specify the size to prevent that.) That probably made sense when monitors were mostly 1024x768, but that was a very long time ago now.

Is there any way that I can make Doxbox automatically scale up to whatever numbers I have in the resolution entries, after performing advmame3x?

This happens in .74, current SVN, and DAUM. Dosbox-X does do it, but seems to have some bugs that hang some games (even without any scaling, so it's not that).