The APIs that DOSBox provides to user programs are the same as those provided by DOS, so it seems like this question is really about DOS programming, not about DOSBox.

Btw, NASM works fine for targeting 8086/8088 - you just need a "cpu 8086" directive at the top of your program. Either way, you will have to use the 16-bit registers (like bp and sp) instead of the 32-bit ones (like ebp and esp) if you want to target 8086.

The 8086 doesn't have a "push iw" instruction, so instead of:

1push msg

you will have to use:

1mov ax,msg
2push ax

.

You'll also need a definition of the printf function, which you can get by assembling your program to an .obj file and then linking it with a C runtime library and startup. Exactly how to do that depends on the exact assembler, linker and C library you're using. It may be as simple as adding c0s.obj and cs.lib to the linker command line.

If you want to write the entire program in assembly and not link with external libraries (like the C runtime library) you'll need to supply your own printing function. If you don't need to do substitutions like %s then it's fairly straightforward to call the BIOS or DOS output interrupts directly. However, none of these work with zero-terminated strings (that's a C thing, not so much an OS thing) so you'll either need to write your own loop to print character by character or count the number of characters to output yourself. Here's some example code for the latter:

1push di ; Standard 8086 calling conventions mandate that di and si are callee-save.
2mov ax,ds
3mov es,ax ; scasb works with es:di. Assume string is at DS:msg
4mov di,msg
5mov al,0
6mov cx,-1
7repnz scasb ; Find end of string in DI
8lea cx,[di-msg] ; Subtract to find length
9mov bx,1 ; Handle for stdout
10mov ah,40h
11mov dx,msg
12int 21h ; DOS call to write to file/stream handle
13pop di

I'm not sure offhand if ES is also a callee-save register on any/all x86 calling conventions - if calling this code from compiled code you should check this.