I'm going to implement my own (Dosbox-copied) VGA interrupt handler by default as a VGA Option ROM (at C000:0000) when no external VGA BIOS is supplied. I know about the checksum set by the initialisation of the ROM. I'm currently modifying the callback handler to write to the VGA Option ROM instead of the BIOS ROM used by the emulator. Currently both the initialisation call (at C000:3) and the normal interrupt handler used (at the end of the set int10 data(tables etc.)) call the relevant handler in the emulator and end with a RETI instruction. How should the initialisation call (at C000:3) return to the calling BIOS? Should it use RETI or a simple RETF instruction?

Edit: I've implemented it using the same method Dosbox uses, but wrapped in a call instruction (entry point calls the actual function, which returns using a RETF instruction, after which a RETF instruction is called to return to the calling BIOS). The same applies to the INT10h handler, but it's finished with a IRET instruction instead of a RETF instruction by the primary handler (these functions are placed at the end of the Video ROM, right after the tables of Dosbox in the ROM (8x8, 8x14 and 8x16 tables at C000:100+).

I do notice something strange when using my INT10h emulation this way: When running the Turbo XT BIOS v2.5, It seems to:
1. Set the display to 40x25 instead of 80x25(which is set using the normal VGA BIOS).
2. Stops doing anything after printing the text "EGA/VGA Graphics".
It also stops outputting blue on white after it gets to printing "n Petrosky (2008-2012). It's displaying gray on black here.

Anyone can see what's going wrong here? It's essentially using an adjusted version of the dosbox int10 handling in my case.

My source code:
http://bitbucket.org/superfury/x86emu.git

The interrupt 10h emulation is at interupts/interrupt10*.c.
The interrupt handlers used in the VGA ROM are generated at cpu/cb_manager.c. It's the cases CB_VIDEOINTERRUPT(int10h handler) and CB_VIDEOENTRY(Video BIOS entry point at C000:3). EMU_VGAROM contains the entire generated VGA ROM (filled with the Dosbox C0000-C7FFF data).