VOGONS


New 16-bit-era benchmark project (TOPBENCH)

Topic actions

First post, by MobyGamer

User metadata
Rank Member
Rank
Member

Hi all, I've finished all milestone work on a new project and would like to get some feedback and testing, if you have some spare time. While I've loved DOSBox for years, especially its ability to "dial" the speed realtime, I've always been frustrated that DOSBox is not (and never can be) cycle-accurate. I never know exactly what kind of machine I'm emulating (286-12? 386-33? etc.) and just have to trust that "lots of cycles" is good enough for the game I'm trying to run.

To address this, as well as other deficiencies in other 1980s benchmarks that have always bugged me, I designed a new benchmark called TOPBENCH. The three main features of TOPBENCH are:

- Scores and timings are gathered from real systems and recorded into a local database. You can compare any two systems to see how they stack up against each other, as well as the current system. (This answered a lot of questions for me, such as exactly how much speedup do you get from an NEC V20/V30, or how wait states affect speed.)

- A "realtime benchmark" mode runs the benchmark continuously and shows you which system in the database most closely matches current performance. This allows you to "dial" DOSBox cycles up and down and see what kind of machine you are most closely emulating.

- All source is fully available (most importantly, the actual assembler metric code and score rationale are fully open, so that when you see a "Score" of "43", or a "3DGame" metric timing of 2314us, you can research exactly what that means). This is meant to help emulator authors (MESS, PCEM, etc.) compare their cycle-exactness to real systems.

TOPBENCH can run on any system that runs DOS, but it is meant for 486s and lower as the metrics are all written in 16-bit realmode assembly (which was deliberate, so that comparisons could be made all the way back to the original IBM PC). Website is http://dosbenchmark.wordpress.com where you can grab the binary+database, browse the source, learn about other 1980s DOS-era benchmarks, etc.

Before I go bothering the folks in the DOSBox forums, I'd like just a little bit more testing and, hopefully, more system entries to add to the database. I've already gotten a lot of help from the vintage-computer forum guys, but I'd love any comments, criticisms, suggestions, etc. from you guys as well. Let me know!

Reply 1 of 32, by sepultribe

User metadata
Rank Newbie
Rank
Newbie

very nice work friend.

Reply 2 of 32, by Stiletto

User metadata
Rank l33t++
Rank
l33t++

Congratulations - this is amazing stuff!

"I see a little silhouette-o of a man, Scaramouche, Scaramouche, will you
do the Fandango!" - Queen

Stiletto

Reply 3 of 32, by elianda

User metadata
Rank l33t
Rank
l33t

Hey trixter, while reading sources may be fun it is rather time consuming. Can you give a brief overview here or in your blog about the ideas/focusing behind the specific benchmark routines. You reference Dosbox as comparison metric and you know the cycle count of 1 per instruction is very different from real world CPUs.
Prepare for some critical review 😉.

Retronn.de - Vintage Hardware Gallery, Drivers, Guides, Videos. Now with file search
Youtube Channel
FTP Server - Driver Archive and more
DVI2PCIe alignment and 2D image quality measurement tool

Reply 4 of 32, by elianda

User metadata
Rank l33t
Rank
l33t

If I load QEMM9 with Stealth on my 386 the MemEA goes down from 100 to 74 while Opcodes goes up from 66 to 71.
Opcodes seem to make sense, why does MemEA goes down?

I would also like to be able to choose a system myself in the live comparison for quick testing. (like with qemm / without qemm) So I don't have to save a result for each setting.

I would also like to see that it remembers the previous content of the input fields for the result entry, so that I do not have to enter every time most of the same data for the same system with slightly changed settings.

And some kind of quicksearch, find as you type or NC style with ALT+name for the result browser would be fine.

UIDF8DE6985,"homebuilt 386 system",91,163,66,224,100,66,AMD Am386DX,40,"R(C)1985-1991,American Megatrends Inc.,All Rights Reserved.,1346 Oakbrook Dr.,#120,GA-30093,USA.(404)-263-8181. (07/07/91, rev. 0)",19910707,F8DE,VGA,"VGA, Tseng ET 4000 with HiColor RAMDAC, DRAM, VESA, 1024kb Video","Am386DX-40/i387-33 ETEQ386-8 32MB/ET4000AX/GUSPnP/SB16+WB1/ELNK3","Michael Huth"

UIDF8DE8143,"homebuilt 386 Turbo Off",29,715,277,398,434,289,AMD Am386DX,10,"R(C)1985-1991,American Megatrends Inc.,All Rights Reserved.,1346 Oakbrook Dr.,#120,GA-30093,USA.(404)-263-8181. (07/07/91, rev. 0)",19910707,F8DE,VGA,"VGA, Tseng ET 4000 with HiColor RAMDAC, DRAM, VESA, 1024kb Video","Am386DX-40/i387/ETEQ386-8/32MB/GUSPnP/SB16+WB1/ELNK3","Michael Huth"

Retronn.de - Vintage Hardware Gallery, Drivers, Guides, Videos. Now with file search
Youtube Channel
FTP Server - Driver Archive and more
DVI2PCIe alignment and 2D image quality measurement tool

Reply 5 of 32, by RacoonRider

User metadata
Rank Oldbie
Rank
Oldbie

Do you need samples from 486DX2-50, 66, 80, 386SX-40?

Reply 6 of 32, by MobyGamer

User metadata
Rank Member
Rank
Member
elianda wrote:

Can you give a brief overview here or in your blog about the ideas/focusing behind the specific benchmark routines. You reference Dosbox as comparison metric and you know the cycle count of 1 per instruction is very different from real world CPUs.

The benchmark wasn't created specifically for DOSBox, but more for every emulator. This is why individual timings of code segments are recorded into the database, so that emulator authors who are striving for cycle-exactness have a reference. As to why there are different metrics, they were divided into various profiles that target what varies the most from system to system and CPU to CPU (memory speed, effective address calculation, etc.) Only the realtime benchmarking, where it compares the current system to what is in the database, was created with DOSBox specifically in mind (because I've always wanted some realtime feedback when I dial DOSBox cycles up and down in realtime).

TOPBENCH is a system benchmark, not a CPU benchmark (although 4 of the five tests are CPU-focused). The five metrics are:

Opcode Exercise: As the name implies, every opcode that doesn't have variable exexecution time is exercised. https://github.com/MobyGamer/TOPBENCH/blob/ma … er/_OPCODES.INC lists all of the opcodes that are NOT exercised and why.

3DGame opcode mix: An instruction mix with the same distribution as some 16-bit era 3D games. I highly recommend looking at https://github.com/MobyGamer/TOPBENCH/blob/ma … er/_3DGAMES.INC to learn how this was created and the rationale behind it.

Memory exercise: System RAM block read and write speeds are measured.

Video RAM write speed: Writes to Video adapter RAM are measured.

Effective address calculation: This varied from CPU to CPU and is one of the more commonly-overlooked areas in getting an emulator cycle-exact, so I felt it necessary to dedicate a section to this.

Prepare for some critical review 😀

That's fine, I'm ready for it. 😀 Just keep in mind that TOPBENCH was intentionally written targeting 16-bit real-mode code, so it is not the correct choice for some research, such as the recent "best 486" and "best 686" mega-research projects I've seen here (which were both mighty impressive!). TOPBENCH doesn't touch a single 32-bit register or operation, and that's deliberate, because if it did, it couldn't be used as a level comparison for 286s and lower.

Reply 7 of 32, by MobyGamer

User metadata
Rank Member
Rank
Member
elianda wrote:

If I load QEMM9 with Stealth on my 386 the MemEA goes down from 100 to 74 while Opcodes goes up from 66 to 71.
Opcodes seem to make sense, why does MemEA goes down?

I honestly don't know, because I would have expected both to go down since V86 management adds overhead. The benchmark is supposed to run with a clean boot, so I'll add a V86 check to the next release.

I would also like to be able to choose a system myself in the live comparison for quick testing. (like with qemm / without qemm) So I don't have to save a result for each setting.

Interesting; I'll add that to the to-do list to see how I can implement it. Also, try /i on the command-line which spits out the Score quickly and then exits, if you were looking for a tweak-test-tweak-test-etc. cycle. The Score is the synthetic number meant to quickly and directly compare systems (meaning, the system with the higher Score is faster).

I would also like to see that it remembers the previous content of the input fields for the result entry, so that I do not have to enter every time most of the same data for the same system with slightly changed settings.

Because I don't accept multiple entries per system (see below), I can't think of any field that would need to persist between entries other than Submitter. It's trivial to persist that in the same session so I'll add that to the to-do list.

And some kind of quicksearch, find as you type or NC style with ALT+name for the result browser would be fine.

ALT+letter should definitely be possible; I'll add that to the to-do list. I'll have it jump to the first system when in sorted-by-alpha mode, and jump to the next system that matches in sorted-by-speed mode.

As for your sample results, if you could post those exactly as they appear in DATABASE.INI (it's a text file), that would help me import them into the master database. I don't accept artifically slow results (like systems with turbo off) or results from emulators (must be real hardware), but otherwise I am definitely interested in more systems for the database 😀 Just because I don't accept them doesn't mean I discourage them though; by all means, experiment.

Reply 8 of 32, by MobyGamer

User metadata
Rank Member
Rank
Member
RacoonRider wrote:

Do you need samples from 486DX2-50, 66, 80, 386SX-40?

Yes, especially the -50, -80, and 386SX-40 (I thought they only made DX-40s)! You can either email your resulting DATABASE.INI file to trixter - at - oldskool.org, or PM it to me, or post it here.

Reply 9 of 32, by RichB93

User metadata
Rank Oldbie
Rank
Oldbie

Oh wow trixter! Sure you've heard it all too many times before but 8088 corruption is fracking awesome. And I've enjoyed reading oldskool.org too 😀

Reply 10 of 32, by MobyGamer

User metadata
Rank Member
Rank
Member

Never get tired of hearing it, thanks 😀

Reply 11 of 32, by MobyGamer

User metadata
Rank Member
Rank
Member

New version available for testing, with some changes as suggested by eliandia:

+ Recompiled stub so it has the new measurement wrapper

+ Added v86 check (and command-line to override)

+ Enhanced /i so that it also spits out usec timings

+ Added [letter] to jump to next system that starts with that letter in the browsing interface

+ Add command-line option to work with a different database file.

+ Added some small "special effects" if working on a color monitor (with command-line option to override)

Snag it at http://dosbenchmark.wordpress.com/downloads/

Reply 12 of 32, by TheLazy1

User metadata
Rank Member
Rank
Member

Nice!
Although it is disturbing to see my M919@150MHz on the same level as a generic DX2-66.
😳

Any other cacheless M919 owners want to check it out?
My score was 170.

Reply 13 of 32, by elianda

User metadata
Rank l33t
Rank
l33t
MobyGamer wrote:

As for your sample results, if you could post those exactly as they appear in DATABASE.INI (it's a text file), that would help me import them into the master database. I don't accept artifically slow results (like systems with turbo off) or results from emulators (must be real hardware), but otherwise I am definitely interested in more systems for the database 😀 Just because I don't accept them doesn't mean I discourage them though; by all means, experiment.

ok here we go:

;AT clone 386 home built by Michael Huth
[UIDF8DE26E1]
MemoryTest=162
OpcodeTest=68
VidramTest=222
MemEATest=81
3DGameTest=64
Score=92
CPU=AMD Am386DX
CPUspeed=40 MHz
BIOSinfo=R(C)1985-1991,American Megatrends Inc.,All Rights Reserved.,1346 Oakbrook Dr.,#120,GA-30093,USA.(404)-263-8181. (07/07/91, rev. 0)
BIOSdate=19910707
BIOSCRC16=F8DE
VideoSystem=VGA
VideoAdapter=VGA, Tseng ET 4000 with HiColor RAMDAC, DRAM, VESA, 1024kb Video
Machine=AT clone 386 home built by Michael Huth
Description=386DX40/i387/32MB/GUSPnP/SB16+WB1/ELNK3
Submitter=Michael Huth

Why are the timings shown in the real time benchmark so different from the timings you get when a system is added to the database?

Retronn.de - Vintage Hardware Gallery, Drivers, Guides, Videos. Now with file search
Youtube Channel
FTP Server - Driver Archive and more
DVI2PCIe alignment and 2D image quality measurement tool

Reply 14 of 32, by elianda

User metadata
Rank l33t
Rank
l33t

Some more results:

;PS/2 Model 80 386-16
[UID2709A18EA]
MemoryTest=537
OpcodeTest=174
VidramTest=1245
MemEATest=191
3DGameTest=173
Score=22
CPU=Intel 80386DX
CPUspeed=16 MHz
BIOSinfo= COPR. IBM 1981, 1987 (03/30/87, rev. 0)
BIOSdate=19870330
BIOSCRC16=2709
VideoSystem=VGA
VideoAdapter=VGA, unknown Chipset, 256kb Video Memory (BIOS)
Machine=PS/2 Model 80 386-16
Description=8580-071 BR55-7568723 /8514A/ELNK3/TC5046/MemExp6MB/Dual16550Serial
Submitter=Michael Huth

;AT clone / home built K6
[UID77C3D4FA1]
MemoryTest=10
OpcodeTest=6
VidramTest=119
MemEATest=4
3DGameTest=4
Score=374
CPU=AMD K6
CPUspeed=233 MHz
BIOSinfo=war,Copyright (C) 1984-98, Award Software, Inc. (04/30/98, rev. 0)
BIOSdate=19980430
BIOSCRC16=77C3
VideoSystem=VGA
VideoAdapter=VGA, unknown Chipset, VESA, 256kb Video Memory (BIOS)
Machine=AT clone / home built K6
Description=K6-233/GA586HX/3dfxBanshee/EWS64XL/GUS/SCB55/2940UW/PX-40TS
Submitter=Michael Huth

;AT clone home built P55C
[UIDCB37DC92B]
MemoryTest=25
OpcodeTest=8
VidramTest=82
MemEATest=4
3DGameTest=5
Score=403
CPU=Intel Pentium with MMX
CPUspeed=166 MHz
BIOSinfo=war,Copyright (C) 1984-98, Award Software, Inc. (10/21/98, rev. 0)
BIOSdate=19981021
BIOSCRC16=CB37
VideoSystem=VGA
VideoAdapter=VGA, VESA, 256kb Video Memory (BIOS)
Machine=AT clone home built P55C
Description=P166MMX/GA586HX/Riva128/MaxiSound64HS/GUS/DB60XG
Submitter=Michael Huth

Retronn.de - Vintage Hardware Gallery, Drivers, Guides, Videos. Now with file search
Youtube Channel
FTP Server - Driver Archive and more
DVI2PCIe alignment and 2D image quality measurement tool

Reply 15 of 32, by RacoonRider

User metadata
Rank Oldbie
Rank
Oldbie

Here are the results (for 486 - same hardware, different bus speed):

;386SX-40 basic build
[UIDEC3DA3D02]
MemoryTest=182
OpcodeTest=82
VidramTest=238
MemEATest=88
3DGameTest=71
Score=76
CPU=AMD Am386SX
CPUspeed=40 MHz
BIOSinfo=MR BIOS (r) Copyright (c) 1992, Microid Research Inc. (11/24/92, rev.0)
BIOSdate=19921124
BIOSCRC16=EC3D
VideoSystem=VGA
VideoAdapter=VGA, Trident TR8900C, VESA, 1024Kb Video Memory (BIOS)
Machine=386SX-40 basic build
Description=386SX-40, cacheless, 16Mb SIMM, generic noname mobo
Submitter=RacoonRider avermakov[at]list.ru

;486DX2-50 with VLB
[UID26053BB3D]
MemoryTest=119
OpcodeTest=48
VidramTest=161
MemEATest=60
3DGameTest=41
Score=146
CPU=Intel i486DX2
CPUspeed=50 MHz
BIOSinfo=(C)1985-1993,American Megatrends Inc.,All Rights Reserved,6145F Northbelt Parkway,GA-30071,USA.(404)-263-8181. (12/15/93, rev. 0)
BIOSdate=19931215
BIOSCRC16=2605
VideoSystem=VGA
VideoAdapter=VGA, Cirrus CL-GD5428, VESA, 256kb Video Memory (BIOS)
Machine=486DX-50 with VLB
Description=486DX2-80, 256Kb cache, 64Mb FPM, VESA local bus mobo
Submitter=RacoonRider avermakov[at]list.ru

;486DX2-80 with VLB
[UID260531125]
MemoryTest=75
OpcodeTest=30
VidramTest=124
MemEATest=37
3DGameTest=25
Score=210
CPU=Intel i486DX2
CPUspeed=80 MHz
BIOSinfo=(C)1985-1993,American Megatrends Inc.,All Rights Reserved,6145F Northbelt Parkway,GA-30071,USA.(404)-263-8181. (12/15/93, rev. 0)
BIOSdate=19931215
BIOSCRC16=2605
VideoSystem=VGA
VideoAdapter=VGA, Cirrus CL-GD5428, VESA, 256kb Video Memory (BIOS)
Machine=486DX-80 with VLB
Description=486DX2-80, 256Kb cache, 64Mb FPM, VESA local bus mobo
Submitter=RacoonRider avermakov[at]list.ru

And thank you for a great job, very cool software!

Reply 16 of 32, by RacoonRider

User metadata
Rank Oldbie
Rank
Oldbie

;P55C 233 at 266Mhz
[UID3AA2CDC43]
MemoryTest=15
OpcodeTest=6
VidramTest=110
MemEATest=2
3DGameTest=4
Score=368
CPU=Intel Pentium with MMX
CPUspeed=266 MHz
BIOSinfo=war,Copyright (C) 1984-95, Award Software, Inc. (07/08/98, rev. 0)
BIOSdate=19980708
BIOSCRC16=3AA2
VideoSystem=VGA
VideoAdapter=Matrox Mystique 220 2Mb + 3dfx Voodoo Graphics
Machine=P55C 233 at 266Mhz
Description=i430HX, 512Kb cache, 64Mb EDO, Mystique220, Voodoo
Submitter=RacoonRider avermakov[at]list.ru

It's curious that 166MMX scored more than 266MMX because of Riva TNT 😀

Reply 17 of 32, by MobyGamer

User metadata
Rank Member
Rank
Member
elianda wrote:
MobyGamer wrote:

Why are the timings shown in the real time benchmark so different from the timings you get when a system is added to the database?

Can you clarify what "so different" looks like? If the Score is off by 1, or the microsecond timings are off by single digits, that is normal interrupt noise. The benchmark deliberately does not disable interrupts or try to get the CPU/video/DRAM refresh/etc. in lockstep because 1. that's only safe for short timing durations, and 2. I didn't want to benchmark the system in a special case, only the system running in its "natural" state.

If the timings are WAY off, that shouldn't be happening because the code is the same, so shoot me an example.

Reply 18 of 32, by MobyGamer

User metadata
Rank Member
Rank
Member
RacoonRider wrote:

It's curious that 166MMX scored more than 266MMX because of Riva TNT 😀

Indeed, which is why video write speed is part of the benchmark 😀 It is a system benchmark, not just a CPU benchmark.

A similar interesting result is the IBM XT-286 vs. the original IBM AT. Both are 6MHz 286s, but because the XT-286 uses 0-wait-state RAM, it is 1.6x faster. In fact, the IBM XT-286 has nearly the same effective throughput as the IBM PS/2 55SX which is a 16MHz 386sx system 😀

Thanks to everyone for the results, I'll add them to the database and put out another update. Keep those results and suggestions (and critcisms, I have a relatively thick skin these days) coming!

Reply 19 of 32, by TheLazy1

User metadata
Rank Member
Rank
Member

Ah, so as it turns out the Mach64 was the reason for the abysmal score...

;PC-Chips M919
[UID252A16E447]
MemoryTest=50
OpcodeTest=35
VidramTest=78
MemEATest=55
3DGameTest=25
Score=292
CPU=AMD Enhanced Am486DX4
CPUspeed=150 MHz
BIOSinfo=(C)1985-1994,American Megatrends Inc.,All Rights Reserved,6145F Northbelt Parkway,GA-30071,USA.(404)-263-8181. (10/10/94, rev. 0)
BIOSdate=19941010
BIOSCRC16=252A
VideoSystem=VGA
VideoAdapter=VGA, S3 86c801/5 D, VESA, 1024kb Video Memory, 1024kb Video Memo
Machine=PC-Chips M919
Description=No L2 Cache
Submitter=