As tested with VBox, NT 3.51 can handle up to 8 CPU cores/threads and ~4GB RAM just fine. One needs to configure this registry variable to enable more CPU cores/threads (the default is 2 for Workstation). This corresponds to bearwindows' information. It seems bearwindows had a private kernel patch that could allow using even more CPU cores/threads (the theoretical limit is 32), but little is known about that, though 8 cores/threads are more than enough for most use cases.

UPDATE: Sadly it looks like the MPS 1.1 HAL may not work on some real hardware configurations. For now multi-core is only something achievable with VBox as it emulates a complaint system... You may have a bit more luck with Intel CPUs/chipsets as the vendor string "GenuineIntel" was being checked on several different occasions.

PS: After some searching looks like it's okay for an ACPI system to have invalid MP tables (thus not MPS complaint). In this case, the issue might be a per-motherboard basis.

The following general incompatibilities apply for NT 3.51. These are the major blockers, and this does not cover programs that don't run for other reasons (like "Cannot open program").
- Explicit SHLWAPI.DLL dependency (and related stuffs).
SHLWAPI.DLL from 32-bit Internet Explorer 4 and onwards cannot function due to the absence of QueueUserAPC which is only available since 95/NT4. This is the case with SHLWAPI.DLL from IE 4.0 up to IE 5.5, and is the only missing symbol. IE 6 version of the library appears to import a lot more missing symbols, which kind of explained the rationale behind the dropping of Win95 support.
Such software generally shipped with newer Internet Explorer versions for install on Windows 95/NT4, and openly stated "IE4 or higher installed" as system requirement.
Note that while some software itself doesn't depend on SHLWAPI.DLL, it might have optional libraries/plugins that do. In that case you'll get errors about SHLWAPI.DLL but otherwise can continue using the program as long as you don't use features that actually involve the libraries/plugins in question.

UPDATE: It seems SHLWAPI.DLL existed since IE 3.x along with WININET.DLL (4.70.xxxx). While NT4 versions of IE3 existed, trying to use SHLWAPI.DLL from IE3 gave me the "not a valid Windows NT image" error, despite it reported no errors from Dependency Walker. Not sure if the matching version of WININET.DLL would work or not.

- Use of DirectX (Designed for Windows 95)
Software/Games that make use of DirectX will usually not work, and in many cases you'd be explicitly blocked from installing (citing unsupported OS).
It seems DirectX is what really made Windows 95 different from the NT family during that period, as these games usually claimed to be "Designed for Windows 95" and nothing else. Some also supported NT4 SP3 which included DirectX 3, though.
An exception to this would be games that can be run optionally using OpenGL or pure software rendering, and Unreal Engine was a popular example. However, some games (or installers) also explicitly check about OS version (>=4.0), and that needs to be worked around as well.

EDIT: I walked over a good amount of DirectX 8.0a DLLs (last Win95 supported version) and found that almost all DirectSound/DirectMusic 8.0a libraries have no dependency issues on NT 3.51 and might be able to get it working to some extent if there's a trivial method to insert all the necessary registry values in the respective INF files (dsound.inf/dmusic.inf).
DirectInput 8 can't work (dependency issues), but a portion of DirectPlay 8 DLLs may be usable.

Additionally, there's an alternative DDRAW called DDHACK, and this German forum has a custom build of it that can run on NT 3.51. You need a usable Mesa 3D library for it to work, though. Walking dependencies with this DDHACK present resolves dependencies with DirectX 3 D3DIM.DLL and up to DirectX 8 D3DRM.DLL. I haven't tested this in detail and it doesn't appear to work correctly on VM, maybe it can work with VBEMP on physical systems to some extent.

Should note that the discovery might also benefit Windows NT 4.0 (preferrably SP6a) to some extent. NT4 has obviously more APIs than 3.51 and should theoretically be able to run a lot more DirectX 8.0a libraries than 3.51, contradicting what M$ officially stated at that time (no new DirectX was ever officially released for NT4 after all).

- 64-bit integer arithmetic
Right now this problem is mostly encountered in device drivers. NT 3.51 doesn't natively support this in kernel and NTDLL, but NT3.51 DDK had a static library for the purpose. The symbols were introduced since NT4, and with NT4 DDK there's a chance the produced driver binary may accidentally import them. While it may be worked around when building the drivers, I'm not sure if it's possible for existing driver binaries to work around this issue.
A while ago I examined several audio cards' NT4 drivers and found quite a few have imported _alldiv mainly in their DLL portion. Most drivers' .sys parts do not import this symbol, fortunately. Taking Aureal's drivers (AU88NT30.DLL) for example, initial inspection with Ghidra revealed the _alldiv was introduced in a MIDI-related subroutine where it was trying to divide something by 10000 and that was the only place _alldiv was used. Aside from _alldiv, some other drivers may involve different symbols.
Interestingly, similar issues also affected UniATA since v0.47. The driver imported those symbols and not usable on 3.51, and building with NT4 DDK as instructed always produced a binary importing those missing symbols. I was able to work that around by substituting the LIB folder of NT4 DDK with that from NT3.51 DDK when building the driver, which produced a binary that doesn't import symbols not present on 3.51 and can be used there.

- USB Input Issues
NT 3.51 doesn't have a USB stack so USB input devices will behave just the same as they would in DOS (or other places that don't have USB stack drivers), if you have Legacy USB Support enabled in BIOS.
While USB keyboards can work fine, USB mice cannot work when using MPS Multiprocessor PC. See this thread (also on MSFN).
Should note that the behavior of USB input devices depends entirely on the devices' own legacy emulation capabilities. If the USB keyboard or mouse in question misbehaves or does not work at all in DOS, then the same will apply to NT 3.51.

Windows NT 3.51 installs fonts to C:\WINNT35\SYSTEM (depending on your installation path) unlike newer Windows, but it appears some software are hardcoded to look for fonts at C:\WINNT35\FONTS which in this case would throw errors (such as failing assertions). In that case, simply create a FONTS folder and make a copy of the fonts already present in SYSTEM there should be enough to address it. Be sure to sync the fonts in both directories so all programs can correctly access them.

An example of this issue is the Modern skin of Winamp (up to 5.13). After addressing this issue I can use Modern skin without major issues, although some non-essential libraries may throw errors due to other incompatibilities, including SHLWAPI.DLL dependency.

UPDATE: It seems there isn't a convenient way to properly install fonts on NT 3.51 unlike newer Windows, where you can install and register new fonts to the system just by pasting into the FONTS folder. The fonts have to be added into the registry by hand.

It seems there are some other issues with NT 3.51 APIs that could cause programs to misbehave, but not critical enough to lead to crashes.

For example, Total Commander (up to latest version, 10.0) 32-bit can run on NT 3.51, but only the upper left part (about 1/4) of its dialog windows (such as configuration) is visible for some reasons, so it can't be easily used. One can still make configuration through INI file, though. (EDIT: This, as well as some other graphical glitches, may be somewhat related to video drivers, so your mileage may vary on different physical systems or VM hosts)

Also, while Dependency Walker (which I've been using extensively to inspect program compatibility) works fine on NT 3.51, I always have to drag and drop the files I want to inspect (from File Manager), because its open dialog doesn't work for some reasons (nothing happens when I clicked Open).

Refer to this for VC++ history as well as versions that can be run or target Windows NT 3.51.
Additionally, it appears that Open Watcom 1.9/2.0 x86 can also be installed and used for Windows NT 3.51 development.
DOSBox Compilation Guide may contain hints on how to target NT 3.51 with MinGW as well as making compatible SDL-based programs.

While NTVDM has been proven sufficient to run a good amount of 16-bit software and games, there indeed are a few that don't work, though rare in overall.

Spiderweb's Exile Trilogy is one of such cases I discovered: Exile 1 and 2 can't run (causes an exception in the Win16 subsystem). Exile 3 and Blades of Exile runs okay. Additionally, it seems Exile 2 and 3 would change the current titlebar color to something gray-ish upon startup (despite Exile 2 can't run). The change of titlebar color is permanent so I have to manually change it back from Control Panel.

Probably something is missing or flawed in NT 3.51's VDM implementation that has been addressed in later Windows versions, as when I tested the games on Windows 2000 both ran okay there. Additionally, Exile 2/3's titlebar color change does not affect the rest of the system on Windows 2000.

A while ago I found this interesting medium post about getting a C# program running on as low as NT 3.51 (using an experimental .NET feature called CoreRT, now NativeAOT). During the progress he encountered an issue that there were SSE opcodes in the executable, which NT 3.51 cannot run out-of-box, as the CPU needs to have some registers set before it could run SSE instructions, and that requires kernel mode code (such as drivers). After setting those registers, he was able to properly run the test program.

Actually he modified the VBEMP driver to inject the SSE initialization code, though if using VirtualBox, it's also possible to insert the initialization in the boxvnt driver as inline assembly. The boxvnt driver can be compiled using Open Watcom 1.9/2.0 and the build process is straightforward.

I suspect the "Cannot Run Program" error is probably due to the presence of SSE/SSE2 instructions in the program. It was already known that many development toolchains started enforcing SSE2 requirements at some point.

NOTE: Sadly this trick only works for single core/thread scenario as it only changes the registers on a single core/thread. If running NT 3.51 with 2 or more cores/threads, this will NOT work, as you need to somehow write your own driver to set the registers across ALL cores/threads.

I'm still new to driver development and I've been looking for documentations on how this can be done on NT 3.51. Linux has this on_each_cpu function that could do the job, but I haven't found anything similar for NT 3.51 yet.

On the other hand, Windows NT4 has a intlfxsr.sys driver that does this, but it is not compatible with NT 3.51, and it appeared to have a hard GenuineIntel check, which roytam1 managed to have it removed so it can be used on any x86 CPU.

After some experiments I think I should indeed put this just in case. Some software or runtime libraries actively check your system version with varying extents, even though the executables or libraries involved have no dependency issues. As a result, care must be taken when attempting to use a higher version of a given library and be prepared to restore the known good version should anything goes wrong.

Simple ones just check if your system version is >= 4 which is enough to just bar NT3.51 from using it. Complex ones might also check your system minor version, build number and service pack level.

If Dependency Walker reports no issues then it should be trivial to work around by hooking the executable or library in question into a debugger/disassembler, look for version related checks such as comparisons against dwMajorVersion, and adjust (hex-edit) those comparisons accordingly to work around the version check (may also need to use EDITBIN to recalculate checksum just in case).

I can confirm that MSVBVM60.DLL will work on NT3.51 after modification. Previously it checks against 4.00.1381 Service Pack 3 for WinNT platform.

Windows Installer 1.x (from Office 2000/XP) can theoretically run on NT3.51 as it has no dependency issues, but its installer (MSIINST.EXE) contained a OS major version check, and it doesn't seem to be able to correctly write into the registry even after working around it by hex-editing. The registry entries as well as file association might have to be done by hand... maybe inspecting how the installer works from a NT4 environment could provide some useful hints.

Windows Installer 2.0, however, cannot work on NT3.51 due to a missing import NdrClientCall2 from RPCRT4.DLL. There was a MSFN question about this that didn't appear to have gone anywhere.

NOTE: NewShell apparently modified the version reporting mechanism to make Windows report a 4.00 version instead of 3.51. How it achieved this needs further investigation but that should be enough to bypass basic (major only) version checks without hex-editing anything (untested though). Complex checks may still fail as they could require you to be at build 1381 (real NT4). While NewShell does work as intended on NT3.51, I don't really recommend this as an actual NT4 would do much better.