None of my oldest working computers have their original Hard Drives, but aside from that these are all in working order.
I haven't owned all these from new, was given most of them for free in the early 2000s, aside from 1 PC I bought 2nd hand as noted:
My oldest would be my Olivetti PCS 11 (not sure when it was released, either late 1980s or early 1990s):

Intel 386SX @ 16MHz
10MB RAM (this is an upgrade, this system came with just RAM soldered to the board and empty slots to expand it)
Onboard Western Digital 9131 Graphics
1x ISA Slot (currently fitted with a 3Com 3C509B Ethernet Card)
1.2GB IDE HDD (original Conner 40MB IDE HDD died)
OS: MS-DOS 6.22 with Windows for Workgroups 3.11 (it had MS-DOS 5 when I originally got it).
2nd oldest is this, originally released in 1993:

This PC has had a hard life, scratches & scuffs and was exposed to the elements outside. Took a lot of cleaning up to get it looking this good again, but could still do with a bit of work. It's mostly working, though may have an issue with it's motherboard as there are some issues occasionally that I haven't figured out yet.
Dell OptiPlex 425S/L
Original CPU that I still have: Intel 486SX @ 25MHz / Current CPU that is installed: Intel 486DX2 @ 66MHz
16MB RAM (can't recall how much RAM it originally came with, probably 4MB)
1GB CF Card + 120GB IDE HDD (original HDD died)
Onboard Tseng Labs ET 4000 1MB Graphics
3x ISA Slots (currently fitted with a SoundBlaster AWE64 Value & a 3Com 3C509B Ethernet Card with XTIDE boot ROM)
OS: Windows 95
3rd oldest, a PC I actually bought 2nd hand in the early 2000s, originally released in either 1995 or 1996:

Digital Venturis 575
Socket 5 Motherboard
Original CPU that I still have: Intel Pentium @ 75MHz / Current CPU that is installed: IDT Winchip C6 @ 200MHz
72MB RAM (originally came with just 8MB RAM soldered onboard and empty SIMM slots. Performance is good with 72MB RAM, despite it only being able to cache 64MB. The board can take 128MB but runs like crap due to the cache limitation)
640MB IDE HDD
Onboard S3 Trio64 1MB Graphics
2x ISA Slots & 2x PCI Slots (currently fitted with S3 ViRGE 325 2MB PCI Graphics, RealTek RTL8139B PCI Ethernet Card & ESS ES1868 ISA Sound Card)
OS: Windows 95
The oldest PC I own that I've owned from brand new is my Advent 8737 bought at PC World in late 1999:
Old picture from the early 2000s, but I still have everything pictured except for the monitor & printer which died:

Specs:
Gigabyte GA-6BXE Rev. 2, Slot 1 Motherboard
Intel 440BX Chipset
Intel Pentium III @ 650MHz
100MHz FSB
512MB PC100 SDRAM (originally came with 128MB RAM)
40GB IDE HDD
Nvidia TNT2 32MB AGP Graphics
Sound Blaster Live! 5.1 PCI Sound Card (originally had an Aureal Vortex 2 AU8330 PCI Sound Card, which I put into another PC)
Intel Pro 100/S PCI 10/100Mbps Ethernet
Current OS: Zeta 1.21 (later version of BeOS) / Originally Installed OS: Windows 98 Second Edition
Aside from the RAM, Sound Card & Ethernet Card, all the rest of the hardware is the same as back in 1999! Also originally had a crappy 56K PCI WinModem, which I still have somewhere but is practically useless nowadays.
I have loads of Windows 98 machines, so this is my retro BeOS machine.
The motherboard needs recapping, there is at least 1 bloated capacitor and this seems to cause a non-working floppy drive. Aside from that, it's working as good as it did in 1999.