Their CRT image quality was unmatched by anyone except maybe Number Nine for a long time and G400 wasn't a bad card, it just had the misfortune of being released so close to the GeForce 256. And of course, they also had driver issues for a long time the same as ATi would have. They had similar issues with Parhelia being released so close to Radeon 9700 and the drivers still sucked. Having made a big investment in Parhelia only to have it bomb was too much,I think, and knocked them out of the mainstream video card arena for good. A lot of their problems were related to their release dates conflicting with those of their competitors. They didn't have enough lead time over nVidia or ATi to establish a foothold in the market with either G400 or Parhelia before something better became available.
I've seen crisp output from 3dfx, ATI, Diamond, STB, and #9. Granted some cards were terrible and you had to research what you were buying. Even some NVIDIA cards are very good.
Matrox drivers were a problem for sure. There were even annoyances with Millennium and Mystique because of how the driver was doing some PCI hacks to get more Winbench score.
After G400, there seemed to be management issues at Matrox. They lost engineers to the competition. I think they couldn't get the products done fast enough. G800 was probably canned because it wouldn't have been competitive and the G550 was the only tech that came from that R&D. Parhelia must have been around a year late seeing how it launched with DirectX 9 nearing release.