Modular software that interfaces with other components (plug-ins, add-ons, codecs, dynamic libraries) etc. will require matching 'bitness' across the board, and if some of those things do not (yet) exist in 64-bit form, you may lose whatever functionality that they provide if you go for 64-bit versions. This is true with browsers, media players/creation tools and a host of other things, but that's just a question of availability, not an inherent problem.

Of course, doubling the width of instructions and operands will create performance and memory overheads for any executing program, at least in theory - but I haven't looked at any numbers regarding that, or at the (possible) overhead of the WoW64 subsystem itself...

"It depends"(tm)...
64-bit mode gives you larger and more registers, and when used properly, you can get more efficient code that way.
When used improperly, the code is less efficient, because you'll be using 64-bit values where you were using 32-bit ones otherwise, so you're using twice the bandwidth/space in cache etc.
As a rule, code that is merely recompiled from 32-bit to 64-bit is slower on x86.
Code that is carefully analyzed and optimized for 64-bit can be faster.

Irfan Skiljan says on Infarview's page that not all plugins are ported to 64 bit yet.

Starting with IrfanView 4.40 there is also a 64-bit version available. […]
Show full quote

Starting with IrfanView 4.40 there is also a 64-bit version available.

Advantages of IrfanView 64-bit over 32-bit version:

It can load VERY large files/images (image RAM size over 1.3 GB, for special users)
Faster for very large images (25+ Megapixels, loading or image operations)
Runs only on a 64-bit Windows (Vista, Win7, Win8, Win10)

Advantages of IrfanView 32-bit over 64-bit version:

Runs on a 32-bit and 64-bit Windows
Loads all files/images for normal needs (max. RAM size is about 1.3 GB)
Needs less disc space
All PlugIns will work: not all PlugIns are ported (yet) to 64-bit (like OCR) and some 32-bit PlugIns must be still used in the 64-bit version, some with limitations (see the "Plugins32" folder)
The Unicode and many Adobe 8BF PlugIns can't work on 64-bit
The TWAIN drivers with 64-bit support are rare

So I am sticking with 32 bits version. Thanks for the heads up about Irfanview though.

Beware of codecs. AFAIK a 64-bit player needs 64-bi codecs and 32-bit players also need 32-bit codecs. That is why I usually install both 32 and 64-bit versions of FFDShow in my machines. Some old codecs might not have 64-bit codecs (I think RMVB is like that), so you'll have to play them in a 32-bit player.