When you say your "entire collection", what is its origin? Did you rip it from CD yourself, download it, record it, etc?
If you ripped from CD (or recorded from vinyl) direct @ 44/16 and encoded those at 320kbps, you're going to hear any difference if you switch to a lossless format. 320 is "transparent" for just about everything.
Downloads are another story - some dodgy places (even ones selling 'legit' tracks, not just pirates) advertise 320k downloads but transcode that from whatever the hell they have - sometimes even converting a low-bitrate file *up* to high-bitrate. Obviously that's going to sound like a shit sandwich. Unfortunately it's really hard to tell who does that.
Bandcamp is fairly reputable and I believe their standard is to upload lossless files and then the site encodes them downwards for distribution. Soundcloud is more hit and miss, some artists themselves upload as mp3s (including me, on any tracks I made before I knew what the hell I was doing!) and the site transcodes them all to hell. Usually if you hit the download link you get the exactly file the artist uploaded but it's not always consistent. And then you get 3rd party download gates or click-throughs or whatever doing who-knows-what to the music.
3rd party "compilations", torrents, pirates, etc. - assume they don't know or care how to encode properly at best, or are deliberately misleading at worst. Sometimes you get lucky, usually not.
Rendering out tracker music or game music is an entirely different can of worms - don't get me started on all the ways that can go wrong.
Tl;dr if you're already at 320 the source of your music is gonna make way more difference in quality than the compressor/encoding format.
twitch.tv/oldskooljay - playing the obscure, forgotten & weird - most Tuesdays & Thursdays @ 6:30 PM PDT. Bonus streams elsewhen!