VOGONS


managing big data

Topic actions

First post, by chris2021

User metadata
Rank Oldbie
Rank
Oldbie

Some people may chuckle at this one. Especially if you're professionally involved with really big data. But 4 - 5 tb is a lot for me, probably most anyone. I bought 2 x 4tb usb hard drives, thinking 4tb would be sufficient to store everything I've stored up over tbe years and wish to hold on to. The 2nd drive is for redundancy. 4tb may actually not be enough, but whatever. I know for a fact there's a measure of redundancy already, in that moving loads of stuff over from wherever results in multiples of whatever winding up on the same drive. Obviously I wish to remove the duplicate files. I just would like suggestions on the best utilities (hopefully free or cheap) to do this. The most efficient manner I'm thinking is to spot files that have a duplicate file of the same size, and manually delete them. I used a utility some time ago and it's use wasn't very intuitive and I wound up losing some data.

Big files are easy to deal with, iso's and whatnot. In my case most of the redundancy will be folders full of jpegs, usually hundreds, that somehow got saved twice (or more then twice). Looking for suggestions.

Reply 1 of 1, by Errius

User metadata
Rank l33t
Rank
l33t

For duplicate images I use Visipics. Beware however that it hasn't been updated in years and so (1) malfunctions for very many files (tens of thousands), (2) ignores very large images (e.g. 4000x4000), and (3) doesn't recognize modern formats such as WEBP and AVIF.

I'm looking for a replacement program with similar interface. (I tried a different program, but it had a confusing interface, and like you I fear I may have deleted the wrong files by mistake, so I went back to Visipics.)

Is this too much voodoo?