VOGONS


Finding dup files

Topic actions

First post, by DaveDDS

User metadata
Rank Oldbie
Rank
Oldbie

Just in case anyone else is interested:

In dealing with collecting/recovering a multi-file project that I've somehow allowed various partially-modified versions of to spread out over MANY directorties over the years, I've cranked out:

Scan Same Files
use: SSF path [[:]filespec...] [options]
opts: -Q Quiet (inhibit scan progress) *1
-Ofile Output to file
Scans for and reports all duplicate filenames in path(+subs)
if no filespec given, uses "*" -like Unix: "*"all "*.*"mustContain'.'
if lead ':', means NOT this spec!
Output:
Lead ' ' marks the first file of a specific name (easy to see in list)
'-' "" files that are the same as ^^ *2
'0-9'A-Z'a-z' "" "" "" with each other *2
'?' "" more tha 62 different files
*1 This reads/calculate CRC of every file - it can take a while on large tree
without -Q it shows each directory being scanned.
*2 "same" is determined by size and 16-bit CRC of file content.
VERY unlikely to identify as "same" files that are actually
different - but you should confirm in critical cases!

Using this test directory containing:

26-09-19  3:12a        5,120 R:\TST\TST1.TXT
26-09-19 3:15a 5,120 R:\TST\TST2.DAT
26-09-19 3:25a 3,072 R:\TST\TST4.BIN
26-09-19 3:12a 5,120 R:\TST\SUB1\TST1.TXT
26-09-19 3:27a 3,072 R:\TST\SUB1\TST4.BIN
26-09-19 3:13a 5,120 R:\TST\SUB2\TST1.TXT
26-09-19 3:15a 5,120 R:\TST\SUB2\TST2.DAT
26-09-19 3:17a 2,048 R:\TST\SUB2\TST3.BIN
26-09-19 3:27a 3,072 R:\TST\SUB2\TST4.BIN
26-09-19 3:14a 5,120 R:\TST\SUB3\TST1.TXT
26-09-19 3:15a 5,120 R:\TST\SUB3\TST2.DAT
26-09-19 3:20a 2,048 R:\TST\SUB3\TST3.BIN
26-09-19 3:28a 3,072 R:\TST\SUB3\TST4.BIN
26-09-19 3:14a 5,120 R:\TST\SUB4\TST1.TXT
26-09-19 3:19a 2,048 R:\TST\SUB4\TST3.BIN
26-09-19 3:28a 3,072 R:\TST\SUB4\TST4.BIN

> SFF R:\TST

 2026-09-19   3:12:22        5120 \TST1.TXT
-2026-09-19 3:12:44 5120 \SUB1\TST1.TXT
12026-09-19 3:13:54 5120 \SUB2\TST1.TXT
-2026-09-19 3:14:24 5120 \SUB3\TST1.TXT
12026-09-19 3:14:46 5120 \SUB4\TST1.TXT
2026-09-19 3:15:32 5120 \TST2.DAT
-2026-09-19 3:15:46 5120 \SUB2\TST2.DAT
-2026-09-19 3:15:58 5120 \SUB3\TST2.DAT
2026-09-19 3:25:44 3072 \TST4.BIN
12026-09-19 3:27:10 3072 \SUB1\TST4.BIN
12026-09-19 3:27:30 3072 \SUB2\TST4.BIN
22026-09-19 3:28:12 3072 \SUB3\TST4.BIN
22026-09-19 3:28:24 3072 \SUB4\TST4.BIN
2026-09-19 3:17:16 2048 \SUB2\TST3.BIN
12026-09-19 3:20:30 2048 \SUB3\TST3.BIN
-2026-09-19 3:19:46 2048 \SUB4\TST3.BIN

I will eventually get this up on my site (after I've used/tested it a bit more) ... but if anyone wants it soon, I will expedite!

** This DOES qualify as "old" since SSF.COM is a 16-bit DOS program (I've also made SSF.DVM for "modern" systems) - perhaps we should have a topic for "new" software to help with old systems!

https://dunfield.themindfactory.com ; "Daves Old Computers" ; SW dev addict best known:
ImageDisk: rd/wr ANY floppy PChw can ; Micro-C: compiler for DOS+ManySmallCPU ; DDLINK: simple/small filetransfer(w/o netSW)via Lan/Lpt/Com

Reply 1 of 4, by DaveDDS

User metadata
Rank Oldbie
Rank
Oldbie

Argh!!! - I was so busy tracking and debugging the difference detection that I failed to notice the actual file paths/names shown were incorrect! (I did say wanted to test it more!) - corrected - for above example output is now:

 2026-09-19   3:12:22        5120 R:\TST\TST1.TXT
-2026-09-19 3:12:44 5120 R:\TST\SUB1\TST1.TXT
12026-09-19 3:13:54 5120 R:\TST\SUB2\TST1.TXT
-2026-09-19 3:14:24 5120 R:\TST\SUB3\TST1.TXT
12026-09-19 3:14:46 5120 R:\TST\SUB4\TST1.TXT
2026-09-19 3:15:32 5120 R:\TST\TST2.DAT
-2026-09-19 3:15:46 5120 R:\TST\SUB2\TST2.DAT
-2026-09-19 3:15:58 5120 R:\TST\SUB3\TST2.DAT
2026-09-19 3:25:44 3072 R:\TST\TST4.BIN
12026-09-19 3:27:10 3072 R:\TST\SUB1\TST4.BIN
12026-09-19 3:27:30 3072 R:\TST\SUB2\TST4.BIN
22026-09-19 3:28:12 3072 R:\TST\SUB3\TST4.BIN
22026-09-19 3:28:24 3072 R:\TST\SUB4\TST4.BIN
2026-09-19 3:17:16 2048 R:\TST\SUB2\TST3.BIN
12026-09-19 3:20:30 2048 R:\TST\SUB3\TST3.BIN
-2026-09-19 3:19:46 2048 R:\TST\SUB4\TST3.BIN

https://dunfield.themindfactory.com ; "Daves Old Computers" ; SW dev addict best known:
ImageDisk: rd/wr ANY floppy PChw can ; Micro-C: compiler for DOS+ManySmallCPU ; DDLINK: simple/small filetransfer(w/o netSW)via Lan/Lpt/Com

Reply 2 of 4, by SiBurning

User metadata
Rank Newbie
Rank
Newbie

As soon as I saw the thread title, I immediately throught back to my short time working on antivirus software. One of the things we did was to run a checksum on every file on the system, keeping a list of full path, datetime, size, and checksum. This way we had a basis to compare and repair any files that were modified by a virus. I think the norton utilities (and/or their competitor whose name I forget) had roughly the same capability. I'm not sure if your program has an output file like that, but it would be a tiny modification to just add the checksum on each line to what you posted. With your comparison, it could also be used as half a snapraid system: running it regularly, noting any differences, could notify users of any disk errors, even if the restore part of raid is missing.

I'm pretty sure the version I was using was just a batch script. It called a checksum utility, and with some sorting... you get the idea. But you're 95% there already without the extra utility baggage.

Reply 3 of 4, by DaveDDS

User metadata
Rank Oldbie
Rank
Oldbie
SiBurning wrote on 2026-09-20, 01:21:

As soon as I saw the thread title, I immediately throught back to my short time working on antivirus software. One of the things we did was to run a checksum on every file on the system, keeping a list of full path, datetime, size, and checksum. This way we had a basis to compare and repair any files that were modified by a virus. I think the norton utilities (and/or their competitor whose name I forget) had roughly the same capability. I'm not sure if your program has an output file like that, but it would be a tiny modification to just add the checksum on each line to what you posted. With your comparison, it could also be used as half a snapraid system: running it regularly, noting any differences, could notify users of any disk errors, even if the restore part of raid is missing.

Thanks, it would be quite trivial to add the CRC to the output (I might make it an option), but what you are describing seems like it would be better server by a
different program - SSF scans all files and reports those with same name, size and CRC - basically my way of quickly sorting through 100s of copies of a file spread out "everyware" - output does incude timestamp and size, making it easy to visually figure out "whats what" and where differences occur. - it only reports files which have the same name.

What you are describing sounds like it should output every file with all information (including CRC) - this allows easy detection of changed files.

I kinda already have a tool which does this, using a small(ish) completely printable file which looks like:

>name {entering a subdirector}
{14 printable chars}filename {file information}
....
<[<...] {exiting subdir(s)}
... entries can repeat/nest ...

The 14 printable characters encode 84 bits (6 bits/char) which contain size, timestamp, CRC, flags

I use this to "fast" find a file on the system, detect oldest, newest, biggest, smallest, changed etc.

I've not wrapped it into a virus detector (probably won't - my old DOS systems are in my lab and never connect to the internet), but it certainly could be done!

https://dunfield.themindfactory.com ; "Daves Old Computers" ; SW dev addict best known:
ImageDisk: rd/wr ANY floppy PChw can ; Micro-C: compiler for DOS+ManySmallCPU ; DDLINK: simple/small filetransfer(w/o netSW)via Lan/Lpt/Com

Reply 4 of 4, by SiBurning

User metadata
Rank Newbie
Rank
Newbie

Yes, a full tree list of every file, one per line, human readable. I don't think it should be tied to virus detection. This is more error or change detection, which you might use afterwards. Errors being either from a virus, hardware or software errors, hacking, or even your kid messing around. Or even just to know you purposely modified a file since last time, and maybe you need to back it up. It's definitely a different use case from yours, but so much of the require functionality is the same, it would be fairly trivial part-way towards building one tool given the other. Heck! What I'm talking about is trivial from scratch. just different use cases. In fact, you could use the output as input to the backend of your tool. Anyway, it just so reminded me of it.

OTOH, I can easily imagine people who need to detect duplicates might have the need for a list. If those files are editable, they might want to compare... I don't know, but I can see how people might want additional similar tools. You have to let need and demand be the guide.