This one has a new facility for scanning a texture pack to tell you if you have repeated mappings (e.g. mappings via links as well as via an equiv files).
It doesn't give you much detail about the repeat, but its enough to track down the problem.
How does this tool work out duplicates? I guess it has to actually look at the file contents to get an accurate byte-for-byte comparison of exact same files?
I'm using a tool called Duplicate File Finder to do this work but it can't seem to match all the duplicates. So It might match the passport cover in four folders as the same, then match a diffrerent four as the same, but not match those four to the first four.
Does this simply mean it sucks or that there are in fact subtle differences in the files that it can find but your tool doesn't?
I have been having a 'crocodile' problem and seem to have unearthed something very weird. In an attempt to get to the root of the matter I pasted my eqiv file into Excel and there is a folder listed (5A6C753E8C4C54E57113EF594419A9E5) which is not in the folder I generated the equiv file from. There are in excess of 50 matches listed between this and other folders, but I can't seem to trace this folder anywhere in any of the levels - am I missing something?
wrote:Does this simply mean it sucks or that there are in fact subtle differences in the files that it can find but your tool doesn't?
I made a test folder with just 16 copies of the medipack in, one from each level.
Duplicate File Finder reports that there are only 6 unique bitmaps! I then did a binary compare using the DOS FC command and it also concurs.
There are significant byte differences in these files which to the eye appear to be identical images. I was hoping to be able to do the duplicate removal using a binary comparison but it looks like it's not possible...
...unless we work out why Glidos generates different byte streams for what appears to be the same bitmap data.
What is Glidos doing when it captures bitmap/texture data? Is it looking at the original data stored in the .phd file or is it actually parsing this data into usable bitmap data and maybe changing it somehow?
Can you shed some light on this Paul? Can it be improved?
Otherwise I have to go and do it all by hand -- or rather, eye... 🙁
wrote:I have been having a 'crocodile' problem and seem to have unearthed something very weird. In an attempt to get to the root of the matter I pasted my eqiv file into Excel and there is a folder listed (5A6C753E8C4C54E57113EF594419A9E5) which is not in the folder I generated the equiv file from. There are in excess of 50 matches listed between this and other folders, but I can't seem to trace this folder anywhere in any of the levels - am I missing something?
Very odd! I don't know if we're using the same version of the game, but here are all my folders from an entire game capture that start with a 5.
51B389D9E0B06AD278B39C4F5571A65951C118643282B65E9E5198DAB0215F4F53330699059647CB8082BE02185C0F46534E2C69AE2356740007B528ADB0ECBC53D820497EE7DE4B514BAE608724561F54637C4BF0304439B63F51D3A5EEA3825564D5A2BFD736964A791151589BF74A57A26AB453DED38FE830BB44A64A71F0583727D7561F71FF5DD276E7962C82FD58A5368950CCD7E18A58DFFE1005A4F158F8C08A00E4D4CCE2875E4BCC870BF25DC11F7D67BD718860E4A540983DCF1B5E44F14BFEC8BEAA0F3CC529C899FB5E5F35239CAC7ABF93A6755BFF42F54238
Is your list the same (other than the discrepancy)?
Heh, out of interest, here's what a full game capture has duplicated, according to Duplicate File Finder:
15,333,132 bytes in 7154 files with 9,362,576 bytes duplicated in 4376 files.
Quite a bit....
Yes - my list matches yours! - I'm starting to find other mysterious folder additions to my equiv file....still investigating!
Ah, you must have combined your equiv file with the version-equivalence file. It has a folder that starts with those numbers.... I'm keeping a master equiv file just for my own data at the moment, I haven't even tried combining with the version one yet -- I'll worry about that later!
In the mean time, I think I've found a way of getting all the duplicate files to be reconised correctly by Duplicate File Finder. I'll report back if it works...
I'm trying to finish the hard-work now of getting an entire game capture accurately equivved and de-duped so that I don't have to worry about it any more and that any future work is simply texture creation. 😀
wrote:How does this tool work out duplicates? I guess it has to actually look at the file contents to get an accurate byte-for-byte comparison of exact same files?
It creates an MD5 hash of the bytes of colour data, and then compares hashes. There is a chance that one day the MD5 algorithm will produce a hash that it has produced before, although being given different data as input, but probably the universe will come to an end first.
wrote:I'm using a tool called Duplicate File Finder to do this work but it can't seem to match all the duplicates. So It might match the passport cover in four folders as the same, then match a diffrerent four as the same, but not match those four to the first four.
Does this simply mean it sucks or that there are in fact subtle differences in the files that it can find but your tool doesn't?
There are textures in the game that appear multiply but with subtle differences. The equiv tool can still help you in this case though.
You might have 8 seemingly identical textures, but actually there are two different versions, 3 of one version, and 5 of the other, say.
The tool will generate
BeginEquivABCDEEndEquivBeginEquivFGHEndEquiv
You might provide texture G, and be disappointed that it doesn't handle all the cases you hoped. What you need to do is find one of the textures that isn't being handled (you might find D). The you can create a tiny equiv file containing
BegingEquivDGEndEquiv
and then combine it with the first equiiv file using the tool. Then all the textures will be handled.
wrote:I have been having a 'crocodile' problem and seem to have unearthed something very weird. In an attempt to get to the root of the matter I pasted my eqiv file into Excel and there is a folder listed (5A6C753E8C4C54E57113EF594419A9E5) which is not in the folder I generated the equiv file from. There are in excess of 50 matches listed between this and other folders, but I can't seem to trace this folder anywhere in any of the levels - am I missing something?
That's very mysterious. As yet, I'm at a loss to explain that.
You may not have seen my response above Paul, think we crossed over. It's the version-equiv file that contains the folder in question...
Stupid of me - I forgot about the version equiv file! I'm about to generate a whole game equiv file (without the version equiv) and check out my crocodile problem using that -
wrote:There are textures in the game that appear multiply but with subtle differences. The equiv tool can still help you in this case though.
You might have 8 seemingly identical textures, but actually there are two different versions, 3 of one version, and 5 of the other, say.
etc...
Could this be related to textures being 'forgotten'?
The passport texture isn't completely forgotten in my case - it sort of 'flashes' on and off when the passport closes
Yes, that's the same for me -- I used the term forgotten a little loosely, sorry.
wrote:There are textures in the game that appear multiply but with subtle differences. The equiv tool can still help you in this case though.
You might have 8 seemingly identical textures, but actually there are two different versions, 3 of one version, and 5 of the other, say.
etc...
What kind of differences would these be? Using my medipacks from all levels, I did a difference filter on them in Photoshop and it shows there are no discernable differences in the viewable pixels....
This is going to be a nightmare ensuring all duplicates are removed. JC, I bow down to your superior patience and commitment!
Very stumped now! I took two of the medipacks that are reported as different by DFF and FC, and saved them both out from Photoshop as GIFs. If there are any visual differences, any comparison run on them will show up again.
But now they are identical ad show no differences using DFF or FC.
Something very weird going on here that I can't get my head around. 😒 😕 😵
DOH DOH DOH!!!!! IT'S THE PALETTE!
It's different between levels. Gah! I completely forgot about this. Because we're working with true colour stuff now, I forgot that the original images have only an 8-bit palette.
So files that look exactly the same visually, have a different palette stored in them. Glidos obviously pumps out the palette associated with the level, not just a subset that the image uses.
Would it therefore be possible to change the way your tool generates images and make it so that It creates a custom palette from the colours used in the image itself? Is this a lot of work? If you *could* do it, would it also guarantee that every copy of a given image that is a duplicate has the same custom palette every time?
It's just that it would really help in ensuring all the duplicates are removed from all the folders and it's this that has been holding me up on producing more textures.
I have now retextured all the levels from Caves to Sanctuary of the Scion using the equiv file and for the most part it hasn't created many issues. There have been no problems with medipacks, lara etc. The only things that appear to be missing are:
Mummies (between Qualopec and City of Khamoon) - but the palette between these is noticibly different; Atlanteans (although the models vary between two-legged, centaur and winged types); Larson (between Qualopec and Sanctuary) - some of his textures appear to be identical, but there are some changes, e.g. he loses the waistcoat in Sanctuary; and then there is the weird thing with the crocodile....
What's the weird thing with the crocodile exactly?
You got as far as sanctuary and removed all dupes? You obviously have a lot more patience than me to sit there and work through it all!
