Awesome, thanks for putting it all together in a repo where it can be better tracked for updates :)

I'm still quite busy with my Rust project, soon as I can get around to wrapping that up I'll go through yours and see if I can improve / port it.

Originally posted by Frankelstner:

Let me rephrase: I cannot ignore the unchanged file because the same string might have a corresponding hash in a future update.

The way I understand it is you find a string in file A that references file B, there wasn't anything in file B(but will be in a future update). File B no longer gets checked until an update happens on that file. What is the problem? until file B changes, no point in searching it again.

Are all the links in the first post up to date code? You should really have that amount sitting on github/gitlab/bitbucket :p It is more than I expected ha. I might take a shot at it in a week or so, might need to ask you some questions over chat.

> I cannot ignore the file because the same string might have a corresponding hash in a future update.

If it gets an update in future you'd know when that file has changed? So it wouldn't be ignored if it's got a reference.

If I have some time I might go over your script and see what I can do.

> Suppose a file A is changed to reference a file B that does not exist yet. Then the next update comes along and B is added, but A is not changed. The name for B is not found because A is not checked again

I don't know how your script works, if A gets a reference to B and that's how you decide to look into B, then you'd have that stored as a search path and add a date that you last checked B(an error is thrown or something when you do, you handle it and store the date or some 0 value anyway).

When the update with B arrives, you go through your search logic for files you'd normally check, compare when that file was last checked vs if it was updated. In this case B would have been updated, your check/compare logic for that should deem it fresh and parse it.

Whatever you're doing I'm sure you could could optimize nicely with caching so you don't have to wait an hour each time when not that much data has changed. No point wasting time with a full scan if the bulk of the data can be cached.

For 60GB of data, where you only likey find a subset of that actually produces new strings, doing the optimization I suggested of knowing which files to search should give a huge perf advantage? Say an update happens and the files that were updated that have the new strings you'd extract is only 400mb in total, that would be far less to scan.

I take it the lua source isn't an issue as that'd be much smaller(if you are handling git version you'd be avoiding the .git directory presumably too). But if you have your first run store info like a checksum on the files or modified time or file size(slight chance a file was updated but the same size however) etc, then you can compare much faster without searching each file(the big I/O usage) on the 2nd run to the prior run, again if anything changes you update your reference/database with that new value.

Your first run will still take long, but afterwards should be waaaaay faster :) Cache that information! Here is two quick results from google that should help:

https://stackoverflow.com/questions/1912567/python-library-to-detect-if-a-file-has-changed-between-different-runs
https://ubuntuforums.org/showthread.php?t=1832668

An hour? It's just running regex through text files right? I've not gone back through the older posts in the thread but remember it as a python script? I know there are some variants of python that are meant to perform pretty well, if they're not hard to port maybe that's worth it?

Another thing you can do is perhaps identify if a file has been changed before you parse it, if you're just going through the source code on bitbucket, that can take advantage of git history, if it's extracted content from an update, maybe you can look at the time attributes or get a checksum hash to compare against(can likewise compare against directories if it can tell a change happened to a file within it.

Is there any harm in getting paths for unused/non-existing assets? Could you still keep the functionality and have a toggle/option for it?

As if 30MB is causing a problem? :) Can always reduce the filesize if it was for some reason.

Why do you need to keep it small, just pair the paths with the hashes? Easy to lookup, could even use a GUI tool similar to the bundle modder one

It may be possible to still try the native code approach and work around the hole punching issue if we did message relays ourselves. Last Bullet servers might be able to support this?