Let me rephrase: I cannot ignore the unchanged file because the same string might have a corresponding hash in a future update.

The current algorithm:
1. Get the set of all unknown hashes, i.e. hashes where the original path is not known. This is just the meta of the archives.
2. Get test strings through regex over the archive payload. Hash each string and check for a match.

I do not know the dependencies. If I had perfect knowledge of that, the script performance would not be an issue because it could just jump right past audio or texture data. When file A starts referencing B, I get one more test string out of millions of others. The string does not match any unknown hash and is discarded. I cannot ignore the file because the same string might have a corresponding hash in a future update.

The only way out is a file with all test strings, but that would be many GB large.

I think that strategy is flawed. Suppose a file A is changed to reference a file B that does not exist yet. Then the next update comes along and B is added, but A is not changed. The name for B is not found because A is not checked again. I don't know how frequent something like this occurs, but I do know that it happens occasionally in the Lua. But you are right about filtering files. Right now I do not regex over individual files, but the full archives with no special logic. By running over the files instead I can skip over redundant files (the unique payload is just 28 GB), and also decompress the few compressed files.

@Hassat Sadly I have not got the slightest clue about masks, materials and others. Is the material missing? Or is only the material icon missing? In the latter case, is the string for the material icon missing?

Well, it's 60+ GB of data. Any speedup would put more stress on IO. I have tried pypy which had basically no effect. Then I tried cython and got four problems instead of one.

1. It was still slow.
2. After adding cython specific code (breaking compatibility with plain Python), it was still slow.
3. I lost all the interactivity that I used to have by just crashing the script at any point (e.g. ctrl+c) and playing around with the data that I have gathered in real time.
4. When commenting out some lines but not doing anything else, cython pretended that nothing had changed at all and left me with the old build. Yes. Actually writing or removing code would trigger it to build everything from scratch, but just commenting broke cython entirely. I cannot code if the program does not act according to source.

Worst part is that after I had switched back to plain Python, it would give preference to cython-created libraries (I didn't even realize it would recognize them) and made me waste even more time on debugging.

I suppose the problem falls into the 1-2 day region: https://xkcd.com/1205/ But I don't actually save time but rather free up resources faster. And heavier IO for a shorter time is not necessarily better. I don't know.

I had hoped that I could sit out this update because it takes me an hour to run the script. Thanks Overkill.

Does the current hashlist have everything you need?

Movie path retrieval is now implemented. Keep them coming!

Neat. It's fairly straightforward to retrieve them from the Lua, so I have automatized that (like so many other parts):
Let me know if you happen to know other places in the Lua which calculate filenames that are not included yet.

The converter does:

Remove elements that are never referenced by other elements and that do not execute on startup.

Remove all permanently disabled nodes unless they are referenced by an ElementRandom that does not ignore disabled.

Shift the number of times an element can be toggled further up the tree of on_execs if possible.
Remove elements with no effect unless they are referenced by certain other elements (e.g. ElementRandom). I have a list of element types with effects and some additional rules (remember that one instance with the filter always evaluating true).

Convert elements into MissionScriptElements if possible. MissionScriptElements do nothing but pass execute other nodes. I essentially abstract away other elements into a MissionScriptElement with an effect assigned to it. For the most part anyway.

Inline elements into other elements.

Remove empty ElementToggles.

Merge identical elements (same effect and refs, but different name).

Inline ElementLogicChanceTriggers to make some nice if/else conditions.


That's the simplification step. The converter then uses the element info and walks along the tree of elements. It starts with trigger elements and those with execute on startup and figures out what other elements are referenced and should come next.


Disabled elements are generally removed for clarity. I am quite careful in not removing anything that could be observed in the game, though there is always room for improvement. Let me know if there is a mismatch between the raw and non-raw version that could be observed.

It has an empty definition. It does nothing. Empty elements are valid.

Actually, it should not exist. It should just be ´---´. The idea behind all this is that some ElementRandoms (like "Execute 1:") make a selection between several items, but some of them are dead ends with various names. These dead ends can be selected (and then have no effect), so I cannot just remove them. To indicate that they are dead ends, I rename all of them to ´---´. I will track down the issue with this duplicate element being created when I find the time.

The "position 3750.0, 8125.0, 4102.5" is for the editor (which only the developers have). The editor would show an icon at that location, so the devs can click on it to change it. My mission file converter uses this (I figured it all out manually):

classesWithPositionRotation = set([
"ElementAIArea",
"ElementAlertTrigger",
"ElementAreaTrigger",
"ElementAreaReportTrigger",
"ElementAreaMinPoliceForce",
"ElementBlurZone",
"ElementExplosion",
"ElementExplosionDamage",
"ElementFeedback",
"ElementFleePoint",
"ElementInstancePoint",
"ElementInteraction",
"ElementInventoryDummy",
"ElementLookAtTrigger",
"ElementLootBag",
"ElementPlayerSpawner",
"ElementPrePlanning",
"ElementShape",
"ElementSmokeGrenade",
"ElementSpawnGrenade",
"ElementSpawnCivilian",
"ElementSpawnDeployable",
"ElementSpawnEnemyDummy",
"ElementSpecialObjective",
"ElementSpecialObjectiveGroup",
"ElementSpotter",
"ElementWaypoint",
"ElementWorldCamera",
"MissionScriptElement",
"ElementAccessCamera", #if not camera_u_id
"ElementDialogue", #if use_position
"ElementPlayEffect", #if not screen_space
"ElementPlaySound", #if not use_instigator and not elements
"ElementTeammateComment", #if close_to_element
"ElementPhysicsPush",
"ElementScenarioEvent",
"ElementLootPile",
])

There are 154 different types of mission elements, but only these 37 use the position value.

Also, the levels folder (not levels raw) usually works better for these locations: https://bitbucket.org/Frankelstner/pd2-mission-scripts/src/63b2f566c703a6c3ae7023388737565c92d3d1c9/levels/narratives/elephant/dark_mis%20default%20True.txt?at=master&fileviewer=file-view-default