I've come to the conclusion that based on the way packets are given ids and handled outside of lua, that the header is SteamWorks API based and likely difficult to immitate while also having peers that rely on relaying messages via Steams static IPs for the p2p hole punching, a proxy server to manipulate packets isn't very viable :P

I've considered running my own network code with a native binary library that would be drag/drop DLL to the PD2 directory. I'm not sure how viable setting up FFI is, if even possible it might not be easily distributable in a user friendly way. Another option would be to communicate via REST or websockets to the localhost running a web server to interact with my native code, but again problematic for the common user to deal with probably. Lastly it might be possible to take advantage of the BLT hook technique or through BLT support itself, however I'd still run into the issue of my last point with SteamWorks API relaying messages through static IPs for certain users that need network hole punching. Possible work around would be to enforce port forwarding if needed.

BLT also has a feature to send network messages in JSON format/strings through a hidden chat channel, but when I used this early on in BigLobby I remember it not being reliable in delivery, it either failed to arrive sometimes or was heavily delayed.

So unfortunately, it seems making custom network calls available to everyones mods isn't too feasible. A pdmod will still be required and can break with every update. For supporting multiple mods, generic network methods could be defined with various param combinations to utilize and logic in the handler class to identify intended method that will transform the params to expected format(you might not use all params in their defined order).

Maintenance could be reduced with some automation/tests, where SteamKit or similar is used i na server with a steam account to download the updates when available, extract the asset XML, patch it with generic methods, create the pdmod and commit a PR to a git repo. Webhooks could then forward those to repo's depending on it and create a new release.

See the strings thread, there is work being done to get as many of the hashes figured out and from the looks of it most of them have been. Once those are released it should be easy to find those. Presumably you could do this yourself with the linked tool as well if you're familiar/comfortable with how to run it.

The hash I gave is for the path mentioned and verified to work with my GPU cracking algorithm, I assumed the hashes you provided me are of the same endianess. On the Bundle Modder utility tool, I have `Use Hex` enabled, `Swap Endianess` disabled. I'm not sure what endianess the hashed filenames are in. It's been a while but I remember prior to all this that I needed to swap endianess with hashes I had sourced from somewhere. From memory with the new difficulty textures I did a test on for someone on reddit, the filenames were the correct endianess, they just lacked proper 0 padding(the tool for cracking expects fixed length since it's matching strings not numbers, at least it seems that way as it failed when I allowed smaller lengths for hashes).

This could just be because I'm on Linux and have noticed some issues with Bundle Modder extracting assets with the wrong extensions/hashes, but there appears to be some lua files, they'd need to be decompiled if they really are lua files. I know that I can't find any lua calling crime.net functions that are called when setting up a lobby(eg MenuCallbackHandler:choice_lobby_reputation_permission).

Under shaders directory I only have one file with that same extension, "base". Are there anymore? I'm running the bruteforce on that hash assuming it's the equivalent to "shaders/base" being "37cba50a84a9919b".

So far string lengths 1-9 for files in "shaders/" returns nothing for that hash(`a-z as the char set), Doing a check on length 10 will take an hour per folder which I don't think is worth doing right now. The file paths look long, I haven't set it up to iterate through patterns/prefixes/suffixes yet, so doubt they'll be any luck there.

Just noticed I forgot to include _ in the charset. Small possibility that might hit something, but with the current setup I won't be going through that again to check sorry :( I included it for the texture hash you requested, I enforced a prefix of pattern_ and suffix of _df, no matches with up to 9 chars inbetween.

Either or, my GPU can do 32 billion hashes/sec, doesn't really matter how many word combinations you want to use. It can handle up to 9-10 permutation sequences(as in 9 to 10 elements to mix combinations with to create a single string/path) within a reasonable amount of time,

If I go back to working on it, I plan to use a graph data structure to break up folder hierarchies and words( separate by _ and /), each word would be a graph node linked to the previous, can analyze frequency and occurances for patterns(obvious ones are ene_ or civ_ for prefixes and _1 for suffixes), the graph also provides context of the patterns/words so you could improve performance by only targeting or prioritizing relevant words/patterns to the context(folder hierarchy being tested or file type for example).

Another improvement would be to perhaps use markov chains. Sounds like you got most assets though already :) My solution is only practical with the path hinting described above, and only when a known prefix/suffix is enough combined with the 9-10 brute forced character/word permutations.

Research so far indicates the bind request/respsonses I'm seeing in WireShark are from SteamWorks API calls to handle network hole punching(avoids needing port forwarding) via STUN packets.

Presumably all network communication is being handled via SteamWorks p2p features. Games provide a byte array from the looks of it and Steam is responsible for the header prior I think. Have monitored packets from another game using SteamWorks and it's very similar header structure.

If you want fast dict attacks use the GPU. Very easy to do with Hashcat, I've got the hashing algorithm written in OpenCL using 64-bit vectors.

If I were to go with the proxy method I'd need some way to listen to what the Network class is sending/receiving and bridge that. Not familiar with this, the UDP port ranges also seem random with multiple used per peer and different per peer..

Not sure if this is useful to anyone but I spent a bit of time with WireShark collecting UDP packets and trying to analyze their structure. I had some success, and thought it worth sharing. If you're not aware, network.network_settings asset is an XML file with network messages defined, the engines Network class reads this in to define RPC methods that the Lua NetworkPeer class uses with send() function.

I wanted to see if I could work around the need for pdmod which breaks with each update by supporting my methods with my own Network class potentially replacing the existing one(or at least some of it's functionality being redirected to mine). This would allow for multiple mods to add or change network messages and mixing them into the main source from OVK avoiding breakage considerably.

Unfortuantely from the looks of it, I'm not sure how possible it is, fairly close, not sure if I can get that information for unit param values via lua or if it's engine access only. It may be possible to proxy the packets and modify the packet data with what I do know however. I'm putting this project on hold for now, but would appreciate help in decoding the rest if anyone is interested!

This could potentially pave the way for dedicated servers or improved netcode or porting to another engine with network compatibility to the original game.


RPC network message definition for example packet below:

Example Packet:

• 010000000000 - Seems to be static with all packet data
  
• 00 - Only ever seen 00/01 and once 03, may refer to client/host or status? Potentially peer id
  
• 0eea - Unknown
  
• 00001375 - Increments at a specific amount, value is consistent to target peer for same method, noticed increment values of 11 and 22 for two different peers
  
• 0000ecb7 - Like the previous but decreases by the amount instead
  
• 00193c54 - Consistent between peers, iterates over time, possibly using game time
  
• 5c1f70df - Unknown, 5c was common throughout the same network message, 1f had 4 variations, possibly local peer to target peer identification
  
• 00000017 - Packet Length in bytes following
  
• 0000ba00 - Increments by 0x0100
  
• 02 - Seems to indicate the end of the header? Only seems to occur at the start
  
• 089a - Increments by 0x0100 per packet sent, the value consistent to the target peer, possibly two seperate bytes.
  
• 06 - Denotes the following byte represents Packet ID
  
• 88 - Packet ID based on counting message elements in network.network_settings XML as they appear
  
• 00 - Null after packet ID? Param values for this packet id follow in order of their definition
  
• 9c08 - Value for unit param
  
• 0a - Prior to a string it's length is given
  
• 6d656c65655f6974656d - String, in this case "melee_item"
  
• Packets may contain additional bytes sometimes, they don't seem relevant/needed, may be old memory in the buffer, sometimes a packet also seems to begin incorrectly when heavily stressing network packets

Additional messages in a packet may be appended, they begin with a Packet ID indicator byte followed by packet ID and so forth like listed above, there is no notion of a repeat message indicator byte(exact same messages repeat in full at packet lengths up to 1400ish?), which is very inefficient.

If there is any checksum it is likely in the header, not sure what type is used or how it is computed, possibly one of the unknowns. One of the other unknowns might be used to indicate packet order or timestamps.

Originally posted by Frankelstner:

So there is a limit on the things we can resolve.

If it's just git related couldn't we use one of the decompiled lua git repo's and run the script through each tagged commit? Too bad no one keeps the assets version controlled I guess?

You seem to have the majority of it sorted from the looks of it. Not sure if there is much point in finishing/releasing my GPU cracker :P It's good for guessing 9ish characters, after that length it'll start to become impractical. It'd be fine with guessing audio files that have 32-bit int file names, especially since the folder names are identified by the soundbank files. Beyond that you need to apply knowledge of folder hierarchy/patterns. If I go back to working on it, using your handy script to feed the paths into my pattern analyzer would be good. Dictionary attacks with known words from all the paths could also help I guess.