Map/match analyzer project

[Wormbo]

Nothing to show yet. I experimented with creating a log file instead of dumping a steady stream of information into a database. The resulting log file seems to be similar in size as the SQLite database file, but processing it is more difficult.

[Wormbo]

I think I may pick up this project again, but it would be nice to have some real-world data to work with. Anyone who is interested can run this mutator to generate log files my tool will be able to parse.

Warning: Generated log files will be in the multi megabyte range!
I did my best to reduce the amount of generated data, e.g. by only logging player location updates if the locations cannot be extrapolated with reasonable accuracy from previous data. Still a match with 16 bots on Torlan that went for 18:46 minutes generated a log file of about 6.1MB. That's roughly 21kB per player and minute. Considering the amount of damage/death data will increase more than linearly if there are more players, and that humans are less likely to move in a predictable way, real-world data may require even more log output.

Like any other logging mod, the mutator will create log files in the UserLogs folder. Log file names will start with "UT2k4Analytics" and contain the game mode, map name and date/time of map load. All rounds of a match will be logged to the same file.

[Wormbo]

And another teaser:


What you see here are results from one round of a Jailbreak bot match on JB-TheDecks, overlayed on the scoreboard minimap texture. Left side is player positions, right side is player deaths. Upper images are team-independent in typical heatmap colors, lower images are team-specific plots in team colors.

[Pegasus]

Great news hearing you're interested in resuming work on the Analytical Tool, and we'd certainly be glad to help with that if we can, Wormbo!

In order to understand how best we can do that, though, I do have a couple of questions for you:

- Would an online server lose its white list status with the Master Server while using the AT's stats gathering mutator?
- Already discussed log filesize expansion rate aside, have you perhaps formed any additional impression regarding the kind of resource strain the AT mutator's operation can typically put on its host server's CPU and memory usage? More specifically, could you speculate as to the practical impact that might have, say, on an online server during peak hours where a full complement of players & spectators will be logged on? I sure hope it won't be able to cause perceivable performance deterioration on purpose-built rigs of CEONSS' type, but more info on this would definitely help put our minds at ease when considering hosting the AT during its beta testing phase; you know, provided that the prospect doesn't look daunting for players' experience, that is.
- Assuming the server does remain white listed and amasses a collection of logs from a day's/week's worth of matches, would there be any way to peruse that data at this point in a visually informative manner, just like in your example pics above, or is that meant to be done via a third, standalone [executable] tool outside of the game?


Quite excited for when this project starts bearing fruit that could have a significant impact on future content analysis & management here, so thanks again for [thinking about] reviving this quite promising tool, and I'll be happy to hear your thoughts on those points whenever you can get to 'em, Wormbo .

[Karma_geddon]

I'm checking in just to say that this is AWESOME. Worm, you really make very nice stuff!

I suppose that the mutator that you are linking us would work only on local (player vs bots) games, right? And not when, for example, I play on CEONSS?

[Wormbo]

It's a mutator, so whitelisting will be gone while using it. Implementing it without using a mutator might be possible in exchange for additional CPU usage, but probably not with the same accuracy for certain events.
Additional memory usage should be insignificant, additional CPU usage should be quite low and hopefully not a problem for a server that can handle 32p Onslaught matches. I'm not quite sure about the performance hit of the log file output, though. A 26 minutes match with 32 bots on ONS-ColdFusion-T32 generated almost 13MB log data. I can't really say whether humans will generate less, more or a similar amount of data. Bots tend to run in straight, "extrapolatable" lines, which is good for reducing the amount of data, but then again they do stupid stuff like repeatedly hopping into and out of vehicles or quickly bounce back and forth for a while between two nearby location, both of which may increase the amount of data to log.
The download only contains the data collection mutator. Log files are processed with an external application that is still under development. Currently that tool only generates heat maps, the image above was created manually by superimposing the generated heat map images on the map texture. The tool can use perspective projection like it is used by Jailbreak's scoreboard minimap and isometric projection like it is used by the Onslaught radar map when converting logged data to heat maps.

The goal for now is getting just a few logs with real humans playing a regular match, because bots only stick to their predefined paths. The mutator's package is marked as serverside-only, so it will not be needed by clients or even cause any kind of network problems. It can be used in offline games as well as listen/dedicated server games. It cannot be used on remote clients or for analyzing demos.

There are still some problems that need to be solved.
For example, many stock Onslaught radar maps are created from screenshots and thus are actually perspective. They hardly match the isometric mapping applied when drawing players and objectives on them. Red Planet is one example of a poorly matched radar image. That may not be a problem on the HUD, but the heat maps look way off. I think I'll extend the mutator at some point to allow taking a well-defined map screenshot and logging appropriate perspective mapping information. (Jailbreak is easier to handle in that regard, as it already uses a special actor (JBPanorama) that comes with interactive in-game setup features.) Such a feature would also help creating background images for other game modes' maps, where a radar map isn't commonly used.
Another problem is correctly handling multi-round Onslaught and Assault matches, where teams switch sides. The required information should already be in the log file, but the analysis tool doesn't really use it yet.

[Pegasus]

It's a mutator, so whitelisting will be gone while using it. Implementing it without using a mutator might be possible in exchange for additional CPU usage, but probably not with the same accuracy for certain events.[...]

Hmm, it probably wouldn't be a sensible choice to hobble the tool's performance potential by turning it into a serveractor, but, OTOH, aiming to preserve white list status for online servers considering hosting it (whatever their specific gametype of trade) would significantly improve its chances, both during the beta testing phase as well as beyond. To that end, I was wondering if it'd be possible for the AT to don the guise of some other popular and white listed mutator, even if just for the limited duration of beta testing. If memory serves, the Santa Hats mutator is an example of a seasonal, white listed mutator one wouldn't expect to be active during this time of the year; could the AT temporarily pretend to be that just so that servers seeking to run it wouldn't have to remain off the list for days, or weeks, on end. Only forcing myself to think along such unconventional lines here because, if this idea proves problematic as a whole, I just cannot see how any server would be willing to sacrifice their white list status to facilitate the necessary testing and data generation for periods longer than, say, a day at a time. Speaking in player traffic terms closer to home, even though I'd want to see the tool hosted here, I doubt CEONSS could afford that on a Friday or during the weekends. Anyway, I hope there's an agreeable way to overcome this hurdle.

[...]Additional memory usage should be insignificant, additional CPU usage should be quite low and hopefully not a problem for a server that can handle 32p Onslaught matches. I'm not quite sure about the performance hit of the log file output, though. A 26 minutes match with 32 bots on ONS-ColdFusion-T32 generated almost 13MB log data. I can't really say whether humans will generate less, more or a similar amount of data. Bots tend to run in straight, "extrapolatable" lines, which is good for reducing the amount of data, but then again they do stupid stuff like repeatedly hopping into and out of vehicles or quickly bounce back and forth for a while between two nearby location, both of which may increase the amount of data to log.[...]

If it's just the file writing overhead that's concerning, that still makes the situation hopeful enough in my eyes: while CEONSS is currently running off of a pair of 500gig WDs in RAID config, if the tool were to place a not-insignificant performance strain on the system, that might be something that a simple SSD purchase & upgrade could alleviate. I mean, I doubt a lean OS as Linux would have trouble handling file write rates to the tune of a few dozen per second, and I think CEONSS' ucc still wouldn't be keeping more than 3-4 logs open during operation (runtime log, AntiTCC, EvenMatch [not sure 'bout that one] and the AT's one). Filesize bloat after a whole day's worth of tracking (1-2GB in size?) would need to be taken into account too, of course, but taking care of that should be as easy as moving the logs off the game server at some point during the early morning.

[...]The download only contains the data collection mutator. Log files are processed with an external application that is still under development. Currently that tool only generates heat maps, the image above was created manually by superimposing the generated heat map images on the map texture. The tool can use perspective projection like it is used by Jailbreak's scoreboard minimap and isometric projection like it is used by the Onslaught radar map when converting logged data to heat maps.[...]

Uhhh, I'm afraid to say, I have absolutely no experience with the JB gametype, so I've no idea how its minimap presentation differs to ONS' "node graph overlaid atop an optional radar image" method; any vids (YT, elsewhere) you could link to perhaps that illustrate just that - sorry 'bout that :/. That said, the complex and/or multi-level geometry that ONS maps can often employ could potentially render all those illustrative approaches unhelpful, if not outright pointless (consider, for example, depicting action inside TripleSlap's lower rooms that lack exits toward the map's peripheral walls), leaving ingame visualization (say, through spawning hundreds of coloured st.meshes/emitters) the only remaining and viable data representation option.

[...]There are still some problems that need to be solved.
For example, many stock Onslaught radar maps are created from screenshots and thus are actually perspective. They hardly match the isometric mapping applied when drawing players and objectives on them. Red Planet is one example of a poorly matched radar image. That may not be a problem on the HUD, but the heat maps look way off. I think I'll extend the mutator at some point to allow taking a well-defined map screenshot and logging appropriate perspective mapping information. (Jailbreak is easier to handle in that regard, as it already uses a special actor (JBPanorama) that comes with interactive in-game setup features.) Such a feature would also help creating background images for other game modes' maps, where a radar map isn't commonly used.
Another problem is correctly handling multi-round Onslaught and Assault matches, where teams switch sides. The required information should already be in the log file, but the analysis tool doesn't really use it yet.

Well, I imagine a quick way to address the multi-round issue for now might simply involve generating a new log file for each subsequent round, with the corresponding designation suffixed at the file's end. As to the bigger issue of map-representative pics for illustrative or overlaying purposes - and maybe even making a tool that would automate that process - from my personal (and still evolving) experience I've come to understand that, no matter the accuracy of the vantage point or how one calculated/arrived to it, a simple pic of the map may not always be the most helpful way of laying out its gameplay-significant geometry; again, multiple levels and enclosed spaces become the bane of avid "map photographers" (to wit, my last tests involved Tanks-A-Lot, copious application of the transparent texture and Paint.NET). Additional assistance through multiple "exposures" and external tools, like image processing programs and the use of opacity/layering, may be necessary to give players the best sense possible when the only comprehensive view of the world offered to 'em is from a fixed angle, with perhaps a changing material sequence the only expansive option.
Long rant short, while the tool you're envisioning here certainly sounds interesting and could help in a multitude cases, there would still remain several where extensive amounts of "trickery" would be required before a useful result got achieved, and no automated process could reasonably be expected to deliver that. Anyway, one step at a time, I suppose...


Oh, and since you seem to've been wondering about it, here you go .

[Wormbo]

Here's a video of the JB2004 scoreboard. That video shows the scoreboard in JB-TheDecks, which uses a centered overhead view similar to Onslaught maps. Other maps may have other angles configured.

Creating that minimap usually involves some effort on the mapper's side (removing geometry, post-processing the screenshot, etc.), but is aided by a built-in "panorama setup" feature in the Jailbreak mode. That feature disables the HUD and puts the player in ghost mode. With some temporary keybinds the player/mapper can then take a screenshot and generate clipboard content for the panorama actor to be pasted into UnrealEd. The panorama screenshot also always gets at least some masking applied afterwards, so a screenshot helper feature only makes it easier to line up the screenshot and the actual perspective projection.

[Pegasus]

Ah, okay, yeah, I see what's going on with that panorama feature, thanks for those links. So basically, the mapper pairs the screenshot they snapped in ghostcam mode with the generated isometric data (the 2 angles) and during matches the custom code just needs to apply the necessary math transformations to properly place player dots on the image. While it's certainly neat for ingame purposes, I wouldn't say it offers anything more than the typical overhead representation in terms of the greater visualization demands posed by the 3D data this tool collects - in both cases we get just one angle, with all the inherent flaws that entails. That's kinda why I threw the ingame visualization idea in there in my last post, but I'm already imagining how heavy it might prove to be for the game to keep spawning and destroying hundreds of actors at the click of a button, based on different filters and toggles. Anyway, 'nuff daydreaming 'bout that, I guess.

Any thoughts on the other points in my last post, btw?

[Wormbo]

Thanks to Ragnos from UTzone.de I now have some data of a few TAM matches with humans. The difference to bots is quite obvious:

Left half shows human movements, right part shows bot movements on the same map.