Expecting Dishonesty

ProgrammerDan

Follow

Feb 23, 2017 · 6 min read

An introspective by a Minecraft server administrator on the default relationship between an accused player and the admins, and what admins do to discover the truth.

There are four typical replies to an administrative accusation against a player:

1. I didn’t do it
2. Even if I did, you have no (valid) proof
3. Your rule is stupid, so I ignored it
4. So and so did it, you didn’t ban them, why are you targeting me

Many interactions involve all four, or evolve to include all four.

More advanced interactions leverage additional tactics:

1. Change username on communication media and pretend ignorance — “why am I banned?”
2. Get your friends to harass the admins
3. Disparage the evidence
4. Attempt to start a populist/player revolt

Almost all of these tactics start from a place of insistence on free expression without consequence, and assumed innocence — neither of which exist in an administrated world, unless the admins consent.

Like it or not the server administrators are final arbiters without guaranteed appeal concerning every player’s ability to continue playing.

Fortunately for players, most of the admins I’ve met abide by a code of ethics: Fairness, Adherence to Self-imposed Rules, and Provability of Action.

In other words, we don’t ban arbitrarily, we ban only according to published rules, and we ban based on evidence.

All of which becomes much harder since a principle resource that initiates most investigations is player-led reporting. Why? Similar to “Mom, he hit me!” from yore, the truth is often only half-told. Player-led reporting has an explicit bias in the favor of the player reporting.

Some of my favorites:

1. He X-Rayed my base! There’s No Way he could detect it without X-Ray! Ban him!
2. She beat me in PvP, so she MUST be hacking.
3. That new player knows way too much about everything, they must be an alt or a ban evader.

As an admin abiding by the prior self-imposed Code of Ethics, every report initiates a fairly lengthy, time consuming process.

1. Investigate initial report to understand the full circumstances.
2. Establish lines of dialogue with accuser and accused, separately.
3. If indicated, preventative ban the accused (use sparingly).
4. Dig deeper into logs, communication, and friend networks.

This can take days, or even weeks, depending on the complexity of the issue at hand, which can range from relatively simple hacking accusations to more complex social issues concerning doxxing or other attempts to manipulate players using social engineering.

The core issue is that the only data we can fully rely on is the data tracked in our logs, and logging frameworks. Player reports are absolutely critical, but are often inherently misleading — intentionally or otherwise — and without adequate logs and data to confirm or deny, admins are powerless to achieve their Ethical Code. Instead, they find themselves compromising on those core tenants in a “best effort” to preserve overall community and gameplay.

So the arms race begins, against those who would lie and manipulate the administration intentionally, to achieve fairness with those who stretch the truth to their own advantage, and to protect those who do not lie or have not broken any rules.

…[T]he only data we can fully rely on is the data tracked in our logs…

My career as a Minecraft Administrator has been one of continued escalation in data collection, analysis, and application. Each iteration brings me richer perspectives on the behaviors and activities of players, balanced against performance and available time.

This has always been and will remain a volunteer activity for me; time is scarce, so effective and rapid resolution is always the goal.

My first exposure was Civcraft; long ago the admin team had abandoned Provability of Action in preference to Sufficient Evidence of Action (e.g. a lower threshold of provability necessary to justify administrative action) yet had a massive logging and infrastructure construct to watch thousands of key data points.

The big missing piece was analysis, and a few glaring gaps in terms of data collected (PvP especially was under-served, there was entirely too much reliance on the on-and-off-again updated NoCheatPlus’s Violation Levels).

Civcraft 3.0 attempted to address the analytic gap by augmenting a variety of hand-build shell scripts which wrapped SQL calls, with a full featured Log Aggregator powered by Solr — Graylog. This open source tool allowed the registering of extractions from the logs, the creation of graphs to visualize the data over time, and a number of other handy tools. At our peak, however, we had maybe two dozen extractors. The challenge as always was — which pieces of data to analyze? And the limitation was that Graylog could not apply extractors back in time; only forwards. We were limited by our ability to predict what we should be watching, and we mostly were good at tracking X-Ray during mining, due to extensive issues with it in prior iterations.

Beyond that, we had an amazing tool whose most useful feature was traditional search; a big step up from grep, but we were often left without adequate information to make fully informed administrative decisions. We made due with less, and moved on.

Around that same time I and another senior developer for Devoted, reacting to a much more dramatic dearth of information on Devoted for issue resolution, completed a new plugin — Devotion. It went to an extreme, logging each and every action of every player all the time. The potential use cases were enormous, but I quickly ran into a different issue; storage constraints, index size explosion, and performance impacts of complex, but insightful, queries.

It was a powerful tool, it was very effective at investigating and assisting in divining “what really happened” behind player reports, but it was deeply flawed by its explosive data growth. Cash strapped, Devoted was unable to fund what was really necessary; a powerful secondary server devoted to storing this deep index of data, and able to perform long-running queries without impacting main server performance.

So, I started working on CivSpy. One of the biggest insights from Devotion was that most investigations only required a “certain level” of granularity to satisfy. Reviewing my actions over the past year, I recognized that a granularity of who, which server, which chunk, what action, and how often would have satisfied 90+% of inquiries. Basically, I needed an aggregation engine for statistic sampling along those dimensions.

CivSpy accomplishes this goal. With it, we’ve caught multi-accounters, established baselines for movement, discredited well-meaning but inaccurate x-ray accusations, and confirmed other x-ray accusations. We’ve significantly improved our ability to refund glitches, investigate “truthiness” of player timeline assertions, and powerfully augment our other targeted logging, often saving hours in the process.

The biggest gaps now are accessibility; most of this analysis occurs at the SQL terminal, based on expert knowledge of the aggregated data. While I have plans to build a web-interface for non-gurus, a bigger issue looms, one similar to Devotion’s specter.

For CivSpy to be an effective tool, data analysis must be confined to an external server. Fortunately in Devoted iteration 3, we have the resources to maintain a separate analysis server. Unfortunately, CivSpy doesn’t include on its own a way to easily migrate data to this analysis server. Currently this is done via a series of regular backups and manual migrations and imports, a time-consuming process that often lags behind what is necessary for proper administration.

So, the struggle continues. The tools evolve, the techniques evolve; more data is loaded and more analysis is performed.

Fundamental to all these struggles is a desire to know ground truth, and follow as closely as possible the self-imposed Code of Ethics, to ensure as best as possible the most consistent application of limited rules in an otherwise un-moderated, player-led sandbox.

Interested in CivSpy for your server, or want to contribute? Check out the build server, the github repository, and the API documentation.

Devoted is a Nation-building survival server running a largely custom set of server-side plugins to shape player experience and allow player-led community maintenance in-game. Learn more about us on our wiki pages.