Scoring in rwthCTF 2013
This post will talk about the confusion concerning the scoring at rwthCTF 2013. The intention is to shed some light on what went wrong and our decision-making in the aftermath.
Introduction
rwthCTF 2013 took place on November 9, 2013. It was, like the last two installments, a self-hosted attack-defense CTF. We had more than 110 registered teams, with roughly 65 of them actively participating.
Game Theory (not really)
Scoring for offensive points in rwthCTF behaved almost exactly like in 2011 and 2012: A team would get an attack point if they stole a flag and that flag was still available for the gameserver when it tried to retrieve it later.
This latter part - the check for availability of the flag and thus functional correctness of the service - is both necessary to count defense points (if a service is down / broken, no defense point) and to reduce the viability of Denial-of-Service attacks and other forms of rendering a service unusable. By making it mandatory that the gameserver collects the flag again from the service, the game design kind of forces teams to behave on the network if they want to score points.
Imagine what would happen if this was not the case. A team could either overwrite / delete content in a service after exploiting it and thus render it unusable for other teams. This would be awesome for that team as it is the only one scoring. As there is not only one capable team participating, everyone would try to do the same thing. Either by actual exploits, or other forms of Denial-of-Service if they do not have an exploit.
We’re talking exploit-storm, packet-madness and service overload here.
It is quite obvious that CTF organizers should make sure that the game runs nicely and people won’t complain in the end about unstable network / services as that would just not be fun for anyone.
For this reason we require the “GET” of a flag to succeed in order for it to be accounted in scores.
The mistakes / screw-up
First: We put together the infrastructure and scoreboard, and somehow that explanation / part of the rules, did not make it into our FAQ.
Second: As we experienced some bugs / problems on the gameserver side and were busy with panicing and trying to keep everything running smoothly, we got word that there exists a scoring bug. OMG, we gotta track this down. Our main gameserver guy was unavailable for an hour and so verifying the scores and logic took quite long. Turns out - everything’s working as intended, just people are trying to delete / overwrite flags, and that does not work in our game logic. Very late in the discussions, we announced that to the teams.
Third: Because of our mistake / clarification issues, we thought about recalculating scores and did some rough checks of what that would mean for the scoreboard. It turned out that several teams did in fact overwrite / delete flags. Some of them noticed those were not scored and stopped overwriting and thus recovered from the issue. Others continued overwriting and thus were affected more. This meant that any change of logic / scoring would be unfair to at least someone in the event and thus was not viable.
Conclusion
The mistakes that we were guilty of for rwthCTF 2013 were:
- Not double-checking the description of the scoring scheme in the FAQ.
- Having a misleading “You scored a point” message in the submission-server.
- Prematurely claiming to have a bug in the scoring during the CTF.
- Taking a lot of precious time to figure out stuff and talk to people on IRC only to be insulted.
That being accounted for, the scoring scheme in itself was fair and we did not change it during the CTF. We are not trying to hide our mistakes, we’re acknowledging them and apologizing.
We hope that everyone involved accepts our apology and we can learn from our mistakes and improve for the future. We had very good feedback about the other aspects of the game, if one excludes the scoring discussions / rule mistakes.
rwthCTF, like every year, was organized by a small crew of volunteer students in their free time. Our corporate sponsorship was fully translated into cash prizes for the winning teams. We labored for hours to set everything up and come up with good services, get the prize money organized, and so on. We hope that you enjoyed it despite the issues and hope to see you next year.
-0ldEur0pe / rwthCTF Orga