We call our approach the Scientific Community Game (SCG). SCG generalizes the approach of TopCoder. Our new competition design is desirable from two viewpoints. (1) We minimize the effort on the competition administration by distributing the work of evaluation among participants while avoiding unfair evaluation. (2) We minimize collusion so that the strong participants cannot be outnumbered by colluding participants. Our design is axiomatic in that we formulate axioms (including a collusion-resistant axiom) for ranking functions and we prove a representation theorem.
We describe a proof of concept implementation of SCG which uses a modular construct, called a lab, to group related requirements and to solve labs incrementally through lab relations, which are themselves captured as labs. We report on the usage of earlier versions of SCG for teaching software development and algorithms (2007-2014).
In addition to software development, SCG and its theoretical foundation is also useful for pushing the state-of-the-art in formal sciences and in education in formal sciences. Indeed, the system was developed in the context of education (Algorithms and Software Development Courses) to facilitate fair peer grading and focused communication among students.
Joint work with Ahmed Abdelmeged and Ruiyang Xu.