Contests for innovation

So, here's a simple (perhaps simplistic) idea to accelerate the development of innovative methods for neuromagnetic source separation.

1. Create a simulated dataset for benchmarking purposes with unknown number of sources. It should be hard to find the underlying sources. The hardness is engineered by placing sources in known blind spots of existing algorithms (eg. since we know that ICA cannot separate Gaussian sources, or MUSIC cannot separate correlated sources, we deliberately introduce them).

2. Release the benchmarking dataset online, provide a two-year time frame, and announce a small prize money like $50k [roughly, less than the cost of one grad student, without factoring in the resources needed to create a benchmarking dataset].

3. Provide a minimum set of rules for reporting the method and its results.

4. Incentivize journal editors to make a 'special issue' out of the contest solutions.

This would

a) force the already tiny 'neuroimaging methods' community to take each others' work into account more seriously than merely paying lip service.
b) incentivize the abolishment of 'idea embargoes' non-scientific in root.
c) avoid the repeated publication of existing solutions in lower tier journals ad infinitum.

and thus, save resources, and accelerate solutions.