Post date: Aug 06, 2008 9:26:5 PM
I've been stewing on some topics for a long time. I'm a real zealot for strong experimental design and rigorous empiricism when evaluating my work and the work of others.
In helping to organize the reinforcement learning competition I always encouraged that we take conversations out of private e-mail exchanges and into a more public forum, even if the public forum was still private.
I have also decided to open-source many of the projects that I've put time into, from TD-Networks to RL-Viz, the bt-Recordbook, and others.
I've recently realized that there is a thread that is common to all of these endeavors and I believe it is that I am a strong advocate of radical transparency in research.
From Wikipedia:
Radical transparency is a management method where nearly all decision making is carried out publicly.
All draft documents, all arguments for and against a proposal, the decisions about the decision making process itself, and all final decisions, are made publicly and remain publicly archived.
The only exceptions to full transparency include data related to personal security or passwords or keys necessary for physical access required to carry out publicly negotiated decisions. Any technical actions which are perceived to be controversial or political are considered to lack legitimacy until a clear, radically transparent decision has been made concerning them.
This definition isn't perfect in the context of research, but I think you understand the point. I don't just want the results of an experiment, I want a full account of why that was the experiment to do, and if there were other similar experiments that were tried and may have failed along the way.
This is a plan that I will follow with my own research. I will be posting all of my code (even code that is in progress) in open-source projects, and I will be carefully documenting design decisions, and keeping track of a horrible amount of results through the bt-recordbook.
If you think this is a good or a bad idea, let me know!