How To Perform an Experiment

  1. Choose an algorithm (see section on paper titles that will get you accepted at conference).
  2. Choose the data set.
  3. Get several graduate students to do it for you separately ; pick the best results.
  4. Tune your algorithm by selecting the parameters on the test set. [Add a little noise if need be to make it look less obvious].
  5. Use a data set that has not been used before so that your algorithm will be the best.
  6. Just do toy problems to save time (call them “illustrations”).
  7. If you are lucky enough to work in information retrieval carefully select your target concept out of the wide range available.
  8. Always interpret your results as “comparing favourably with the state of the art”.
  9. If the results are are quite poor, stress they are only preliminary.
  10. Never produce statistically significant experiments.
  11. If you do run multiple experiments, do not quote error bars; simply quote the best result obtain – people are more interested in the best.
  12. Redefine the performance measure until your method looks good.
  13. Do not present your data graphically; it is much better to use a very large table with numbers. Do not truncate to a couple of digits of precision.
  14. Exploit monotonic transformations of the results to make your data look better (logarithms are particularly useful here).
  15. Ensure you omit essential details to prevent competitors being able to reproduce your results.
  16. Make your code available but make sure it is buggy and only compiles on exotic compilers that nobody else has access to.
  17. Instead of conducting a complete experiment, do a partial experiment, and then derive the optimal result from the partial experiment. This is great for variants of error-correcting output codes or other systems which solve many sub-problems.
  18. Use statistics whose assumptions are manifestly false to derive statements of excessive certainty. For example, it is popular to assume that cross validated error is independent.
  19. When you can not experimentally demonstrate the effect your paper is about, include good experiments that illustrate some other point.
  20. Conduct carefully designed, statistically well founded and significant, experiments that are fully and carefully documented sufficiently well for them to be readily reproduced by other researchers.