Update Experiments.md desirable attrbiutes

[jacobkrueger]

Financial incentives facilitate replications and comparisons, and thus should be favored over non-financial ones (e.g., a voucher for a company may have no value for some participants if they dislike the company, course credits are impossible to compare across universities or regarding their value for an individual student).

This does not mean non-financial incentives cannot be used, but they are limiting replications and potentially threaten the validity of an experiment.

[drpaulralph]

I think whether to use financial vs. non-financial incentives is controversial and context-dependent. Calculating appropriate financial incentives for cross-cultural replications in countries with different currencies, standards of living and buying power is for replications is nontrivial, and non-financial incentives may be more cost effective and just better at motivating participation in some contexts. Let's leave this here for now and see if anyone else weighs in.

[jacobkrueger]

Dear Paul,

Fully agreed on this being controversial, and non-financial incentives can of course be used and may be more cost-effective. We also agree that calculating appropriate financial incentives for cross-cultural replications in countries with different currencies (etc.) is difficult.

However, it is actually easier than transferring non-financial incentives in cross-cultural replications.
Just as one example:

Researchers in country A conduct an experiment. The design includes that students receive points for an exam. Now researchers want to replicate the experiment in country B. But the researchers in country B would not know (1) the number of points students could achieve through the experiment compared to the total number of points; (2) whether the course was mandatory or optional; (3) whether students received a grade or only had to pass; (4) how many credits of the total degree the students acquired through the course/experiment; (5) or how motivated the students were to achieve a certain point. Such things heavily depend on the study program and personal motivation. Best (but very rare) case, all information is reported in the original study. Still, researchers in country B would need to find a similar environment, which is almost impossible, leading to worse replications, and threatening the validity and comparability.

Other disadvantages of course credits/exam points as the most commonly used non-financial incentive are:

• Limiting the sample to students of a certain course.
• Using course credits as incentives implies that the lecturer is the experimenter. This leads to Experimenter Demand Effects.
• The students are obviously in a dependent position, no matter what means are taken.

Still, we agree that non-financial incentives are an acceptable alternative, but they are inferior to financial incentives. For this reason, we consider them Desirable, but not Essential.