Why (Not) Do Online Research?

There are compelling reasons to do online research, including:

• Not being dependent on availability of labs and/or experimenters.
• Conducting studies even in times of lockdown
• Quickly collecting large samples
• Targeting very specific target populations (e.g. different nationalities, languages or those with impaired mobility)

However, generally we recommend you don't run web-based experiments when:

• Your experiment needs to connect to hardware (e.g. when recording physiological data).
• You require millisecond accuracy.
• You require strict control over the setting (e.g. lighting, temperature, noise) or time of day

On this page you will find some guidelines on how to start doing online research, e.g. selecting the right software and platform and connecting the various parts.

If you have any further questions about online studies, please mail SOLO lab support at: labsupport@fsw.leidenuniv.nl or check out our dedicated Teams channel:  To request advise regarding GDPR matters, contact privacy@fsw.leidenuniv.nl.

In the video below (recorded during a CogPsy labmeeting) an introduction to the basics of online research is presented. While aimed mostly at novices to online research, some pointers and considerations may be valuable for more experienced researchers as well. This video is a good place to start if you are looking to start your first online research project. You can also see a pdf of the presentation slides.

The Basics

For doing online studies, you need three basic building block (see Figure 1):

1. Task (or survey) builder software (e.g. OSWeb)
2. Study platform (e.g. JATOS or Pavlovia)
3. Participant recruitment and management platform (e.g. SONA and Prolific)

In some cases two or more of the above are combined in one (e.g. Qualtrics and Gorilla are task builder and study hosting platforms).

It is important to consider which combination of building blocks is optimal for you study. This may depend on many different characteristics of the various options, see Figure 2 for examples. For more information about each of these, please see: Platform Choices, Task Builder Choices and Recruitment Choices.

Study Flow

For most online studies, you will need to incorporate multiple elements (e.g. recruitment platforms, multiple surveys and behavioral tasks). A typical flow of an experiment might send a participant from SONA, to an information letter and screening on Qualtrics, then on to a behavioral task programmed in OpenSesame and hosted on Jatos, followed by a debriefing on Qualtrics, and finally back to SONA again for automatic study credit awarding (see Figure 3)

When connecting multiple parts it is often crucial that a common identifier (e.g. a participant number) is shared and stored in all parts, so that you can later match up the data.

Most online experiment software including OSWeb (but also SONA, JATOS, Pavlovia, Qualtrics, Gorilla, etc), has the ability to link to other online platforms / software, and send/receive some information via the URL (so called query strings).

Generally, such linking requires three steps (see Figure 4 and also our general introduction to online testing video):

1. set up to receive query strings from the preceding part
2. if needed, store the values from the query strings in the data
3. set up to send along query string to the next part

Instructions for how to link to and from different platforms or task builders are available on the wiki pages for those platforms and task builders.

Platform Choices

JATOS, Pavlovia and Gorilla have both overlapping features and unique use-cases. Below are some general guidelines to help you find the one best suited for your purposes.

Use JATOS when:

• You have experience with OpenSesame and want to use it to build your experiment, or you have a pre-built OpenSesame experiment you want to convert for online use.
• You have an existing experiment built in HTML, CSS, and JavaScript that you want to host online (this includes jsPsych or lab.js tasks).
• You want to program your task from scratch in HTML, CSS and JavaScript; or using a JavaScript library, such as jsPsych or lab.js.
• You want to build multi-participant interactive tasks, such as a prisoner's dilemma game.
• GDPR considerations require you to store the data you collect internally; i.e. on a server owned and managed by the Leiden University.

Use Pavlovia when:

• You have experience with PsychoPy and want to use it to build your experiment.
• Your task requires scripting, and you necessarily want to use Python instead of JavaScript.
• You can make do with very limited technical support from SOLO.

Use Gorilla when:

• You have experience with Gorilla or have found a preexisting Gorilla experiment that you want to run; or modify and run.
• You want a super easy-to-use platform with integrated experiment-building and survey/questionnaire tools.
• You are fine with your experiment being forever locked-in to the Gorilla platform; experiments built in Gorilla cannot be exported and run without a Gorilla license.
• You do NOT intend to collect data from a large number of participants (>>500). This is due to Gorilla being a pay-per-participant service, unlike JATOS (free) and Pavlovia (flat fee).

Note: when creating experiments for online studies that you intend to convert for use in a physiology lab, SOLO suggests using OpenSesame.