Kathy Layton, Founding Partner at McCallum Layton outlines a best practice guide to help clients ensure they are getting the most from their online research approach.
As a result of the growth in usage of the internet, online research is no longer just appropriate for researching niche markets; it is now considered a mainstream technique and offers various benefits over the more traditional research methodologies. However, simply substituting a CATI or face-to-face survey with an online questionnaire, or a normal focus group for an online group or bulletin board is likely to produce less than satisfactory results. In order to get the most out of online methodologies, it is necessary to understand how techniques need to be modified and adapted to this relatively new research medium. This article covers some things to consider when commissioning an online research study, or managing one in-house.
The Issue of Engagement...
Let’s think first about a qualitative survey. A key difference between a face-to-face discussion and an online group, bulletin board or community is that in the latter respondents do not have any audio or visual cues; we perhaps underestimate the degree to which such cues are used by a skilful moderator in a face-to-face situation to keep respondents interested and engaged and to control the discussion (eg invite hesitant respondents to contribute or ‘stem the flow’ from overly vociferous ones!). Consequently, in an online situation more thought has to be given to how to achieve the same degree of engagement. It should be borne in mind that for a respondent 90 minutes alone in a room typing messages into his or her computer has the potential to be a great deal more tedious than a 90 minute live conversation, Therefore, to reduce the possibility of respondents becoming bored and ‘switching off’ the discussion guide should contain some ‘thought provoking’ questions, or some interesting exercises or games which the moderator can ‘drop in’ to the discussion when s/he senses respondents are becoming disinterested. In addition, full use should be made of the technology in order to maximize engagement – audio clips, video footage, web pages, blogs and so on should be used wherever possible to enliven the discussion and keep respondents interested.
Getting Involved...
One of the hardest aspects for an online moderator is keeping tabs on all the threads of conversation – unlike in a face to face group there can be a delay between a comment and a response, by which time the conversation can have moved in another direction; the moderator has to make it clear which element of the conversation they are picking up on. As inevitably there tends to be more need to clarify issues and responses with an online group, clients need to be realistic as to how much subject matter can be covered; attempting to include too many topics can lead to a disjointed, confused and poor quality output. As well as keeping tabs on the respondents’ dialogue, the moderator has also to read, absorb and decide how to deal with comments from the client. Whilst the ability of clients to be involved in, and influence the direction of the discussion in real time, is a significant advantage of online discussions, clients need to use this facility judiciously if they are not to prejudice the flow and quality of the overall discussion.
Two’s Company...
When comparing quotes for on-line focus groups it is worth checking how many moderators will be used. In our experience it is essential to have two moderators; one can lead the main discussion whilst the other can take and respond to questions from the client, and can prompt and chivvy along respondents who do not appear to be contributing fully. Two moderators are also beneficial when the discussion diverges as is often the case – some respondents may have moved on whilst others (who may have taken more time to consider their response) still have interesting comments to make about a previous topic. It is worth clients bearing in mind when comparing quotes for online focus groups that whilst one quote may be more competitive, if it is based on using only one moderator this may well be false economy as the resultant output could be greatly prejudiced.
Another thing to bear in mind with online qualitative groups, is that it is far easier than with face-to-face groups to incorporate some pre or post exercises. For example before the discussion respondents could be asked to complete an online diary, or provide background on their purchasing and usage behaviour, or upload images/advertisements that resonate with them. Not only does this get respondents more involved in the subject matter, the information gathered then can be used by the moderator as context for the ensuing discussion.
First Impressions...
Moving on to a quantitative online survey, the first thing to be aware of is that on the internet judgements tend to be made in a nano second – an almost instant decision is often taken as to whether or not to participate and if the decision is negative that person is usually lost forever (unlike a self completion paper survey where the questionnaire may be put to one side and picked up again later). Consequently, a well designed online survey should attract attention and should succinctly provide a compelling reason to take part.
The Impact of Incentives...
In some cases the ‘compelling reason’ may take the form of some sort of a monetary incentive – whilst such incentives are used on occasion to incentivise paper based self-completion surveys, they are more prevalent with online surveys as often the only way to obtain sample is to use a panel (and panellists are incentivised to participate in surveys). It is difficult to assess the impact of incentives on the quality of information – however it is almost certainly true that a proportion of those participating will be motivated solely by the financial reward and will give little thought or consideration to their replies. It may be worth considering therefore incorporating some questions designed to identify those who are not engaged. For instance two very similar statements can be asked – one worded positively and one worded negatively – a check for consistency of thought between the two can then be made. Alternatively or in addition, a minimum survey complete time can be set to try and identify those who are not giving consideration to the questions and suspect respondents can then be eliminated from the analysis.
Honesty is Best...
With an online survey it is important to provide up-front an honest statement of how long the survey will take to complete (unlike a paper questionnaire, the potential respondent has no way of gauging this before they decide to take part) and clients should bear in mind that length has a significant impact on propensity to participate – so keep the questionnaire as short as possible and eliminate the ‘nice to knows!’. Unlike with a paper self-completion questionnaire, it is possible to force respondents to answer questions, but however tempting this might be it can be counter-productive; if someone really finds it difficult to answer, forcing them to do so can lead to irritation and the inclination to give ill considered responses – far better to get fewer honest and accurate replies than 100% nonsense!
Keep It Simple...
As with qualitative research, online quantitative surveys lend themselves to creative use of technology – again respondents can be kept engaged by making the survey as visually appealing and interesting as possible. In this context, the incorporation of video and images, pop up boxes to explain and expand on text where appropriate and innovative question types (drag and drop, card sort and slider bars etc) all have a role to play. However, beware of making the survey so technologically complex that it fails to work properly – whilst asking respondents to enter a virtual shop and simulate buying as they would do in real life might seem like an interesting and novel way to obtain survey data, it’s no use if the majority of potential respondents do not have Flash technology on their computer!
In summary, online methodologies are increasingly being used by researchers, but the results they generate have the potential to be as insubstantial as the medium through which they’re gathered. Hopefully the above tips and hints will help to ensure your online research provides rich, insightful and above all actionable information.
Visit Us On Stand D147