Guidelines on Research

Updated on 08 December, 2020

QUESTIONNAIRE DESIGN

Questionnaire design is one of the most crucial exercises, especially for academic dissertations (or market research), whereby data are to be collected via interviews and/or surveys.

Being a quantitative statistician and researcher, I will focus here on quantitatively-oriented questionnaires.

A questionnaire should start with a well-designed cover page containing at least the following:

  • Your research title
  • A brief description of the purpose and objectives of your survey
  • Ethical statements on the voluntary participation of the respondent and his/her right to withdraw at any given moment from the survey (Hesse-Biber and Leavy, 2010)
  • Assurance of anonymity, confidentiality and data protection (Laerd Dissertation, 2012)
  • Your contact details (email address and/or phone number), should there be any query from respondents

Designing a questionnaire is not a mere random assembly of questions or statements, or simply copying/pasting them from some previous research, though replication of research is perfectly licit in an academic context.

Among the various criteria to be taken into consideration, when formulating your questions, you need to:

  • Ascertain that your questions are addressing your research objectives effectively, or will accommodate for testing the validity of your proposed conceptual model, if any
  • Ensure that your proposed answers/options to each question cover the entire range of possibilities. When in doubt, provide a "Other (please specify)" option.
  • Ensure that there are no overlapping options (e.g. it would be incorrect to label age subgroups as "10-20", "20-30", etc, thus creating confusion for a person aged 20)
  • Avoid questions that
    1. are too lengthy
    2. are ambiguous (may be interpreted differently by different respondents)
    3. are double- or multiple-barrelled (e.g. "Are you rich and healthy?", obviously unanswerable if the respondent is rich but not healthy)
    4. are leading (e.g. "Why do you think that marijuana is injurious to health?" - who said that marijuana is injurious to health?)
    5. are irrelevant (e.g. do not include a demographic variable like "Marital status" just for fun, if it is not going to contribute anything to your research)
    6. take things for granted (e.g. "What type of insurance policy do you have?" , which is obviously unanswerable if the respondent doesn't even have any insurance policy)
  • Formulate questions that are suitable to, and understandable by, your target population. For example, do not use high-level English if you are going to survey people with very low academic qualifications, Therefore, do not hesitate to translate your questions into a language that will be clearly understood by your respondents (may it be their mother-tongue), because your prime objective is to retrieve information that is as accurate as possible, so as to obtain reliable findings. Remember, YOU need them. THEY don't need you.
  • Avoid, as far as possible, branching questions (e.g. "If No, go to Section C" or "If No, skip the next two questions")
  • Avoid ranking questions if you have too many options (e.g. "Please rank the following in your order of preference"), while providing the participant with a list of 20 items to rank. This will give the respondent a real headache, which will eventually provoke him/her to just rank items at random in order to finish this painful exercise quickly.
  • Verify whether your questions or statements are measured on appropriate scales, in view of the statistical tests and techniques that you intend to use for data analysis purposes. Some frequently used measurement scales are: dichotomy (e.g. Yes/No or Male/Female), multiple-choice, multiple-response (where more than one proposed answer may be selected), Likert (e.g. "Strongly disagree" to "Strongly agree"), semantic differential (e.g. rating of 1 to 10), among several other possibilities.

With regards to measurement scales, there are a few things that you need to know:

  • If you intend to use Pearson's linear (product-moment) correlation coefficient to determine the extent of the relationship between variables, then the latter must be measured on numerical scale.
  • If you intend to conduct regression analysis (simple or multiple), your dependent variable must be measured on a numerical scale, not categorical like the Likert (ordinal) or multiple-choice (nominal).
  • If you have planned to use parametric testing via the paired t-test, the independent-samples t-test, one-way ANOVA or two-way ANOVA, then your test (or dependent) variable must be measured on a numerical scale.

With regards to (3) above, you are required to perform normality testing by means of the Kolmogorov-Smirnov test or the Shapiro-Wilk test. If your sample size n < 2000, use the Shapiro-Wilk test (Laerd Statistics, 2018). If your test variable fails the normality test, you should use non-parametric tests, like the Mann-Whitney U test, the Kruskal-Wallis H test, amongst others (Campbell and Shantikumar, 2016), to test your research hypotheses.

NOTE

  • Give brief, but clear, instructions to your respondents, wherever and whenever necessary in your questionnaire.
  • Once you have designed your (draft) questionnaire according to the above criteria, you still have to pilot it by selecting at least 15-20 respondents (no particular restrictions on how to select them). The purpose of this standard procedure is to detect any flaws in your design, like typo errors, vocabulary, orthography, missing options, etc. (Saunders et al., 2016). The piloting phase, in fact, verifies the face and content validity (Cooper and Schindler, 2014) of your measuring instrument via respondents' feedback, though subjective.
  • You are also expected to conduct a preliminary reliability analysis of the responses that you obtained. The best way to proceed is design your questionnaire in such a way that your research constructs are defined by sets of Likert-type statements. This will enable you to compute the Cronbach Alpha coefficient for each set of statements to determine whether it exhibits unidimensionality (Ahmad and Sabri, 2013), i.e., whether statements in the set are reasonably intercorrelated for you to deduce that they do "belong" to that set.
  • Besides face and content validity, you should also test the construct validity of each set of statement. This can be carried out by the use of factor validity (in SPSS, for example), which entails subjecting the responses to each set of statement to factor analysis (principal components analysis would be sufficient) and check the significance of Bartlett's test of sphericity (Abraham and Barker, 2014). Software like SPSS would also output the test results for sample adequacy in the process via the Kaiser-Meyer-Olkin (KMO) statistic. To pass these two tests, the p-value for Bartlett's test should be below 5% and the KMO statistic should be at least 0.5 respectively (Field, 2016).
  • It is important that you mention all the above in your Research Methodology chapter, including any modifications that you've had had to make before you finalise your survey questionnaire (to be approved by your academic supervisor before you officially start your survey). Also, publish the results of your preliminary reliability analysis under the heading "Pilot Study" in Research Methodology chapter.

You will now realise that it is not within the reach of anyone to design a survey questionnaire. If you still have any doubt about your ability to design your questionnaire after reading this note, seek the assistance of an expert or a statistician.


REFERENCES


Abraham, J. and Barker, K. (2014). Exploring gender difference in motivation, engagement and enrolment behaviour of senior secondary physics students in New South Wales. Research in Science Education, 45(1), pp.59-73.


Ahmad, N. S. and Sabri, A. (2013). Assessing the unidimensionality, reliability, validity and fitness of influential factors of 8th grades student's Mathematics achievement in Malaysia. International Journal of Advance Research, 1(2), pp.1-7.


Campbell, M. J. and Shantikumar, S. (2016). Parametric and Non-parametric tests for comparing two or more groups. Available from: https://www.healthknowledge.org.uk/public-health-textbook/research-methods/1b-statistical-methods/parametric-nonparametric-tests


Cooper, D. R. and Schindler, P. S. (2014). Business Research Methods. 12th ed. New York: McGraw-Hill.


Field, A. (2016). Discovering Statistics Using IBM SPSS Statistics. 4th ed. London: SAGE Publications Ltd.


Hesse-Biber, S. N. and Leavy, P. (2010). The Practice of Qualitative Research. 2nd ed. London: SAGE.


Laerd Dissertation (2012). Principles of Research Ethics. Available from: http://dissertation.laerd.com/principles-of-research-ethics.php


Laerd Statistics (2018). Testing for Normality using SPSS Statistics. Available from: https://statistics.laerd.com/spss-tutorials/testing-for-normality-using-spss-statistics.php


Saunders, M., Lewis, P. and Thornhill, A. (2016). Research Methods for Business Students. 7th ed. England: Pearson Educated Limited.

Back to Resources »