Validity and Reliability of Questionnaires
Is your questionnaire valid and reliable? Discover easy validation and reliable checks you can apply.
For a questionnaire to be regarded as acceptable, it must possess two very important qualities which are reliability and validity. The former measures the consistency of the questionnaire while the latter measures the degree to which the results from the questionnaire agrees with the real world.
As we go on, we need to first understand what a questionnaire is.
A questionnaire contain sets of questions used for research purposes. These questions aim at collecting demographic information, personal opinions, facts and attitudes from respondents. The questions asked, offer the respondent the ability to air their thoughts on a particular subject matter considered by the questionnaire. Examples of questionnaires are; customer satisfaction questionnaire, product use satisfaction questionnaire and company communications evaluation questionnaire.
The type of information required from the questionnaire directly affects the design of the questionnaire. While qualitative questionnaires are used to collect expository information, quantitative questionnaires are used to validate previously generated hypothesis.
Even though people always confuse a survey for questionnaire, the difference between the two is clear. While surveys always consist of a questionnaire, is much more expensive to execute and often have standard answers that are used to compile data, Questionnaires on the other hand are limited by the fact that the respondents must be able to read the questions and understand them perfectly; to be able to respond well.
Types of Questionnaires
There are 2 major types of questionnaires that exist namely;
Structured Questionnaires
A structured questionnaire is used to collect quantitative data. This type of questionnaire is designed in such a way that it collects intended and specific information. It can also be used to initiate formal inquiry, supplement data and check data that have been formerly accumulated and also, to validate hypothesis.
Unstructured Questionnaires
This type of questionnaire is used to collect qualitative information. The questions contained in this type of questionnaires have basic structure and some branching questions but contain no questions that may limit the responses of a respondent. Simply put, the questions here are more open-ended.
Qualities of a good Questionnaire
Reliability and validity are two very important qualities of a questionnaire. There are different statistical ways to measure the reliability and validity of your questionnaire. The statistical choice often depends on the design and purpose of the questionnaire.
Questionnaire Reliability
This also describes consistency. It is the extent to which that same questionnaire would produce the same results if the study was to be conducted again under the same conditions. Reliability is assessed by;
Test-retest reliability
This involves giving the questionnaire to the same group of respondents at a later point in time and repeating the research. Then, comparing the responses at the two time points. This type of reliability test has a disadvantage caused by memory effects. If the respondents respond to the questions in the way they remembered answering it the first time, it may provide the researcher with an artificial reliability. Thus, to reduce memory effects, the time between the first test and the retest should be increased.
Inter-rater reliability
Like the test-retest reliability, it is conducted under different conditions; the raters are different with one been systematically “harsher” than the other.
Parallel form reliability
Here, parallel equivalent forms of the questionnaire are developed (A and B). Both forms would be used to get the same information but the questions would be constructed differently. Respondents are to fill both forms of questionnaires. Based on the assumption that both forms are interchangeable, the correlation of the 2 forms estimates the reliability of the questionnaire. A disadvantage of this checker is that it is expensive.
Split-half reliability
Split-half reliability measures the extent to which the questions all measure the same underlying construct. Here, the questions are split in two halves and then, the correlation of the scores on the scales from the two halves is calculated. Afterwards, the calculated correlation is run through the Spearman Brown formula.
Questionnaire Validity
This measures the degree of agreement of the results or conclusions gotten from the research questionnaire with the real world. Steps in validating a questionnaire include;
Establish face validity
First, have people who understand your topic go through your questionnaire. They should check if your questionnaire has captured the topic under investigation effectively. Secondly, get an expert on questionnaire construction to check your questionnaire for double, confusing and leading questions.
Conduct a pilot test
Sample size for pilot test varies. You can decide to use a small sample size or a large one. Say you are going for 20 participants per question, if your questionnaire has 30 questions that means you would need a total of 600 respondents. After the respondents have filled out the form, you can then determine what questions are irrelevant and those that are not. Drop the irrelevant questions.
Enter the pilot test in a spreadsheet
Enter the data into a spreadsheet and clean the data. Have one person read the values while the other enters them. This would reduce mistakes that may happen if one person reads and enters the data. Using reversed code, negatively paraphrase questions to determine if the respondents answered recklessly. If the questions were answered correctly, their responses to the negative paraphrased questions will match similar positively phrased questions. If any inconsistency is found, the person’s questionnaire should be tossed out. Check for minimum and maximum value for the entire data sets. If you are getting a response of 6 from a 5-point Likert style scale you have identified an error.
Use principal component analysis (PCA)
This is used to identify underlying components. These components or factor loadings tell you what factors your questions measure. Factor loadings have values ranging from -1.0 to 1.0. When grouping factor loadings, you are advised to look for values that are ±0.60 or higher. You are advised not to attempt conducting PCA if you are inexperienced.
Check the internal consistency of questions loading onto the same factors
This step is used to determine the correlation between questions loading onto the same factor and checks if the responses are consistent. A standard test is Cronbach’s Alpha (CA). ca values range from 0-1.0. A value from 0.60-0.70 is also accepted. If you have a low value, you should consider removing a question; CA value may dramatically increase when you do so.
Revise the questionnaire based on information from your PCA and CA
You can decide to analyze a particular question that does not adequately load onto a factor separately, especially because you think the question is important. If the question that doesn’t load onto a factor is unimportant, you can remove it from the questionnaire. Also, if removing a question increases the CA of a group question then, you can also remove it from the factor loading group. If your questionnaire undergoes major changes then you would have to conduct the pilot test again.
A good questionnaire should be able to establish qualities of reliability and validity for it to be able to produce correct information concerning a particular topic. If a questionnaire used to conduct a study lacks these two very important characteristics, then the conclusion drawn from that particular study can be referred to as invalid. This often means, the study needs to be conducted again.