Don't be afraid

Explain, give details, substantiate

17 December 2014

Validity & Reliability

For your test to be reliable it also has to be valid. And all research should be reliable and valid... but what does that mean? Nobody expects your research to be perfect. It is impossible to do research without any errors. You are allowed to make mistakes, and you will, but that is not a problem as long as you acknowledge it. You probably have to make choices that affect the outcome of your research because you do not have enough resources. Doing research costs time and money most people do not have, so you have to make choices.

You might not have a sample frame (that is a list with everybody from your population, so ALL people you want to say something about are on that list. It can be a database, a class list, email addresses of all members, etc.) so you cannot do a simple random sample and have to draw a selective sample. Instead of randomly picking your sample from a list, giving everybody on that list an equal chance to be picked, you might choose using Facebook to try to find respondents or the snowball method (you ask one person who refers you to someone else who again refers you to the next respondent, etc.). You can do that, but you have to realise it affects the findings of your research.

Important is that you describe the choices you made and the errors you made. You might do qualitative research without a lot of experience in interviewing people face-to-face. You do your best but when you are transcribing the interviews you might find out you did not probe or maybe you asked leading questions. We are all humans and everybody can make mistakes. Be honest about those mistakes and describe them in your report.

Choices you make are systematic errors; when you choose Facebook or select your respondents, you know in advance it was not perfect, but it was probably the best you could do.

Reliability refers to repeatability of your research. You describe everything you have done in detail so other people can repeat your research and get similar results. If you want to know how much people are willing to spend on your product, you measure in euros. If you ask many times or other people ask the same question in similar circumstances, the results should be (as good as) equal. The other researcher should know exactly how you asked your questions, in what location, at what time, etc. so they can copy what you did. Even if you made a mistake, you describe it so the other person is aware of that.

Research can be reliable but not valid. The most used example is that if you want to measure the average height of a group, you can measure as often as you want or have other people doing the measuring. As long as the conditions are the same every time the outcome should be the same. So always with or without shoes and at the same time of the day. For other people to repeat it exactly, they need to know what you have done.

But what if you measure height in kilos or in euros? Then you can repeat it a hundred times and the outcome will be the same (so it is reliable) but it is not valid because you did not measure what you intend to measure! You cannot measure height in kilos or in euros, like you cannot measure willingness to pay in centimeters.

Validity is about systematic errors. The choices that you might make because of lack of resources. If you invite people to fill in a questionnaire through sending an email, it should be clear that they have access to internet. People without internet access never got the chance to fill it in, because they did not receive the invitation. If you post a request to complete a survey on Facebook and/or Twitter, it should be no surprise that most respondents are in your age group, have similar level of education, and perhaps also live in your area. Posting things on Facebook is not a probability (random) sample, but it might be the best you can do. You write about this in your research report; show that you are aware of these errors and that you know it affects your outcome. You cannot generalise your results for the whole population but only draw conclusions for the group of people that took part in your research, the respondents.

The external validity has everything to do with the generalisability. And you can only generalise your findings with a random representative sample. This is always a challenge, so don't worry if your sample is selective or purposive - just describe that in your report.

You can increase the internal validity by using literature (journals, theoretical models, research reports) to operationalise the most important terms of your research. Use models or constructs already used by other researchers. Do not just make up definitions or measurement criteria yourself, but check what is already out there. It is likely that someone studied your subject for many years (maybe in a different context, but that is oké) and developed a model you can use to construct your research. Sometimes you can even use a questionnaire that has been made by someone who studied your problem before. Instead of designing your own survey, you can use a mix of your own new ideas and already existing validated instruments (like SERVQUAL). The internal validity of your research increases by using existing theories or measurement criteria based on research that you found in ejournals or books. You make operational models of all important terms based on scales or indicators that have been predefined.

If you do a test twice, the outcome should be very similar the second time. Seems logic, right? Reliability refers to the repeatability of findings. If the study would be done a second time, would it yield the same results? If more than 1 person observes behaviour of visitors at a festival, all observers should agree on what data is being recorded to claim the data to be reliable. Or if you observe behaviour at festival A, you should record the same behaviour in similar circumstances on Festival B if you want to compare them. And if your colleagues are to do the same at festivals C and D, they should know what you have done; where, when and why, and they can find that in your research report because you described everything in detail of course ;)

Whatever you do, always elaborate on your choices and on your mistakes in your report. It is your research, you make your own choices and you are allowed to. Let the reader know why you made certain choices (again, arguing with the use of sources is always better than your own personal opinion) - substantiate!!! Why did you pick one theoretical model over another? Why did you use a selective sampling technique while you know a random non-selective sample is better for the external validity. Explain, give details, substantiate and don't be afraid to make choices or to make mistakes. Nobody is perfect and I dare to say, no research is perfect!

Do you want me to comment on your operational models and paragraphs about reliability and validity? You can send me a message and I will post your text with my comments on MasterYourAbility.com - so the deal is that I give feedback, but your text will be online for others to see.