psychology research limitations

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• Limitations of the Study
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The limitations of the study are those characteristics of design or methodology that impacted or influenced the interpretation of the findings from your research. Study limitations are the constraints placed on the ability to generalize from the results, to further describe applications to practice, and/or related to the utility of findings that are the result of the ways in which you initially chose to design the study or the method used to establish internal and external validity or the result of unanticipated challenges that emerged during the study.

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67; Theofanidis, Dimitrios and Antigoni Fountouki. "Limitations and Delimitations in the Research Process." Perioperative Nursing 7 (September-December 2018): 155-163. .

Importance of...

Always acknowledge a study's limitations. It is far better that you identify and acknowledge your study’s limitations than to have them pointed out by your professor and have your grade lowered because you appeared to have ignored them or didn't realize they existed.

Keep in mind that acknowledgment of a study's limitations is an opportunity to make suggestions for further research. If you do connect your study's limitations to suggestions for further research, be sure to explain the ways in which these unanswered questions may become more focused because of your study.

Acknowledgment of a study's limitations also provides you with opportunities to demonstrate that you have thought critically about the research problem, understood the relevant literature published about it, and correctly assessed the methods chosen for studying the problem. A key objective of the research process is not only discovering new knowledge but also to confront assumptions and explore what we don't know.

Claiming limitations is a subjective process because you must evaluate the impact of those limitations . Don't just list key weaknesses and the magnitude of a study's limitations. To do so diminishes the validity of your research because it leaves the reader wondering whether, or in what ways, limitation(s) in your study may have impacted the results and conclusions. Limitations require a critical, overall appraisal and interpretation of their impact. You should answer the question: do these problems with errors, methods, validity, etc. eventually matter and, if so, to what extent?

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67; Structure: How to Structure the Research Limitations Section of Your Dissertation. Dissertations and Theses: An Online Textbook. Laerd.com.

Descriptions of Possible Limitations

All studies have limitations . However, it is important that you restrict your discussion to limitations related to the research problem under investigation. For example, if a meta-analysis of existing literature is not a stated purpose of your research, it should not be discussed as a limitation. Do not apologize for not addressing issues that you did not promise to investigate in the introduction of your paper.

Here are examples of limitations related to methodology and the research process you may need to describe and discuss how they possibly impacted your results. Note that descriptions of limitations should be stated in the past tense because they were discovered after you completed your research.

Possible Methodological Limitations

• Sample size -- the number of the units of analysis you use in your study is dictated by the type of research problem you are investigating. Note that, if your sample size is too small, it will be difficult to find significant relationships from the data, as statistical tests normally require a larger sample size to ensure a representative distribution of the population and to be considered representative of groups of people to whom results will be generalized or transferred. Note that sample size is generally less relevant in qualitative research if explained in the context of the research problem.
• Lack of available and/or reliable data -- a lack of data or of reliable data will likely require you to limit the scope of your analysis, the size of your sample, or it can be a significant obstacle in finding a trend and a meaningful relationship. You need to not only describe these limitations but provide cogent reasons why you believe data is missing or is unreliable. However, don’t just throw up your hands in frustration; use this as an opportunity to describe a need for future research based on designing a different method for gathering data.
• Lack of prior research studies on the topic -- citing prior research studies forms the basis of your literature review and helps lay a foundation for understanding the research problem you are investigating. Depending on the currency or scope of your research topic, there may be little, if any, prior research on your topic. Before assuming this to be true, though, consult with a librarian! In cases when a librarian has confirmed that there is little or no prior research, you may be required to develop an entirely new research typology [for example, using an exploratory rather than an explanatory research design ]. Note again that discovering a limitation can serve as an important opportunity to identify new gaps in the literature and to describe the need for further research.
• Measure used to collect the data -- sometimes it is the case that, after completing your interpretation of the findings, you discover that the way in which you gathered data inhibited your ability to conduct a thorough analysis of the results. For example, you regret not including a specific question in a survey that, in retrospect, could have helped address a particular issue that emerged later in the study. Acknowledge the deficiency by stating a need for future researchers to revise the specific method for gathering data.
• Self-reported data -- whether you are relying on pre-existing data or you are conducting a qualitative research study and gathering the data yourself, self-reported data is limited by the fact that it rarely can be independently verified. In other words, you have to the accuracy of what people say, whether in interviews, focus groups, or on questionnaires, at face value. However, self-reported data can contain several potential sources of bias that you should be alert to and note as limitations. These biases become apparent if they are incongruent with data from other sources. These are: (1) selective memory [remembering or not remembering experiences or events that occurred at some point in the past]; (2) telescoping [recalling events that occurred at one time as if they occurred at another time]; (3) attribution [the act of attributing positive events and outcomes to one's own agency, but attributing negative events and outcomes to external forces]; and, (4) exaggeration [the act of representing outcomes or embellishing events as more significant than is actually suggested from other data].

Possible Limitations of the Researcher

• Access -- if your study depends on having access to people, organizations, data, or documents and, for whatever reason, access is denied or limited in some way, the reasons for this needs to be described. Also, include an explanation why being denied or limited access did not prevent you from following through on your study.
• Longitudinal effects -- unlike your professor, who can literally devote years [even a lifetime] to studying a single topic, the time available to investigate a research problem and to measure change or stability over time is constrained by the due date of your assignment. Be sure to choose a research problem that does not require an excessive amount of time to complete the literature review, apply the methodology, and gather and interpret the results. If you're unsure whether you can complete your research within the confines of the assignment's due date, talk to your professor.
• Cultural and other type of bias -- we all have biases, whether we are conscience of them or not. Bias is when a person, place, event, or thing is viewed or shown in a consistently inaccurate way. Bias is usually negative, though one can have a positive bias as well, especially if that bias reflects your reliance on research that only support your hypothesis. When proof-reading your paper, be especially critical in reviewing how you have stated a problem, selected the data to be studied, what may have been omitted, the manner in which you have ordered events, people, or places, how you have chosen to represent a person, place, or thing, to name a phenomenon, or to use possible words with a positive or negative connotation. NOTE :   If you detect bias in prior research, it must be acknowledged and you should explain what measures were taken to avoid perpetuating that bias. For example, if a previous study only used boys to examine how music education supports effective math skills, describe how your research expands the study to include girls.
• Fluency in a language -- if your research focuses , for example, on measuring the perceived value of after-school tutoring among Mexican-American ESL [English as a Second Language] students and you are not fluent in Spanish, you are limited in being able to read and interpret Spanish language research studies on the topic or to speak with these students in their primary language. This deficiency should be acknowledged.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Senunyeme, Emmanuel K. Business Research Methods. Powerpoint Presentation. Regent University of Science and Technology; ter Riet, Gerben et al. “All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies.” PLOS One 8 (November 2013): 1-6.

Structure and Writing Style

Information about the limitations of your study are generally placed either at the beginning of the discussion section of your paper so the reader knows and understands the limitations before reading the rest of your analysis of the findings, or, the limitations are outlined at the conclusion of the discussion section as an acknowledgement of the need for further study. Statements about a study's limitations should not be buried in the body [middle] of the discussion section unless a limitation is specific to something covered in that part of the paper. If this is the case, though, the limitation should be reiterated at the conclusion of the section.

If you determine that your study is seriously flawed due to important limitations , such as, an inability to acquire critical data, consider reframing it as an exploratory study intended to lay the groundwork for a more complete research study in the future. Be sure, though, to specifically explain the ways that these flaws can be successfully overcome in a new study.

But, do not use this as an excuse for not developing a thorough research paper! Review the tab in this guide for developing a research topic . If serious limitations exist, it generally indicates a likelihood that your research problem is too narrowly defined or that the issue or event under study is too recent and, thus, very little research has been written about it. If serious limitations do emerge, consult with your professor about possible ways to overcome them or how to revise your study.

When discussing the limitations of your research, be sure to:

• Describe each limitation in detailed but concise terms;
• Explain why each limitation exists;
• Provide the reasons why each limitation could not be overcome using the method(s) chosen to acquire or gather the data [cite to other studies that had similar problems when possible];
• Assess the impact of each limitation in relation to the overall findings and conclusions of your study; and,
• If appropriate, describe how these limitations could point to the need for further research.

Remember that the method you chose may be the source of a significant limitation that has emerged during your interpretation of the results [for example, you didn't interview a group of people that you later wish you had]. If this is the case, don't panic. Acknowledge it, and explain how applying a different or more robust methodology might address the research problem more effectively in a future study. A underlying goal of scholarly research is not only to show what works, but to demonstrate what doesn't work or what needs further clarification.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Ioannidis, John P.A. "Limitations are not Properly Acknowledged in the Scientific Literature." Journal of Clinical Epidemiology 60 (2007): 324-329; Pasek, Josh. Writing the Empirical Social Science Research Paper: A Guide for the Perplexed. January 24, 2012. Academia.edu; Structure: How to Structure the Research Limitations Section of Your Dissertation. Dissertations and Theses: An Online Textbook. Laerd.com; What Is an Academic Paper? Institute for Writing Rhetoric. Dartmouth College; Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University.

Writing Tip

Don't Inflate the Importance of Your Findings!

After all the hard work and long hours devoted to writing your research paper, it is easy to get carried away with attributing unwarranted importance to what you’ve done. We all want our academic work to be viewed as excellent and worthy of a good grade, but it is important that you understand and openly acknowledge the limitations of your study. Inflating the importance of your study's findings could be perceived by your readers as an attempt hide its flaws or encourage a biased interpretation of the results. A small measure of humility goes a long way!

Another Writing Tip

Negative Results are Not a Limitation!

Negative evidence refers to findings that unexpectedly challenge rather than support your hypothesis. If you didn't get the results you anticipated, it may mean your hypothesis was incorrect and needs to be reformulated. Or, perhaps you have stumbled onto something unexpected that warrants further study. Moreover, the absence of an effect may be very telling in many situations, particularly in experimental research designs. In any case, your results may very well be of importance to others even though they did not support your hypothesis. Do not fall into the trap of thinking that results contrary to what you expected is a limitation to your study. If you carried out the research well, they are simply your results and only require additional interpretation.

Lewis, George H. and Jonathan F. Lewis. “The Dog in the Night-Time: Negative Evidence in Social Research.” The British Journal of Sociology 31 (December 1980): 544-558.

Yet Another Writing Tip

Sample Size Limitations in Qualitative Research

Sample sizes are typically smaller in qualitative research because, as the study goes on, acquiring more data does not necessarily lead to more information. This is because one occurrence of a piece of data, or a code, is all that is necessary to ensure that it becomes part of the analysis framework. However, it remains true that sample sizes that are too small cannot adequately support claims of having achieved valid conclusions and sample sizes that are too large do not permit the deep, naturalistic, and inductive analysis that defines qualitative inquiry. Determining adequate sample size in qualitative research is ultimately a matter of judgment and experience in evaluating the quality of the information collected against the uses to which it will be applied and the particular research method and purposeful sampling strategy employed. If the sample size is found to be a limitation, it may reflect your judgment about the methodological technique chosen [e.g., single life history study versus focus group interviews] rather than the number of respondents used.

Boddy, Clive Roland. "Sample Size for Qualitative Research." Qualitative Market Research: An International Journal 19 (2016): 426-432; Huberman, A. Michael and Matthew B. Miles. "Data Management and Analysis Methods." In Handbook of Qualitative Research . Norman K. Denzin and Yvonna S. Lincoln, eds. (Thousand Oaks, CA: Sage, 1994), pp. 428-444; Blaikie, Norman. "Confounding Issues Related to Determining Sample Size in Qualitative Research." International Journal of Social Research Methodology 21 (2018): 635-641; Oppong, Steward Harrison. "The Problem of Sampling in qualitative Research." Asian Journal of Management Sciences and Education 2 (2013): 202-210.

• << Previous: 8. The Discussion
• Next: 9. The Conclusion >>
• Last Updated: Sep 4, 2024 9:40 AM
• URL: https://libguides.usc.edu/writingguide

Limitations of Psychological Enquiry Explained

Psychologists study various mental processes, experiences, and human behavior, and to study all these factors, they rely on various methods of enquiry such as case study, content analysis, correlational researches, psychological testing, survey research, experimental, and observational. These enquiry methods help psychologists study various phenomena of the human mind in a systematic, organized, and well-structured manner. The main goals of the psychological enquiry involve describing the human behavior, predicting the effects of various variables, explaining & controlling various factors that change the behavior, and then finally applying the gained information to understand and formulate the psychological theories. The systematic approach of the psychological enquiry methods is the reason that various psychological theories are often termed scientifically proven. However, there exist some limitations also of the psychological enquiry methods. We’ll discuss these limitations in this article.

1. Lack of True Zero Point

A true zero point indicates that any other factor or trait is absent or null at that point. Whenever we measure or weigh anything, we start measuring it from the zero value, say 5 inches in length or 15 kg of weight. In fact, in physical sciences, every measurement starts from zero, but psychological measurements lack the availability of true zero points. For instance, we cannot say that a particular person has a zero value of Intelligence Quotient (IQ) or zero effect of outward stimuli on his/her behavior because every person has some level of intelligence, and outward stimuli always impact the human behavior to a certain extent. So, to measure various psychological factors, psychologists arbitrarily assign certain points as zero points and measure various phenomena with respect to these arbitrarily assigned zero points.

2. Relative Nature of Psychological Tests

The scores we obtain through the psychological tests are not absolute, rather these are relative scores because we score various psychological attributes by comparing them with the other relative factors. However, relative scores do not tell the actual level of the specific factor in the subject. For example, if we rank three subjects, say A, B, and C, as first, second, and third respectively on the basis of the marks they obtained in the IQ test, the major problem of this ranking would be that the intelligence difference of the subject A and subject B may or may not be same to the intelligence difference of the subject B and subject C. Let’s say, out of 200, the first rank holder (A) obtained a 140 IQ score, the second rank holder (B) obtained 130, and the third rank holder (C) obtained 90. Clearly, the IQ score difference of subjects A and B is much less than the subjects B and C. This shows that the subjects are not ranked according to their actual level of intelligence; in fact, they are ranked by the comparison methods.

3. Lack of Generalizability

Another major problem with the psychological enquiry method or, specifically, psychological testing is that they lack generalizability. Every psychological test is designed by keeping in mind certain factors related to a particular context or area. In fact, the majority of the psychological tests that we encounter today were designed by conducting various researches in the European and American regions, and most of the psychological theories are formulated by taking samples from these western populations. These psychological tests may not give accurate results if these are applied to the Asian or the African population because there are cultural and behavioral differences in different regions of the world. So, before conducting any psychological test, it should be properly checked that whether the particular test is generalizable or not because every single psychological test can not be used universally.

4. Subjective Interpretation of Qualitative Research Data

Psychological enquiry involves the collection of a vast amount of qualitative data, which includes feelings, behavior, perception, and other psychological factors of the subjects under study, and there is not any proper degree to measure these factors. Every psychologist interprets the research data in his/her own way. So, there may exist differences in the results obtained by different psychologists about the same factor or event. Hence, it is always preferred that the particular psychological enquiry or case study should be done by a panel of psychologists, and there should be a proper discussion between the psychologists before deducing the final results or formulating any psychological theory.

5. Other Issues

As psychological researches involve inspection of human behavior, the researchers have to follow various ethical guidelines while doing the psychological enquiry. Ethical guidelines may limit the psychological enquiry, but psychologists should respect the privacy of the subject and follow these ethical guidelines. There are proper ethical guidelines provided by the ‘American Psychological Association,’ which emphasize various aspects that need to be followed by the researcher like participation of the subject in the research should be voluntary, consent should be taken before conducting the research, proper debriefing about the test should be provided to the subjects, and confidentiality of the subjects should be maintained. As every research can not be conducted on humans, researchers conduct various studies on animals too. It should be noted that only those animals should be considered for the research, which are scientifically suitable for that particular research according to the ethical guidelines. Although it’s unethical to conduct researches on animals, it’s a necessity in every field of sciences to conduct researches on animals to discover new phenomena and mysteries of living beings. However, proper care of the animals should be taken, and wherever possible, mental or physical harm to the animals should be avoided.

Related Posts

15 Examples of Psychology in Everyday Life

Moral Equilibrium

15 Acts of Altruism Examples

Examples of Ray in Real Life

22 Altruism Examples in Animals

Cannon’s Theory of Emotion and Stress Explained

Add comment cancel reply.

Identifying and Overcoming Limitations in Psychology

Understanding the limitations in psychology research is crucial for the accuracy and validity of the findings. In this article, we will explore the various types of limitations in psychology, such as methodological, sampling, measurement, generalizability, and ethical limitations.

We will also discuss the importance of identifying these limitations and how researchers can overcome them through strategies like using multiple methods, increasing sample size, improving measurement tools, considering different populations, and following ethical guidelines.

We will provide examples of how researchers have successfully overcome limitations in psychology research. Join us as we delve into the realm of identifying and overcoming limitations in psychology research.

• Limitations in psychology research can hinder the accuracy and validity of findings.
• Identifying limitations is crucial for improving the quality and credibility of psychological research.
• Addressing methodological, sampling, measurement, generalizability, and ethical limitations can enhance the reliability of results.
• 1 What Are Limitations in Psychology?
• 2 Why Is It Important to Identify Limitations in Psychology?
• 3.1 Methodological Limitations
• 3.2 Sampling Limitations
• 3.3 Measurement Limitations
• 3.4 Generalizability Limitations
• 3.5 Ethical Limitations
• 4.1 Use Multiple Methods
• 4.2 Increase Sample Size
• 4.3 Improve Measurement Tools
• 4.4 Consider Different Populations
• 4.5 Follow Ethical Guidelines
• 5 Examples of Overcoming Limitations in Psychology Research
• 6 Conclusion
• 7.1 1. What are limitations in psychology and why is it important to identify them?
• 7.2 2. How can I identify limitations in psychology?
• 7.3 3. What are some common limitations in psychology research?
• 7.4 4. How can limitations in psychology be overcome?
• 7.5 5. Can personal biases and assumptions be a limitation in psychology?
• 7.6 6. Are there ethical limitations in psychology research?

What Are Limitations in Psychology?

Limitations in psychology refer to the boundaries or constraints that hinder the field’s research, understanding, and applicability.

These limitations can manifest in various forms, impacting the accuracy and generalizability of research findings. For instance, one common challenge is the issue of sample size, where small sample sizes may lead to results that are not statistically significant or representative of the larger population. External validity can be compromised when studies are conducted in controlled environments that do not capture the complexity of real-world situations.

Theoretical frameworks in psychology may be limited by the constructs and variables that researchers choose to include or exclude, which can influence the conclusions drawn from the research. Control theory, for example, emphasizes the importance of controlling for extraneous variables to establish causal relationships, but this may not always be feasible in practice.

Mental barriers, such as biases and preconceptions, can introduce limitations in the interpretation of results and the application of psychological principles in real-life settings. These mental barriers can cloud judgment, skew observations, and impede the development of effective interventions or treatments. Understanding and addressing these limitations is crucial for advancing the field of psychology and ensuring that research outcomes are robust, reliable, and actionable.

Why Is It Important to Identify Limitations in Psychology?

Identifying limitations in psychology is crucial as it enables researchers and practitioners to acknowledge shortcomings, improve methodologies, and foster self-compassion in the face of failures.

Recognizing these limitations not only leads to a deeper understanding of human emotions and behavior but also encourages a growth mindset, allowing individuals to view setbacks as opportunities for learning and development. By embracing a positive self-image and understanding that success is often accompanied by failures, individuals can navigate obstacles with resilience and determination. Through this process, valuable lessons are learned that contribute to personal and professional growth, shaping more effective and empathetic researchers and practitioners.

Types of Limitations in Psychology

Various types of limitations exist in psychology research, including methodological, sampling, measurement, generalizability, and ethical constraints that can impact the validity and reliability of findings.

Methodological limitations refer to flaws in the design or execution of research studies. For instance, if a study lacks a control group, it may lead to inaccurate conclusions. Sampling limitations occur when the sample size is too small or not representative of the population being studied, compromising the study’s ability to generalize findings. Measurement limitations involve issues with the tools used to measure variables, such as unreliable tests or subjective rating scales. Generalizability limitations arise when findings from a specific study can’t be applied universally. Ethical constraints involve ensuring the protection and rights of participants involved in research.

Methodological Limitations

Methodological limitations in psychology research pertain to constraints related to study design, data collection methods, and analytical approaches that may impact the validity and generalizability of results.

One common pitfall in psychology research is the reliance solely on self-report measures, which can introduce response biases and social desirability effects.

Experts often caution against overgeneralizing findings from a single study, emphasizing the importance of replicating results to ensure reliability.

Triangulating data from multiple sources, such as observational, self-report, and physiological measures, can enhance the robustness of findings.

Researchers must be mindful of potential biases in participant selection, sampling methods, and data interpretation to minimize errors and enhance the credibility of their conclusions.

Sampling Limitations

Sampling limitations in psychology research refer to issues concerning the selection, representation, and size of study participants, influencing the generalizability and external validity of findings.

When considering collaboration in research, some challenges may arise in recruiting a diverse and representative sample. For instance, researchers like Aditi Paul often face biases in participant selection due to factors such as location, culture, or access to technology. To mitigate these biases, strategies such as targeted recruitment through various channels, inclusive language in study materials, and leveraging community partnerships can help enhance the representativeness of the sample. Diligent data analysis techniques, like statistical adjustments or subgroup analyses, can provide insights into the impact of sampling biases on research outcomes.

Measurement Limitations

Measurement limitations in psychology research encompass issues related to the accuracy, reliability, and validity of assessment tools, impacting the precision and consistency of data collected.

Instrument reliability is crucial in ensuring that the measurements taken remain consistent over time and across different conditions. An instrument with high reliability will produce similar results when used repeatedly. Construct validity, on the other hand, focuses on whether the assessment tool is actually measuring the intended construct accurately.

Emotions, being complex and multifaceted, pose challenges in measurement due to their subjective nature and variability. Researchers must employ control theory to minimize external influences that can introduce biases and distort the results. Implementing robust research methodologies can help mitigate potential sources of measurement error and enhance the overall quality of data.

Generalizability Limitations

Generalizability limitations in psychology research relate to the extent to which findings and conclusions can be applied or generalized beyond the specific study sample or context, affecting the broader implications of research outcomes.

When considering the generalizability of research findings, it is crucial to acknowledge the influence of various factors such as cultural differences, geographical locations, and socio-economic backgrounds on the transferability of results. Dr. Jean-Charles Lebeau emphasizes the importance of ethical guidelines in research to ensure the protection and well-being of participants. The scope of the study, sample size, and research methodology are essential considerations when evaluating the external validity of a study’s results.

Ethical Limitations

Ethical limitations in psychology research encompass challenges and constraints related to participant consent, confidentiality, data handling, and research conduct, emphasizing the importance of ethical considerations in psychological studies.

When conducting research, it is crucial to navigate the delicate balance between acquiring valuable data and respecting the rights and well-being of participants. One must ensure that participants provide informed consent, understand how their data will be used and stored, and have the option to withdraw from the study at any point. Researchers must uphold stringent confidentiality measures to protect participants’ privacy and prevent any breach of trust. Understanding ethical dilemmas that may arise and having a clear ethical review process in place guides researchers in maintaining integrity and fostering a culture of collaboration and self-compassion in the field of psychology.

How to Overcome Limitations in Psychology Research?

Overcoming limitations in psychology research requires adopting multifaceted strategies, including methodological enhancements , increased sample sizes, and ethical considerations to mitigate biases and enhance research quality.

One effective way to address the challenge of small sample sizes within psychological research is to prioritize diverse sampling techniques . By utilizing a variety of sampling methods such as convenience sampling, stratified sampling, or snowball sampling, researchers can access a more representative and varied pool of participants. This broader range of participants can lead to more nuanced and reliable findings, improving the overall success of the study and the accuracy of conclusions drawn.

Use Multiple Methods

Employing a diverse range of research methods and approaches can help researchers triangulate data, validate findings, and enhance the robustness of research outcomes in psychology.

By combining quantitative surveys with qualitative interviews, researchers can gain a deeper understanding of complex phenomena and behaviors. This mixed-method approach allows for a more comprehensive examination of research questions, providing richer insights and more nuanced interpretations.

Interdisciplinary collaborations with experts from various fields can also bring fresh perspectives and innovative techniques to the research process, fostering creativity and stimulating new ideas.

Increase Sample Size

Expanding the sample size in psychology research can improve statistical power, reduce sampling errors, and enhance the generalizability of research findings to larger populations or contexts.

Determining the optimal sample size is crucial to ensure the study’s results are reliable and valid. Conducting a power analysis helps researchers ascertain the minimum number of participants needed to detect a significant effect accurately. By increasing the sample size, researchers can enhance the study’s ability to detect true effects, reducing the likelihood of Type I and Type II errors. Incorporating strategies to ensure sample representativeness, such as random sampling methods and proper participant recruitment techniques, can enhance the study’s external validity.

Improve Measurement Tools

Enhancing measurement tools and assessment instruments in psychology research can bolster the reliability, validity, and sensitivity of data collection, leading to more precise and accurate research outcomes.

When conducting research, it is crucial to ensure that the measurement tools used are robust and valid. By focusing on refining psychometric properties, researchers can better capture the intricacies of emotions and other psychological constructs within their studies. Developing a well-constructed scale involves meticulous attention to detail and thorough testing to guarantee its efficacy. Validation processes play a significant role in establishing the credibility and accuracy of the instrument, paving the way for insightful findings related to self-compassion and self-image . Careful consideration of these aspects contributes to the overall strength and impact of psychological research.”

Consider Different Populations

Exploring diverse populations and demographic groups in psychology research can enrich study findings, uncover unique insights, and enhance the external validity and applicability of research outcomes.

Incorporating diverse samples is crucial in ensuring that research findings are not only comprehensive but also reflective of human diversity worldwide. When researchers limit themselves to a narrow participant pool, they risk producing biased results that may not be generalizable to a broader population. By stepping out of the comfort zone and actively seeking to collaborate with individuals from different cultural backgrounds, researchers open up avenues for richer data collection and nuanced interpretations.

Follow Ethical Guidelines

Adhering to ethical guidelines and principles in psychology research is paramount to safeguarding participant rights, ensuring data integrity, and upholding the moral and professional standards of the research community.

In the realm of psychology studies, ethical conduct acts as a guiding compass, steering researchers towards responsible and respectful interactions with study participants. Central to this principle is the concept of informed consent, where individuals are provided with clear information about the study’s purpose, procedures, and potential risks before agreeing to participate. Confidentiality protocols play a pivotal role in protecting the privacy and anonymity of participants, fostering a safe space for them to share their experiences.

Examples of Overcoming Limitations in Psychology Research

Several exemplary cases illustrate successful strategies for overcoming limitations in psychology research, showcasing innovative methodologies, collaborative efforts, and adaptive practices that lead to groundbreaking discoveries.

One such remarkable case involved a team of researchers who encountered challenges in dealing with the emotional aspects of their study on trauma healing. Recognizing the significance of understanding and addressing emotions in the research process, they integrated self-compassion practices into their data collection methods. This innovative approach not only improved participant engagement but also provided valuable insights into the impact of emotions on trauma recovery.

Recognizing and addressing limitations in psychology research is fundamental to advancing the field’s knowledge, fostering innovation, and promoting evidence-based practices for future studies.

Embracing failures as learning opportunities can lead to greater success in research endeavors. By learning from setbacks, researchers can refine methodologies, develop new insights, and contribute more effectively to the academic community.

• Overcoming obstacles in psychology research requires a growth mindset that values perseverance and resilience.
• Continuous improvement through further research and collaboration not only benefits individual researchers but also enhances the credibility and impact of the entire field.
• Engaging in dialogue with peers, seeking constructive feedback, and staying abreast of the latest developments are essential strategies for navigating challenges and achieving breakthroughs in psychology research.

Frequently Asked Questions

1. what are limitations in psychology and why is it important to identify them.

Limitations in psychology refer to factors that may affect the accuracy or generalizability of research findings. It is important to identify these limitations as they can impact the validity and reliability of psychological studies, ultimately affecting our understanding and application of psychological principles.

2. How can I identify limitations in psychology?

One way to identify limitations is to critically evaluate the research methods and designs used in a study. Look for potential biases, flaws in the data collection process, or limitations in the sample size or characteristics. Additionally, comparing the results with other studies in the same area can help identify any discrepancies or limitations.

3. What are some common limitations in psychology research?

Some common limitations in psychology research include small sample sizes, biased samples, subjective measurements, and lack of control over extraneous variables. Additionally, the generalizability of findings to the larger population may be limited by the characteristics of the sample used in the study.

4. How can limitations in psychology be overcome?

One way to overcome limitations is through replication and conducting further research. By replicating a study with a larger and more diverse sample, researchers can test the generalizability of findings and address any potential biases. Additionally, using a variety of research methods and incorporating multiple perspectives can help overcome limitations.

5. Can personal biases and assumptions be a limitation in psychology?

Yes, personal biases and assumptions can be a limitation in psychology. Researchers’ personal beliefs and assumptions can influence the way they conduct and interpret their research, potentially leading to biased results. It is important for researchers to be aware of their own biases and use objective measures to minimize their impact on the study.

6. Are there ethical limitations in psychology research?

Yes, there can be ethical limitations in psychology research. Researchers must adhere to ethical principles and guidelines when conducting studies involving human participants, such as informed consent, protection of participants’ privacy, and avoiding harm or coercion. These ethical limitations may restrict the type of research that can be conducted, but they are necessary to protect the well-being of participants.

Dr. Henry Foster is a neuropsychologist with a focus on cognitive disorders and brain rehabilitation. His clinical work involves assessing and treating individuals with brain injuries and neurodegenerative diseases. Through his writing, Dr. Foster shares insights into the brain’s ability to heal and adapt, offering hope and practical advice for patients and families navigating the challenges of cognitive impairments.

Similar Posts

Understanding the Psychology of Habits: How They Form and Influence Behavior

The article was last updated by Dr. Emily Tan on February 8, 2024. Have you ever wondered why we have certain behaviors that seem almost…

The Art of Designing an Observational Study in Psychology: Guidelines and Best Practices

The article was last updated by Samantha Choi on February 5, 2024. Are you curious about how psychologists conduct observational studies to better understand human…

The Role of Vignettes in Psychology: Understanding the Power of Narratives

The article was last updated by Julian Torres on February 8, 2024. Vignettes are powerful tools used in psychology to convey complex concepts, evoke emotions,…

Signal Detection Theory in Psychology: Explanation and Applications

The article was last updated by Sofia Alvarez on February 8, 2024. Signal Detection Theory is a fundamental concept in psychology that aims to understand…

Understanding Confabulation in Psychology

The article was last updated by Dr. Henry Foster on February 6, 2024. Confabulation is a fascinating phenomenon in the field of psychology that involves…

Understanding Baseline in Psychology

The article was last updated by Lena Nguyen on February 4, 2024. Baseline in psychology is a crucial concept that serves as a reference point…

Instant insights, infinite possibilities

How to present limitations in research

Last updated

30 January 2024

Reviewed by

Short on time? Get an AI generated summary of this article instead

Limitations don’t invalidate or diminish your results, but it’s best to acknowledge them. This will enable you to address any questions your study failed to answer because of them.

In this guide, learn how to recognize, present, and overcome limitations in research.

• What is a research limitation?

Research limitations are weaknesses in your research design or execution that may have impacted outcomes and conclusions. Uncovering limitations doesn’t necessarily indicate poor research design—it just means you encountered challenges you couldn’t have anticipated that limited your research efforts.

Does basic research have limitations?

Basic research aims to provide more information about your research topic . It requires the same standard research methodology and data collection efforts as any other research type, and it can also have limitations.

• Common research limitations

Researchers encounter common limitations when embarking on a study. Limitations can occur in relation to the methods you apply or the research process you design. They could also be connected to you as the researcher.

Methodology limitations

Not having access to data or reliable information can impact the methods used to facilitate your research. A lack of data or reliability may limit the parameters of your study area and the extent of your exploration.

Your sample size may also be affected because you won’t have any direction on how big or small it should be and who or what you should include. Having too few participants won’t adequately represent the population or groups of people needed to draw meaningful conclusions.

Research process limitations

The study’s design can impose constraints on the process. For example, as you’re conducting the research, issues may arise that don’t conform to the data collection methodology you developed. You may not realize until well into the process that you should have incorporated more specific questions or comprehensive experiments to generate the data you need to have confidence in your results.

Constraints on resources can also have an impact. Being limited on participants or participation incentives may limit your sample sizes. Insufficient tools, equipment, and materials to conduct a thorough study may also be a factor.

Common researcher limitations

Here are some of the common researcher limitations you may encounter:

Time: some research areas require multi-year longitudinal approaches, but you might not be able to dedicate that much time. Imagine you want to measure how much memory a person loses as they age. This may involve conducting multiple tests on a sample of participants over 20–30 years, which may be impossible.

Bias: researchers can consciously or unconsciously apply bias to their research. Biases can contribute to relying on research sources and methodologies that will only support your beliefs about the research you’re embarking on. You might also omit relevant issues or participants from the scope of your study because of your biases.

Limited access to data : you may need to pay to access specific databases or journals that would be helpful to your research process. You might also need to gain information from certain people or organizations but have limited access to them. These cases require readjusting your process and explaining why your findings are still reliable.

• Why is it important to identify limitations?

Identifying limitations adds credibility to research and provides a deeper understanding of how you arrived at your conclusions.

Constraints may have prevented you from collecting specific data or information you hoped would prove or disprove your hypothesis or provide a more comprehensive understanding of your research topic.

However, identifying the limitations contributing to your conclusions can inspire further research efforts that help gather more substantial information and data.

• Where to put limitations in a research paper

A research paper is broken up into different sections that appear in the following order:

Introduction

Methodology

The discussion portion of your paper explores your findings and puts them in the context of the overall research. Either place research limitations at the beginning of the discussion section before the analysis of your findings or at the end of the section to indicate that further research needs to be pursued.

What not to include in the limitations section

Evidence that doesn’t support your hypothesis is not a limitation, so you shouldn’t include it in the limitation section. Don’t just list limitations and their degree of severity without further explanation.

• How to present limitations

You’ll want to present the limitations of your study in a way that doesn’t diminish the validity of your research and leave the reader wondering if your results and conclusions have been compromised.

Include only the limitations that directly relate to and impact how you addressed your research questions. Following a specific format enables the reader to develop an understanding of the weaknesses within the context of your findings without doubting the quality and integrity of your research.

Identify the limitations specific to your study

You don’t have to identify every possible limitation that might have occurred during your research process. Only identify those that may have influenced the quality of your findings and your ability to answer your research question.

Explain study limitations in detail

This explanation should be the most significant portion of your limitation section.

Link each limitation with an interpretation and appraisal of their impact on the study. You’ll have to evaluate and explain whether the error, method, or validity issues influenced the study’s outcome and how.

Propose a direction for future studies and present alternatives

In this section, suggest how researchers can avoid the pitfalls you experienced during your research process.

If an issue with methodology was a limitation, propose alternate methods that may help with a smoother and more conclusive research project . Discuss the pros and cons of your alternate recommendation.

Describe steps taken to minimize each limitation

You probably took steps to try to address or mitigate limitations when you noticed them throughout the course of your research project. Describe these steps in the limitation section.

• Limitation example

“Approaches like stem cell transplantation and vaccination in AD [Alzheimer’s disease] work on a cellular or molecular level in the laboratory. However, translation into clinical settings will remain a challenge for the next decade.”

The authors are saying that even though these methods showed promise in helping people with memory loss when conducted in the lab (in other words, using animal studies), more studies are needed. These may be controlled clinical trials, for example. 

However, the short life span of stem cells outside the lab and the vaccination’s severe inflammatory side effects are limitations. Researchers won’t be able to conduct clinical trials until these issues are overcome.

• How to overcome limitations in research

You’ve already started on the road to overcoming limitations in research by acknowledging that they exist. However, you need to ensure readers don’t mistake weaknesses for errors within your research design.

To do this, you’ll need to justify and explain your rationale for the methods, research design, and analysis tools you chose and how you noticed they may have presented limitations.

Your readers need to know that even when limitations presented themselves, you followed best practices and the ethical standards of your field. You didn’t violate any rules and regulations during your research process.

You’ll also want to reinforce the validity of your conclusions and results with multiple sources, methods, and perspectives. This prevents readers from assuming your findings were derived from a single or biased source.

• Learning and improving starts with limitations in research

Dealing with limitations with transparency and integrity helps identify areas for future improvements and developments. It’s a learning process, providing valuable insights into how you can improve methodologies, expand sample sizes, or explore alternate approaches to further support the validity of your findings.

Should you be using a customer insights hub?

Do you want to discover previous research faster?

Do you share your research findings with others?

Do you analyze research data?

Start for free today, add your research, and get to key insights faster

Editor’s picks

Last updated: 18 April 2023

Last updated: 27 February 2023

Last updated: 22 August 2024

Last updated: 5 February 2023

Last updated: 16 August 2024

Last updated: 9 March 2023

Last updated: 30 April 2024

Last updated: 12 December 2023

Last updated: 11 March 2024

Last updated: 4 July 2024

Last updated: 6 March 2024

Last updated: 5 March 2024

Last updated: 13 May 2024

Latest articles

Related topics, .css-je19u9{-webkit-align-items:flex-end;-webkit-box-align:flex-end;-ms-flex-align:flex-end;align-items:flex-end;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-box-flex-wrap:wrap;-webkit-flex-wrap:wrap;-ms-flex-wrap:wrap;flex-wrap:wrap;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;row-gap:0;text-align:center;max-width:671px;}@media (max-width: 1079px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}}@media (max-width: 799px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}} decide what to .css-1kiodld{max-height:56px;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}@media (max-width: 1079px){.css-1kiodld{display:none;}} build next, decide what to build next, log in or sign up.

Get started for free

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Perspect Med Educ
• v.8(4); 2019 Aug

Limited by our limitations

Paula t. ross.

Medical School, University of Michigan, Ann Arbor, MI USA

Nikki L. Bibler Zaidi

Study limitations represent weaknesses within a research design that may influence outcomes and conclusions of the research. Researchers have an obligation to the academic community to present complete and honest limitations of a presented study. Too often, authors use generic descriptions to describe study limitations. Including redundant or irrelevant limitations is an ineffective use of the already limited word count. A meaningful presentation of study limitations should describe the potential limitation, explain the implication of the limitation, provide possible alternative approaches, and describe steps taken to mitigate the limitation. This includes placing research findings within their proper context to ensure readers do not overemphasize or minimize findings. A more complete presentation will enrich the readers’ understanding of the study’s limitations and support future investigation.

Introduction

Regardless of the format scholarship assumes, from qualitative research to clinical trials, all studies have limitations. Limitations represent weaknesses within the study that may influence outcomes and conclusions of the research. The goal of presenting limitations is to provide meaningful information to the reader; however, too often, limitations in medical education articles are overlooked or reduced to simplistic and minimally relevant themes (e.g., single institution study, use of self-reported data, or small sample size) [ 1 ]. This issue is prominent in other fields of inquiry in medicine as well. For example, despite the clinical implications, medical studies often fail to discuss how limitations could have affected the study findings and interpretations [ 2 ]. Further, observational research often fails to remind readers of the fundamental limitation inherent in the study design, which is the inability to attribute causation [ 3 ]. By reporting generic limitations or omitting them altogether, researchers miss opportunities to fully communicate the relevance of their work, illustrate how their work advances a larger field under study, and suggest potential areas for further investigation.

Goals of presenting limitations

Medical education scholarship should provide empirical evidence that deepens our knowledge and understanding of education [ 4 , 5 ], informs educational practice and process, [ 6 , 7 ] and serves as a forum for educating other researchers [ 8 ]. Providing study limitations is indeed an important part of this scholarly process. Without them, research consumers are pressed to fully grasp the potential exclusion areas or other biases that may affect the results and conclusions provided [ 9 ]. Study limitations should leave the reader thinking about opportunities to engage in prospective improvements [ 9 – 11 ] by presenting gaps in the current research and extant literature, thereby cultivating other researchers’ curiosity and interest in expanding the line of scholarly inquiry [ 9 ].

Presenting study limitations is also an ethical element of scientific inquiry [ 12 ]. It ensures transparency of both the research and the researchers [ 10 , 13 , 14 ], as well as provides transferability [ 15 ] and reproducibility of methods. Presenting limitations also supports proper interpretation and validity of the findings [ 16 ]. A study’s limitations should place research findings within their proper context to ensure readers are fully able to discern the credibility of a study’s conclusion, and can generalize findings appropriately [ 16 ].

Why some authors may fail to present limitations

As Price and Murnan [ 8 ] note, there may be overriding reasons why researchers do not sufficiently report the limitations of their study. For example, authors may not fully understand the importance and implications of their study’s limitations or assume that not discussing them may increase the likelihood of publication. Word limits imposed by journals may also prevent authors from providing thorough descriptions of their study’s limitations [ 17 ]. Still another possible reason for excluding limitations is a diffusion of responsibility in which some authors may incorrectly assume that the journal editor is responsible for identifying limitations. Regardless of reason or intent, researchers have an obligation to the academic community to present complete and honest study limitations.

A guide to presenting limitations

The presentation of limitations should describe the potential limitations, explain the implication of the limitations, provide possible alternative approaches, and describe steps taken to mitigate the limitations. Too often, authors only list the potential limitations, without including these other important elements.

Describe the limitations

When describing limitations authors should identify the limitation type to clearly introduce the limitation and specify the origin of the limitation. This helps to ensure readers are able to interpret and generalize findings appropriately. Here we outline various limitation types that can occur at different stages of the research process.

Study design

Some study limitations originate from conscious choices made by the researcher (also known as delimitations) to narrow the scope of the study [ 1 , 8 , 18 ]. For example, the researcher may have designed the study for a particular age group, sex, race, ethnicity, geographically defined region, or some other attribute that would limit to whom the findings can be generalized. Such delimitations involve conscious exclusionary and inclusionary decisions made during the development of the study plan, which may represent a systematic bias intentionally introduced into the study design or instrument by the researcher [ 8 ]. The clear description and delineation of delimitations and limitations will assist editors and reviewers in understanding any methodological issues.

Data collection

Study limitations can also be introduced during data collection. An unintentional consequence of human subjects research is the potential of the researcher to influence how participants respond to their questions. Even when appropriate methods for sampling have been employed, some studies remain limited by the use of data collected only from participants who decided to enrol in the study (self-selection bias) [ 11 , 19 ]. In some cases, participants may provide biased input by responding to questions they believe are favourable to the researcher rather than their authentic response (social desirability bias) [ 20 – 22 ]. Participants may influence the data collected by changing their behaviour when they are knowingly being observed (Hawthorne effect) [ 23 ]. Researchers—in their role as an observer—may also bias the data they collect by allowing a first impression of the participant to be influenced by a single characteristic or impression of another characteristic either unfavourably (horns effect) or favourably (halo effort) [ 24 ].

Data analysis

Study limitations may arise as a consequence of the type of statistical analysis performed. Some studies may not follow the basic tenets of inferential statistical analyses when they use convenience sampling (i.e. non-probability sampling) rather than employing probability sampling from a target population [ 19 ]. Another limitation that can arise during statistical analyses occurs when studies employ unplanned post-hoc data analyses that were not specified before the initial analysis [ 25 ]. Unplanned post-hoc analysis may lead to statistical relationships that suggest associations but are no more than coincidental findings [ 23 ]. Therefore, when unplanned post-hoc analyses are conducted, this should be clearly stated to allow the reader to make proper interpretation and conclusions—especially when only a subset of the original sample is investigated [ 23 ].

Study results

The limitations of any research study will be rooted in the validity of its results—specifically threats to internal or external validity [ 8 ]. Internal validity refers to reliability or accuracy of the study results [ 26 ], while external validity pertains to the generalizability of results from the study’s sample to the larger, target population [ 8 ].

Examples of threats to internal validity include: effects of events external to the study (history), changes in participants due to time instead of the studied effect (maturation), systematic reduction in participants related to a feature of the study (attrition), changes in participant responses due to repeatedly measuring participants (testing effect), modifications to the instrument (instrumentality) and selecting participants based on extreme scores that will regress towards the mean in repeat tests (regression to the mean) [ 27 ].

Threats to external validity include factors that might inhibit generalizability of results from the study’s sample to the larger, target population [ 8 , 27 ]. External validity is challenged when results from a study cannot be generalized to its larger population or to similar populations in terms of the context, setting, participants and time [ 18 ]. Therefore, limitations should be made transparent in the results to inform research consumers of any known or potentially hidden biases that may have affected the study and prevent generalization beyond the study parameters.

Explain the implication(s) of each limitation

Authors should include the potential impact of the limitations (e.g., likelihood, magnitude) [ 13 ] as well as address specific validity implications of the results and subsequent conclusions [ 16 , 28 ]. For example, self-reported data may lead to inaccuracies (e.g. due to social desirability bias) which threatens internal validity [ 19 ]. Even a researcher’s inappropriate attribution to a characteristic or outcome (e.g., stereotyping) can overemphasize (either positively or negatively) unrelated characteristics or outcomes (halo or horns effect) and impact the internal validity [ 24 ]. Participants’ awareness that they are part of a research study can also influence outcomes (Hawthorne effect) and limit external validity of findings [ 23 ]. External validity may also be threatened should the respondents’ propensity for participation be correlated with the substantive topic of study, as data will be biased and not represent the population of interest (self-selection bias) [ 29 ]. Having this explanation helps readers interpret the results and generalize the applicability of the results for their own setting.

Provide potential alternative approaches and explanations

Often, researchers use other studies’ limitations as the first step in formulating new research questions and shaping the next phase of research. Therefore, it is important for readers to understand why potential alternative approaches (e.g. approaches taken by others exploring similar topics) were not taken. In addition to alternative approaches, authors can also present alternative explanations for their own study’s findings [ 13 ]. This information is valuable coming from the researcher because of the direct, relevant experience and insight gained as they conducted the study. The presentation of alternative approaches represents a major contribution to the scholarly community.

Describe steps taken to minimize each limitation

No research design is perfect and free from explicit and implicit biases; however various methods can be employed to minimize the impact of study limitations. Some suggested steps to mitigate or minimize the limitations mentioned above include using neutral questions, randomized response technique, force choice items, or self-administered questionnaires to reduce respondents’ discomfort when answering sensitive questions (social desirability bias) [ 21 ]; using unobtrusive data collection measures (e.g., use of secondary data) that do not require the researcher to be present (Hawthorne effect) [ 11 , 30 ]; using standardized rubrics and objective assessment forms with clearly defined scoring instructions to minimize researcher bias, or making rater adjustments to assessment scores to account for rater tendencies (halo or horns effect) [ 24 ]; or using existing data or control groups (self-selection bias) [ 11 , 30 ]. When appropriate, researchers should provide sufficient evidence that demonstrates the steps taken to mitigate limitations as part of their study design [ 13 ].

In conclusion, authors may be limiting the impact of their research by neglecting or providing abbreviated and generic limitations. We present several examples of limitations to consider; however, this should not be considered an exhaustive list nor should these examples be added to the growing list of generic and overused limitations. Instead, careful thought should go into presenting limitations after research has concluded and the major findings have been described. Limitations help focus the reader on key findings, therefore it is important to only address the most salient limitations of the study [ 17 , 28 ] related to the specific research problem, not general limitations of most studies [ 1 ]. It is important not to minimize the limitations of study design or results. Rather, results, including their limitations, must help readers draw connections between current research and the extant literature.

The quality and rigor of our research is largely defined by our limitations [ 31 ]. In fact, one of the top reasons reviewers report recommending acceptance of medical education research manuscripts involves limitations—specifically how the study’s interpretation accounts for its limitations [ 32 ]. Therefore, it is not only best for authors to acknowledge their study’s limitations rather than to have them identified by an editor or reviewer, but proper framing and presentation of limitations can actually increase the likelihood of acceptance. Perhaps, these issues could be ameliorated if academic and research organizations adopted policies and/or expectations to guide authors in proper description of limitations.

Research Limitations 101 📖

A Plain-Language Explainer (With Practical Examples)

By: Derek Jansen (MBA) | Expert Reviewer: Dr. Eunice Rautenbach | May 2024

Research limitations are one of those things that students tend to avoid digging into, and understandably so. No one likes to critique their own study and point out weaknesses. Nevertheless, being able to understand the limitations of your study – and, just as importantly, the implications thereof – a is a critically important skill.

In this post, we’ll unpack some of the most common research limitations you’re likely to encounter, so that you can approach your project with confidence.

Overview: Research Limitations 101

• What are research limitations ?
• Access – based limitations
• Temporal & financial limitations
• Sample & sampling limitations
• Design limitations
• Researcher limitations
• Key takeaways

What (exactly) are “research limitations”?

At the simplest level, research limitations (also referred to as “the limitations of the study”) are the constraints and challenges that will invariably influence your ability to conduct your study and draw reliable conclusions .

Research limitations are inevitable. Absolutely no study is perfect and limitations are an inherent part of any research design. These limitations can stem from a variety of sources , including access to data, methodological choices, and the more mundane constraints of budget and time. So, there’s no use trying to escape them – what matters is that you can recognise them.

Acknowledging and understanding these limitations is crucial, not just for the integrity of your research, but also for your development as a scholar. That probably sounds a bit rich, but realistically, having a strong understanding of the limitations of any given study helps you handle the inevitable obstacles professionally and transparently, which in turn builds trust with your audience and academic peers.

Simply put, recognising and discussing the limitations of your study demonstrates that you know what you’re doing , and that you’ve considered the results of your project within the context of these limitations. In other words, discussing the limitations is a sign of credibility and strength – not weakness. Contrary to the common misconception, highlighting your limitations (or rather, your study’s limitations) will earn you (rather than cost you) marks.

So, with that foundation laid, let’s have a look at some of the most common research limitations you’re likely to encounter – and how to go about managing them as effectively as possible.

Need a helping hand?

Limitation #1: Access To Information

One of the first hurdles you might encounter is limited access to necessary information. For example, you may have trouble getting access to specific literature or niche data sets. This situation can manifest due to several reasons, including paywalls, copyright and licensing issues or language barriers.

To minimise situations like these, it’s useful to try to leverage your university’s resource pool to the greatest extent possible. In practical terms, this means engaging with your university’s librarian and/or potentially utilising interlibrary loans to get access to restricted resources. If this sounds foreign to you, have a chat with your librarian 🙃

In emerging fields or highly specific study areas, you might find that there’s very little existing research (i.e., literature) on your topic. This scenario, while challenging, also offers a unique opportunity to contribute significantly to your field , as it indicates that there’s a significant research gap .

All of that said, be sure to conduct an exhaustive search using a variety of keywords and Boolean operators before assuming that there’s a lack of literature. Also, remember to snowball your literature base . In other words, scan the reference lists of the handful of papers that are directly relevant and then scan those references for more sources. You can also consider using tools like Litmaps and Connected Papers (see video below).

Limitation #2: Time & Money

Almost every researcher will face time and budget constraints at some point. Naturally, these limitations can affect the depth and breadth of your research – but they don’t need to be a death sentence.

Effective planning is crucial to managing both the temporal and financial aspects of your study. In practical terms, utilising tools like Gantt charts can help you visualise and plan your research timeline realistically, thereby reducing the risk of any nasty surprises. Always take a conservative stance when it comes to timelines, especially if you’re new to academic research. As a rule of thumb, things will generally take twice as long as you expect – so, prepare for the worst-case scenario.

If budget is a concern, you might want to consider exploring small research grants or adjusting the scope of your study so that it fits within a realistic budget. Trimming back might sound unattractive, but keep in mind that a smaller, well-planned study can often be more impactful than a larger, poorly planned project.

If you find yourself in a position where you’ve already run out of cash, don’t panic. There’s usually a pivot opportunity hidden somewhere within your project. Engage with your research advisor or faculty to explore potential solutions – don’t make any major changes without first consulting your institution.

Limitation #3: Sample Size & Composition

As we’ve discussed before , the size and representativeness of your sample are crucial , especially in quantitative research where the robustness of your conclusions often depends on these factors. All too often though, students run into issues achieving a sufficient sample size and composition.

To ensure adequacy in terms of your sample size, it’s important to plan for potential dropouts by oversampling from the outset . In other words, if you aim for a final sample size of 100 participants, aim to recruit 120-140 to account for unexpected challenges. If you still find yourself short on participants, consider whether you could complement your dataset with secondary data or data from an adjacent sample – for example, participants from another city or country. That said, be sure to engage with your research advisor before making any changes to your approach.

A related issue that you may run into is sample composition. In other words, you may have trouble securing a random sample that’s representative of your population of interest. In cases like this, you might again want to look at ways to complement your dataset with other sources, but if that’s not possible, it’s not the end of the world. As with all limitations, you’ll just need to recognise this limitation in your final write-up and be sure to interpret your results accordingly. In other words, don’t claim generalisability of your results if your sample isn’t random.

Limitation #4: Methodological Limitations

As we alluded earlier, every methodological choice comes with its own set of limitations . For example, you can’t claim causality if you’re using a descriptive or correlational research design. Similarly, as we saw in the previous example, you can’t claim generalisability if you’re using a non-random sampling approach.

Making good methodological choices is all about understanding (and accepting) the inherent trade-offs . In the vast majority of cases, you won’t be able to adopt the “perfect” methodology – and that’s okay. What’s important is that you select a methodology that aligns with your research aims and research questions , as well as the practical constraints at play (e.g., time, money, equipment access, etc.). Just as importantly, you must recognise and articulate the limitations of your chosen methods, and justify why they were the most suitable, given your specific context.

Limitation #5: Researcher (In)experience 

A discussion about research limitations would not be complete without mentioning the researcher (that’s you!). Whether we like to admit it or not, researcher inexperience and personal biases can subtly (and sometimes not so subtly) influence the interpretation and presentation of data within a study. This is especially true when it comes to dissertations and theses , as these are most commonly undertaken by first-time (or relatively fresh) researchers.

When it comes to dealing with this specific limitation, it’s important to remember the adage “ We don’t know what we don’t know ”. In other words, recognise and embrace your (relative) ignorance and subjectivity – and interpret your study’s results within that context . Simply put, don’t be overly confident in drawing conclusions from your study – especially when they contradict existing literature.

Cultivating a culture of reflexivity within your research practices can help reduce subjectivity and keep you a bit more “rooted” in the data. In practical terms, this simply means making an effort to become aware of how your perspectives and experiences may have shaped the research process and outcomes.

As with any new endeavour in life, it’s useful to garner as many outsider perspectives as possible. Of course, your university-assigned research advisor will play a large role in this respect, but it’s also a good idea to seek out feedback and critique from other academics. To this end, you might consider approaching other faculty at your institution, joining an online group, or even working with a private coach .

Key Takeaways

Understanding and effectively navigating research limitations is key to conducting credible and reliable academic work. By acknowledging and addressing these limitations upfront, you not only enhance the integrity of your research, but also demonstrate your academic maturity and professionalism.

Whether you’re working on a dissertation, thesis or any other type of formal academic research, remember the five most common research limitations and interpret your data while keeping them in mind.

• Access to Information (literature and data)
• Time and money
• Sample size and composition
• Research design and methodology
• Researcher (in)experience and bias

If you need a hand identifying and mitigating the limitations within your study, check out our 1:1 private coaching service .

Psst… there’s more!

This post is an extract from our bestselling short course, Methodology Bootcamp . If you want to work smart, you don't want to miss this .

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Journal Club
• Published: 19 January 2024

Meta-science

Psychology remains marginally valid

• Kerem Oktar   ORCID: orcid.org/0000-0002-0118-5065 1  

Nature Reviews Psychology volume  3 ,  page 144 ( 2024 ) Cite this article

494 Accesses

1 Citations

20 Altmetric

Metrics details

• Research management
• Scientific community

In his 1990 paper, Paul Meehl argues that much psychology research is doomed to fail. He identifies ten reasons why accumulating evidence fails to provide decisive verdicts on psychological theories. To address these issues, he suggests many reforms — from mandating power analyses to abandoning entire lines of research.

Although Meehl’s critique targeted particular subfields in 1990, his ideas explain why psychology struggles to produce predictive, generalizable insights to this day. The core problem is that behaviour emerges from an extremely dense causal web. This observation has profound implications: theories that hypothesize relations across a few nodes cannot predict complex real-world behaviour; experiments that use a few stimuli cannot validate comprehensive theories; and analyses that compare effects to null baselines are uninformative. Moreover, meta-analyses cannot recover the web of cognition by collating such research findings.

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 12 digital issues and online access to articles

55,14 € per year

only 4,60 € per issue

Buy this article

• Purchase on SpringerLink
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Original article

Meehl, P. E. Why summaries of research on psychological theories are often uninterpretable. Psychol. Rep. 66 , 195–244 (1990)

Article   Google Scholar  

Related articles

Yarkoni, T. The generalizability crisis. Behav. Brain Sci. 45 , e1 (2022)

Almaatouq, A. et al. Beyond playing 20 questions with nature: Integrative experiment design in the social and behavioral sciences. Behav. Brain Sci. https://doi.org/10.1017/S0140525X22002874 (2022)

Article   PubMed   Google Scholar  

Download references

Author information

Authors and affiliations.

Department of Psychology, Princeton University, Princeton, NJ, USA

Kerem Oktar

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Kerem Oktar .

Ethics declarations

Competing interests.

The author declares no competing interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Oktar, K. Psychology remains marginally valid. Nat Rev Psychol 3 , 144 (2024). https://doi.org/10.1038/s44159-024-00281-5

Download citation

Published : 19 January 2024

Issue Date : March 2024

DOI : https://doi.org/10.1038/s44159-024-00281-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Evaluating Psychology Research

Many famous psychological studies cannot be reproduced..

Updated August 16, 2024

Studies in psychology often find different results. Even in fields like medicine, where one might think there to be a direct relationship between the intervention being tested and its effects, results can vary.

For example, a study found that drinking one glass of orange juice a day could increase a person’s risk of getting Type 2 diabetes by 18 percent. Researchers at the University of California, Davis, however, found that drinking 100% juice reduced the risk for several chronic diseases, including cancer.

But many think the situation is worse in psychology.

A recent New York Times article mentions some famous psychology studies of human behavior that cannot be reproduced, including the famous Stanford Prison Experiment that showed how people role-playing as guards quickly acted cruelly to mock prisoners, as well as the famed "marshmallow test" that showed that young children who could delay gratification demonstrated greater educational achievement years later than those who could not.

Why do research results vary and fail to replicate?

The relationship between an intervention and its effects may depend on many factors. And differences in context or implementation can have a large impact on a study's results. There are other reasons that studies might report different effects: Chance errors could affect a study’s results. Researchers may also consciously or inadvertently sway their results.

All these sources of variability have led to fears of a “ replication crisis ” in psychology and other social sciences. Given this concern, how should we evaluate psychology and social science research?

The first rule of thumb is to not rely solely on any one study. If possible, review meta-analyses or systematic reviews that combine the results from multiple studies. Meta-analyses can provide more credible evidence. Meta-analyses can suggest reasons why results differ.

A meta-analysis is a statistical analysis that combines the results of multiple research studies. The basic principle behind meta-analysis is that there is a common truth behind all conceptually similar research studies, but each individual study has been measured with a certain error within individual studies. The aim is to use statistics to get a pooled estimate, closest to the unknown common truth. A meta-analysis, then, yields a weighted average from the results of all the individual studies.

Aside from providing an estimate of the unknown common truth, meta-analysis also can contrast results from different studies and identify patterns among study results. It can also identify sources of disagreement among these results. And it can identify other interesting relationships that pop out in the context of multiple studies. A key benefit of the meta-analytic approach is the aggregation of information leading to a higher statistical power and more robust point estimate than is possible from the measure derived from many individual studies.

Still, there are some limitations of the meta-analytic approach to consider, too. The researcher must make choices about what studies to include that can affect the results of the meta-analysis (e.g. only published studies). The researcher must decide how to search for the studies. And the researcher must decide how to deal with incomplete data, analyze the data, and account for publication bias .

Sometimes, however, we want to evaluate a single, individual psychology study. So how should we go about that? When considering how much weight to give to a study and its results, focus on sample size. Studies are more likely to fail to replicate if they used small samples. The most positive and negative results are often those with the smallest samples or widest confidence intervals. Smaller studies are more likely to fail to replicate in part due to chance, but effects may also be smaller as sample size increases, for numerous reasons. If the study was testing an intervention, there may be capacity constraints that prevent high-quality implementation at scale. Smaller studies also often target the exact desirable sample that would yield the biggest effects.

There is a line of reasoning to this: If, for example, you have an expensive diversity educational program that you can only use with a limited amount of students, you might only have one class and have students who could benefit from it the most. That means the effect would likely be smaller if you implemented the diversity education in a larger group. So more generally, it can be helpful to think about what things might be different if the educational program was scaled up. For example, small diversity educational programs are unlikely to affect the broader institution, community, or society. But if scaled up, the institutional, community, or societal culture might change in response.

Similarly, consider specific features of the sample, context, and implementation. How did the researchers come to study the diversity educational program including the institution and the students they did? Would you expect this sample to have performed better or worse than the sample you are interested in? For example, if I was interested in testing the outcome of the teaching method I use in my web conference course the Psychology of Diversity at Harvard Summer School ( https://scholar.harvard.edu/weissmark/classes) the setting and format (e.g. Harvard Summer School, web conference, Harvard campus) could have affected the results, too. Was there anything unique about the setting and format that could have made the results larger?

If the study was evaluating a diversity educational course, how that course was implemented is important, too. For example, suppose you hear that a web conference course on diversity can improve students’ feelings of belonging and inclusion. If you were considering implementing a similar course, you would probably want to know the format of the web conference course and the course content and the training of the teaching staff in order to gauge whether you might have different results.

You may also have more confidence in the results of a study if there is some clear mechanism that explains the findings and is constant across settings. Some results in behavioral economics , for instance, suggest that certain rules of human behavior are hardwired. But these mechanisms can be difficult to uncover. And many experiments in behavioral economics that initially seemed to reflect a hardwired rule have failed to replicate, such as finding that happiness increases patience and learning.

But, if there is a convincing reason that we might expect to see the results that a study has found, or if there is a strong theoretical reason that we might expect a specific result to generalize, that should lead us to trust the results from a single study a little more. But we should take care to examine why we think there is a convincing reason. Finally, if it appears too good to be true, it probably is.

In conclusion, all psychology research is subject to error, and hence the results may vary and fail to replicate. It is far better to be aware of this than to be uninformed of the errors potentially concealed in the research and in the mind of the researcher. The scientific method was developed to draw on empirical reasoning to help us resolve cases in which studies vary or fail to replicate. Applying the scientific method to the study of human behavior and psychology has not simplified human behavior; instead, it has suggested how complex human behavior is (Weissmark, M. S. , February 18, 2020. "Advice to Students: Learn to Think Scientifically". The Harvard Gazette https://news.harvard.edu/gazette/story/2020/02/embrace-logic-to-improve-both-education-and-society/ ).

Copyright © 2020; Mona Sue Weissmark All Rights Reserved; Do not copy or distribute this article without permission.

Weissmark,M. (2020). The Science of Diversity . Oxford University Press.

https://www.amazon.com/Science-Diversity-Mona-Sue-Weissmark/dp/0190686340

Weissmark,M. (2020). The Science of Diversity . Oxford University Press, USA. https://www.amazon.com/Science-Diversity-Mona-Sue-Weissmark/dp/0190686340

Weissmark, M. (2004). J ustice Matters:Legacies of the Holocaust and World War II . Oxford University Press, USA. https://www.amazon.com/Justice-Matters-Legacies-Holocaust-World/dp/0195157575

Weissmark, M. & Giacomo, D. (1998). Doing Psychotherapy Effectively. U niversity of Chicago Press, USA.

https://www.amazon.com/Doing-Psychotherapy-Effectively-Mona-Weissmark/dp/0226891674

Mona Sue Weissmark, Ph.D. , is a psychology professor and founder of the Program Initiative for Global Mental Health Studies at Northwestern University.

• Find a Therapist
• Find a Treatment Center
• Find a Psychiatrist
• Find a Support Group
• Find Online Therapy
• United States
• Brooklyn, NY
• Chicago, IL
• Houston, TX
• Los Angeles, CA
• New York, NY
• Portland, OR
• San Diego, CA
• San Francisco, CA
• Seattle, WA
• Washington, DC
• Asperger's
• Bipolar Disorder
• Chronic Pain
• Eating Disorders
• Passive Aggression
• Personality
• Goal Setting
• Positive Psychology
• Stopping Smoking
• Low Sexual Desire
• Relationships
• Child Development
• Self Tests NEW
• Therapy Center
• Diagnosis Dictionary
• Types of Therapy

It’s increasingly common for someone to be diagnosed with a condition such as ADHD or autism as an adult. A diagnosis often brings relief, but it can also come with as many questions as answers.

• Emotional Intelligence
• Gaslighting
• Affective Forecasting
• Neuroscience

Psychology: Research and Review

• Open access
• Published: 25 January 2017

Scale development: ten main limitations and recommendations to improve future research practices

• Fabiane F. R. Morgado 1 ,
• Juliana F. F. Meireles 2 ,
• Clara M. Neves 2 ,
• Ana C. S. Amaral 3 &
• Maria E. C. Ferreira 2  

Psicologia: Reflexão e Crítica volume  30 , Article number:  3 ( 2018 ) Cite this article

104k Accesses

364 Citations

2 Altmetric

Metrics details

An Erratum to this article was published on 03 March 2017

The scale development process is critical to building knowledge in human and social sciences. The present paper aimed (a) to provide a systematic review of the published literature regarding current practices of the scale development process, (b) to assess the main limitations reported by the authors in these processes, and (c) to provide a set of recommendations for best practices in future scale development research. Papers were selected in September 2015, with the search terms “scale development” and “limitations” from three databases: Scopus, PsycINFO, and Web of Science, with no time restriction. We evaluated 105 studies published between 1976 and 2015. The analysis considered the three basic steps in scale development: item generation, theoretical analysis, and psychometric analysis. The study identified ten main types of limitation in these practices reported in the literature: sample characteristic limitations, methodological limitations, psychometric limitations, qualitative research limitations, missing data, social desirability bias, item limitations, brevity of the scale, difficulty controlling all variables, and lack of manual instructions. Considering these results, various studies analyzed in this review clearly identified methodological weaknesses in the scale development process (e.g., smaller sample sizes in psychometric analysis), but only a few researchers recognized and recorded these limitations. We hope that a systematic knowledge of the difficulties usually reported in scale development will help future researchers to recognize their own limitations and especially to make the most appropriate choices among different conceptions and methodological strategies.

Introduction

In recent years, numerous measurement scales have been developed to assess attitudes, techniques, and interventions in a variety of scientific applications (Meneses et al. 2014 ). Measurement is a fundamental activity of science, since it enables researchers to acquire knowledge about people, objects, events, and processes. Measurement scales are useful tools to attribute scores in some numerical dimension to phenomena that cannot be measured directly. They consist of sets of items revealing levels of theoretical variables otherwise unobservable by direct means (DeVellis 2003 ).

A variety of authors (Clark and Watson 1995 ; DeVellis 2003 ; Nunnally 1967 ; Pasquali 2010 ) have agreed that the scale development process involves complex and systematic procedures that require theoretical and methodological rigor. According to these authors, the scale development process can be carried out in three basic steps.

In the first step, commonly referred as “item generation,” the researcher provides theoretical support for the initial item pool (Hutz et al. 2015 ). Methods for the initial item generation can be classified as deductive, inductive, or a combination of the two. Deductive methods involve item generation based on an extensive literature review and pre-existing scales (Hinkin 1995 ). On the other hand, inductive methods base item development on qualitative information regarding a construct obtained from opinions gathered from the target population—e.g., focus groups, interviews, expert panels, and qualitative exploratory research methodologies (Kapuscinski and Masters 2010 ). The researcher is also concerned with a variety of parameters that regulate the setting of each item and of the scale as a whole. For example, suitable scale instructions, an appropriate number of items, adequate display format, appropriate item redaction (all items should be simple, clear, specific, ensure the variability of response, remain unbiased, etc.), among other parameters (DeVellis 2003 ; Pasquali 2010 ).

In the second step, usually referred to as the “theoretical analysis,” the researcher assesses the content validity of the new scale, ensuring that the initial item pool reflects the desired construct (Arias et al. 2014 ). A content validity assessment is required, since inferences are made based on the final scale items. The item content must be deemed valid to instill confidence in all consequent inferences. In order to ensure the content validity, the researcher seeks other opinions about the operationalized items. The opinions can be those of expert judges (experts in the development scales or experts in the target construct) or target population judges (potential users of the scale), enabling the researcher to ensure that the hypothesis elaborated in the research appropriately represents the construct of interest (Nunnally 1967 ).

In the last step, psychometric analysis, the researcher should assess whether the new scale has construct validity and reliability. Construct validity is most directly related to the question of what the instrument is in fact measuring—what construct, trait, or concept underlies an individual’s performance or score on a measure (Churchill 1979 ). This refers to the degree to which inferences can be legitimately made from the observed scores to the theoretical constructs about which these observations are supposed to contain information (Podsakoff et al. 2013 ). Construct validity can be assessed with the use of exploratory factor analysis (EFA), confirmatory factor analysis (CFA), or with convergent, discriminant, predictive/nomological, criterion, internal, and external validity. In turn, reliability is a measure of score consistency, usually measured by use of internal consistency, test-retest reliability, split-half, item-total correlation/inter-item reliability, and inter-observer reliability (DeVellis 2003 ). To ensure construct validity and reliability, the data should be collected in a large and appropriately representative sample of the target population. It is a common rule of thumb that there should be at least 10 participants for each item of the scale, making an ideal of 15:1 or 20:1 (Clark and Watson 1995 ; DeVellis 2003 ; Hair Junior et al. 2009 ).

Although the literature on theoretical and methodological care in scale development is extensive, many limitations have been identified in the process. These include failure to adequately define the construct domain, failure to correctly specify the measurement model, underutilization of some techniques that are helpful in establishing construct validity (MacKenzie et al. 2011 ), relatively weak psychometric properties, applicability to only a single form of treatment or manual, extensive time required to fill out the questionnaire (Hilsenroth et al. 2005 ), inappropriate item redaction, too few items and participants in the construction and analysis, an imbalance between items that assess positive beliefs and those that assess negative beliefs (Prados 2007 ), social desirability bias (King and Bruner 2000 ), among others.

These limitations in the scale development process weaken the obtained psychometric results, limiting the future applicability of the new scale and hindering its generalizability. In this sense, knowledge of the most often reported limitations is fundamental in providing essential information to help develop best practices for future research in this area. The purpose of this article is threefold: (a) to provide a systematic review of the published literature regarding some current practices of the scale development process, (b) to assess the main limitations reported by the authors in this process, and (c) to provide a set of recommendations for best practices in future scale development research.

This systematic review identified and selected papers from three databases: Scopus, PsycINFO, and Web of Science. There was no time restriction in the literature search, which was completed in September 1, 2015. The following search term was used: “scale development.” In the set of databases analyzed, the search was done inclusively in “Any Field” (PsycINFO), in “Article Title, Abstract, Keywords” (Scopus), or in any “Topic” (Web of Science). In addition, we used an advanced search to filter the articles in (search within results), with the search term “limitations” identified in “Any Field” in all databases. Both terms were used in English only. Four reviewers evaluated the papers in an independent and blinded way. Any disagreements on eligibility of a particular study were resolved through consensus among reviewers.

Figure  1 shows a flowchart summarizing the strategy adopted for identification and selection of studies. We used only one inclusion criteria for the evaluation of the studies: (a) articles that aim to develop and validate self-administered measurement scales for humans. We excluded (a) unavailable full-text papers in the analyzed databases, (b) papers in languages other than English, Portuguese, or Spanish, (c) articles which were not clearly aimed at the development of a new scale (i.e., we excluded articles investigating only the reliability, validity, or revisions of existing scales and studies that describe the validation of instruments for other languages), (d) papers with unvalidated scales, and (e) articles that did not declare the limitations of the study.

Flowchart showing summary of the systematic process of identifying and selecting article

In all, this systematic review evaluated 105 studies published between 1976 and 2015. Most (88.5%) was published between 2005 and 2015, and only two studies date from the last century. We analyzed two major issues: (a) current practices of the scale development process —considering the three steps usually reported in the literature (step 1—item generation, step 2—theoretical analysis, step 3—psychometric analysis), the number of participants in step 3, the number of items in the beginning scale, and the number of items in the final scale; (b) m ain limitations reported by the authors in the scale development process —considering the limitations observed and recorded by the authors during the scale development process. The description of these results can be found in Table  1 .

Current practices of the scale development process

Step 1—item generation.

In the first step, 35.2% ( n  = 37) of the studies reported using exclusively deductive methods to write items, 7.6% ( n  = 8) used only inductive methods, and 56.2% ( n  = 59) combined deductive and inductive strategies. The majority of the studies used a literature review (84.7%, n  = 89) as the deductive method in item generation. In inductive methods, 26.6% of studies ( n  = 28) chose to conduct an interview.

Step 2—theoretical analysis

In order to theoretically refine the items, several studies used opinions of experts (74.2%, n  = 78), whereas others used target population opinions (43.8%, n  = 46). In addition, 63.8% ( n  = 67) of the studies used only one of these approaches (expert or population judges).

Step 3—psychometric analysis

The most common analyses that have been used to assess construct validity are EFA (88.6%, n  = 93), CFA (72.3%, n  = 76), convergent validity (72.3%, n  = 76), and discriminant validity (56.2%, n  = 59). Most studies opted to combine EFA and CFA (65.7%, n  = 69). Only 4.7% ( n  = 5) failed to use factor analysis in their research. In relation to study reliability, internal consistency checks were used by all studies and test-retest reliability was the second most commonly used technique (22.8%, n  = 24).

Sample size in step 3 and number of items

Interestingly, 50.4% ( n  = 53) of the studies used sample sizes smaller than the rule of thumb, which is a minimum of 10 participants for each item in the scale. Regarding number of items, the majority of the studies (49.6%, n  = 52) lost more than 50% of the initial item pool during the validation process.

Table  2 summarizes and provides more details on our findings regarding the current practices in the scale development.

Main limitations reported in the scale development process

As result of this systematic review, we found ten main limitations commonly referenced in the scale development process: (1) sample characteristic limitations—cited by 81% of the studies, (2) methodological limitations—33.2%, (3) psychometric limitations—30.4%, (4) qualitative research limitations—5.6%, (5) missing data—2.8%, (6) social desirability bias—1.9%, (7) item limitations—1.9%, (8) brevity of the scale—1.9%, (9) difficulty controlling all variables—0.9%, and (10) lack of manual instructions—0.9%. Table  3 summarizes these findings.

This systematic review was primarily directed at identifying the published literature regarding current practices of the scale development. The results show a variety of practices that have been used to generate and assess items, both theoretically and psychometrically. We evaluated these current practices, considering three distinct steps (item generation, theoretical analysis, and psychometric analysis). We also considered the relationship between sample size and number of items, since this is considered an important methodological aspect to be evaluated during the scale development process. The results are discussed together with recommendations for best practices in future scale development research.

Current practices of the scale development process—findings and research implications

Regarding step 1, item generation, our results show that, although several studies used exclusively deductive methods (e.g., Henderson-King and Henderson-King 2005 ; Kim et al. 2011 ), the majority (e.g., Bakar and Mustaffa 2013 ; Uzunboylu and Ozdamli 2011 ) combined deductive and inductive methods, a combination consistent with the recommended strategy for the creation of new measures (DeVellis 2003 ). These findings, however, differ from previous critical reviews of scale development practices, which found that most of the reported studies used exclusively deductive methods (Hinkin 1995 ; Kapuscinski and Masters 2010 ; Ladhari 2010 ). This is particularly important since the quality of generated items depends on the way that the construct is defined. Failing to adequately define the conceptual domain of a construct causes several problems related to poor construct definition, leading to, for example, (a) confusion about what the construct does and does not refer to, including the similarities and differences between it and other constructs that already exist in the field, (b) indicators that may either be deficient or contaminated, and (c) invalid conclusions about relationships with other constructs (MacKenzie et al. 2011 ). Considering that item generation may be the most important part of the scale development process, future measures should be developed using the appropriate definition of the conceptual domain based on the combination of both deductive and inductive approaches.

Our results suggest that literature review was the most widely used deductive method (e.g., Bolton and Lane 2012 ; Henderson-King and Henderson-King 2005 ). This is consistent with the views of several other researchers who have systematically reviewed scales (Bastos et al. 2010 ; Ladhari 2010 ; Sveinbjornsdottir and Thorsteinsson 2008 ). Nevertheless, this finding differs from another study (Kapuscinski and Masters 2010 ) that found that the most common deductive strategies were reading works by spiritual leaders, theory written by psychologists, and discussion among authors. Literature review should be considered central for the enumeration of the constructs. It also serves to clarify the nature and variety of the target construct content. In addition, literature reviews help to identify existing measures that can be used as references to create new scales (Clark and Watson 1995 ; DeVellis 2003 ). In this sense, future research should consider the literature review as the initial and necessary deductive step foundational to building a new scale.

This review also highlights the fact that interviews and focus groups were the most widely used inductive methods (e.g., Lin and Hsieh 2011 ; Sharma 2010 ). Similar results were found in the systematic review by Kapuscinski and Masters ( 2010 ), Sveinbjornsdottir and Thorsteinsson ( 2008 ), and Ladhari ( 2010 ). These findings have particular relevance to future researchers, since they emphasize the importance of using methodological strategies that consider the opinions of the target population. Despite the fact that a panel of experts contributes widely to increasing the researchers’ confidence in the content validity of the new scale, it is important to also consider the most original and genuine information about the construct of interest, which can be best obtained through reports obtained from interviews and focus groups with the target population.

Related to step 2, theoretical analysis, the results of this review indicate that expert judges have been the most widely utilized tool for analyzing content validity (e.g., Uzunboylu and Ozdamli 2011 ; Zheng et al. 2010 ). Previous studies have also found expert opinion to be the most common qualitative method for the elimination of unsuitable items (Kapuscinski and Masters 2010 ; Ladhari 2010 ). In the literature review conducted by Hardesty and Bearden ( 2004 ), the authors highlighted the importance of these experts to carefully analyze the initial item pool. They suggested that any research using new, changed, or previously unexamined scale items, should at a minimum be judged by a panel of experts. However, the authors also point out the apparent lack of consistency in the literature in terms of how researchers use the opinions of expert judges in aiding the decision of whether or not to retain items for a scale. Given this inconsistency, the authors developed guidelines regarding the application of different decision rules to use for item retention. For example, the “sumscore decision rule,” defined as the total score for an item across all judges, is considered by the authors to be the most effective in predicting whether an item should be included in a scale and appears, therefore, to be a reasonable rule for researchers to employ.

Future research in developing scales should be concerned, not only with opinions from experts but also with the opinions of the target population. The results of this review show that only a minority of studies considered the review of the scales’ items by members of the target population (e.g., Uzunboylu and Ozdamli 2011 ; Zheng et al. 2010 ). In addition, a smaller minority combined the two approaches in the assessment of item content (e.g., Mahudin et al. 2012 ; Morgado et al. 2014 ). The limited use of target population opinions is a problem. A previous study of systematic scale development reviews found that the opinion of these people is the basis for content validity (Bastos et al. 2010 ). As highlighted by Clark and Watson ( 1995 ) and Malhotra ( 2004 ), it is essential for the new scale to undergo prior review by members of the target population. Pre-test or pilot study procedures make it possible to determine respondents’ opinions of, and reactions to, each item on the scale, enabling researchers to identify and eliminate potential problems in the scale before it is applied at large.

Another problem noted in this systematic review was that some studies failed to clearly report how they performed the theoretical analysis of the items (e.g., Glynn et al. 2015 ; Gottlieb et al. 2014 ). We hypothesized that the authors either did not perform this analysis or found it unimportant to record. Future research should consider this analysis, as well as all subsequent analyses, necessary and relevant for reporting.

Almost all studies (95.3%) reported using at least one type of factor analysis—EFA or CFA—in step 3, psychometric analysis (e.g., Sewitch et al. 2003 ; Tanimura et al. 2011 ). Clark and Watson ( 1995 ) consider that “unfortunately, many test developers are hesitant to use factor analysis, either because it requires a relatively large number of respondents or because it involves several perplexing decisions” (p. 17). They emphasized the importance of the researcher’s need to understand and apply this analysis, “it is important that test developers either learn about the technique or consult with a psychometrician during the scale development process” (Clark and Watson 1995 , p. 17). This question seems to have been almost overcome in recent studies, since the vast majority of the analyzed studies used the factor analysis method.

Among the studies than used factor analysis, the majority chose to use EFA (e.g., Bakar and Mustaffa 2013 ; Turker 2009 ). Similar to our findings, Bastos et al. ( 2010 ) and Ladhari ( 2010 ) found EFA to be the more commonly utilized construct validity method when compared to CFA. EFA has extensive value because it is considered to be effective in identifying the underlying latent variables or factors of a measure by exploring relationships among observed variables. However, it allows for more subjectivity in the decision-making process than many other statistical procedures, which can be considered a problem (Roberson et al. 2014 ).

For more consistent results on the psychometric indices of the new scale, DeVellis ( 2003 ) indicates the combined use of EFA and CFA, as was performed with most studies evaluated in this review. In CFA, the specific hypothesized factor structure proposed in EFA (including the correlations among the factors) is statistically evaluated. If the estimated model fits the data, then a researcher concludes that the factor structure replicates. If not, the modification indices are used to identify where constraints placed on the factor pattern are causing a misfit (Reise et al. 2000 ). Future studies should consider the combined use of EFA and CFA during the evaluation of construct validity of the new measure, and should also apply a combination of multiple fit indices (e.g., modification indices) in order to provide more consistent psychometric results.

After EFA and CFA, convergent validity was the preferred technique used in the vast majority of the studies included in this review (e.g., Brun et al. 2014 ; Cicero et al. 2010 ). This finding is consistent with prior research (Bastos et al. 2010 ). Convergent validity consists in examining whether a scale’s score is associated with the other variables and measures of the same construct to which it should be related. It is verified either by calculating the average variance extracted for each factor when the shared variance accounted for 0.50 or more of the total variance or by correlating their scales with a measure of overall quality (Ladhari 2010 ). In the sequence of convergent validity, the following methods were identified as favorites in the assessment of construct validity: discriminant validity (the extent to which the scale’s score does not correlate with unrelated constructs) (e.g., Coker et al. 2011 ), predictive/nomological validity (the extent to which the scores of one construct are empirically related to the scores of other conceptually related constructs) (e.g., Sharma 2010 ), criterion validity (the empirical association that the new scale has with a gold standard criterion concerned with the prediction of a certain behavior) (e.g., Tanimura et al. 2011 ), internal (signifies whether the study results and conclusions are valid for the study population), and external validity (generalizability of study) (e.g., Bolton and Lane 2012 ; Khorsan and Crawford 2014 ). Considering the importance of validity to ensure the quality of the collected data and the generalized potential of the new instrument, future studies should allow different ways to assess the validity of the new scale, thus increasing the psychometric rigor of the analysis.

With regard to reliability, all studies reported internal consistency statistics (Cronbach’s alpha) for all subscales and/or the final version of the full scale (e.g., Schlosser and McNaughton 2009 ; Sewitch et al. 2003 ). These findings are consistent with those of previous review studies (Bastos et al. 2010 ; Kapuscinski and Masters 2010 ). DeVellis ( 2003 ) explains that internal consistency is the most widely used measure of reliability. It is concerned with the homogeneity of the items within a scale. Given its importance, future studies should to consider alpha evaluation as a central point of measurement reliability, and yet, as much as possible, involve the assessment of internal consistency with other measures of reliability. In the sequence of internal consistency, the following methods were identified by this review: test-retest reliability (analysis of the temporal stability; items are applied on two separate occasions, and the scores could be correlated) (e.g., Forbush et al. 2013 ), item-total/inter-item correlation reliability (analysis of the correlation of each item with the total score of the scale or subscales/analysis of the correlation of each item with another item) (e.g., Rodrigues and Bastos 2012 ), split-half reliability (the scale is split in half and the first half of the items are compared to the second half) (e.g., Uzunboylu and Ozdamli 2011 ), and inter-judge reliability (analysis of the consistency between two different observers when they assess the same measure in the same individual) (e.g., Akter et al. 2013 ; DeVellis 2003 ; Nunnally 1967 ).

Regarding sample size in step 3 and number of items, a particularly noteworthy finding was that most studies utilized sample sizes smaller than the rule of thumb that the minimum required ratio should be 10:1 (e.g., Turker 2009 ; Zheng et al. 2010 ). DeVellis ( 2003 ) and Hair Junior et al. ( 2009 ) comment that the sample size should be as large as possible to ensure factor stability. The ‘observations to variables’ ratio is ideal at 15:1, or even 20:1. However, most of the studies included in this review failed to adopt this rule. Some studies looked for justification on evidence related to the effectiveness of much smaller observations to variables ratios. For example, Nagy et al. ( 2014 ) justified the small sample size used in their investigation based on the findings of Barrett and Kline ( 1981 ), concluding that the difference in ratios 1.25:1 and 31:1 was not a significant contributor to results obtained in the factor stability. Additionally, Arrindell and van der Ende ( 1985 ) concluded that ratios of 1.3:1 and 19.8:1 did not impact the factor stability. Although the rules of thumb vary enormously, ten participants to each item has widely been considered safe recommended (Sveinbjornsdottir and Thorsteinsson 2008 ).

Finally, several studies had their number final of items reduced by more than 50%. For example, Flight et al. ( 2011 ) developed an initial item pool composed of 122 items and finished the scale with only 43. Pommer et al. ( 2013 ) developed 391 initial items and finished with only 18. Our findings clearly indicate that a significant amount of items can get lost during the development of a new scale. These results are consistent with previous literature which states both that the initial number of items must be twice the desired number in the final scale, since, during the process of analysis of the items, many may be excluded for inadequacy (Nunnally 1967 ), and that the initial set of items should be three or four times more numerous than the number of items desired, as a good way to ensure internal consistency of the scale (DeVellis 2003 ). Future research should consider these issues and expect significant loss of items during the scale development process.

Ten main limitations reported in the scale development process—findings and research implications

In addition to identifying the current practices of the scale development process, this review also aims to assess the main limitations reported by the authors. Ten limitations were found, which will be discussed together with recommendations for best practices in future scale development research (Table  3 ).

Sample characteristic limitations

The above-mentioned limitations were recorded in the majority of the studies, in two main ways. The first and the most representative way was related to the sample type. Several studies used homogeneous sampling (e.g., Forbush et al. 2013 ; Morean et al. 2012 ), whereas others used convenience sampling (e.g., Coker et al. 2011 ; Flight et al. 2011 ). Both homogeneous and convenience samples were related to limitations of generalization. For example, Atkins and Kim ( 2012 ) pointed out that “the participants for all stages of the study were US consumers; therefore, this study cannot be generalized to other cultural contexts.” Or indeed, “convenience samples are weaknesses of this study, as they pose generalizability questions,” as highlighted by Blankson et al. ( 2012 ). Nunnally ( 1967 ) suggested that, to extend the generalizability of the new scale, sample diversification should be considered in terms of data collection, particularly in the psychometric evaluation step. Future studies should consider this suggestion, recruiting heterogeneous and truly random samples for the evaluation of construct validity and the reliability of the new measure.

The second way was related to small sample size. As previously described, most of the analyzed studies utilized sample sizes less than 10:1. Only some of the authors recognized this flaw. For example, Nagy et al. ( 2014 ) reported that “the sample size employed in conducting the exploratory factor analysis is another potential limitation of the study,” Rosenthal ( 2011 ) described, “the current study was limited by the relatively small nonprobability sample of university students,” and Ho and Lin ( 2010 ) recognized that “the respondent sample size was small.” Based in these results, we emphasize that future research should seek a larger sample size (minimum ratio of 10:1) to increase the credibility of the results and thus obtain a more exact outcome in the psychometric analysis.

Methodological limitations

Cross-sectional methods were the main methodological limitations reported by other studies (e.g., Schlosser and McNaughton 2009 ; Tombaugh et al. 2011 ). Data collected under a cross-sectional study design contains the typical limitation associated with this type of research methodology, namely inability to determine the causal relationship. If cross-sectional methods are used to estimate models whose parameters do in fact vary over time, the resulting estimation may fail to yield statistically valid results, fail to identify the true model parameters, and produce inefficient estimates (Bowen and Wiersema 1999 ). In this way, different authors (e.g., Akter et al. 2013 ; Boyar et al. 2014 ) recognized that employing instruments at one point in time limits the ability to assess causal relationships. With the goal of remediating these issues and gaining a deeper understanding of the construct of interest, different studies (e.g., Morean et al. 2012 ; Schlosser and McNaughton 2009 ) suggest conducting a longitudinal study during the scale development. Using the longitudinal studies in this process may also allow the assessment of the scale’s predictive validity, since longitudinal designs evaluate whether the proposed interpretation of test scores can predict outcomes of interest over time. Therefore, future studies should consider the longitudinal approach in the scale development, both to facilitate greater understanding of the analyzed variables and to assess the predictive validity.

Self-reporting methodologies were also cited as limitations in some studies (e.g., Fisher et al. 2014 ; Pan et al. 2013 ). Mahudin et al. ( 2012 ) clarified that the self-reporting nature of quantitative studies raises the possibility of participant bias, social desirability, demand characteristics, and response sets. Such possibilities may, in turn, affect the validity of the findings. We agree with the authors’ suggestion that future research may also incorporate other objective or independent measures to supplement the subjective evaluation of the variables studied in the development of the new scale and to improve the interpretation of findings.

In addition, web-based surveys were another methodological limitation reported in some studies (e.g., Kim et al. 2011 ; Reed et al. 2011 ). Although this particular method has time- and cost-saving elements for data collection, its limitations are also highlighted. Researchers have observed that important concerns include coverage bias (bias due to sampled individuals not having—or choosing not to access—the Internet) and nonresponse bias (bias due to participants of a survey differing from those who did not respond in terms of demographic or attitudinal variables) (Kim et al. 2011 ). Alternatives to minimize the problem in future research would be in-person surveys or survey interviews. Although more costly and more time consuming, these methods reduce problems related to concerns about confidentiality and the potential for coverage and nonresponse bias (Reed et al. 2011 ). Therefore, whenever possible, in-person surveys or survey interviews should be given priority in future research rather than web surveys.

Psychometric limitations

Consistent with previous reports (MacKenzie et al. 2011 ; Prados 2007 ), this systematic review found distinct psychometric limitations reported in the scale development process. The lack of a more robust demonstration of construct validity and/or reliability was the most often mentioned limitation in the majority of the analyzed studies. For example, Alvarado-Herrera et al. ( 2015 ) reported the lack of a more robust demonstration of the predictive validity whereas Kim et al. ( 2011 ) of the nomological validity. Caro and Garcia ( 2007 ) noted that the relationships of the scale with other constructs were not analyzed. Saxena et al. ( 2015 ) and Pan et al. ( 2013 ) described the lack of demonstrable temporal stability (e.g., test-retest reliability). Imprecise or incomplete psychometric procedures that are employed during scale development are likely to obscure the outcome. Therefore, it is necessary for future research to consider adverse consequences for the reliability and validity of any construct, caused by poor test-theoretical practices. Only through detailed information and explanation of the rationale for statistical choices can the new measures be shown to have sufficient psychometric adjustments (Sveinbjornsdottir and Thorsteinsson 2008 ).

Additionally, the inadequate choice of the instruments or variables to be correlated with the variable of interest was another psychometric limitation cited in some studies (e.g., Bakar and Mustaffa 2013 ; Tanimura et al. 2011 ). This kind of limitation directly affects the convergent validity, which is a problem since, as has already been shown in this review, this type of validity has been one of the most recurrent practices in scale development. One hypothesis for this limitation may be the lack of gold standard measures to assess similar constructs as those of a new scale. In such cases, a relatively recent study by Morgado et al. ( 2014 ) offers a valid alternative. The authors used information collected on sociodemographic questionnaires (e.g., level of education and intensity of physical activity) to correlate with the constructs of interest. Future researchers should seek support from the literature on the constructs that would be theoretically associated with the construct of interest, searching for alternatives in information collected on, for example, sociodemographic questionnaires, to assess the convergent validity of the new scale.

Another psychometric limitation reported in some studies was related to factor analysis. These limitations were identified in five main forms: (1) EFA and CFA were conducted using the data from the same sample (Zheng et al. 2010 )—when this occurs, good model fit in the CFA is expected, as a consequence, the added strength of the CFA in testing a hypothesized structure for a new data set based on theory or previous findings is lost (Khine 2008 ); (2) lack of CFA (Bolton and Lane 2012 )—if this happens, the researcher loses the possibility of assigning items to factors, testing the hypothesized structure of the data, and statistically comparing alternative models (Khine 2008 ); (3) a certain amount of subjectivity was necessary in identifying and labeling factors in EFA (Lombaerts et al. 2009 )—since a factor is qualitative, it is common practice to label each factor based on an interpretation of the variables loading most heavily on it; the problem is that these labels are subjective in nature, represent the authors’ interpretation, and can vary typically from 0.30 to 0.50 (Gottlieb et al. 2014 ; Khine 2008 ); (4) the initial unsatisfactory factor analysis output (Lombaerts et al. 2009 ); and (5) lack of a more robust CFA level (Jong et al. 2014 ) taken together—when the study result distances itself from statistical results expected for EFA (e.g., KMO, Bartlett test of sphericity) and/or CFA (e.g., CFI, GFI, RMSEA), it results in an important limitation, since the tested exploratory and theoretical models are not considered valid (Khine 2008 ). Taking these results, future studies should consider the use of separate samples for EFA and CFA, the combination of EFA and CFA, the definition of objective parameters to label factors, and about the consideration for unsatisfactory results of EFA and CFA, seeking alternatives to better fit the model.

Qualitative research limitations

This review also found reported limitations on the qualitative approach of the analyzed studies. The first limitation was related to the exclusive use of the deductive method to generate items. It is noteworthy that, although most of the studies included in this review used exclusively deductive methods to generate items, only two studies recognized this as a limitation (Coleman et al. 2011 ; Song et al. 2011 ). Both studies used only the literature review to generate and operationalize the initial item pool. The authors recognized the importance of this deductive method to theoretically operationalize the target construct, but they noted that, “for further research, more diverse views should be considered to reflect more comprehensive perspectives of human knowledge-creating behaviors to strengthen the validity of the developed scales” (Song et al. 2011 , p. 256) and, “a qualitative stage could have been used to generate additional items […]. This could also have reduced measurement error by using specific language the population used to communicate” (Coleman et al. 2011 ; p. 1069). Thus, the combination of deductive and inductive approaches (e.g., focus groups or interviews) in item generation is again suggested in future research.

In addition, it is also necessary that the researcher consider the quality of the reviewed literature. Napoli et al. ( 2014 , p. 1096) reported limitations related to the loss of a more robust literature review, suggesting that the scale developed in the study may have been incorrectly operationalized: “Yet some question remains as to whether cultural symbolism should form part of this scale. Perhaps the way in which the construct was initially conceptualized and operationalized was incorrect.” The incorrect operation of the construct compromises the psychometric results of scale and its applicability in future studies.

Another limitation involves the subjective analysis of the qualitative research. Fisher et al. ( 2014 , p. 488) pointed out that the qualitative methods (literature reviews and interviews) used to develop and conceptualize the construct were the main weaknesses of the study, “this research is limited by […] the nature of qualitative research in which the interpretations of one researcher may not reflect those of another.” The authors explained that, due to the potential for researcher bias when interpreting data, it has been recognized that credible results are difficult to achieve. Nevertheless, subjective analysis is the essence and nature of qualitative studies. Some precautions in future studies can be taken to rule out potential researcher bias, such as attempts at neutrality. This is not always possible, however, and this limitation will remain a common problem in any qualitative study.

In turn, Sewitch et al. ( 2003 , p. 260) reported that failure to formally assess content validity was a limitation. The reason given was budgetary constraints. It is worthwhile to remember that the content validity is an important step to ensure confidence in any inferences made using the final scale form. Therefore, it is necessarily required in any scale development process.

An additional limitation was reported by Lucas-Carrasco et al. ( 2011 ) in the recruitment of a larger number of interviewers, which may have affected the quality of the data collected. In order to minimize this limitation, the authors reported, “all interviewers had sufficient former education, received training on the study requirements, and were provided with a detailed guide” (p. 1223). Future studies planning the use of multiple interviewers should consider potential resulting bias.

Missing data

In connection, missing data was another issue reported by some studies included in this systematic review (e.g., Glynn et al. 2015 ; Ngorsuraches et al. 2007 ). Such limitations typically occur across different fields of scientific research. Missing data includes numbers that have been grouped, aggregated, rounded, censored, or truncated, resulting in partial loss of information (Schafer and Graham 2002 ). Collins et al. ( 2001 ) clarified that when researchers are confronted with missing data, they run an increased risk of reaching incorrect conclusions. This is because missing data may bias parameter estimates, inflate type I and type II error rates, and degrade the performance of confidence intervals. The authors also explained that, “because a loss of data is nearly always accompanied by a loss of information, missing values may dramatically reduce statistical power” (p. 330). Therefore, future researchers who wish to mitigate these risks during the scale development must pay close attention to the missing data aspect of the analysis and choose their strategy carefully.

Statistical methods to solve the problem of missing data have improved significantly, as demonstrated by Schafer and Graham ( 2002 ), although misconceptions still remain abundant. Several methods to deal with missing data were reviewed, issues raised, and advice offered for those that remain unresolved. Considering the fact that a more detailed discussion of the statistics dealing with missing data is beyond of the scope of this article, more details about missing data analysis can be found in Schafer and Graham ( 2002 ).

Social desirability bias

Another limitation reported in some studies (Bova et al. 2006 ; Ngorsuraches et al. 2007 ) and identified in this systematic review is social desirability bias. This type of bias is considered to be a systematic error in self-reporting measures resulting from the desire of respondents to avoid embarrassment and project a favorable image to others (Fisher 1993 ). According to King and Bruner ( 2000 ), social desirability bias is an important threat to the validity of research employing multi-item scales. Provision of socially desirable responses in self-reported data may lead to spurious correlations between variables, as well as the suppression or moderation of relationships between the constructs of interest. Thus, one aspect of scale validity, which should be of particular concern to researchers, is the potential threat of contamination due to social-desirability response bias. To remedy this problem, we agree with the authors that it is incumbent upon researchers to identify situations in which data may be systematically biased toward the respondents’ perceptions of what is socially acceptable, to determine the extent to which this represents contamination of the data, and to implement the most appropriate methods of control. Details on methods for identifying, testing for, and/or preventing social desirability bias are beyond the scope of this article, but can be found at King and Bruner ( 2000 ).

Item limitations

In comparison with at least one previous study (Prados 2007 ), our findings reflect some potential item limitations. Firstly, items that were ambiguous or difficult to answer were the main weaknesses reported by Gottlieb et al. ( 2014 ). On this issue, the literature dealing with the necessary caution in wording the items is extensive. For example, items must clearly define the problem being addressed, must be as simple as possible, express a single idea, and use common words that reflect the vocabulary level of the target population. Items should not be inductors or have alternative or underlying assumptions. They must be free of generalizations and estimates, and be written to ensure the variability of responses. In writing the items, the researcher should avoid using fashionable expressions and colloquialisms or other words or phrases that impair understanding for groups of varying ages, ethnicities, religions, or genders. Furthermore, the items should be organized properly. For example, the opening questions should be simple and interesting to win the trust of the subjects. The most delicate, complex, or dull questions should be asked at the end of the sequence (Clark and Watson 1995 ; Malhotra 2004 ; Pasquali 2010 ).

Furthermore, Cicero et al. ( 2010 ) reported that the main limitation of their study was the fact that none of the items were reverse-scored. Although some methodologists claim that reverse scoring is necessary to avoid acquiescence among participants, this advice should be taken with caution. There are reports that the reverse-scored items may be confusing to participants, that the opposite of a construct reverse-scored may be fundamentally different than the construct, that reverse-scored items tend to be the worst fitting items in factor analyses, or that the factor structure of scales includes a factor with straightforward wording compared to a reverse-scored factor (Cicero et al. 2010 ). Awareness of these issues is necessary for future researchers to choose between avoiding acquiescence among participants or preventing a number of other problems related to the use of reverse scores.

Brevity of the scale

Limitations on the scale size were also identified in this review. Studies by Negra and Mzoughi ( 2012 ) and Tombaugh et al. ( 2011 ) mentioned the short version of the scale as their main limitation. In both studies, the final version of the new scale included only five items. Generally, short scales are good, because they require less time from respondents. However, very short scales can in fact seriously compromise the reliability of the instrument (Raykov 2008 ). To the extent that the researcher removes items of the scale, the Cronbach’s alpha tends to decrease. It is valuable to remember that the minimum acceptable alpha should be at least 0.7, while an alpha value between 0.8 and 0.9 is considered ideal. Scales with many items tend to be more reliable, with higher alpha values (DeVellis 2003 ). In this context, future researchers should prioritize scales with enough items to keep the alpha within the acceptable range. Although many items may be lost during theoretical and psychometric analysis, an alternative already mentioned in this study would be to begin the initial item pool with at least twice the desired items of the final scale.

Difficulty controlling all variables

In addition to all limitations reported, Gottlieb et al. ( 2014 ) mentioned a common limitation in different research fields—the difficulty of controlling all the variables that could influence the central construct of the study. The authors reported that “it may be that there are other variables that influence visitors’ perception of trade show effectiveness that were not uncovered in the research” and suggest “future research might yield insights that are not provided here” (p. 104). The reported limitation calls attention to the importance of the first step—item generation—in the scale development process. A possible remedy to this issue would be to know the target construct in detail during the item generation, allowing for all possible and important variables to be investigated and controlled. However, this is not always possible. Even using inductive and deductive approaches to generate items (literature review and interview), the authors still reported that limitation. In this light, future researchers must use care in hypothesizing and testing potential variables that could be controlled during construction of the scale development process.

Lack of manual instructions

Finally, this review found a weakness reported on the loss of manualized instructions that regulate the data analysis. Saxena et al. ( 2015 , p. 492) pointed out that the initial version of the new scale “did not contain manualized instructions for raters, so it lacked objective anchor points for choosing specific ratings on many of its questions”. Therefore, an important detail that should have the attention of future researchers are instructions that determine the application methods of the new scale. Pasquali ( 2010 ) suggests that when drafting the instructions, the researcher should define the development of operational strategies that will enable the application of the instrument and the format in which it will be presented and decide both how the subject’s response will be given for each item and the way that the respondent should answer each item. The researcher should also define how the scale scores would be analyzed. In addition, the instructions need to be as short as possible without confusion to the subjects of the target population, should contain one or more examples of how the items should be answered, and should ensure that the subject is free of any related tension or anxiety.

Study limitations and strengths

This review itself is subject to some limitations that should be taken into consideration. First, during the selection of the articles included in the analysis, we may have missed some studies that could have been identified by using other terms related to “scale development.” This may have impacted our findings. However, application of this term alone was recommended by its widespread use by researchers in the area (Clark and Watson 1995 ; DeVellis 2003 ; Hinkin 1995 ; Nunnally 1967 ) and by the large number of publications identified with this descriptor in the period evaluated, as compared with those screened with correlates (e.g., “development of questionnaire” and “development of measure”). In the same way, we may also have missed numerous studies that, despite recording their weaknesses, did not have the search term “limitations” indexed in the analyzed databases. We could have reduced this limitation by also using the search term ‘weakness’ or a similar word for selection and inclusion of several other articles. However, a larger number of included studies would hinder the operationalization of our findings.

Second, particularly regarding analysis of items and reliability, we lost information about the basic theories that support the scale development process: classical test theory (CTT)—known as classical psychometry—and item response theory (IRT)—known as modern psychometry (PASQUALI 2010 ). Although it was beyond the scope of this article to examine these theories, information on the employability of one or the other could contribute to a deeper understanding of their main limitations. Future studies could focus on CTT and IRT, compare the applicability of both, and identify their main limitations in the scale development process.

Still, our review is current with studies published until September 2015. As new evidence emerges on current practices and limitations reported in the scale development process, revisions to this systematic review and practice guideline would be required in future studies.

Despite its weaknesses, the strengths of this study should be highlighted. First, this study reviews the updated and consistent literature on scale development practices to be applied in, not only a specific field of knowledge as carried out in most systematic review studies, but across various fields. With this variety of conceptions, we hope to assist future researchers in different areas of human and social sciences in making the most appropriate choice between strategies.

Second, this study differs from most studies of scale development revision, since it primarily considers the conceptions of the authors themselves about the main difficulties and mistakes made during the scale development process in their own studies. We hope to contribute to the efforts of future researchers, based on the knowledge of previous mistakes. While several weaknesses in scale development research were identified, specific recommendations for future research relevant to particular previously dimensions discussed were embedded within the appropriate sections throughout the article.

We observe that, although some weaknesses have been clearly identified in the scale development practices of many studies, only a few researchers recognized and recorded these limitations. This was evidenced in the large number of studies using exclusively deductive approaches to generate the initial item pool and the limited number of studies that recognized this as a limitation, or there were a large number of studies using smaller sample sizes than recommended in the literature for psychometric analysis and the limited number of studies that reported this issue as a limitation. Considering the observed distance between the limitation and its recognition, it is important that future researchers are comfortable with the detailed process of developing a new measure, especially as it pertains to avoiding theoretical and/or methodological mistakes, or at least, if they occur, to mention them as limitations.

Conclusions

In conclusion, the present research reviews numerous studies that both proposed current practices of the scale development process and also reported its main limitations. A variety of conceptions and methodological strategies and ten mains limitations were identified and discussed along with suggestions for future research. In this way, we believe that this paper makes important contributions to the literature, especially because it provides a comprehensive set of recommendations to increase the quality of future practices in the scale development process.

Aagja, J. P., & Garg, R. (2010). Measuring perceived service quality for public hospitals (PubHosQual) in the Indian context. International Journal of Pharmaceutical and Healthcare Marketing, 4 (10), 60–83. http://dx.doi.org/10.1108/17506121011036033 .

Article   Google Scholar  

Ahmad, N., Awan, M. U., Raouf, A., & Sparks, L. (2009). Development of a service quality scale for pharmaceutical supply chains. International Journal of Pharmaceutical and Healthcare Marketing, 3 (1), 26–45. http://dx.doi.org/10.1108/17506120910948494 .

Akter, S., D’Ambra, J., & Ray, P. (2013). Development and validation of an instrument to measure user perceived service quality of mHealth. Information and Management, 50 , 181–195. http://dx.doi.org/10.1016/j.im.2013.03.001 .

Alvarado-Herrera, A, Bigne, E, Aldas-Manzano, J, & Curras-Perez, R. (2015). A scale for measuring consumer perceptions of corporate social responsibility following the sustainable development paradigm. Journal of Business Ethics , 1-20. doi: http://dx.doi.org/10.1007/s10551-015-2654-9

Arias, M. R. M., Lloreda, M. J. H., & Lloreda, M. V. H. (2014). Psicometría . S.A.: Alianza Editorial

Armfield, J. M. (2010). Development and psychometric evaluation of the Index of Dental Anxiety and Fear (IDAF-4C + ). Psychological Assessment, 22 (2), 279–287. http://dx.doi.org/10.1037/a0018678 .

Article   PubMed   Google Scholar  

Arrindell, W. A., & van der Ende, J. (1985). An empirical-test of the utility of the observations-to-variables ratio in factor and components analysis. Applied Psychological Measurement, 9 (2), 165–178. http://dx.doi.org/10.1177/014662168500900205 .

Atkins, K. G., & Kim, Y. (2012). Smart shopping: conceptualization and measurement. International Journal of Retail and Distribution Management, 40 (5), 360–375. http://dx.doi.org/10.1108/09590551211222349 .

Bagdare, S., & Jain, R. (2013). Measuring retail customer experience. International Journal of Retail and Distribution Management, 41 (10), 790–804. http://dx.doi.org/10.1108/IJRDM-08-2012-0084 .

Bakar, H. A., & Mustaffa, C. S. (2013). Organizational communication in Malaysia organizations. Corporate Communications: An International Journal, 18 (1), 87–109. http://dx.doi.org/10.1108/13563281311294146 .

Barrett, P. T., & Kline, P. (1981). The observation to variable ratio in factor analysis. Personality Study and Group Behavior, 1 , 23–33.

Google Scholar  

Bastos, J. L., Celeste, R. K., Faerstein, E., & Barros, A. J. D. (2010). Racial discrimination and health: a systematic review of scales with a focus on their psychometric properties. Social Science and Medicine, 70 , 1091–1099. http://dx.doi.org/10.1016/j.socscimed.2009.12.20 .

Beaudreuil, J, Allard, A, Zerkak, D, Gerber, RA, Cappelleri, JC, Quintero, N, Lasbleiz, S, … Bardin, T. (2011). Unite’ Rhumatologique des Affections de la Main (URAM) Scale: development and validation of a tool to assess Dupuytren’s disease–specific disability. Arthritis Care & Research, 63 (10), 1448-1455. doi: http://dx.doi.org/10.1002/acr.20564

Bhattacherjee, A. (2002). Individual trust in online firms: scale development and initial test. Journal of Management Information Systems, 19 (1), 211–241. http://dx.doi.org/10.1080/07421222.2002.11045715 .

Blankson, C., Cheng, J. M., & Spears, N. (2007). Determinants of banks selection in USA, Taiwan and Ghana. International Journal of Bank Marketing, 25 (7), 469–489. http://dx.doi.org/10.1108/02652320710832621 .

Blankson, C., Paswan, A., & Boakye, K. G. (2012). College students’ consumption of credit cards. International Journal of Bank Marketing, 30 (7), 567–585. http://dx.doi.org/10.1108/02652321211274327 .

Bolton, D. L., & Lane, M. D. (2012). Individual entrepreneurial orientation: development of a measurement instrument. Education + Training, 54 (2/3), 219–233. http://dx.doi.org/10.1108/00400911211210314 .

Bova, C., Fennie, K. P., Watrous, E., Dieckhaus, K., & Williams, A. B. (2006). The health care relationship (HCR) trust scale: development and psychometric evaluation. Research in Nursing and Health, 29 , 477–488. http://dx.doi.org/10.1002/nur.20158 .

Bowen, H. P., & Wiersema, M. F. (1999). Matching method to paradigm in strategy research: limitations of cross-sectional analysis and some methodological alternatives. Strategic Management Journal, 20 , 625–636.

Boyar, S. L., Campbell, N. S., Mosley, D. C., Jr., & Carson, C. M. (2014). Development of a work/family social support measure. Journal of Managerial Psychology, 29 (7), 901–920. http://dx.doi.org/10.1108/JMP-06-2012-0189 .

Brock, J. K., & Zhou, Y. (2005). Organizational use of the internet. Internet Research, 15 (1), 67–87. http://dx.doi.org/10.1108/10662240510577077 .

Brun, I., Rajaobelina, L., & Ricard, L. (2014). Online relationship quality: scale development and initial testing. International Journal of Bank Marketing, 32 (1), 5–27. http://dx.doi.org/10.1108/IJBM-02-2013-0022 .

Butt, M. M., & Run, E. C. (2010). Private healthcare quality: applying a SERVQUAL model. International Journal of Health Care Quality Assurance, 23 (7), 658–673. http://dx.doi.org/10.1108/09526861011071580 .

Caro, L. M., & García, J. A. M. (2007). Measuring perceived service quality in urgent transport service. Journal of Retailing and Consumer Services, 14 , 60–72. http://dx.doi.org/10.1016/j.jretconser.2006.04.001 .

Chahal, H., & Kumari, N. (2012). Consumer perceived value. International Journal of Pharmaceutical and Healthcare Marketing, 6 (2), 167–190. http://dx.doi.org/10.1108/17506121211243086 .

Chen, H., Tian, Y., & Daugherty, P. J. (2009). Measuring process orientation. The International Journal of Logistics Management, 20 (2), 213–227. http://dx.doi.org/10.1108/09574090910981305 .

Choi, S. W., Victorson, D. E., Yount, S., Anton, S., & Cella, D. (2011). Development of a conceptual framework and calibrated item banks to measure patient-reported dyspnea severity and related functional limitations. Value in Health, 14 , 291–306. http://dx.doi.org/10.1016/j.jval.2010.06.001 .

Christophersen, T., & Konradt, U. (2012). Development and validation of a formative and a reflective measure for the assessment of online store usability. Behaviour and Information Technology, 31 (9), 839–857. http://dx.doi.org/10.1080/0144929X.2010.529165 .

Churchill, G. (1979). A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16 (1), 64–73. http://dx.doi.org/10.2307/3150876 .

Cicero, D. C., Kerns, J. G., & McCarthy, D. M. (2010). The Aberrant Salience Inventory: a new measure of psychosis proneness. Psychological Assessment, 22 (3), 688–701. http://dx.doi.org/10.1037/a0019913 .

Clark, L. A., & Watson, D. (1995). Constructing validity: basic issues in objective scale development. Psychological Assessment, 7 (3), 309–319. http://dx.doi.org/10.1037/1040-3590.7.3.309 .

Coker, B. L. S., Ashill, N. J., & Hope, B. (2011). Measuring internet product purchase risk. European Journal of Marketing, 45 (7/8), 1130–1151. http://dx.doi.org/10.1108/03090561111137642 .

Coleman, D., Chernatony, L., & Christodoulides, G. (2011). B2B service brand identity: scale development and validation. Industrial Marketing Management, 40 , 1063–1071. http://dx.doi.org/10.1016/j.indmarman.2011.09.010 .

Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6 (4), 330–351. http://dx.doi.org/10.1037/1082-989X.6.4.330 .

Colwell, S. R., Aung, M., Kanetkar, V., & Holden, A. L. (2008). Toward a measure of service convenience: multiple-item scale development and empirical test. Journal of Services Marketing, 22 (2), 160–169. http://dx.doi.org/10.1108/08876040810862895 .

Cossette, S., Cara, C., Ricard, N., & Pepin, J. (2005). Assessing nurse–patient interactions from a caring perspective: report of the development and preliminary psychometric testing of the Caring Nurse–Patient Interactions Scale. International Journal of Nursing Studies, 42 , 673–686. http://dx.doi.org/10.1016/j.ijnurstu.2004.10.004 .

Dennis, R. S., & Bocarnea, M. (2005). Development of the servant leadership assessment instrument. Leadership and Organization Development Journal, 26 (8), 600–615. http://dx.doi.org/10.1108/01437730510633692 .

DeVellis, R. F. (2003). Scale development: theory and applications (2nd ed.). Newbury Park: Sage Publications.

Devlin, J. F., Roy, S. K., & Sekhon, H. (2014). Perceptions of fair treatment in financial services. European Journal of Marketing, 48 (7/8), 1315–1332. http://dx.doi.org/10.1108/EJM-08-2012-0469 .

Dunham, A., & Burt, C. (2014). Understanding employee knowledge: the development of an organizational memory scale. The Learning Organization, 21 (2), 126–145. http://dx.doi.org/10.1108/TLO-04-2011-0026 .

Edwards, J. R., Knight, D. K., Broome, K. M., & Flynn, P. M. (2010). The development and validation of a transformational leadership survey for substance use treatment programs. Substance Use and Misuse, 45 , 1279–1302. http://dx.doi.org/10.3109/10826081003682834 .

Article   PubMed   PubMed Central   Google Scholar  

Feuerstein, M., Nicholas, R. A., Huang, G. D., Haufler, A. J., Pransky, G., & Robertson, M. (2005). Workstyle: development of a measure of response to work in those with upper extremity pain. Journal of Occupational Rehabilitation, 15 (2), 87–104. http://dx.doi.org/10.1007/s10926-005-3420-0 .

Fisher, R. J. (1993). Social desirability bias and the validity of indirect questioning. Journal of Consumer Research, 20 (2), 303–315. http://dx.doi.org/10.1086/209351 .

Fisher, R., Maritz, A., & Lobo, A. (2014). Evaluating entrepreneurs’ perception of success. International Journal of Entrepreneurial Behavior and Research, 20 (5), 478–492. http://dx.doi.org/10.1108/IJEBR-10-2013-0157 .

Flight, R. L., D’Souza, G., & Allaway, A. W. (2011). Characteristics-based innovation adoption: scale and model validation. Journal of Product and Brand Management, 20 (5), 343–355. http://dx.doi.org/10.1108/10610421111157874 .

Forbush, KT, Wildes, JE, Pollack, LO, Dunbar, D, Luo, J, Patterson, P, Petruzzi, L, … Watson, D. (2013). Development and validation of the Eating Pathology Symptoms Inventory (EPSI). Psychological Assessment, 25 (3), 859-878. doi: http://dx.doi.org/10.1037/a0032639 .

Foster, J. D., McCain, J. L., Hibberts, M. F., Brunell, A. B., & Johnson, B. (2015). The grandiose narcissism scale: a global and facet-level measure of grandiose narcissism. Personality and Individual Differences, 73 , 12–16. http://dx.doi.org/10.1016/j.paid.2014.08.042 .

Franche, R., Corbière, M., Lee, H., Breslin, F. C., & Hepburn, G. (2007). The readiness for return-to-work (RRTW) scale: development and validation of a self-report staging scale in lost-time claimants with musculoskeletal disorders. Journal of Occupational Rehabilitation, 17 , 450–472. http://dx.doi.org/10.1007/s10926-007-9097-9 .

Gesten, E. L. (1976). A health resources inventory: the development of a measure of the personal and social competence of primary-grade children. Journal of Consulting and Clinical Psychology, 44 (5), 775–786. http://dx.doi.org/10.1037/0022-006X.44.5.775 .

Gibbons, C. J., Kenning, C., Coventry, P. A., Bee, P., Bundy, C., Fisher, L., & Bower, P. (2013). Development of a Multimorbidity Illness Perceptions Scale (MULTIPleS). PloS One, 8 (12), e81852. http://dx.doi.org/10.1371/journal.pone.0081852 .

Gligor, D. M., & Holcomb, M. (2014). The road to supply chain agility: an RBV perspective on the role of logistics capabilities. The International Journal of Logistics Management, 25 (1), 160–179. http://dx.doi.org/10.1108/IJLM-07-2012-0062 .

Glynn, N. W., Santanasto, A. J., Simonsick, E. M., Boudreau, R. M., Beach, S. R., Schulz, R., & Newman, A. B. (2015). The Pittsburgh fatigability scale for older adults: development and validation. Journal of American Geriatrics Society, 63 , 130–135. http://dx.doi.org/10.1111/jgs.13191 .

Gottlieb, U., Brown, M., & Ferrier, L. (2014). Consumer perceptions of trade show effectiveness. European Journal of Marketing, 48 (1/2), 89–107. http://dx.doi.org/10.1108/EJM-06-2011-0310 .

Hair Junior, J. F., Black, W. C., Babin, N. J., Anderson, R. E., & Tatham, R. L. (2009). Análise multivariada de dados (6ath ed.). São Paulo: Bookman.

Hall, M. A., Camacho, F., Dugan, E., & Balkrishnan, R. (2002). Trust in the medical profession: conceptual and measurement issues. Health Services Research, 37 (5), 1419–1439. http://dx.doi.org/10.1111/1475-6773.01070 .

Han, H., Back, K., & Kim, Y. (2011). A multidimensional scale of switching barriers in the full-service restaurant industry. Cornell Hospitality Quarterly, 52 (1), 54–-63. http://dx.doi.org/10.1177/1938965510389261 .

Hardesty, D. M., & Bearden, W. O. (2004). The use of expert judges in scale development Implications for improving face validity of measures of unobservable constructs. Journal of Business Research, 57 , 98–107. http://dx.doi.org/10.1016/S0148-2963(01)00295-8 .

Henderson-King, D., & Henderson-King, E. (2005). Acceptance of cosmetic surgery: scale development and validation. Body Image, 2 , 137–149. http://dx.doi.org/10.1016/j.bodyim.2005.03.003 .

Hernandez, J. M. C., & Santos, C. C. (2010). Development-based trust: proposing and validating a new trust measurement model for buyer-seller relationships. Brazilian Administration Review, 7 (2), 172–197. http://dx.doi.org/10.1590/S1807-76922010000200005 .

Hildebrandt, T., Langenbucher, J., & Schlundt, D. G. (2004). Muscularity concerns among men: development of attitudinal and perceptual measures. Body Image, 1 , 169–181. http://dx.doi.org/10.1016/j.bodyim.2004.01.001 .

Hilsenroth, M. J., Blagys, M. D., Ackerman, S. J., Bonge, D. R., & Blais, M. A. (2005). Measuring psychodynamic-interpersonal and cognitive-behavioral techniques: development of the comparative psychotherapy process scale. Psychotherapy: Theory, Research, Practice, Training, 42 (3), 340–356. http://dx.doi.org/10.1037/0033-3204.42.3.340 .

Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21 (5), 967–988. http://dx.doi.org/10.1177/014920639502100509 .

Ho, C. B., & Lin, W. (2010). Measuring the service quality of internet banking: scale development and validation. European Business Review, 22 (1), 5–24. http://dx.doi.org/10.1108/09555341011008981 .

Hutz, CS, Bandeira, DR, & Trentini, CM. (Org.). (2015). Psicometria . Porto Alegre, Artmed

Jong, N., Van Leeuwen, R. G. J., Hoekstra, H. A., & van der Zee, K. I. (2014). CRIQ: an innovative measure using comparison awareness to avoid self-presentation tactics. Journal of Vocational Behavior, 84 , 199–214. http://dx.doi.org/10.1016/j.jvb.2014.01.003 .

Kapuscinski, A. N., & Masters, K. S. (2010). The current status of measures of spirituality: a critical review of scale development. Psychology of Religion and Spirituality, 2 (4), 191–205. http://dx.doi.org/10.1037/a0020498 .

Khine, M. S. (2008). Knowing, knowledge and beliefs: epistemological studies across diverse cultures . New York: Springer.

Book   Google Scholar  

Khorsan, R, & Crawford, C. (2014). External validity and model validity: a conceptual approach for systematic review methodology. Evidence-Based Complementary and Alternative Medicine, 2014 , Article ID 694804, 12 pages. doi: http://dx.doi.org/10.1155/2014/694804

Kim, S., Cha, J., Knutson, B. J., & Beck, J. A. (2011). Development and testing of the Consumer Experience Index (CEI). Managing Service Quality: An International Journal, 21 (2), 112–132. http://dx.doi.org/10.1108/09604521111113429 .

Kim, D., Lee, Y., Lee, J., Nam, J. K., & Chung, Y. (2014). Development of Korean smartphone addiction proneness scale for youth. PloS One, 9 (5), e97920. http://dx.doi.org/10.1371/journal.pone.0097920 .

King, M. F., & Bruner, G. C. (2000). Social desirability bias: a neglected aspect of validity testing. Psychology and Marketing, 17 (2), 79–103. http://dx.doi.org/10.1002/(SICI)1520-6793(200002)17:2<79::AID-MAR2>3.0.CO;2-0 .

Kwon, W., & Lennon, S. J. (2011). Assessing college women’s associations of American specialty apparel brands. Journal of Fashion Marketing and Management: An International Journal, 15 (2), 242–256. http://dx.doi.org/10.1108/13612021111132663 .

Ladhari, R. (2010). Developing e-service quality scales: a literature review. Journal of Retailing and Consumer Services, 17 , 464–477. http://dx.doi.org/10.1016/j.jretconser.2010.06.003 .

Lin, J. C., & Hsieh, P. (2011). Assessing the self-service technology encounters: development and validation of SSTQUAL scale. Journal of Retailing, 87 (2), 194–206. http://dx.doi.org/10.1016/j.jretai.2011.02.006 .

Lombaerts, K., Backer, F., Engels, N., Van Braak, J., & Athanasou, J. (2009). Development of the self-regulated learning teacher belief scale. European Journal of Psychology of Education, 24 (1), 79–96. http://dx.doi.org/10.1007/BF03173476 .

Lucas-Carrasco, R., Eser, E., Hao, Y., McPherson, K. M., Green, A., & Kullmann, L. (2011). The quality of care and support (QOCS) for people with disability scale: development and psychometric properties. Research in Developmental Disabilities, 32 , 1212–1225. http://dx.doi.org/10.1016/j.ridd.2010.12.030 .

MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavioral research: integrating new and existing techniques. MIS Quarterly, 35 (2), 293–334.

Mahudin, N. D. M., Cox, T., & Griffiths, A. (2012). Measuring rail passenger crowding: scale development and psychometric properties. Transportation Research Part, F 15 , 38–51. http://dx.doi.org/10.1016/j.trf.2011.11.006 .

Malhotra, N. K. (2004). Pesquisa de marketing: Uma orientação aplicada (4ath ed.). Porto Alegre: Bookman.

Medina-Pradas, C., Navarro, J. B., López, S. R., Grau, A., & Obiols, J. E. (2011). Further development of a scale of perceived expressed emotion and its evaluation in a sample of patients with eating disorders. Psychiatry Research, 190 , 291–296. http://dx.doi.org/10.1016/j.psychres.2011.06.011 .

Meneses, J., Barrios, M., Bonillo, A., Cosculluela, A., Lozano, L. M., Turbany, J., & Valero, S. (2014). Psicometría . Barcelona: Editorial UOC.

Morean, M. E., Corbin, W. R., & Treat, T. A. (2012). The anticipated effects of alcohol scale: development and psychometric evaluation of a novel assessment tool for measuring alcohol expectancies. Psychological Assessment, 24 (4), 1008–1023. http://dx.doi.org/10.1037/a0028982 .

Morgado, F. F. R., Campana, A. N. N. B., & Tavares, M. C. G. C. F. (2014). Development and validation of the self-acceptance scale for persons with early blindness: the SAS-EB. PloS One, 9 (9), e106848. http://dx.doi.org/10.1371/journal.pone.0106848 .

Nagy, B. G., Blair, E. S., & Lohrke, F. T. (2014). Developing a scale to measure liabilities and assets of newness after start-up. International Entrepreneurship and Management Journal, 10 , 277–295. http://dx.doi.org/10.1007/s11365-012-0219-2 .

Napoli, J., Dickinson, S. J., Beverland, M. B., & Farrelly, F. (2014). Measuring consumer-based brand authenticity. Journal of Business Research, 67 , 1090–1098. http://dx.doi.org/10.1016/j.jbusres.2013.06.001 .

Negra, A., & Mzoughi, M. N. (2012). How wise are online procrastinators? A scale development. Internet Research, 22 (4), 426–442. http://dx.doi.org/10.1108/10662241211250971 .

Ngorsuraches, S., Lerkiatbundit, S., Li, S. C., Treesak, C., Sirithorn, R., & Korwiwattanakarn, M. (2007). Development and validation of the patient trust in community pharmacists (TRUST-Ph) scale: results from a study conducted in Thailand. Research in Social and Administrative Pharmacy, 4 , 272–283. http://dx.doi.org/10.1016/j.sapharm.2007.10.002 .

Nunnally, J. C. (1967). Psychometric theory . New York: McGraw Hill.

Oh, H. (2005). Measuring affective reactions to print apparel advertisements: a scale development. Journal of Fashion Marketing and Management: An International Journal, 9 (3), 283–305. http://dx.doi.org/10.1108/13612020510610426 .

Olaya, B, Marsà, F, Ochoa, S, Balanzá-Martínez, V, Barbeito, S, González-Pinto, A, … Haro, JM. (2012). Development of the insight scale for affective disorders (ISAD): modification from the scale to assess unawareness of mental disorder . Journal of Affective Disorders, 142 , 65-71. doi: http://dx.doi.org/10.1016/j.jad.2012.03.041 .

Omar, N. A., & Musa, R. (2011). Measuring service quality in retail loyalty programmes (LPSQual). International Journal of Retail and Distribution Management, 39 (10), 759–784. http://dx.doi.org/10.1108/09590551111162257 .

Pan, J., Wong, D. F. K., & Ye, S. (2013). Post-migration growth scale for Chinese international students: development and validation. Journal of Happiness Studies, 14 , 1639–1655. http://dx.doi.org/10.1007/s10902-012-9401-z .

Pasquali, L. (2010). Instrumentação psicológica: fundamentos e práticas . Porto Alegre: Artmed.

Patwardhan, H., & Balasubramanian, S. K. (2011). Brand romance: a complementary approach to explain emotional attachment toward brands. Journal of Product and Brand Management, 20 (4), 297–308. http://dx.doi.org/10.1108/10610421111148315 .

Pimentel, C. E., Gouveia, V. V., & Pessoa, V. S. (2007). Escala de Preferência Musical: construção e comprovação da sua estrutura fatorial. Psico-USF, 12 (2), 145–155.

Podsakoff, N. P., Podsakoff, P. M., MacKenzie, S. B., & Klinger, R. L. (2013). Are we really measuring what we say we’re measuring? Using video techniques to supplement traditional construct validation procedures. Journal of Applied Psychology, 98 (1), 99–113. http://dx.doi.org/10.1037/a0029570 .

Pommer, AM, Prins, L, van Ranst, D, Meijer, J, Hul, AV, Janssen, J, … Pop, VJM. (2013). Development and validity of the Patient-Centred COPD Questionnaire (PCQ). Journal of Psychosomatic Research, 75 , 563-571. doi: http://dx.doi.org/10.1016/j.jpsychores.2013.10.001

Prados, J. M. (2007). Development of a new scale of beliefs about the worry consequences. Annals of Psychology, 23 (2), 226–230.

Raykov, T. (2008). Alpha if item deleted: a note on loss of criterion validity in scale development if maximizing coefficient alpha. British Journal of Mathematical and Statistical Psychology, 61 , 275–285. http://dx.doi.org/10.1348/000711007X188520 .

Reed, L. L., Vidaver-Cohen, D., & Colwell, S. R. (2011). A new scale to measure executive servant leadership: development, analysis, and implications for research. Journal of Business Ethics, 101 , 415–434. http://dx.doi.org/10.1007/s10551-010-0729-1 .

Reise, S. P., Waller, N. G., & Comrey, A. L. (2000). Factor analysis and scale revision. Psychological Assessment, 12 (3), 287–297. http://dx.doi.org/10.1037//1040-3590.12.3.287 .

Rice, S. M., Fallon, B. J., Aucote, H. M., & Möller-Leimkühler, A. M. (2013). Development and preliminary validation of the male depression risk scale: Furthering the assessment of depression in men. Journal of Affective Disorders, 151 , 950–958. http://dx.doi.org/10.1016/j.jad.2013.08.013 .

Riedel, M., Spellmann, I., Schennach-Wolff, R., Obermeier, M., & Musil, R. (2011). The RSM-scale: a pilot study on a new specific scale for self- and observer-rated quality of life in patients with schizophrenia. Quality of Life Research, 20 , 263–272. http://dx.doi.org/10.1007/s11136-010-9744-z .

Roberson, R. B., III, Elliott, T. R., Chang, J. E., & Hill, J. N. (2014). Exploratory factor analysis in rehabilitation psychology: a content analysis. Rehabilitation Psychology, 59 (4), 429–438. http://dx.doi.org/10.1037/a0037899 .

Rodrigues, A. C. A., & Bastos, A. V. B. (2012). Organizational entrenchment: scale development and validation. Psicologia: Reflexão e Crítica, 25 (4), 688–700. http://dx.doi.org/10.1590/S0102-79722012000400008 .

Rodríguez, I., Kozusznik, M. W., & Peiró, J. M. (2013). Development and validation of the Valencia Eustress-Distress Appraisal Scale. International Journal of Stress Management, 20 (4), 279–308. http://dx.doi.org/10.1037/a0034330 .

Rosenthal, S. (2011). Measuring knowledge of indoor environmental hazards. Journal of Environmental Psychology, 31 , 137–146. http://dx.doi.org/10.1016/j.jenvp.2010.08.003 .

Saxena, S., Ayers, C. R., Dozier, M. E., & Maidment, K. M. (2015). The UCLA Hoarding Severity Scale: development and validation. Journal of Affective Disorders, 175 , 488–493. http://dx.doi.org/10.1016/j.jad.2015.01.030 .

Schafer, J. L., & Graham, J. W. (2002). Missing data: our view of the state of the Art. Psychological Methods, 7 (2), 147–177. http://dx.doi.org/10.1037//1082-989X.7.2.147 .

Schlosser, F. K., & McNaughton, R. B. (2009). Using the I-MARKOR scale to identify market-oriented individuals in the financial services sector. Journal of Services Marketing, 23 (4), 236–248. http://dx.doi.org/10.1108/08876040910965575 .

Sewitch, M. J., Abrahamowicz, M., Dobkin, P. L., & Tamblyn, R. (2003). Measuring differences between patients’ and physicians’ health perceptions: the patient–physician discordance scale. Journal of Behavioral Medicine, 26 (3), 245–263. http://dx.doi.org/10.1023/A:1023412604715 .

Sharma, P. (2010). Measuring personal cultural orientations: scale development and validation. Journal of the Academy of Marketing Science, 38 , 787–806. http://dx.doi.org/.1007/s11747-009-0184-7 .

Sharma, D., & Gassenheimer, J. B. (2009). Internet channel and perceived cannibalization. European Journal of Marketing, 43 (7/8), 1076–1091. http://dx.doi.org/10.1108/03090560910961524 .

Shawyer, F., Ratcliff, K., Mackinnon, A., Farhall, J., Hayes, S. C., & Copolov, D. (2007). The Voices Acceptance and Action Scale (VAAS): pilot data. Journal of Clinical Psychology, 63 (6), 593–606. http://dx.doi.org/10.1002/jclp.20366 .

Sin, L. Y. M., Tse, A. C. B., & Yim, F. H. K. (2005). CRM: conceptualization and scale development. European Journal of Marketing, 39 (11/12), 1264–1290. http://dx.doi.org/10.1108/03090560510623253 .

Sohn, D., & Choi, S. M. (2014). Measuring expected interactivity: scale development and validation. New Media and Society, 16 (5), 856–870. http://dx.doi.org/10.1177/1461444813495808 .

Song, J. H., Uhm, D., & Yoon, S. W. (2011). Organizational knowledge creation practice. Leadership and Organization Development Journal, 32 (3), 243–259. http://dx.doi.org/10.1108/01437731111123906 .

Staines, Z. (2013). Managing tacit investigative knowledge: measuring “investigative thinking styles”. Policing: An International Journal of Police Strategies and Management, 36 (3), 604–619. http://dx.doi.org/10.1108/PIJPSM-07-2012-0072 .

Sultan, P., & Wong, H. (2010). Performance-based service quality model: an empirical study on Japanese universities. Quality Assurance in Education, 18 (2), 126–143. http://dx.doi.org/10.1108/09684881011035349 .

Sveinbjornsdottir, S., & Thorsteinsson, E. B. (2008). Adolescent coping scales: a critical psychometric review. Scandinavian Journal of Psychology, 49 (6), 533–548. http://dx.doi.org/10.1111/j.1467-9450.2008.00669.x .

Swaid, S. I., & Wigand, R. T. (2009). Measuring the quality of E-Service: scale development and initial validation. Journal of Electronic Commerce Research, 10 (1), 13–28.

Tanimura, C., Morimoto, M., Hiramatsu, K., & Hagino, H. (2011). Difficulties in the daily life of patients with osteoarthritis of the knee: scale development and descriptive study. Journal of Clinical Nursing, 20 , 743–753. http://dx.doi.org/10.1111/j.1365-2702.2010.03536.x .

Taute, H. A., & Sierra, J. (2014). Brand tribalism: an anthropological perspective. Journal of Product and Brand Management, 23 (1), 2–15. http://dx.doi.org/10.1108/JPBM-06-2013-0340 .

Tombaugh, J. R., Mayfield, C., & Durand, R. (2011). Spiritual expression at work: exploring the active voice of workplace spirituality. International Journal of Organizational Analysis, 19 (2), 146–170. http://dx.doi.org/10.1108/19348831111135083 .

Turker, D. (2009). Measuring corporate social responsibility: a scale development study. Journal of Business Ethics, 85 , 411–427. http://dx.doi.org/10.1007/s10551-008-9780-6 .

Uzunboylu, H., & Ozdamli, F. (2011). Teacher perception for m-learning: scale development and teachers’ perceptions. Journal of Computer Assisted Learning, 27 , 544–556. http://dx.doi.org/10.1111/j.1365-2729.2011.00415.x .

Van der Gaag, M, Schütz, C, ten Napel, A, Landa, Y, Delespaul, P, Bak, M, … Hert, M. (2013). Development of the Davos Assessment of Cognitive Biases Scale (DACOBS). Schizophrenia Research, 144 , 63-71. doi: http://dx.doi.org/10.1016/j.schres.2012.12.010

Von Steinbüchel, N, Wilson, L, Gibbons, H, Hawthorne, G, Höfer, S, Schmidt, S, … Truelle, J. (2010). Journal of Neurotrauma, 27 , 1167-1185. doi: http://dx.doi.org/10.1089/neu.2009.1076

Voon, B. H., Abdullah, F., Lee, N., & Kueh, K. (2014). Developing a HospiSE scale for hospital service excellence. International Journal of Quality and Reliability Management, 31 (3), 261–280. http://dx.doi.org/10.1108/IJQRM-10-2012-0143 .

Walshe, M., Peach, R. K., & Miller, N. (2009). Dysarthria Impact Profile: development of a scale to measure psychosocial effects. International Journal of Language and Communication Disorders, 44 (5), 693–715. http://dx.doi.org/10.1080/13682820802317536 .

Wang, C. L., & Mowen, J. C. (1997). The separateness-connectedness self-schema: scale development and application to message construction. Psychology and Marketing, 14 (2), 185–207. http://dx.doi.org/10.1002/(SICI)1520-6793(199703)14:2<185::AID-MAR5>3.0.CO;2-9 .

Wepener, M., & Boshoff, C. (2015). An instrument to measure the customer-based corporate reputation of large service organizations. Journal of Services Marketing, 29 (3), 163–172. http://dx.doi.org/10.1108/JSM-01-2014-0026 .

Williams, Z., Ponder, N., & Autry, C. W. (2009). Supply chain security culture: measure development and validation. The International Journal of Logistics Management, 20 (2), 243–260. http://dx.doi.org/10.1108/09574090910981323 .

Wilson, N. L., & Holmvall, C. M. (2013). The development and validation of the incivility from customers scale. Journal of Occupational Health Psychology, 18 (3), 310–326. http://dx.doi.org/10.1037/a0032753 .

Yang, M., Weng, S., & Hsiao, P. (2014). Measuring blog service innovation in social media services. Internet Research, 24 (1), 110–128. http://dx.doi.org/10.1108/IntR-12-2012-0253 .

Zhang, X., & Hu, D. (2011). Farmer-buyer relationships in China: the effects of contracts, trust and market environment. China Agricultural Economic Review, 3 (1), 42–53. http://dx.doi.org/10.1108/17561371111103534 .

Zheng, J, You, L, Lou, T, Chen, N, Lai, D, Liang, Y, … Zhai, C. (2010). Development and psychometric evaluation of the dialysis patient-perceived exercise benefits and barriers scale. International Journal of Nursing Studies, 47 , 166-180. doi: http://dx.doi.org/10.1016/j.ijnurstu.2009.05.023

Download references

Authors’ contributions

FFRM is responsible for all parts of this manuscript, from its conception to the final writing. JFFM, CMN, ACSA and MECF participated in the data collection, analysis and interpretation of data and critical review of the manuscript. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interest.

Author information

Authors and affiliations.

Institute of Education, Universidade Federal Rural do Rio de Janeiro, BR-465, km 7, Seropédica, Rio de Janeiro, 23890-000, Brazil

Fabiane F. R. Morgado

Faculty of Psychology, Universidade Federal de Juiz de Fora, Rua José Lourenço Kelmer, s/n—Campus Universitário Bairro São Pedro, Juiz de Fora, Minas Gerais, 36036-900, Brazil

Juliana F. F. Meireles, Clara M. Neves & Maria E. C. Ferreira

Faculty of Physical Education of the Instituto Federal de Educação, Ciência e Tecnologia do Sudeste de Minas Gerais, Av. Luz Interior, n 360, Estrela Sul, Juiz de Fora, Minas Gerais, 36030-776, Brazil

Ana C. S. Amaral

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Fabiane F. R. Morgado .

Additional information

An erratum to this article is available at http://dx.doi.org/10.1186/s41155-017-0059-7 .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Morgado, F.F.R., Meireles, J.F.F., Neves, C.M. et al. Scale development: ten main limitations and recommendations to improve future research practices. Psicol. Refl. Crít. 30 , 3 (2018). https://doi.org/10.1186/s41155-016-0057-1

Download citation

Received : 03 August 2016

Accepted : 22 December 2016

Published : 25 January 2017

DOI : https://doi.org/10.1186/s41155-016-0057-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Measurement
• Psychometrics
• Reliability