which type of exam promotes higher level critical thinking skills

“It entails the examination of those structures or elements of thought implicit in all reasoning: purpose; problem, or question-at-issue; assumptions; concepts; empirical grounding; inferences; implications and consequences; objections from alternative viewpoints, and frame of reference.” (National Council Draft Statement)

It entails larger-scale abilities of integrating elementary skills in such a way as to be able to apply, synthesize, analyze, and evaluate complicated and multidimensional issues. These include such abilities as clarifying issues, transferring insights into new contexts, analyzing arguments, questioning deeply, developing criteria for evaluation, assessing solutions, refining generalizations, and evaluating the credibility of sources of information. Among the abilities are included also the central forms of communication: critical reading, writing, speaking, and listening. Each of them is a large-scaled mode of thinking which is successful to the extent that it is informed, disciplined, and guided by critical thought and reflection. (Paraphrased from National Council Draft Statement.)

Critical thinking entails the possession and active use of a set of traits of mind and affective dimensions: independence of thought, fair-mindedness, intellectual humility, intellectual courage, intellectual perseverance, intellectual integrity, curiosity, confidence in reason, and the willingness to see objections, to enter sympathetically into another’s point of view, and to recognize one’s own egocentricity or ethnocentricity. (Paraphrased from National Council Draft Statement.)
Critical thinking — in being responsive to variable subject areas, issues, and purposes — is incorporated in a family of interrelated modes of thinking, among them: scientific thinking, mathematical thinking, historical thinking, anthropological thinking, economic thinking, moral thinking, and philosophical thinking (National Council Draft Statement).

CRITERION #19 Can critical thinking be assessed nationally in a way that is financially affordable? To make it affordable, the constructed response segment of the assessment should be administered not to the population of students as a whole, but rather to a representative sample of the student population of a school system. The assessment should be a) paid for by school systems that contract to have their students tested, and b) constructed, monitored, administered, and graded by a private agency with critical thinking credentials, or at least under the direction of scholars with a solid grounding in the research into critical thinking.

CRITERIA #20 AND #21 Can critical thinking be assessed so as to gauge the improvement of students over the course of their education and to measure the achievement of students against national standards? To evaluate students in both these dimensions requires:      a) an assessment administered as a pre-test at the 6th grade and then as a follow up at the 9th and 12th grades (to provide for value-added judgments)      b) a criterion-referenced assessment that is built on clear, consistently applied quality-norms that are derived from a rich and substantive concept of critical thinking (to provide for the measuring of national progress)

What, Specifically, Are the Dangers of a Non-Substantive Concept of Critical Thinking?

It is important to be alert to the dangers posed by a non-substantive concept of critical thinking. Such a concept exists when, separate from a consideration of the research in the field, a person or institution presupposes a) that the meaning or terminology of critical thinking is intuitively obvious (hence not in need of scholarly analysis), or b) that each concept underlying critical thinking (such as assumption, inference, implication, reasoning . . . ) can be analyzed separately from a theory that accounts for the interrelation of these concepts, or c) that the skills of critical thinking can be adequately cultivated without reference to the values, traits of mind, and dispositions that underlie those skills.

There Are At Least Three Serious Problems That May Result From the Use of a Theoretically Superficial Concept of Critical Thinking

    a) important critical thinking concepts, which must be clearly defined to be used effectively in assessment, may be used vaguely,  inconsistently, incorrectly, or misleadingly       b) a false, misleading, or simplistic over-arching concept of critical thinking may be fostered     c) an unrealistic strategy for the assessment and cultivation of critical thinking may be incorporated into testing and teaching

Many examples of the unwitting use of a non-substantive concept of critical thinking could be cited — such as “thinking skills” programs devoid of intellectual standards (which, for example, systematically confuse “inferences” with “valid inferences” and “analogies” with “sound analogies”), or testing personnel who lack adequate grounding in critical thinking theory (and so, for example, frequently confuse assumptions with inferences or inferences with implications). The most far-reaching danger occurs when influential educational systems or institutions, like state departments of education, inadvertently incorporate a non-substantive concept of critical thinking into statewide curriculum standards or into statewide testing programs. This can result in significant, unintended negative consequences, for example: thousands of teachers encouraged to follow a misconceived model for the assessment of reasoning, leading to mis-instruction on a grand scale.

ILLUSTRATION: THE CALIFORNIA DIRECT WRITING ASSESSMENT

We shall look at one important case. Unfortunately, given the brevity of this paper, one case must stand for all. The case we have chosen concerns the Integrated Language Arts Assessment of the California Assessment Program, a massive statewide program that has impact not only on every student in the public schools of California, but also, because of the leadership role of California in assessment, on national teaching and testing practices as well. It appears that three fundamental mistakes occurred in the design of the direct writing assessment:

Consider Figure 1 and Figure 2 used as illustrations of the nature and quality of the writing assessment program in an article authorized and developed by the staff of the California Assessment Program. It is entitled “California: The State of Assessment” and was written for an important national anthology, Developing Minds (more than 150,000 copies disseminated by ASCD). The show-piece article, in which these figures occur, argues that the examples illustrate a “state-of-the-art teacher-developed writing assessment” that is sophisticated in “its testing, scoring, and reporting systems” and designed to “include only those tasks that will stimulate high-quality instruction.”

There are a number of problems illustrated in these figures that a substantive understanding of critical thinking would have avoided:

• A description of subjective reactions was systematically confused with sound evaluative reasoning. It is important to distinguish questions like, “Is rock music good music?” or “Does rock music excel as a form of music?” (which call for objective evaluation) from questions like, “Do you enjoy rock music?” or “Does rock music stir powerful emotions in you?” (which call, not for reasoning, but for the description of subjective reactions). Apparently the test developers were unclear about this distinction.
• The assessing teachers did not notice that the student failed to respond to the directions. The student did not develop evaluative reasoning, did not support his judgment with reasons and evidence, did not consider possible criteria on which to base his judgment, did not analyze the subject in the light of the criteria, and did not select evidence that clearly supported his judgment. Instead the student described an emotional exchange, asserted — without evidence — some questionable claims, and expressed a variety of subjective preferences (a fuller critique of the student essay is available on page 170 of, “Why Students — and Teachers — Don’t Reason Well”). The assessing teachers were apparently too confused about the nature of evaluative reasoning or the basic notions of criteria, evidence, reasons, and well-supported judgment to notice the discrepancy.
• The California State Department of Education assessment staff did not notice these errors once they were made. Instead of catching the errors once made, the California Department of Education chose to use the mis-graded student essay as a showcase model to disseminate nationally as illustrating “exceptional achievement” in reasoned evaluation, and as a model of their assessment of reasoned writing. We conclude that the California Assessment Program does not use scholars with a background in critical thinking research, any of whom would surely have recognized the problem.

Fundamental misconceptions of the nature of critical thinking and reasoned discourse, such as those documented above, must not be replicated in a national assessment program. Steps should be taken to insure that a substantive concept of critical thinking and a well-supervised implementation of that concept form the basis of the finished assessment program.

Section Three: The Four Domains of Critical Thinking

What are the four component domains of critical thinking and their implications for the assessment of higher order thinking?

Elements of Thought

As soon as we move from thought which is purely associational and undisciplined, to thinking which is conceptual and inferential, thinking which attempts in some intelligible way to figure something out, to use the power of reason, then it is helpful to think about what can be called “the elements of thought.” The elements of thought are the basic building blocks of thinking, essential dimensions of reasoning whenever and wherever it occurs. Working together, they shape reasoning and provide a general logic to reason. We can articulate these elements by paying close attention to what is implicit in the attempt on the part of the mind to figure anything out whatsoever. Once we make them clear, it will be obvious that each of them can serve as an important touchstone or point of assessment in critical analysis and in the assessment of thinking.

For each of the elements of thought there is a cluster of attendant basic thinking skills. Because they involve fundamental structures of thought, these skills can be characterized as micro-skills, those skills out of which larger-domained critical thinking abilities are built. Being able to think critically about a particular issue, then, will include the ability to identify, clarify, and argue for and against alternative formulations of the elements of thought.

The basic conditions implicit whenever we gather, conceptualize, apply, analyze, synthesize, or evaluate information the elements of thought are as follows:

• Purpose, Goal, or End in View. Whenever we reason, we reason to some end, to achieve some objective, to satisfy some desire or fulfill some need. One source of problems in reasoning is traceable to defects at the level of goal, purpose, or end. If the goal is unrealistic, for example, or contradictory to other goals we have, confused or muddled in some way, then the reasoning used to achieve it is problematic. An assessment of critical thinking, then, would test, at the appropriate educational level, skills of being able to state an author’s purpose, to identify a plausible statement of an author’s goals from a list provided, to rank formulations of an author’s objectives according to which are more or less reasonable in light of a particular passage, to distinguish clearly between purposes, consequences, assumptions, and other elements of thought.
• Question at Issue, or Problem to be Solved. Whenever we attempt to reason something out, there is at least one question at issue, at least one problem to be solved. One area of concern for reasoners, therefore, will be the formulation of the question to be answered or problem to be solved, whether with respect to their own reasoning or to that of others. Assessing skills of mastery of this element of thought would test students’ ability to formulate a problem in a clear and relevant way, to choose from among alternative formulations, to discuss the merits of different versions of the question at issue, to recognize key common elements in statements of different problems, to structure the articulation of problems so as to make possible lines of solution more apparent.
• Point of View, or Frame of Reference. Whenever we reason, we must reason within some point of view or frame of reference. Any “defect” in that point of view or frame of reference is a possible source of problems in the reasoning. A point of view may be too narrow, too parochial, may be based on false or misleading analogies or metaphors, may contain contradictions, and so forth. Levels of skill here would be tested with reference to being able to enunciate an author’s point of view in a passage, to adjudicate between different statements of that point of view, to recognize bias, narrowness, and contradictions when they occur in the point of view, to recognize relations between the frame of reference being used and its implications, assumptions, and main concepts.
• The Empirical Dimension of Reasoning. Whenever we reason, there is some “stuff”, some phenomena about which we are reasoning. Any “defect”, then, in the experiences, data, evidence, or raw material upon which a person’s reasoning is based is a possible source of problems. Students would be tested, again, based on their level, on their ability to distinguish evidence from conclusions based on that evidence, to give evidence themselves, to identify from a pre-selected list data that would support an author’s positions, data that would oppose it, data that would be neutral, to notice the presence or lack of relevant evidence, to recognize, to be intellectually courageous in recognizing (and labeling as such) mere speculation that goes beyond the evidence.
• The Conceptual Dimension of Reasoning. All reasoning uses some ideas or concepts and not others. These concepts can include the theories, principles, axioms and rules implicit in our reasoning. Any “defect” in the concepts or ideas of the reasoning is a possible source of problems. The assessment of the relevant higher order thinking would test the ability to identify main concepts of a passage, to choose among different versions of those concepts (some perhaps equally good), to see relations among concepts, to reason about the similarity of points of view on the basis of similarity of fundamental concepts, to distinguish central from peripheral concepts, derived concepts from basic concepts, to see the implications of using one concept rather than another.
• Assumptions. All reasoning must begin somewhere, must take some things for granted. Any “defect” in the assumptions or presuppositions with which the reasoning begins is a possible source of problems. Assessing skills of reasoning about assumptions would test the ability to identify assumptions underlying given inferences, points of view, and goals, to evaluate the accuracy of different formulations of the assumptions, to distinguish between assumptions and inferences, to rank assumptions with respect to their plausibility, to be intellectually fair-minded by choosing the most plausible version of assumptions underlying points of view with which they disagree.
• Implications and Consequences. No matter where we stop our reasoning, it will always have further implications and consequences. As reasoning develops, statements will logically be entailed by it. Any “defect” in the implications or consequences of our reasoning is a possible source of problems. Skills to be assessed would include the ability to identify important implications, to do so by selecting from a list of possible implications, to make fine discriminations among necessary, probable, and improbable consequences, to distinguish between implications and assumptions, to recognize the weakness of an author’s position as shown by the implausibility of its implications, to exercise intellectual fair-mindedness in discriminating between the likelihood of dire and mild consequences of an action to which one is opposed.
• Inferences. Reasoning proceeds by steps in which we reason as follows: “Because this is so, that also is so (or probably so),” or “Since this, therefore that.” Any “defect” in such inferences is a possible problem in our reasoning.

Assessment would test, in a way geared to their educational level, students’ ability to recognize faulty and justified inferences in a passage, to rank inferences with respect to both their plausibility and their relevance, to make good inferences in their own reasoning, to discriminate among various formulations of an author’s inferences with respect to which is most accurate, to take something they do not believe but to entertain it for the sake of argument and draw reasonable inferences from it.

Assessment of Elements of Thought Any program for the assessment of critical thinking skills must itself be assessed in terms of its validity and reliability in testing for the ability to think about, and in terms of, the elements of thought. These abilities can be successfully assessed in three related ways: by a restricted use of standard multiple-choice items, by multiple-rating items, and by short essay items. Both multiple-choice and multiple-rating items are machine-gradable, while essay items are not.

Although our recommendations about the content of the assessment will be spelled out in detail in Section Four, some of these can be anticipated here with respect to the assessment of reasoning abilities centering on the elements of thought. Multiple-choice testing (as in the existing Watson-Glaser Critical Thinking Appraisal or the Cornell Critical Thinking Tests) is an important part of an assessment of critical thinking, but its legitimate use is restricted to testing only the most basic skills of identifying and recognizing elements of thought, and then only as they occur in relatively short and unambiguous excerpts.

Within this domain, multiple-choice questions will require students:

• to identify an author’s purpose in a passage;
• to rate selected inferences as justified, probably true, insufficiently evidenced, probably false, unjustified;
• to select among formulations of the problem at issue in a passage those that are clearly reasonable, probably reasonable, probably unreasonable, clearly unreasonable;
• to recognize unstated assumptions;
• to distinguish evidence from hypotheses and conclusions;
• to rate described evidence as reliable, probably reliable, probably not reliable, unreliable.

The elements of thought do not exist in isolation from one another, nor — more importantly for the concept of an assessment procedure — do they exist outside a particular context of application. In the practice of good critical thinking, skills more closely associated with elements of thought are orchestrated into larger-domained abilities which are applied to thinking about complex and sometimes ambiguous issues, problems, decisions, theories, states of affairs, social institutions, and human artifacts. These critical thinking abilities include being skillful at:

• refining generalizations and avoiding over-simplifications,
• comparing analogous situations: transferring insights into new contexts,
• developing one’s perspective: creating or exploring the implications of beliefs, arguments, or theories,
• clarifying issues, conclusions, or beliefs,
• clarifying and analyzing the meanings of words and phrases,
• developing criteria for evaluation: clarifying values and standards,
• evaluating the credibility of sources of information,
• questioning deeply: raising and pursuing root or significant questions,
• analyzing or evaluating arguments, interpretations, beliefs, or theories,
• generating or assessing solutions,
• analyzing or evaluating actions or policies,
• reasoning dialogically: comparing perspectives, interpretations, or theories,
• reasoning dialectically: evaluating perspectives, interpretations, or theories,
• reading critically: constructing an accurate interpretation of, understanding the elements of thought in, and evaluating, the reasoning of a text,
• listening critically: constructing an accurate interpretation of, understanding the elements of thought in, and evaluating, the reasoning of an oral communication,
• writing critically: creating, developing, clarifying, and conveying, in written form, the logic of one’s thinking,
• speaking critically: creating, developing, clarifying, and conveying, in spoken form, the logic of one’s thinking.

Abilities like these play a central role in a rich and substantive concept of critical thinking. They are essential to approaching actual issues, problems, and situations rationally. Understanding the rights and duties of citizenship, for example, requires that one at least have the ability to compare perspectives and interpretations, to read and listen critically, to analyze and evaluate policies. In fact, there is no macro-ability on the list that would not be relevant or even crucial to thinking deeply about the rights and duties of citizenship.

Similarly, the capacity to make sound decisions, to participate knowledgeably in the work-place, to function as part of a global economy, to master the content in anything as complex as the academic disciplines, to apply those subject area insights to real-life situations, to make insightful cross-disciplinary connections, to communicate effectively — each of these relies in a fundamental way on having a significant number of the abilities listed. Take, for example, the capacity to make sound decisions: such decision-making is hardly possible without an attendant ability to (going down the list of abilities in order) refine generalizations, compare analogous situations, develop one’s perspective, clarify issues, and so forth.

The last four abilities listed — the ability to read, write, listen, and speak, each in a critical, informed, constructive way — are best considered not as in the usual model, not as manifestations of thinking already accomplished, but as being themselves actual modes of constructive thinking. As such, they are structured amalgams of elementary skills together with any number of other abilities.

Assessment of Abilities The assessment of abilities, too often neglected, is essential to assessment of critical thinking. Since these are the abilities implicit in the realistic use of thinking, no assessment tool that fails to assess a significant number of these abilities could justifiably be called an assessment of higher order thinking. The assessment, moreover, needs to address such abilities directly (rather than through secondary indicators), systematically (rather than haphazardly as a result of an attempt to assess other variables like academic achievement), and in settings as authentic as possible given the requirement of uniform, relevant grading. Assessment of abilities that meets these four criteria cannot be accomplished within the confines of a standard multiple-choice-type test. It can be accomplished, however, for all of the abilities (except those having to do with oral communication), by means of a combination of machine-gradable multiple-rating items and essay items.

For any macro-ability, there will be dimensions of the ability that are generative and other dimensions of it that are selective. In trying to solve a real problem, for example, much of one’s thinking is devoted to generating a formulation of the problem that will make it more susceptible to solution. Another, and quite different, aspect of problem solving, is the ability to select, from among a large variety of possibilities, that avenue of thought which will most likely result in a solution. Students who are trained using a rich, substantive concept of critical thinking tend to improve in both dimensions of this ability, and both are genuine dimensions of real problem-solving.

The selective dimensions of an ability can be assessed accurately, even in complex, ambiguous, and subtle cases, using multiple-rating items. The generative dimension, on the other hand, cannot. Since it requires students to come up with their own critical thinking approaches within that macro-ability, this dimension can be assessed adequately only by carefully constructed and carefully graded essay tests. Details of the assessment and samples of assessment items will be presented in Section Four.

Affective Dimensions

Higher order thinking requires more than higher order thinking skills. Critical thinking, in any substantive sense, includes more than abilities. The concept also includes, in a crucial way, certain attitudes, dispositions, passions, traits of mind. These affective dimensions are not merely important to critical thinking; they are essential to the effective use of higher order thinking in real settings. These affective dimensions include:

• thinking independently,
• exercising fair-mindedness,
• developing insight into egocentricity and socio-centricity,
• developing intellectual humility and suspending judgment,
• developing intellectual courage,
• developing intellectual good faith and integrity,
• developing intellectual perseverance,
• developing confidence in reason,
• exploring thoughts underlying feelings and feelings underlying thoughts,
• developing intellectual curiosity.

Without intellectual perseverance, one could not solve the complicated, multi-faceted problems one confronts in industry. Without intellectual courage, one could not maintain a defense of citizenship rights in the face of scare tactics. Without fair-mindedness, one could not enter into another’s point of view and thus would lack that empathetic understanding necessary for a reasonable approach to living in a pluralistic society. Without developing insight into egocentricity and socio-centricity one could employ one’s reasoning skills in a merely self-serving and prejudiced way. Without confidence in reason one could not adequately address those complex and frequently ambiguous real-life problems that require reasonable decisions in the face of crucial uncertainties.

Assessment of Affective Dimensions The assessment of affective dimensions of critical thinking is an important part of an assessment of higher order thinking. An initial problem is that from the fact that all these dimensions are essential, it does not follow that all are directly testable, nor does it follow that any of them is easily testable. For some of these affective dimensions (intellectual perseverance, for example), any testing would have to take place over an appropriately long period of time and thus could not be legitimately assessed at all during a time frame suitable for a national test. Nevertheless, a number of affective dimensions can be assessed in a relatively straightforward way using essay items and, especially, machine-gradable multiple-rating items.

“Reasoning Within Conflicting Points of View,” a central aspect of the disposition of fair-mindedness, is already being assessed on a machine scorable test developed by Dr. Richard Paul. This Appraisal asks students to select the strongest (that is, the most defensible) argument in favor of each side of a pair of conflicting and sometimes emotionally charged points of view. Proficiency on these items indicates a fair-minded willingness to distinguish the concept of reasonable defensibility from that of personal belief.

Multiple-rating items are currently being prepared that address aspects of intellectual courage, other aspects of fair-mindedness, aspects of intellectual humility, and aspects of the development of insight into one’s own egocentricity and socio-centricity.

Intellectual Standards

In any domain where assessment is taking place, there are standards implicit in that assessment. Higher order thinking is thinking that meets universal intellectual standards. Thus, when assessing a student’s ability to compare and evaluate perspectives (a macro-ability) and to do so with fair-mindedness (a trait of mind), we would judge whether she had made such evaluations in a relevant and consistent way, with attention to accuracy, fairness, and completeness in describing each perspective, and with a sensitivity to the degree of precision appropriate to the topic.

We would assess critical thinking about and in terms of the elements of thought in very much the same way: to judge a person’s skill at recognizing the frame of reference underlying a position, we would want to judge whether she could see relevant alternatives, whether the frame of reference she identified fits the available evidence, whether her answer was deep or merely mechanical, clear or vague, fair or biased. Intellectual standards apply to thinking in every subject. The process of learning to teach so as to foster critical thinking is the very process by means of which one establishes intellectual standards for assessing thinking, and, by extension, for assessing instruction itself. Such standards are more useful if they are made explicit — to the students who are taking the test, to those doing the assessing, and to classroom teachers. Making standards explicit benefits student test-takers because they can then see that there are standards, that the standards are not arbitrary, and that understanding the standards gives them insight into what good critical thinking is. It benefits those doing the assessing because, in addition to the reasons already mentioned, it fosters both uniformity in grading and a strong correlation between the grade and the skills being graded. Judging a response by how clearly and completely it states a position, for example, is using a critical thinking standard and dictates a certain level of assessment; judging a response by how concisely or how elegantly it states a position, on the other hand, is using a standard that is inappropriate to critical thinking assessment. Explicit standards — part of a rich and substantive concept of critical thinking — might have avoided at least some of the mistaken assessment on the California Assessment Program, cited earlier. Thus, making standards explicit promotes both the reliability and the validity of the assessment. Finally, it benefits classroom teachers because such standards can readily be built into classroom instruction. The standards, after all, are those implicit in teaching for higher order thinking; they are therefore invaluable both for teachers to use explicitly with their classes and — an essential feature of critical thinking-internalized — for students to learn to use as part of assessing themselves.

Section Four:

Recommendations of the center for critical thinking and moral critique.

What is the most workable solution to the design of a process to assess higher order thinking? In this section we will 1) briefly survey existing assessment tools; 2) make recommendations regarding the substance and format of a national assessment tool — the critical thinking domains to be assessed, the varieties of assessment strategies to be used (including sample test items), and the dual interdisciplinary and intradisciplinary scope of the assessment — 3) appraise the value of the proposed assessment strategy for the reform of instruction, and 4) make recommendations regarding the implementation of the assessment.

Existing Assessment Tools There are limitations in all twelve of the commercially available critical thinking tests as instruments for assessing higher order thinking:

Cornell Class Reasoning Test, Form X (1964) Cornell Conditional Reasoning Test, Form X (1964) Cornell Critical Thinking Test, Level X (1985) Cornell Critical Thinking Test, Level Z (1985) The Ennis–Weir Critical Thinking Essay Test (1985) Judgment: Deductive Logic and Assumption Recognition (1971) Logical Reasoning (1955) New Jersey Test of Reasoning Skills (1983) Ross Test of Higher Cognitive Processes (1976) Test on Appraising Observations (1983) Test of Enquiry Skills (1979) Watson–Glaser Critical Thinking Appraisal (1980)

In addition there are limitations in all of the other available “higher studies” tests which might be taken as a possible model for the assessing of higher order thinking: the SAT, LSAT, the Test of Academic Aptitude (British), ACT, the Graduate Record Exam, the Commonwealth Secondary Scholarships Exam (Australia). We do not have the space here to review each of these tests one-by-one. Instead we will summarize the general situation as we see it.

Though aspects and dimensions of critical thinking are tested, some more and some less, in all of the above tests, none has been designed with the 21 criteria in Sections one and two in mind. Most importantly, none was designed to serve as a national assessment tool which establishes national standards in higher order thinking and as a motivation for and guide to instruction.

Behind none of these tests was there a comprehensive model for the elements of thought, the abilities of critical thinking, or the affective dispositions (as we have here provided). The relative recentness of the bulk of scholarship in critical thinking makes it unlikely that long-established tests will fill the bill. Of course any new test for assessing higher order thinking should be based on a thorough review of established test strategies to incorporate those with significant application.

Given the need for assessment on the basis of a rich and substantive concept of critical thinking, there are two areas where competing values and objectives come into play. The first concerns the substance and format of the test itself: Which domains exactly are to be covered, and with what emphases? What kinds of question will be asked? Will it include both interdisciplinary and intradisciplinary items? What kind of assessment questions best test for skills of citizenship and the challenges of the work place? The second area concerns the implementation of the test and how it is conceived: Should it be value-added or simply criterion-referenced? Who will do the assessing and who will be assessed? How much will the assessment cost and who will pay for it? How often will the test be given? Some of these are difficult questions, with genuine values and goals on different sides, where reasonable cases can be made for more than one position. Others of these questions are clearer, especially once the objectives of the test as a whole are brought into focus.

Substance and Format

The overall recommendations of the Center for Critical Thinking are set forward below.

1) DOMAINS TO BE ASSESSED The national assessment of higher order thinking must test for a rich and substantive concept of critical thinking, and this testing must be geared to assessment within all four domains of critical thinking.

A) ELEMENTS OF THOUGHT Skills of identifying, explicating, and using the elements of thought need to be assessed. They are necessary for any of the abilities to be employed with precision, depth, or accuracy. They are required if essential affective traits are to be rooted in solid, locatable, intellectual skills and the concepts they presuppose. Lack of a solid grounding in these skills, and the concepts behind them, results in thinking which, good intentions notwithstanding, is far removed from the close, careful reasoning demanded by the rigors of higher order thinking. Among testing personnel, lack of the informed use of these concepts is part of what results in such poor assessment tools and grading as we found in the California Direct Writing Assessment.

Critical thinking in students requires them to be able to perform well, with an expertise appropriate to their grade level, on items testing a list of skills that center around the elements of thought:

• identify a plausible statement of a writer’s purpose;
• rank formulations of an author’s objectives;
• distinguish clearly between purposes, consequences, assumptions, and inferences;
• choose the most reasonable statement of the problem an author is addressing;
• discuss reasonably the merits of different versions of the question at issue;
• recognize key common elements in formulations of different problems;
• give a clear articulation of an author’s point of view;
• identify the most reasonable statement of an author’s point of view;
• recognize bias, narrowness, and contradictions in the point of view behind an excerpt;
• identify assumptions and implications of a writer’s point of view;
• distinguish evidence from conclusions based on that evidence;
• give evidence to back up their position in an essay;
• recognize data that would support, data that would oppose, and data that would be neutral with respect to, an author’s position;
• recognize conclusions that go beyond the evidence;
• note, in an evaluative essay, the presence, or the absence, of evidence in an excerpt;
• identify the main concepts in a passage;
• distinguish central from peripheral concepts;
• identify the assumption underlying a given inference;
• evaluate the aptness of different versions of an assumption;
• choose the most reasonable statement of a background theory involved in a passage;
• distinguish between inferences and assumptions;
• rank different formulations of assumptions with respect to which is the most reasonable;
• identify crucial implications of a passage;
• discriminate between consequences that are necessary, probable, and improbable;
• evaluate an author’s inferences;
• make, in an evaluative essay, justified inferences;
• choose the most accurate version of an author’s inferences;
• draw reasonable inferences from positions they disagree with.

B) ABILITIES Abilities, grounded in a thorough familiarity with the elements of thought, are the activities we actually use to perform our higher order thinking. Abilities like clarifying values and standards, comparing analogous situations, generating and assessing solutions, analyzing and evaluating actions or policies are the stuff of reasoning. They are the means whereby decisions are to be made, problems are to be solved, thinking in the work-place is to be strengthened, and understanding of rights and responsibilities deepened.

The abilities of critical reading and critical writing are keystones of any process to assess higher order thinking in that each of them, when considered at any level, is permeated by other critical thinking abilities. It is not as if we read and clarify values, read and compare analogous situations, write and generate solutions. To read critically is to clarify values, compare analogous situations, and to exercise the other abilities as well; to write is to generate solutions and much more besides. Assessment of proficiency in the abilities can be keyed to student performance on test items that are geared to as many of the abilities listed on p. 127 as is feasible given the time constraints of the test.

C) AFFECTIVE TRAITS Without assessing affective traits, only a diminished idea of critical thinking will be addressed. What allows us to confront our prejudices and analytically break them down is not just abilities but a commitment to use them for this purpose. What allows us to solve our problems in a sufficiently diligent way as to address complicated and intricate real-life problems, is again not just cognitive abilities. It is intellectual perseverance — a drive, a disposition, an affective trait. A similar point can be made for each of the intellectual traits which are the driving force behind sound and penetrating reasoning.

Assessment of the affective dimensions will concentrate on those aspects it is plausible to test for within the constraints imposed by a national assessment. These will include aspects of fair-mindedness, of the willingness to suspend judgment, of intellectual courage and intellectual integrity.

D) INTELLECTUAL STANDARDS Assessment has to involve explicit universal standards. If we are not testing students’ abilities to be relevant, precise, logical, consistent, and the rest, then we are not assessing students’ abilities to engage in higher order thinking. And if testing personnel do not employ these same explicit standards, then they are grading for something other than higher order thinking.

Relative mastery of these intellectual standards requires students to be able to

• recognize clarity vs. unclear;
• distinguish accurate from inaccurate accounts;
• decide when a statement is relevant or irrelevant to a given point;
• identify inconsistent positions as well as (relatively) consistent ones;
• discriminate deep, complete, and significant accounts from those that are superficial, fragmentary, and trivial;
• evaluate responses with respect to their fairness;
• prefer well-evidenced accounts to accounts that are unsupported by evidence;
• tell good reasons from bad.

2) VARIETIES OF ASSESSMENT STRATEGIES The assessment should contain three kinds of items: a) machine-gradable multiple-choice items; b) machine-gradable multiple-rating items; c) essay items.

A) MULTIPLE-CHOICE ITEMS Legitimate use of multiple-choice items on the assessment is limited. This type of item is geared toward relatively straightforward skills of reasoning, particularly with respect to recognizing elements of thought, distinguishing one element of thought from another, and recognizing clear examples of faulty reasoning.

Two detailed samples of assessment items follow (the first, Figure 3, is on Inferences, the second, Figure 4, on Recognition of Assumptions).

Other abbreviated samples of appropriate multiple-choice items are as follows:

• In the following excerpt, mark E for each item that is a piece of empirical evidence; mark C for each item that is a conclusion based on evidence; mark N for each item that is neither.…
• In this test, each exercise consists of several statements (premises) followed by several suggested conclusions…. If you think the conclusion necessarily follows from the statements given, make a heavy black mark under “Conclusion Follows”; if you think it is not a necessary conclusion, put a mark under “Conclusion Does Not Follow.”
• The following is a list of possible findings in relation to the experiment quoted above. For each, say whether it would support the author’s hypothesis, oppose the author’s hypothesis, or be neutral with respect to the author’s hypothesis….
• Below is a series of questions. Each question is followed by several reasons. For the purpose of this test, you are to regard each reason as true. The problem then is to decide whether it is a strong reason or a weak reason….
• Which of the following conclusions is C completely supported by the stated evidence, P partially supported by the stated evidence, or U unsupported by the stated evidence?
• Which of the following is an implication of the author’s position in the passage cited?

B) MULTIPLE-RATING ITEMS Though the use of multiple-choice questions is justified in assessing some micro-skills, the bulk of the machine-gradable items will be multiple rating rather than multiple-choice. Multiple-rating items allow one to ask questions where any number of answers from a provided list may be correct, or incorrect. It further allows students to rank, from a number of possibilities provided, those that are more correct. Thus students can be tested on their ability to arrange items on a continuum of reasonability. This allows much more subtle testing and grading.

The same list of possible answers can pertain to any number of independent test items. Thus, a list of twenty possibilities can be provided, and students can be asked to choose the appropriate response from that list to six different questions. There is no restriction on the number of times a given answer may be correct. Nor is there any guarantee that there will be a reasonable answer on the list to every question. Guessing, using the process of elimination, and scoring well because of test-taking skills are all but impossible.

By including clearly unreasonable choices among the multiple-rating possibilities, a grade can be much more sensitive to the degree of a macro-ability or to the intensity of an affective dimension. Thus, if there are five possible answers to a given question, they need not be graded 5, 4, 3, 2, 1. Rather, they may be graded, say, 5, 4, 1, 1, -3.

We have provided two detailed samples of multiple-rating items: Figure 5 is on Reasoning Within Conflicting Points of View (and thus is an assessment of an aspect of the affective trait of fair-mindedness) and Figure 6 is on Comparing Analogous Situations (and is thus an assessment of a macro-ability). Each sample is limited here by having only four possible answers, a limitation that would not obtain on an actual test.

The following is a list of abbreviated samples of multiple-rating items, having to do with elements of thought, with abilities, with affective dimensions, and with intellectual standards.

MULTIPLE-RATING ITEMS, ELEMENTS OF THOUGHT

• Here is a list of formulations of the writer’s objectives in this excerpt. Rank them from 1 to 5 with respect to which is the most reasonable in the light of the quoted passage….
• For each of the underlined passages in the excerpts below, mark P on the answer sheet if it is a statement of the writer’s Purpose, C if it is a statement of the Consequences, A if it is a statement of the writer’s Assumptions, and I if it is an Inference the writer is making.
• Which of the following would the author most likely give as the statement of the problem she is attempting to solve?
• Read the excerpt; then, from the following list, identify the most plausible statement of the writer’s purpose.
• Of the following statements of the author’s point of view in this passage, select the one from the following list that is both most reasonable and most relevant to the passage….
• List A below is a list of various possible statements of the writer’s point of view in the quoted passage; List B is a list that includes possible assumptions and implications of those points of view. Match the items on list A with the items on list B…
• Which of the following are main concepts in the passage cited; which are peripheral concepts?
• For each inference below, decide whether the accompanying statement is U an unstated assumption, A an assertion, or N neither…
• Rank the following items on a scale of 1 to 5 according to how reasonable it is as a statement of the author’s assumptions…
• Look at each of the statements below as a possible consequence of the writer’s position in the excerpt cited. Rank each statement on a scale of 1 to 7, where 7 means that you consider the statement a necessary consequence of the passage, and 1 means that you consider the statement a highly unlikely consequence of the passage.
• Each of the following is an inference one might draw from the passage. Rank each one on a scale from 1 to 5, according to whether it is completely justified (5) or completely unjustified (1)…
• Which of the following is the most accurate formulation of the author’s inference in the cited passage?

MULTIPLE-RATING ITEMS, ABILITIES

• Which of the following would be relevant to deciding whether A is a credible source of information on the topic…?
• Here is a list of observations about the behavior of X’s, made by a responsible investigator. Which of the items from the following list would be a justified generalization about X’s?
• A has the following beliefs about astrology. Which of the questions below would be root or significant questions that A would have to answer to claim his beliefs about astrology were rational?
• A refuses to refund a customer’s money and, when asked, defends her action by stating that it is “dictated by store policy”. Which of the following would be relevant to deciding whether her action was indeed “dictated by store policy”? Which of the questions would be relevant to deciding if the store policy was rational?
• Judge A makes the following ruling in a case… Which of the following is the clearest statement of the standards Judge A is using?
• A compares the relation between managers and employees to the relation between teachers and students. Which of the following would A have to answer in order to continue using the analogy rationally?
• A gives the following argument for…. Which of the listed comments would be the strongest objection to her argument?
• Listen to the accompanying excerpt from an audiotape of a lecture by A. Which of the following questions would be of most help in clarifying A’s views?

MULTIPLE-RATING ITEMS, AFFECTIVE TRAITS

• Here are position-statements from both sides, A and B, of a controversial and inflammatory debate. From list X below, choose those items which are the most reasonable inferences to draw from position A; then choose those items which are the most reasonable inferences to draw from position B.
• Here are position-statements from both sides, A and B, of a controversial and inflammatory debate. From list X below, choose those items which state the most reasonable assumptions underlying position A; then choose those items which state the most reasonable assumptions underlying position B.
• For each of the items below, tell which is the most reasonable action to take under the circumstances described. If, in your view, there is not enough information to make a reasonable decision, you may choose the action of suspending judgment as the most reasonable response.
• A disposition to take a measured response rather than an exaggerated, disproportionate response will be measured by requiring students to discriminate between the likelihood of dire versus mild consequences of positions they dislike.

MULTIPLE-RATING ITEMS, INTELLECTUAL STANDARDS

• The following are four definitions from Webster’s New World Dictionary. Which of them gives the clearest definition of…? • Rank the following definitions for their precision on a scale of 1 to 7. 1 means “not precise at all”; 7 means “too precise for the subject matter”; and 4 means “exactly as precise as it should be”. • Here is a list of data and a series of accounts summarizing the data. Which of the accounts is the most accurate summary of the data? • For each statement below, tell whether it is relevant or irrelevant to the hypothesis in the passage cited. • Which of the following is the fairest restatement of the author’s position [where the author is stating a highly controversial position]? • Rank the following statements according to which are the best-evidenced and which are the least-evidenced. • Which of the following is a good reason for believing the statement in question? Which is a bad reason? Which is somewhere in the middle?

C) ESSAY ITEMS

The full range of the use of critical thinking cannot be assessed without requiring writing on the part of the student. To confront real issues, balance competing interests, weigh objections and alternatives, and make a reasonable decision about a matter of some consequence — this is a major part of what it is to think critically. The ability and the disposition to engage in full-fledged critical thinking is measured only in part by a person’s ability to choose from among a pre-selected list. A true measure of critical thinking, and thus of a program’s capacity to improve critical thinking, can be obtained only by including in the assessment generative as well as selective dimensions. Neither multiple-rating nor, obviously, multiple-choice items are adequate for testing this dimension.

Essay items will require proficiency in handling the elements of thought, in using appropriate abilities, in applying intellectual standards, and, what is more, it will require integrating these and bringing them to bear on a substantive issue. Three detailed samples of essay items follow on the next page. Each has the same set of general directions.

In addition to full-blown essay tests, a series of short-justification items are currently being prepared. These would not ask students to write an essay on a topic, but would rather have them choose an answer from a pre-selected multiple-rating list and then justify their answer in a sentence of their own writing.

This type of test, if it were sufficiently developed, would have several advantages: it could be administered, because of the brevity and straightforwardness of students’ written answers, to the student population as a whole rather than merely to a representative sample (see #1, under “Implementation”, below); it would assess some, though not all, generative dimensions of critical thinking; it would allow flexibility in grading the machine-gradable keyed answers (thus, one could adjust the rating of an item up or down depending on the justification); it would be no more difficult to grade by trained personnel than the math work on currently administered standardized calculus tests.

Interdisciplinary and Subject-Specific

SCOPE OF THE ASSESSMENT

An assessment of the results of critical thinking instruction ought to focus both on thinking within the framework of particular academic subjects, and on thinking in the interdisciplinary contexts that are so important to functioning as an autonomous, well-informed, productive member of a democracy.

A basic principle of critical thinking instruction, as applied to teaching subject matter in an area, is that (to quote the National Council for Excellence in Critical Thinking Instruction) “to achieve knowledge in any domain, it is essential to think critically”. A related principle is that in any domain where one is thinking well, one is thinking critically. Any example of good scientific thinking, or good historical thinking, or good anthropological thinking, or thinking in any other subject, will necessarily be an example of critical thinking: It will involve basic skills dealing with elements of thought; it will involve at least some, and probably many, of the abilities; it will involve affective traits like independent thinking and intellectual perseverance. And as far as instruction is concerned, there is a real sense in which learning biology is learning to think within and about the logic of biology.

Including critical thinking items taken from individual subject areas would also properly test those thinking skills that are more subject-specific, and it would do so in the context of presupposing a good deal of specialized knowledge. A critical thinking test in nursing or in history of art or in geology might well (in their different ways) test for skills of critical observation, while a test in sociology might assess thinking skills involved in constructing an unbiased questionnaire; a critical thinking test in English literature might well presuppose a knowledge of who Milton was, while a thinking test in physics might justifiably ask about a problem for which a knowledge of the second law of thermodynamics was taken for granted.

Even if we already had a series of critical thinking items within the various subject areas, however, we would not be testing for many of the interdisciplinary abilities we most want critical thinking for. Many of these have already been mentioned: the ability to make sound decisions in the context of understanding our rights and responsibilities as citizens, in the context of the work-place, as well-informed and thinking consumers, as members of our families, as participants in what is becoming a symbiotic and fragile world economy — the ability to reason about the gaps between subject areas, the bridges between them, and the generalizability of subjects to other areas.

To test critical thinking abilities, as they apply to these areas, what is needed are interdisciplinary questions. These are questions of broad interest, ones that shed light on the quality of and improvement in student thinking about realistic and fundamental issues; they ought to be the kind of questions that can be at least partially illuminated by well-integrated knowledge in any number of academic areas. The national assessment we are proposing would offer a range of subject-specific items, from which students would choose those relevant to their subject-matter knowledge. The interdisciplinary items, on the other hand, would not provide choices because of the desirability of avoiding the loss of equivalency that is almost always involved. (That loss would have to be minimized in the case of subject-specific items by field testing and rewriting.)

The interdisciplinary part is constructable by experts well versed in a rich and substantive concept of critical thinking. Subject-specific critical thinking assessment items will be constructed by members of the discipline working in consultation with experts in critical thinking, perhaps the standing committees on the various disciplines of the National Council for Excellence in Critical Thinking Instruction. Both groups would work in conjunction with grade-level experts to construct appropriate levels of items, from the 6th-grade test through the college-graduate test.

The Value of the Proposed Assessment Strategy for the Reform of Instruction Since higher order thinking has always been considered an important object of education, and since this assessment would furnish a measure of that concept, and since performance on this assessment would have a significant impact on the standing of the school not only in the eyes of the intellectual community but in the eyes of the public as well, administrators and teachers would have a strong motivation to become familiar with the concepts and program behind the assessment. Most importantly, teachers and others in charge of instruction and the formulation of educational goals would find in it a clear model for the articulation and integration of higher order thinking across the curriculum. Note the following:

• The concept of the elements of thought not only provides a realistic analysis of the common dimensions of reasoning in every domain, it also encourages the explicit use in instruction of those critical/analytic terms which are the common possession of the intellectual community (question-at-issue, problem, evidence, data, concept, inference, assumption, implication, conclusion, point of view, frame of reference, etc.) and makes explicit the intellectual standards implicit in every subject as well as in the closely reasoned professional work in business and industry (clarity, precision, accuracy, logic, consistency . . . )
• By highlighting reading, writing, speaking, and listening as modes of critical reasoning, the necessity of having instruction go beyond mere didactic coverage of content would become more intelligible. As long as reading, writing, speaking, and listening skills appear the sole province of specialized subjects and at specialized levels rather than modes of reasoning intrinsic to the construction and mastery of knowledge in any subject at any level, there will continue to be a significant lack of fit between modes of instruction and modes of necessary learning.
• By highlighting the other abilities of critical thinking, each analyzed into the same elements of thought, there would be significant transfer of emphasis to important modes of higher order thinking within a larger number of student assignments. At present, many teachers fail to notice the extent to which they either presuppose that students already grasp the nature of fundamental intellectual processes, or they make assignments which, though they appear to call for such processes, can be successfully completed by simply repeating to the teacher what was said in lecture or written in the text.
• By highlighting a common critical/analytic language across the curriculum, students are encouraged to seek to transfer learning and intellectual discipline emphasized in one domain of learning to other domains of learning and application. The fragmentation of the subject areas, in the minds of the students if not in fact, is now a serious problem in education. This problem is mirrored, of course, in business, industry, and government in the tendency to engage in fragmented, over-specialized problem solving which fails to address the multi-dimensional nature of many complex problems.
• By highlighting the importance of intellectual discipline and grounding it in specific skills and abilities, teachers and other educational leaders will be given a reasonable impetus to help students make connections of a broader, more interdisciplinary nature. This will also be strongly re-enforced by the inclusion of everyday, multi-logical, interdisciplinary essay questions.

Implementation of the Proposed Assessment

Our recommendations about implementation can be summarized as follows:

• The essay assessment should be administered to a representative sample of the student population at each educational institution, the machine-gradable items to the total student population;
• it should be administered at the 6th, 9th, and 12th grades, and three times during a student’s college career — at entrance, at the start of the junior year, and just prior to graduation — and thus yield value-added information to schools;
• the test should be constructed to be roughly three-hours long;
• test items should be constructed from item shells, rather than from a simple pool of actual items;
• it should be administered by a private agency with critical thinking credentials;
• it should be paid for by school districts, colleges, and universities that contract to have their students tested;
• it should provide educational institutions with detailed information about central aspects of their students’ higher order thinking;
• it should be developed according to the costs and timetables listed below.

Details of our recommendations center around the answers to five practical questions about the administration of the test:

WHO WILL BE ASSESSED? Our minimal recommendation is that all portions of the assessment be given to, at the very least, a representative sample of the student population at each educational institution. Since the problems implicit in testing a random sample can be easily worked out, this recommendation avoids the expense of administering an essay test to the student population as a whole.

The assessment strategies we have proposed include two broad areas of testing: a machine-gradable portion that includes multiple-choice items and multiple-rating items and an essay portion. Both portions will assess, in their different ways and with their different emphases, micro-skills, abilities, affective traits, and intellectual standards.

There are, therefore, really two options with respect to who is assessed using the strategies we propose. First, the machine gradable portion of the assessment can be administered to the student population as a whole, while the essay portion can be administered to a representative sample of students at each institution. Second, both portions could be given only to a representative sample of the population at each institution. Both options will hold down costs, though the latter will clearly be less expensive than the former. Which option is ultimately chosen will depend on the amount of detail desired, the precise role the assessment is to play, and the funds available.

HOW OFTEN WILL THE ASSESSMENT TAKE PLACE? The maximum benefit to educational institutions will be provided to the extent that they are enabled to measure the progress of their students’ higher order thinking during the course of their educational career. This will enable school systems not only to gauge their contribution to their students’ progress, but also to measure the success of attempts to re-design their instruction so as to increase critical thinking capabilities.

These objectives can be accomplished by having students assessed often enough to reflect such progress, optimally: at the 6th, 9th, and 12th grades, and at the time of their college entrance, at the beginning of their junior year, and just before graduation from college.

HOW LONG WILL THE TEST TAKE? The test should last about three hours in order to cover multiple-choice, multiple-rating, and essay items without becoming a speeded test to an inappropriate degree. To span all difficulty levels, it would be best to have a total of at least 30 items. While two of these could be short essay items requiring 20 minutes each to answer, the machine-gradable items would be faster to answer, and hence could be handled in 3–8 minutes.

HOW WILL A SUFFICIENTLY LARGE POOL OF ITEMS BE CONSTRUCTED? While it might be possible to release a pool of items which would provide the equivalent of 6 tests at each level, hence 6 x 6 x 30, it would be better to increase flexibility by using item shells, which would be items that include identified variables, each of which could be replaced from a list of acceptable values. This would greatly increase the number of items that could be generated, but without “surprises”. A pool of shells would generate over a thousand items at each level, possibly several thousand.

WHO WILL DO THE ASSESSING? In order to avoid problems in the reliability of the assessment (like those we have seen occur in the California Direct Writing Assessment), the assessment needs to be monitored, administered, and graded by a private agency whose personnel have critical thinking credentials or are at least under the direction of scholars with a solid grounding in research in critical thinking.

WHO WILL BEAR THE COSTS OF THE ASSESSMENT? The assessment should be paid for by the school systems, colleges, and universities that contract to have their students tested. This not only puts least burden on the public but also represents an established precedent in distributing costs of testing.

WHAT WILL INSTITUTIONS BE ABLE TO LEARN FROM THE RESULTS OF THE ASSESSMENT? We anticipate that educational institutions will receive an analytic report that will document all of the following:

• where their students are strongest and weakest with respect to particular micro-skills;
• where their students are strongest and weakest with respect to important abilities;
• how students stand in each of the school’s subject-matter areas;
• how their students stand in relation to students at other institutions;
• how their students at one educational level stand in relation to their students at other educational levels;
• how their students stand with respect to established performance criteria.

This information would enable institutions to target instruction to remediate weaknesses and build on strengths, as well as to measure what students are gaining as a result of attending their classes.*

Back to top

{Taken from Paul, R. (1993). Critical Thinking: What Every Student Needs to Survive in A Rapidly Changing World , Dillon Beach, CA: Foundation For Critical Thinking).

35 Higher-Order Thinking Questions

Chris Drew (PhD)

Dr. Chris Drew is the founder of the Helpful Professor. He holds a PhD in education and has published over 20 articles in scholarly journals. He is the former editor of the Journal of Learning Development in Higher Education. [Image Descriptor: Photo of Chris]

Learn about our Editorial Process

Higher-order thinking questions are questions that you can ask in order to stimulate thinking that requires significant knowledge mastery and data manipulation.

Generally, higher-order thinking involves thinking from the top 3 levels of bloom’s taxonomy: analysis, evaluation, and knowledge creation.

The term “higher-order” is used because these forms of thinking require strong command of information and the ability to work with it to develop complex understanding (Stanley, 2021).

Generally, a higher-order thinking question will be open-ended and require the student to demonstrate their ability to analyze and evaluate information.

Higher-Order Thinking Questions

Below are some useful questions for stimulating higher-order thinking.

Questions for Teachers to Ask Students

• Encourage compare and contrast: How would you compare and contrast these two concepts/ideas?
• Seek alternatives: Can you provide an alternative solution to this problem?
• Apply an ethical lens: What ethical considerations are involved in this situation or decision?
• Categorize and classify: How would you categorize or classify these items based on their shared characteristics?
• Sort by priority: How would you prioritize these tasks, and what factors did you consider?
• Real-world connections: How can you apply this concept to a real-world situation?
• Rephrase and reframe: How would you rephrase this question or problem from a different perspective?
• Identify trends: Can you identify any trends or developments that may influence this issue in the future?
• Seek solutions: How would you design a solution to address this challenge?
• Use evidence: What evidence supports your point of view or conclusion?
• Find relationships: Can you explain the relationship between these two events or phenomena?
• Change a variable: How would this situation change if we altered this variable or factor?
• Compare to prior knowledge: In what ways does this concept challenge your previous understanding or beliefs?
• Identify connections: Can you explain how these two seemingly unrelated ideas are connected or interdependent ?
• Re-conextualize: How would you adapt this solution to work in a different context or environment?
• Identifying consequences: What are the potential consequences of this decision or action?
• Evaluate: What criteria would you use to evaluate the effectiveness or success of this approach?
• Interdisciplinary connections: How can you apply principles from another discipline to enhance your understanding of this topic?
• Distil key factors: What factors may have contributed to this outcome or result, and how might they be addressed?
• Identifying bias: Can you identify any biases or assumptions in this argument?
• Find weaknesses: How would you argue against your own position or point of view?
• Steelman: Can you think of likely criticisms of your position and identify ways you would respond?
• Make judgments about best practices: Can you develop a set of guidelines or best practices based on this information?
• Seek next steps: What questions would you ask to further investigate or explore this topic?
• Reflect on process: What did you learn about how you went about this task and how would you make changes next time for improvements?

Questions for Students to Ask Themselves

• K-W-L: What do I already know about this topic, what do I still need to learn, and what have I learned today?
• Compare and contrast with prior knowledge: How does this new information relate to what I already know?
• Identify assumptions : What assumptions am I making, and are they justified?
• Organize: How can I organize this information in a way that makes sense to me?
• Identify trends: What patterns or connections can I identify between these concepts or ideas?
• Think from another perspective: Am I considering multiple perspectives or viewpoints in my analysis?
• Brainstorm implications : What are the potential implications of my conclusions or decisions?
• Hypothesize: How can I use my current knowledge to predict or hypothesize about future events?
• Identify inconsistency: Can I recognize any logical fallacies or inconsistencies in my reasoning?
• Seek new strategies: What strategies can I employ to improve my understanding and retention of this material?

Higher-Order Thinking vs Lower-Order Thinking

Benefits of Higher-Order Thinking

Higher-order thinking offers numerous benefits to learners, including:

• Enhanced problem-solving skills : Higher-order thinking develops a student’s ability to tackle complex problems by breaking them down, analyzing different aspects, and putting the information back together to find new solutions. This is highly valued in 21st Century workplaces (Saifer, 2018).
• Critical thinking and reasoning : Students who engage in higher-order thinking are better equipped to evaluate information, question assumptions, and identify biases. This helps them to have better media literacy and enables them to form independent conclusions rather than being easily swayed by flawed information (Richland & Simms, 2015).
• Creativity and innovation : Higher-order thinking fosters creativity by encouraging students to think beyond the obvious. Students are encouraged to explore alternative perspectives and find alternative ways to approach common problems. This creative thinking is highly valuable in various academic and professional fields, including STEM and the arts.
• Deeper understanding and retention: Lower-order thinking prioritizes memorization, but because the information is not sufficiently contextualized and learned though knowledge construction, it tends to be lost with time. Higher-order thinking, on the other hand,  promotes a more profound understanding of subjects. This deeper comprehension leads to better long-term retention of knowledge and better ability to manipulate information (Ghanizadeh, Al-Hoorie & Jahedizadeh, 2020).
• Greater self-awareness and metacognition : Higher-order thinking fosters self-reflection and metacognition. Students who have learned skills like critique, identifying flaws and biases, and logical analysis, are able to apply those skills to their own thinking to reflect on how they can improve their own rational meaning-making.

How to Stimulate Higher-Order Thinking in the Classroom

• Cultivate inquisitive minds: Encourage students to ask questions – regularly. Create a classroom culture where questioning is encouraged and there are “no wrong questions.” Encourage questions that delve deeper into subjects, challenge assumptions, or stimulate further cuiriosity. This will foster their critical thinking by constantly making them peel back the layers of knowledge on any topic (Yen & Halili, 2015).
• Tackle real-life challenges: Create lesson plans that root the learning content in real-world situations (i.e. situated learning ). Require students to apply their knowledge and skills to new situations rather than just on worksheets. By addressing genuine issues that, ideally, are relevant to students’ lives, students can start to work with and manipulate the knowledge they have received in the classroom (Saifer, 2018).
• Encourage collaboration and active learning : Promote group discussions, debates, and cooperative problem-solving activities. Group work helps with higher-order thinking because students are exposed to diverse perspectives and new ways of doing things from their peers. By seeing others’ thought processes, we can enhance our own (Ghanizadeh, Al-Hoorie & Jahedizadeh, 2020).
• Reflect and build self-awareness : Nurture the habit of self-reflection in students. Here, we’re referring to the concept of metacognition which refers to ‘thinking about thinking’. This encourages students to evaluate how they went about learning and continually work on improving their learning process. This plays a vital role in recognizing my strengths and weaknesses and refining my learning strategies (Yen & Halili, 2015).
• Interweave interdisciplinary connections: Combine ideas, concepts, and techniques from various disciplines to encourage a comprehensive understanding of complex subjects. One discipline may shed light on the topic in a way that another discipline is completely blind to. By establishing connections between different fields, students can sharpen their analytical and creative thinking abilities (Richland & Simms, 2015).

Higher-Order Thinking on Bloom’s Taxonomy

Higher and lower-order thinking skills are most famously presented in Bloom’s Taxonomy .

This taxonomy is used to categorize levels of understanding , starting from shallow knowledge and ending with deep understanding.

Below is an image demonstrating the Bloom’s Taxonomy hierarchy of knowledge :

As shown in the above image, Bloom distils 6 forms of knowledge and understanding. The bottom 3 (remember, understand, and apply) relate to lower-order thinking that doesn’t require deep knowledge. The top 3 (analyze, evaluate, create) represent higher-oreder thinking.

Each is explained below:

1. Remembering (Lower-Order)

Definition: This is the most fundamental level of understanding that involves remembering basic information regarding a subject matter. This means that students will be able to define concepts, list facts, repeat key arguments, memorize details, or repeat information.

Example Question: “What is 5×5?”

2. Understanding (Lower-Order)

Definition: Understanding means being able to explain. This can involve explaining the meaning of a concept or an idea. This is above remembering because it requires people to know why , but it is not yet at a level of analysis or critique.

Example Question: “Can you show me in a drawing what 5×5 looks like?”

3. Applying (Middle-Order)

Definition: Applying refers to the ability to use information to do work. Ideally, it will occur in situations other than the situation in which it was learned. This represents a deeper level of understanding.

Example Question: “If you buy five chocolates worth $5 each, how much will you have to pay?”

4. Analyzing (Higher-Order)

Definition: This is generally considered to be the first layer of higher-order thinking. It involves conducting an analysis independently. This includes the ability to make connections between ideas, explore the logic of an argument, and compare various concepts.

Example Question: “Based on what you’ve learned, can you identify five key themes?”

5. Evaluating (Higher-Order)

Definition: Evaluating means determining the correctness, morality, or rationality of a perspective. At this level, students can identify the merits of an argument or point of view and weigh the relative strengths of each point. It requires analysis, but steps-up to making judgments about what you’re seeing.

Example Question: “Based on all the information you’ve gathered, what do you think is the most ethical course of action?”

6. Creating (Higher-Order)

Definition: The final level of Bloom’s taxonomy is when students can create knowledge by building on what they already know. This may include, for example, formulating a hypothesis and then testing it through rigorous experimentation.

Example Question: “Now you’ve mastered an understanding of accounting, could you make an app that helps an everyday person manage their bookkeeping?”

Higher-order thinking is a necessary skill for the 21st Century. It promotes those thinking skills that are required for high-paying jobs and allows people to think critically, be more media literate, and come to better solutions to problems both in their personal and professional lives. By encouraging this sort of thinking in school, educators can help their students get better grades now and live a better life into the future.

Ghanizadeh, A., Al-Hoorie, A. H., & Jahedizadeh, S. (2020).  Higher order thinking skills in the language classroom: A concise guide . New York: Springer International Publishing.

Richland, L. E., & Simms, N. (2015). Analogy, higher order thinking, and education.  Wiley Interdisciplinary Reviews: Cognitive Science ,  6 (2), 177-192. doi: https://doi.org/10.1002/wcs.1336

Saifer, S. (2018).  HOT skills: Developing higher-order thinking in young learners . London: Redleaf Press.

Stanley, T. (2021). Promoting rigor through higher level questioning practical strategies for developing students’ critical thinking. New York: Taylor & Francis.

Yen, T. S., & Halili, S. H. (2015). Effective teaching of higher order thinking (HOT) in education.  The Online Journal of Distance Education and e-Learning ,  3 (2), 41-47.

• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 25 Number Games for Kids (Free and Easy)
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 25 Word Games for Kids (Free and Easy)
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 25 Outdoor Games for Kids
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 50 Incentives to Give to Students

Leave a Comment Cancel Reply

Your email address will not be published. Required fields are marked *

Our websites may use cookies to personalize and enhance your experience. By continuing without changing your cookie settings, you agree to this collection. For more information, please see our University Websites Privacy Notice .

Center for Excellence in Teaching and Learning

• Critical Thinking and other Higher-Order Thinking Skills

Critical thinking is a higher-order thinking skill. Higher-order thinking skills go beyond basic observation of facts and memorization. They are what we are talking about when we want our students to be evaluative, creative and innovative.

When most people think of critical thinking, they think that their words (or the words of others) are supposed to get “criticized” and torn apart in argument, when in fact all it means is that they are criteria-based. These criteria require that we distinguish fact from fiction; synthesize and evaluate information; and clearly communicate, solve problems and discover truths.

Why is Critical Thinking important in teaching?

According to Paul and Elder (2007), “Much of our thinking, left to itself, is biased, distorted, partial, uninformed or down-right prejudiced.  Yet the quality of our life and that of which we produce, make, or build depends precisely on the quality of our thought.”  Critical thinking is therefore the foundation of a strong education.

Using Bloom’s Taxonomy of thinking skills, the goal is to move students from lower- to higher-order thinking:

• from knowledge (information gathering) to comprehension (confirming)
• from application (making use of knowledge) to analysis (taking information apart)
• from evaluation (judging the outcome) to synthesis (putting information together) and creative generation

This provides students with the skills and motivation to become innovative producers of goods, services, and ideas.  This does not have to be a linear process but can move back and forth, and skip steps.

How do I incorporate critical thinking into my course?

The place to begin, and most obvious space to embed critical thinking in a syllabus, is with student-learning objectives/outcomes.  A well-designed course aligns everything else—all the activities, assignments, and assessments—with those core learning outcomes.

Learning outcomes contain an action (verb) and an object (noun), and often start with, “Student’s will....” Bloom’s taxonomy can help you to choose appropriate verbs to clearly state what you want students to exit the course doing, and at what level.

• Students will define the principle components of the water cycle. (This is an example of a lower-order thinking skill.)
• Students will evaluate how increased/decreased global temperatures will affect the components of the water cycle. (This is an example of a higher-order thinking skill.)

Both of the above examples are about the water cycle and both require the foundational knowledge that form the “facts” of what makes up the water cycle, but the second objective goes beyond facts to an actual understanding, application and evaluation of the water cycle.

Using a tool such as Bloom’s Taxonomy to set learning outcomes helps to prevent vague, non-evaluative expectations. It forces us to think about what we mean when we say, “Students will learn…”  What is learning; how do we know they are learning?

The Best Resources For Helping Teachers Use Bloom’s Taxonomy In The Classroom by Larry Ferlazzo

Consider designing class activities, assignments, and assessments—as well as student-learning outcomes—using Bloom’s Taxonomy as a guide.

The Socratic style of questioning encourages critical thinking.  Socratic questioning  “is systematic method of disciplined questioning that can be used to explore complex ideas, to get to the truth of things, to open up issues and problems, to uncover assumptions, to analyze concepts, to distinguish what we know from what we don’t know, and to follow out logical implications of thought” (Paul and Elder 2007).

Socratic questioning is most frequently employed in the form of scheduled discussions about assigned material, but it can be used on a daily basis by incorporating the questioning process into your daily interactions with students.

In teaching, Paul and Elder (2007) give at least two fundamental purposes to Socratic questioning:

• To deeply explore student thinking, helping students begin to distinguish what they do and do not know or understand, and to develop intellectual humility in the process
• To foster students’ abilities to ask probing questions, helping students acquire the powerful tools of dialog, so that they can use these tools in everyday life (in questioning themselves and others)

How do I assess the development of critical thinking in my students?

If the course is carefully designed around student-learning outcomes, and some of those outcomes have a strong critical-thinking component, then final assessment of your students’ success at achieving the outcomes will be evidence of their ability to think critically.  Thus, a multiple-choice exam might suffice to assess lower-order levels of “knowing,” while a project or demonstration might be required to evaluate synthesis of knowledge or creation of new understanding.

Critical thinking is not an “add on,” but an integral part of a course.

• Make critical thinking deliberate and intentional in your courses—have it in mind as you design or redesign all facets of the course
• Many students are unfamiliar with this approach and are more comfortable with a simple quest for correct answers, so take some class time to talk with students about the need to think critically and creatively in your course; identify what critical thinking entail, what it looks like, and how it will be assessed.

Additional Resources

• Barell, John. Teaching for Thoughtfulness: Classroom Strategies to Enhance Intellectual Development . Longman, 1991.
• Brookfield, Stephen D. Teaching for Critical Thinking: Tools and Techniques to Help Students Question Their Assumptions . Jossey-Bass, 2012.
• Elder, Linda and Richard Paul. 30 Days to Better Thinking and Better Living through Critical Thinking . FT Press, 2012.
• Fasko, Jr., Daniel, ed. Critical Thinking and Reasoning: Current Research, Theory, and Practice . Hampton Press, 2003.
• Fisher, Alec. Critical Thinking: An Introduction . Cambridge University Press, 2011.
• Paul, Richard and Linda Elder. Critical Thinking: Learn the Tools the Best Thinkers Use . Pearson Prentice Hall, 2006.
• Faculty Focus article, A Syllabus Tip: Embed Big Questions
• The Critical Thinking Community
• The Critical Thinking Community’s The Thinker’s Guides Series and The Art of Socratic Questioning

Quick Links

• Developing Learning Objectives
• Creating Your Syllabus
• Active Learning
• Service Learning
• Case Based Learning
• Group and Team Based Learning
• Integrating Technology in the Classroom
• Effective PowerPoint Design
• Hybrid and Hybrid Limited Course Design
• Online Course Design

Consult with our CETL Professionals

Consultation services are available to all UConn faculty at all campuses at no charge.

Home → Features → Resources → School and Study

What is Bloom’s Taxonomy: the pyramid of true learning

Exploring the depths and implications of Bloom's Taxonomy in learning.

Imagine a seed, tucked into the soil, ready to start its growth. It will go through stages, transforming from a tiny seed into a sprouting plant, and eventually into a blooming flower.

This process is not unlike the journey your mind embarks on when learning something new. That’s where Bloom’s Taxonomy comes in—a framework for understanding the stages of learning.

A Seed Planted: What is Bloom’s Taxonomy?

Bloom’s Taxonomy is a hierarchical model designed to classify learning objectives. Created by educational psychologist Benjamin Bloom and his colleagues in the 1950s, it’s a ladder of sorts—a way to understand how learning progresses from simple facts to complex analysis. Each rung represents a different level of understanding, a new layer of complexity, a new bloom on the flower of knowledge.

Bloom’s Taxonomy, also known as The Taxonomy of Educational Objectives, provides a roadmap for the journey of learning, outlining the steps from basic knowledge to complex understanding. The aim is to promote higher forms of thinking in education, moving beyond rote memorization to foster critical thinking and problem-solving skills .

It’s not just about memorizing — it’s about understanding information, applying it, analyzing it, evaluating it, and creating something new with it. It’s a model that recognizes that learning is not a one-size-fits-all process but a layered, multifaceted journey.

Bloom’s Taxonomy has been widely embraced in education and training contexts around the globe, and it remains one of the most well-known and frequently used educational frameworks. In essence, it provides a blueprint for learning, offering a structured approach to the development of educational programs and lesson plans, and a systematic method for assessing learners’ understanding.

“At the time it was introduced, the term taxonomy was unfamiliar as an education term. Potential users did not understand what it meant, therefore, little attention was given to the original Taxonomy at first. But as readers saw its potential, the framework became widely known and cited, eventually being translated into 22 languages,” wrote American psychologist David R. Krathwohl, who would later greatly influence the revision of Bloom’s Taxonomy.

Original Bloom’s Taxonomy: The First Blossom

The original Bloom’s Taxonomy had six stages, each representing a different cognitive skill. Let’s take a walk through this garden of growth.

The first level of the taxonomy is Knowledge. This is where your learning journey starts. It’s about remembering and recalling basic facts and information—names, dates, places, and ideas. This is the seed of knowledge, the foundation upon which all other learning is built.

Comprehension

Next comes Comprehension. It’s more than just knowing facts—it’s understanding them. It’s the moment when a seedling first breaks through the surface and reaches towards the sunlight, starting to take in its environment.

Application

At the Application level, things start getting interesting. Here, you begin to use what you’ve learned in new ways. It’s the stage when the plant begins to spread its leaves, putting to use the sunlight it’s absorbing.

The fourth stage, Analysis, is where the understanding becomes deeper. It’s about breaking information down into parts and understanding how those parts relate to each other—the way a botanist might dissect a flower to better understand its structure.

Synthesis, the second-to-last stage, is where creativity comes into play. It’s about combining information in new ways like a gardener arranging different flowers to create a beautiful bouquet.

The final stage, Evaluation, is about making judgments and forming opinions about what you’ve learned. It’s the full bloom of the flower, ready to be admired and critiqued.

Revised Bloom’s Taxonomy: A New Bloom

Over time, our understanding of learning has evolved, and so too has Bloom’s Taxonomy. Just as a gardener might prune and adjust a plant to help it grow more effectively, a group of cognitive psychologists are improving and adapting the taxonomy.

Led by one of Bloom’s former students, Lorin Anderson, and educational psychologist David R. Krathwohl , researchers revisited the original taxonomy in the early 2000s . They made some key changes to better reflect our contemporary understanding of learning processes. The revised taxonomy maintains the essence of the original—still presenting a hierarchical model of cognitive processes. However, there were two significant adjustments.

Changing Nouns to Verbs

The first change was linguistic but impactful: the six categories were renamed from nouns to verbs. This shift emphasizes that learning is an active process. Each category now represents an action that a learner engages in, rather than a static stage of understanding.

• Remembering: The base level, previously known as “Knowledge,” is now “Remembering.” This involves retrieving, recognizing, and recalling relevant knowledge from long-term memory.
• Understanding: Previously “Comprehension,” this level involves constructing meaning from oral, written, and graphic messages.
• Applying: This remains unchanged in name but now emphasizes the use of procedures to solve problems, not just in routine tasks.
• Analyzing: This level is about breaking material into constituent parts and detecting how the parts relate to one another and to an overall structure or purpose.
• Evaluating: Previously the highest level in the original taxonomy, “Evaluation” is now the penultimate stage. It involves making judgments based on criteria and standards.
• Creating: This final stage, previously “Synthesis,” is now “Creating.” It’s about putting elements together to form a coherent or functional whole or reorganizing elements into a new pattern or structure.

Swapping the Top Two Levels

The second significant change in the revised Bloom’s Taxonomy is the swapping of the top two levels. In the original, “Evaluation” was the highest level of cognitive work. In the revised model, “Creating” takes this spot. This change was made to reflect the idea that creating a new product or point of view requires a higher level of understanding and mastery than making a judgment or evaluation.

These adjustments in the revised Bloom’s Taxonomy were not made to replace the original model but to update it, recognizing that our understanding of cognitive processes and learning behaviors have evolved.

Like the original, the revised Bloom’s Taxonomy serves as a powerful tool that can enhance teaching and learning strategies, emphasizing the active, dynamic nature of learning.

The Importance of Bloom’s Taxonomy: Why it Matters

Why should we care about this taxonomy? Why not just learn and let learn? Well, Bloom’s Taxonomy offers a structured way to approach learning that helps ensure a deep, thorough understanding of a topic. It’s like a map guiding us through the territory of knowledge, ensuring we don’t miss any important landmarks along the way.

The taxonomy also provides a useful framework for teachers and educators to design their curriculum and assessments. By addressing each level of the taxonomy, they can ensure they’re not just teaching facts, but also fostering understanding, application, analysis, evaluation, and creativity. It’s a way to nurture the whole garden of learning, not just a single plant.

Bloom’s Taxonomy Verbs: The Language of Learning

Each level of Bloom’s Taxonomy is associated with specific verbs that describe the cognitive processes involved. These verbs serve as signposts, guiding learners and educators alike through the stages of understanding.

• Remembering: This level involves verbs like define, list, recall, and recognize. You’re dealing with the basic elements of knowledge here—the seeds of understanding.
• Understanding: Verbs at this level include describing, explaining, summarizing, and interpreting. They’re about making sense of the knowledge you’ve gained.
• Applying: Here, you’ll see verbs like use, demonstrate, solve, and implement. These actions are all about using your knowledge in practical ways.
• Analyzing: At this level, verbs such as compare, differentiate, examine, and relate come into play. They’re about dissecting knowledge, and understanding its inner workings.
• Evaluating: Here, verbs, like assess, critique, judge, and defend, are used. These actions are about forming judgments based on your understanding.
• Creating: The highest level of Bloom’s Taxonomy involves verbs like design, construct, produce, and invent. These are the actions of a fully bloomed learner, able to use their knowledge in new and innovative ways.

These verbs aren’t just descriptors—they’re actions that learners actively engage in. They provide a clear structure to follow, guiding learners and educators alike on the journey from the simple act of recalling information to the complex process of creating something new.

It’s not just about the destination—it’s also about the journey, and these verbs help illuminate the path.

Bloom’s Taxonomy Questions: Probing the Depths of Knowledge

Bloom’s Taxonomy doesn’t just guide the learning process—it also provides a framework for asking questions. Each level of the taxonomy corresponds to a different type of question , allowing teachers and learners to probe the depths of understanding.

• Remembering: Questions at this level might include “What is…?” or “Can you recall…?” They’re about retrieving basic information.
• Understanding: Here, questions might be “Can you explain why…?” or “How would you summarize…?” They’re about comprehending the information.
• Applying: Questions at this level could include “How would you use…?” or “What examples can you find…?” They’re about applying knowledge to new situations.
• Analyzing: Questions like “What are the differences between…?” or “How does…relate to…?” come into play here. They’re about breaking down complex ideas.
• Evaluating: Questions at this level might be “What do you think about…?” or “Can you defend your position on…?” They’re about making judgments based on knowledge.
• Creating: Finally, questions like “What would you invent…?” or “How would you improve…?” are used. These are the questions that push learners to create something new from their understanding.

Bloom’s Taxonomy List of Objectives: Guiding the Learning Journey

Just as a gardener plans out a garden, setting objectives for what they want to grow and where, educators use Bloom’s Taxonomy to plan out the learning journey. Each level of the taxonomy can be associated with specific learning objectives , forming a roadmap for growth.

• Remembering: The objective at this level is to recall information accurately.
• Understanding: The goal here is to comprehend the meaning of the information.
• Applying: The aim at this level is to use the information in a new context.
• Analyzing: The objective here is to break down complex information into its constituent parts and understand how they relate.
• Evaluating: The goal at this level is to make informed judgments based on the acquired understanding.
• Creating: The final objective is to use the accumulated knowledge to create something new or original.

Each objective serves as a stepping stone, guiding the learner from one level of understanding to the next, ensuring a comprehensive grasp of the subject matter.

Using and Applying Bloom’s Taxonomy: Charting the Course of Learning

Like a compass in a vast forest, Bloom’s Taxonomy provides a sense of direction in the learning journey. But how do you use it in practice?

Start by identifying the current level of understanding. Are you or your students just beginning to learn about a topic, or do you already have a solid foundation to build upon? This will determine your starting point on the taxonomy.

From there, use the taxonomy as a guide to structure your learning or teaching. Plan activities and assessments that address each level of the taxonomy, ensuring a well-rounded understanding.

Bloom’s Taxonomy Activities: From Seed to Blossom

Let’s take a look at some examples of activities that correspond to each level of Bloom’s Taxonomy. Remember, these are just starting points—feel free to adapt and expand upon them to suit your specific learning needs.

Remembering: Create flashcards with key terms and concepts. Use them to practice recalling information.

Understanding: Write a summary or an explanation of a topic in your own words. This can help solidify your understanding.

Applying: Solve a problem or carry out a task using the information you’ve learned. This could be a math problem, a science experiment, or a real-world scenario.

Analyzing: Compare and contrast two related concepts or ideas. How are they similar? How are they different? This can help you see the nuances of the topic.

Evaluating: Form an opinion about a topic and defend it with evidence. This encourages critical thinking and helps you form your own perspectives.

Creating: Design a project or create a product that incorporates what you’ve learned. This could be a piece of artwork, a business plan, a research proposal, or anything else that allows you to use your knowledge in a new and creative way.

Critique and Limitations of Bloom’s Taxonomy

Bloom’s Taxonomy has been applied by generations of  K – 12  teachers and college instructors in their teaching. It is a model that greatly enhances learning objectives compared to unstructured learning. But like all models, it is not perfect.

In a 2018 article , Ron Berger, Chief Academic Officer at EL Education, critiques the traditional view of Bloom’s Taxonomy. He argues that the hierarchical pyramid graphic misrepresents the nature of learning by suggesting that cognitive processes are discrete and ranked in importance.

Berger contends that learning does not occur in a linear progression, starting with remembering and moving upward. Instead, he asserts that understanding is built through the application and creation of knowledge. Just as adults learn new skills by applying and creating, students also need opportunities to apply their knowledge and engage in the creation process to develop a deep understanding. Berger suggests that the integrated, circular, and iterative process of creating, analyzing, revising, and improving is how learners truly build understanding.

While acknowledging the usefulness of frameworks like Bloom’s Taxonomy in organizing thinking and reminding educators to address a range of skills, Berger argues that the taxonomy encourages instructional practices that do not align with how learning actually happens. He emphasizes the importance of providing students with ample opportunities to create and analyze throughout their learning journey, integrating active, hands-on experiences with the acquisition of knowledge.

Final words

In the end, Bloom’s Taxonomy is more than just a hierarchy of learning—it’s a celebration of the human mind’s capacity to grow, to understand, and to create. It’s a reminder that learning is not a straight line but a blooming flower , each petal a new layer of understanding. So the next time you embark on a journey of learning, remember to take a moment to stop and smell the roses.

FAQ about Bloom’s Taxonomy

Bloom’s Taxonomy is a hierarchical framework used to classify educational objectives and skills into different levels of complexity and cognitive processes.

Bloom’s Taxonomy was developed by Benjamin Bloom, an educational psychologist, in collaboration with a group of educators in the 1950s.

The purpose of Bloom’s Taxonomy is to provide educators with a structured and systematic approach to design and assess learning outcomes. It helps in setting clear learning objectives, planning instructional strategies, and evaluating student performance.

The six levels of Bloom’s Taxonomy, in ascending order of complexity, are: Remembering, Understanding, Applying, Analyzing, Evaluating, and Creating.

The “Remembering” level focuses on recalling or retrieving factual information, such as facts, terms, and concepts. It involves tasks like listing, defining, naming, and describing.

The “Understanding” level requires students to demonstrate comprehension and interpretation of information. It involves tasks like summarizing, explaining, comparing, and paraphrasing.

The “Applying” level involves using acquired knowledge and skills in new situations or contexts. It includes tasks like solving problems, demonstrating techniques, and implementing procedures.

The “Analyzing” level focuses on breaking down information into its constituent parts and examining relationships between them. It involves tasks like categorizing, organizing, comparing, and contrasting.

The “Evaluating” level requires making judgments and assessments based on criteria and standards. It involves tasks like critiquing, appraising, justifying, and defending.

The “Creating” level involves generating new ideas, products, or interpretations by combining existing knowledge and skills. It includes tasks like designing, constructing, composing, and planning.

Bloom’s Taxonomy can be used by teachers to design effective lesson plans, develop appropriate assessment methods, and scaffold student learning. It encourages higher-order thinking skills and promotes deep understanding.

Yes, Bloom’s Taxonomy is applicable to all subjects and grade levels. It can be adapted and customized according to the specific needs and learning objectives of different disciplines and age groups.

Absolutely! Bloom’s Taxonomy can be effectively applied in online or distance learning environments. It provides a framework for designing engaging and interactive activities that promote active learning and critical thinking.

Some critics argue that Bloom’s Taxonomy oversimplifies the complexity of learning and that it places too much emphasis on the cognitive domain while neglecting other important aspects, such as affect

Was this helpful?

Related posts.

• What is oxytocin: unraveling the ‘love hormone’
• Making Mistakes while Learning facilitates Memory
• Pollution linked to memory loss
• Girls aren’t good at math: the stereotype

Recent news

This European satellite can track nitrogen dioxide — and it wasn’t meant to

Researchers in Singapore have uncovered a new Candida clade. It could be a global threat

How Good (or Bad) You Look In High School Yearbook Photos May Predict Lifespan

• Editorial Policy
• Privacy Policy and Terms of Use
• How we review products

© 2007-2023 ZME Science - Not exactly rocket science. All Rights Reserved.

• Science News
• Environment
• Natural Sciences
• Matter and Energy
• Quantum Mechanics
• Thermodynamics
• Periodic Table
• Applied Chemistry
• Physical Chemistry
• Biochemistry
• Microbiology
• Plants and Fungi
• Planet Earth
• Earth Dynamics
• Rocks and Minerals
• Invertebrates
• Conservation
• Animal facts
• Climate change
• Weather and atmosphere
• Diseases and Conditions
• Mind and Brain
• Food and Nutrition
• Anthropology
• Archaeology
• The Solar System
• Asteroids, meteors & comets
• Astrophysics
• Exoplanets & Alien Life
• Spaceflight and Exploration
• Computer Science & IT
• Engineering
• Sustainability
• Renewable Energy
• Green Living
• Editorial policy
• Privacy Policy

• Become a Member
• Artificial Intelligence
• Computational Thinking
• Digital Citizenship
• Edtech Selection
• Global Collaborations
• STEAM in Education
• Teacher Preparation
• ISTE Certification
• School Partners
• Career Development
• ISTELive 24
• Solutions Summit
• Leadership Exchange
• 2024 ASCD Leadership Summit
• 2025 ASCD Annual Conference
• Edtech Product Database
• Solutions Network
• Sponsorship & Advertising
• Sponsorship & Advertising
• Learning Library

3 types of questions to boost higher-order thinking

• Teaching Strategies

In Julie Jaeger’s experience, enhancing students’ creativity and critical thinking doesn’t come from having all the right answers. It comes from asking more questions.

Jaeger, a gifted education teacher for Minot Public Schools in North Dakota, finds that when you create an environment where students have time to ponder and ask deep questions, it encourages them to think more creatively and more critically. And that’s the kind of thinking that brings students to the highest levels of learning.

“During my master’s program, I was introduced to Costa and Kallick’s 16 Habits of Mind for self-directed learners and realized this was a framework and foundation for much of what I believed should happen for students. The questioning strategy they designed fell right in line with how I wanted my students to think.”

In fact, Jaeger uses questioning strategies with both young and adult learners to elevate thinking and creativity. It’s an approach that should be a natural part of what teachers do.

In addition to Costa and Kallick’s work, she points to Blooms Taxonomy , a framework teachers can use to focus on higher-order thinking. By providing a hierarchy of levels, it assists teachers in designing performance tasks, crafting questions for conferring with students and providing feedback on student work.

While Bloom’s points to six levels of questioning, Jaeger’s a fan of a three-story house model that connects to both the Habit’s of Mind and Bloom’s Taxonomy. “There are images of the three-story model that are posted in many classrooms to inspire students to think beyond the ‘ground floor’ and shoot for the ‘attic’ or ‘rooftop’ with their questioning and thinking,” Jaeger says.

First floor questions . Students use questions to gather information: What is ___? When did___? Which one ___? How would you show ___?

Second-floor questions . Students tackle questions that help them process information: How would you categorize ___? What can we infer from ___? How would you summarize ___? What is the function of ___? What conclusions can you draw?

Third floor or attic questions . Students ask questions that require them to apply the information they’ve learned: What would you predict if ___? How would you prioritize ___? How could you prove or disprove __? What evidence supports ___?

Jaeger admits that posing lots of questions can be uncomfortable for students at first, but when teachers ask higher-level questions, students deepen their knowledge and create connections to the material being presented. She also encourages teachers to leave time for “creative pauses” after posing questions to give students think time and to prevent them from raising their hands before you’ve even finished asking the question. Then seek many, varied and unusual responses. Educators looking for more tips to help students become critical thinkers can watch a recording of the  ISTE Professional Learning Series webinar “Elevate Critical Thinking and Creativity With Technology”  Viewers will:

• Get tips for elevating students’ critical thinking skills with help from the latest tech tools.
• Learn how to create a classroom environment where students are comfortable with pondering, thinking and guessing.
• Get resources and tools you can use to help students practice critical thinking.

ISTE members can sign up now for the ISTE Professional Learning Series that includes the recorded webinar “Elevate Critical Thinking and Creativity With Technology.” Not a member? Join ISTE today .

• artificial intelligence

• Academic Success
• Higher Order Thinking

Higher Order Thinking: Bloom's Taxonomy

Many students start college using the study strategies they used in high school, which is understandable—the strategies worked in the past, so why wouldn’t they work now? As you may have already figured out, college is different. Classes may be more rigorous (yet may seem less structured), your reading load may be heavier, and your professors may be less accessible. For these reasons and others, you’ll likely find that your old study habits aren’t as effective as they used to be. Part of the reason for this is that you may not be approaching the material in the same way as your professors. In this guide, we provide information on Bloom’s Taxonomy—a way of thinking about your schoolwork that can change the way you study and learn to better align with how your professors think (and how they grade).

Why Higher Order Thinking Leads to Effective Study

Most students report that high school was largely about remembering and understanding large amounts of content and then demonstrating this comprehension periodically on tests and exams. Bloom’s Taxonomy is a framework that starts with these two levels of thinking as important bases for pushing our brains to five other higher order levels of thinking—helping us move beyond remembering and recalling information and move deeper into application, analysis, synthesis, evaluation, and creation—the levels of thinking that your professors have in mind when they are designing exams and paper assignments. Because it is in these higher levels of thinking that our brains truly and deeply learn information, it’s important that you integrate higher order thinking into your study habits.

The following categories can help you assess your comprehension of readings, lecture notes, and other course materials. By creating and answering questions from a variety of categories, you can better anticipate and prepare for all types of exam questions. As you learn and study, start by asking yourself questions and using study methods from the level of remembering. Then, move progressively through the levels to push your understanding deeper—making your studying more meaningful and improving your long-term retention.

Level 1: Remember

This level helps us recall foundational or factual information: names, dates, formulas, definitions, components, or methods.

Level 2: Understand

Understanding means that we can explain main ideas and concepts and make meaning by interpreting, classifying, summarizing, inferring, comparing, and explaining.

Level 3: Apply

Application allows us to recognize or use concepts in real-world situations and to address when, where, or how to employ methods and ideas.

Level 4: Analyze

Analysis means breaking a topic or idea into components or examining a subject from different perspectives. It helps us see how the “whole” is created from the “parts.” It’s easy to miss the big picture by getting stuck at a lower level of thinking and simply remembering individual facts without seeing how they are connected. Analysis helps reveal the connections between facts.

Level 5: Synthesize

Synthesizing means considering individual elements together for the purpose of drawing conclusions, identifying themes, or determining common elements. Here you want to shift from “parts” to “whole.”

Level 6: Evaluate

Evaluating means making judgments about something based on criteria and standards. This requires checking and critiquing an argument or concept to form an opinion about its value. Often there is not a clear or correct answer to this type of question. Rather, it’s about making a judgment and supporting it with reasons and evidence.

Level 7: Create

Creating involves putting elements together to form a coherent or functional whole. Creating includes reorganizing elements into a new pattern or structure through planning. This is the highest and most advanced level of Bloom’s Taxonomy.

Pairing Bloom’s Taxonomy with Other Effective Study Strategies

While higher order thinking is an excellent way to approach learning new information and studying, you should pair it with other effective study strategies. Check out some of these links to read up on other tools and strategies you can try:

• Study Smarter, Not Harder
• Simple Study Template
• Using Concept Maps
• Group Study

Works Consulted

Anderson, L. W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., & Wittrock, M.C (2001). A taxonomy of learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Longman.

“Bloom’s Taxonomy.” (PDF).

Overbaugh, R., and Schultz, L. (n.d.). “Image of two versions of Bloom’s Taxonomy.” Norfolk, VA: Old Dominion University.

Developed and shared by The Learning Center , University of North Carolina at Chapel Hill.

• Class Attendance
• Semester-At-A-Glance Calendar
• Semester Action Plan
• Using Calendars and Planners
• Studying 101

Watch Asking for Help Video

Ask for Help - We Want You to Succeed

View All Student Tips

815-753-6636 [email protected]

Connect with us

Teach Starter, part of Tes Teach Starter, part of Tes

Search  everything  in all resources

Teaching Higher-Order Thinking Skills: Here's Why It Matters Matters So Much

Written by Alison Smith

Have you been hearing a lot about teaching and learning higher-order thinking skills lately? It may not be a new topic in educational circles, but higher-order thinking remains a hot topic of discussion, and there’s a real need to address ways to build higher-order thinking into your already crammed school year.

Imagine students leaving school without any number sense or reading comprehension skills . Imagine the outcry. Now, suppose for a moment students leave school without the life skills that they need to succeed in the 21st century. The sad reality is that many students do leave school without the ability to think for themselves and without the experience of thinking critically and creatively.

In order for our students to be equipped and prepared to live in the 21st Century, there is a very real need to teach our students to:

• think about the problems that we face in life
• explore possibilities
• come up with creative solutions to problems
• consider and appreciate other points of view
• critically evaluate what we read and hear
• make reasonable judgments

So how do you teach higher-order thinking? And for that matter, how do you define higher-order thinking? The Teach Starter teacher team did a deep dive into the educational research to save you time and get you primed on all the things you need to know to help your students!

What Is Higher-Order Thinking?

Maybe you’ve got this definition rattling around in your head, but it seemed right to start at the beginning here. When we talk about higher-order thinking, we are talking about the ability to think abstractly and make connections between concepts.

In that sense, higher-order thinking includes critical thinking skills such as analysis, synthesis, and evaluation, as well as the ability to problem-solve and make decisions. Just as the name would imply, it is considered a more advanced level of cognitive processing than lower-order thinking, which mainly involves the recall of facts and information.

Bloom’s Taxonomy and Higher-Order Thinking

So where did higher-order thinking come from?

The classical Greek philosopher, Socrates, is often thought of as the founder of critical thinking skills. In a nutshell, Socrates introduced the idea of teaching by not providing answers but instead, teaching by asking questions: questions that explore, investigate, probe, stimulate, and engage.

Far more recently, Bloom’s taxonomy was created by Benjamin Bloom in 1956. In one sentence, Bloom’s taxonomy is a set of six cognitive skills (in a specific order) that teachers, students, and anyone can use to promote higher-order thinking. Bloom’s framework was revised in 2001 by Lorin Anderson and David Krathwohl to produce the framework used by educators today. The six levels of the revised Bloom’s taxonomy are:

• remembering — recalling known facts
• understanding — explaining ideas or concepts
• applying — use information in new situations
• analyzing — drawing connections among ideas
• evaluating — justifying a point of view or decision
• creating  — producing something new or original

Explore Bloom’s Taxonomy Teaching Tools for your classroom!

Higher-Order Thinking Skills in Kids

Children develop higher-order thinking skills at different rates, but generally, it is a gradual process that begins in early childhood and continues through adolescence and into adulthood.

According to Piaget’s theory of cognitive development, kids begin to develop higher-order thinking skills during the concrete operational stage, which typically occurs in upper elementary school, sometime between the ages of 7 and 11. During this stage, kids are able to think logically and understand cause-and-effect relationships .

As with all areas of the curriculum, there is a learning continuum for critical and creative thinking. Typically by the end of fourth grade , students will be able to:

• pose questions to expand their knowledge about the world
• identify main ideas and select and clarify information from a range of sources
• collect, compare and categorize facts and opinions found in a widening range of sources.

In early adolescence, children begin to develop formal operational thinking, which is characterized by the ability to think abstractly and make logical deductions. This stage is typically reached as they’re heading out of elementary school and into high school — sometime around ages 11 to 15.

Typically by the end of sixth grade, students should be able to:

• pose questions to clarify and interpret information and probe for causes and consequences
• identify and clarify relevant information and prioritize ideas
• analyze, condense, and combine relevant information from multiple sources

Focusing on helping your students build these skills has real benefits. Research has shown that teaching higher-order thinking skills can improve student achievement and prepare them for success in the 21st century.

One study by famed psychologist Richard Herrnstein (best known for his book The Bell Curve) and colleagues looked at 400 seventh graders and found that the students who were given critical thinking lessons made substantial and statistically significant improvements in language comprehension, inventive thinking, and IQ as compared to a control group. Other studies have found that students who receive instruction in higher-order thinking skills have better problem-solving abilities and are more likely to transfer their learning to novel situations.

How to Teach Higher-Order Thinking Skills

Teaching higher-order thinking skills in elementary school truly comes down to providing students with opportunities to question, connect concepts, and make inferences — much of the work you’re likely doing in your classroom right now. That said, here are some strategies you might want to employ in your classroom:

Use Graphic Organizers

Using graphic organizers, your students can actively engage with the material and make connections between different concepts, which can improve their critical thinking and problem-solving skills. The visual tools can be vital for kids as they learn to analyze, synthesize, and evaluate information, which are all important aspects of higher-order thinking.

Download concept maps, Venn Diagrams, and more graphic organizers for your students!

Use Socratic Questioning

Asking a series of open-ended and probing questions to encourage critical thinking, problem-solving, and deep learning will help students to challenge assumptions, clarify concepts, and promote reasoning — all keys to improving their higher-order thinking skills.

Socratic questions can help your students to generate and test hypotheses — an important aspect of scientific thinking — as well as encouraging kids to synthesize information.

Using Question-Answer-Relationships or QARs helps students learn to make connections between the information they find in a text and their prior knowledge, boosting their higher-order thinking skills.

Use Cooperative Learning

When small groups of students work together to complete a task or achieve a goal, they have opportunities to engage in active, collaborative, and constructive learning. They’re more likely to ask questions, share ideas, and engage in critical thinking, which can lead to a deeper understanding of the material. On top of that, the social interactions that take place in a cooperative learning setting can help to build student motivation, engagement, and self-esteem, which can also contribute to the development of higher-order thinking skills.

Use Problem-Based Learning

Present your students with real-world problems that require them to apply critical thinking and problem-solving skills to find a solution. Students will also benefit from learning to use step-by-step methods for solving problems as it presents them with a methodology for tackling problems in alternative ways.

Encourage Elaboration

When students provide answers in class or while completing tasks, encourage them to move beyond the basic answer and elaborate on the why with facts and ideas to support their answer.

Explore more higher-order thinking skills resources for teachers!

Banner image via Shutterstock/bernatets photo

30 Buzzing Facts About Bees to Excite Kids About Nature

Everyone benefits from the busyness of bees which is why these bee facts will help inspire your students to appreciate and protect them!

21 Epic Last Day of School Activities to Kick Off Summer Break

Looking for easy last day of school ideas for elementary or middle school? These quick and fun end of year classroom activities will help you finish off your year just right!

70 1st Grade Books to Add to Your Classroom Reading Corner This Year

Wondering which 1st grade books you should add to your classroom library? Look no further! We have a list of 70 that are teacher (and student) approved!

22 Fun Groundhog Facts to Share With Your Class on Groundhog Day

Need fun groundhog facts to share with your students this Groundhog Day? Find out what groundhogs eat, where they live and why we let them predict the weather!

How to Teach 1st Grade — The Ultimate Guide to What to Do, What to Buy and What to Teach

Looking for tips on teaching first grade? Our comprehensive 1st grade teacher guide will answer all your questions from what to buy to how to prepare for the first day of school and what to do throughout the school year.

10 Best Pencil Sharpeners for the Classroom — Recommended by Teachers

Need new pencil sharpeners in your classroom? We've rounded up 10 best pencil sharpeners recommended by teachers to get you started.

Thank you for this much needed teacher resource. I feel so much better equipped and prepared for this coming school year!

Hi Patricia. thanks for your positive feedback. I hope that you and your class have lots of higher-order thinking fun. You are amazing!!! Have a great day! Ali

Get more inspiration delivered to your inbox!

Sign up for a free membership and receive tips, news and resources directly to your email!

Join Our Back to School PD Webinar | Friday, 8/16 at 10AM PDT | Learn More

• Staff Portal
• Consultant Portal

Toll Free 1-800-495-1550

Local 559-834-2449

• Articles & Books
• Explicit Direct Instruction
• Student Engagement
• Checking for Understanding
• ELD Instruction
• About Our Company
• DataWORKS as an EMO
• About Our Professional Development
• English Learner PD
• Schedule a Webinar
• Higher-order Questions
• Classroom Strategy

After reading The Diary of Anne Frank , a student is asked, “ Who is Anne Frank ?” To answer the question, the student simply recalls the information he or she memorized from the reading.

With the implementation of Common Core, students are expected to become critical thinkers instead of just recalling facts and ideas from text. In order for students to reach this potential and be prepared for success, educators must engage students during instruction by asking higher-order questions.

Higher-order Questions (HOQ)

Higher-order questions are those that the students cannot answer just by simple recollection or by reading the information “ verbatim ” from the text. Higher-order questions put advanced cognitive demand on students. They encourage students to think beyond literal questions.

Higher-order questions promote critical thinking skills because these types of questions expect students to apply, analyze, synthesize, and evaluate information instead of simply recalling facts. For instance, application questions require students to transfer knowledge learned in one context to another; analysis questions expect students to break the whole into component parts such as analyze mood, setting, characters, express opinions, make inferences, and draw conclusions; synthesis questions have students use old ideas to create new ones using information from a variety of sources; and evaluation questions require students to make judgments, explain reasons for judgments, compare and contrast information, and develop reasoning using evidence from the text.

Higher-order Questions Research

According to research, teachers who effectively use a variety of higher-order questions can overcome the brain’s natural tendency to develop mental routines and patterns to limit information, which is called neural pruning . As a result, student’s brains may become more open-minded, which strengthens the brain.

According to an article in Educational Leadership (March 1997), researchers Thomas Cardellichio and Wendy Field discovered that higher-order questions increase neural branching , the opposite of neural pruning. In addition, these researchers found that teachers can promote the process of neural branching through seven types of questions.

• Hypothetical thinking . This form of thinking is used to create new information. It causes a person to develop an answer based on generalizations related to that situation. These questions follow general forms such as What if this happened ? What if this were not true ?, etc.
• Reversal thinking . This type of thinking expects students to turn a question around and look for opposite ideas. For example, What happens if I reverse the addends in a math problem ? What caused this? How does it change if I go backward ?, etc.
• Application of different symbol systems . This way of thinking is to apply a symbol system to a situation for which it is not usually used, such as writing a math equation to show how animal interaction is related.
• Analogy . This process of thinking is to compare unrelated situations such as how is the Pythagorean Theorem related to cooking. These questions typically ask How is this like ___?
• Analysis of point of view . This way of thinking requires students to consider and question other people’s perspective, belief, or opinion in order to extend their minds. For instance, a teacher may ask a student, What else could account for this ? or How many other ways could someone look at this ?
• Completion . This form of thinking requires students to finish an incomplete project or situation that would normally be completed. For example, removing the end of a story and expecting the students to create their own ending.
• Web analysis . With web analysis, students must synthesize how events are related in complex ways instead of simply relying on the brain’s natural ability to develop a simple pattern. For example, How extensive were the effects of _____? Or Track the relationship of events following from ___ are types of web analysis questions.

The researchers concluded that this type of questioning can lead to better critical thinking skills. “ They can analyze, synthesize, evaluate, and interpret the text they are reading at complex levels. They can process text at deep levels, make judgments, and detect shades of meaning. They can make critical interpretations and demonstrate high levels of insight and sophistication in their thinking. They are able to make inferences, draw relevant and insightful conclusions, use their knowledge in new situations, and relate their thinking to other situations and to their own background knowledge. These students fare well on standardized tests and are considered to be advanced. They will indeed be prepared to function as outstanding workers and contributors in a fast-paced workplace where the emphasis is on using information rather than just knowing facts .”

Higher-order Questions and Explicit Direct Instruction

The Explicit Direct Instruction (EDI) model incorporates a variety of higher-order questions in order to encourage and increase critical thinking skills.

The LEARNING OBJECTIVE component in EDI is the only question that is at a low level of Bloom’s Taxonomy. The reason for this is because the content during this portion of the lesson is not at a high level. Also, the students have not been taught the high-level content. Typically, the question asked to students is “ What are we doing today ? or What is our Learning Objective ?”

The CONCEPT DEVELOPMENT component includes a variety of higher-order concept-related questions because the content is at a high level. Here is a list of higher-order questions that are asked during this EDI component:

• In your own words, what is (insert the concept being taught)?
• Which is an example of ________? Why?
• What is the difference between the example and the non-example?
• Why is this an example of ______?
• Give me an example of ______.
• Draw an example of ______.
• Match the examples to the definition of ______.
• Which picture/poster shows an example of _______?

The SKILL DEVELOPMENT component asks higher-level thinking-process questions after modeling the skill.

• How did I know how to (insert skill modeled)?
• How did I know that this was the correct answer?
• How did I use to ensure that I knew how to find the _____?
• How did I know how to interpret the answer?

The GUIDED PRACTICE asks higher-level process questions that require the students to show their thought process when performing the skill.

• How did you know how to __________?
• How did you know that this was the correct answer?
• How did you use to ensure that you knew how to find the _____?
• How did you know how to interpret the answer?
• Which steps was most difficult for you? Why?

The RELEVANCE component includes higher-level evaluation questions.

• Does anyone have any other reason as to why this is important?
• Which reason is the most relevant to you? Why?

The CLOSURE component includes high-level questions such as:

• What did you learn today?
• How did the lesson meet the Learning Objective ?
• How will this lesson benefit you in the future ?

If higher-order questions promote critical thinking skills, as research shows, then higher-order questions should be included throughout instruction. The EDI model offers a good way to do just that!

Educational Leadership, Seven Strategies That Encourage Neural Branching , March 1997

How do you incorporate higher-order questions during instruction?

Additional Resources:

• Educeri; Ready Made Lessons with Higher-Order Questions
• The Secret to Differentiation with EDI: Making Better Decisions at Choice Points
• How to Differentiate and Scaffold – after teaching including RTI
• How to Differentiate and Scaffold – Before the Lesson
• How to Differentiate and Scaffold – while teaching
• Differentiation Strategies: Teaching Grade-Level Content to ALL Students
• Explicit Direct Instruction (EDI) Resource Page

Author:  Patricia Bogdanovich

Patricia has held various positions with DataWORKS since 2002. She currently works as a Curriculum Specialist. Patricia helped develop and create many of the early resources and workshops designed by DataWORKS, and she is an expert in analysis of standards. Patricia plans to blog about curriculum and assessments for CCSS and NGSS, classroom strategies, and news and research from the world of education.

Related posts

Leave a Reply Cancel reply

You must be logged in to post a comment.

The eLearning Coach

For designing effective learning experiences

Writing Multiple Choice Questions For Higher Order Thinking

by Connie Malamed

One of the biggest criticisms of multiple choice questions is that they only test factual knowledge. But it doesn’t have to be that way. We can also use multiple choice questions to assess higher-order thinking.

Higher Order Thinking in a Nutshell

Higher order thinking goes beyond memorizing and recalling facts and data. It even goes beyond comprehension. Higher-order thinking refers to cognitive processes that involve analytical, critical and creative thinking. The concept is based on various learning taxonomies, such as application, analysis, evaluation, creation, problem-solving, connecting ideas and making decisions. Keep in mind that many of these cognitive tasks, including the recall of information, appear to occur simultaneously. See Alternatives to Bloom’s Taxonomy for criticisms of the hierarchical classifications.

Because test items must be aligned with performance objectives, you’ll need to include higher-order thinking skills from the start. And yes, these may be better measured through open ended questions, essays and discussions. But if you find yourself needing to use multiple choice tests, you can make the best of this situation with these three approaches.

1. Real-world Scenarios

One of the most effective ways to assess higher-order thinking is to use scenario-based questions, particularly ones that simulate real work experiences. The sample question pairs below demonstrate how to transform a question from the simple recall to a higher-order thinking skill using the scenarios approach. Note: Only the question stems are shown. Assume there are multiple choice options beneath each question.

• Before: What symbol does a formula always start with in Excel?
• After: To total the first ten data cells in column B, which one of the following formulas should you use?
• Before: What is the first concern of an emergency worker?
• After: An emergency technician (ET) arrived at the scene of an accident. Three people were injured. One person had scrapes and bruises, a second was moaning in pain and a third was quickly losing blood from a skull injury. Why did the ET triage the skull injury as the highest priority?
• Before: When writing a CSS style sheet, what property is used to position an element to the left or right?
• After: How will the text and photo be aligned using the code below? CSS: .photo {float: right;}   HTML: My Cat <img src=”cat.jpg” alt=”photo of stuff” class=”photo” />

2. Analysis of Visuals

You can also assess critical thinking skills by asking learners to analyze or interpret information from visuals, which are provided as part of the question stem or the answer choices. In many cases, visuals such as job aids, diagrams and graphs simulate workplace tasks. See a few examples below. Note: Assume there are multiple choice options beneath each question and that visuals are displayed.

• Before: What is/are the function(s) of a computer’s projecting mechanism? (Select all that apply.)
• After: Using the repair flowchart shown here, what should you check if the monitor stops working?
• Before: Choose the best definition of rate card.
• After: Using the rate card, what is the best time for a customer in the United States to make an international call to Brazil?
• Before: Which country will have the largest population in the year 2030?
• After: What does this graph predict about world population in the year 2030?
• Before: Select the best definition of active listening .
• After: Which video best demonstrates active listening during a call with an irate customer?

3. The Answer Plus The Reason Why

A third approach to measuring critical or creative thinking is to ask learners to synthesize what they’ve learned into an explanation. Although these test items are more difficult to write, they can be quite satisfactory for assessments. The possible responses include the answer and then a variety of reasons that support the answer. Of course, only one reason is logical and correct, based on the knowledge and skills being assessed.

• Before: What are three signs of edema?
• After: A patient entered the hospital with edema of both lower extremities. What action should the nurse take and why?
• Before: Select the most effective tone for writing technical documentation.
• After: Select the paragraph and reasoning that best demonstrate how to start a technical document.
• Before: Name three qualities of effective entrepreneurs.
• After: Review the financials of these three startup companies. Which one would you invest in and why?

Also see 10 Rules for Writing Multiple Choice Questions for ways to make your tests more accurate and error-free. Plus see Patti Shank’s book: Write Better Multiple-Choice Questions to Assess Learning for more great tips.

June 7, 2022 at 2:03 pm

Thank you so much for your article. I’m currently a MSc student doing developmental and educational psychology research and have been suggested a dissertation topic on visual learning strategies and higher-order thinking. The scope of study was narrowed to using the argumentation vie-diagram (AVD) (a diagram which is similar to mind-map but focuses on helping one to creative critical solutions for both arguments and counterargument as well as to create compromises/solutions to solve a case scenario/problem based question/statement etc. by integrating the ideas from both sides) as the learning strategy and the HOT skill would be creative thinking — the highest level of HOT skills according to the Bloom’s Taxonomy. My supervisor suggested me to use MCQs to test creative thinking/creative solutions using MCQs as the test format with the use of AVD as the learning strategy. However, since the nature of MCQ consists of close-ended questions so it provides little to no space for creativity. I’ve been struggling to find related rating scale and MCQs for assessing creativity based on this matter. May I ask if you have any ideas in creating MCQs to suit this case scenario?

January 11, 2022 at 10:29 pm

Glad it was helpful, Carlton.

January 8, 2022 at 3:50 am

Greate article Connie. I’m not always a fan of quizzes etc as the way they are written often means they are just testing short term memory. However, multiple-choice questions written in the way you have described can be brilliant for testing so much more. Thanks for sharing.

November 5, 2021 at 9:30 pm

Hi Amy, I do have three examples there, such as “A patient entered the hospital with edema of both lower extremities. What action should the nurse take and why?” Maybe you’d like to see an answer choice? Even though I’m not a medical person, I can imagine an answer like this: a. The nurse calls a resident heart specialist to examine the patient because she wants to rule out congestive heart failure, which can be an underlying condition of edema.

Is that what you were looking for?

November 4, 2021 at 9:59 am

Can you give an example of an “answer plus the reason why” question?

September 25, 2016 at 7:24 am

Hi Dick, I think your advice is sound as long as we avoid the appearance of “tricking” the student and that the test questions are tied to the original learning objectives. Thanks for your thoughtful comment. Connie

September 24, 2016 at 2:10 am

The questions should be written very generally (“Which alternative(s) is/are correct regarding [subject]-“) Then all the logic should be in the alternatives. Yes, it’s a pain then to come up with 4-5 alternatives that not all are correct, BUT all make sense to a student – IF the student hasn’t studied the subject well enough.

And of course one should always use “multiple answers” of the quiz making tool has this implemented in a smart way.

January 19, 2013 at 7:48 pm

Writing a good multiple choice question is a difficult task starting at grade K and continuing all the way through adult learning. Just for fun, I goggled “How to cheat at Multiple Choice Questions” and found 3,590,000 hits. The top ones included: “when in doubt choose c” and “all of the above” is usually the correct answer. I found your post very helpful when thinking about writing multiple choice questions. The scenario option is perfect for the instructional designer. It seems as if it will almost write itself.

I took a stab at a before and after. Before: What is the best app to use to find the definition of a word? a) dictionary.com b) webster word of the day c) color dictionary After: Your students are reading “The Whirligig” by Paul Fleischman and come across the word “inadvertently”. What is the best way for them to find the definition on their tablet? a) look at dictionary.com b) open the app “Webster word of the day” c) use the “color dictionary” app

I believe that as an Instructional Designer assessment will be the most difficult part of our job. It will be easy to design a course and sit back and marvel at our work. The real “test” will be in the “tests” and the more authentic we can design them the better we will be.

[…] Planning < Education and Teaching < steben45 Get flash to fully experience Pearltrees Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning One of the biggest criticisms of multiple choice questions is that they only test factual […]

[…] thinking [Malamed] On 10/19/2012, in elearning, instructional design, by Daniel Christian Writing multiple choice questions for higher order thinking — from theelearningcoach.com by Connie […]

[…] e-Learning Coach offers three tricks for asking good multiple choice questions. Read ‘em and use ‘em. She knows what she’s talking […]

[…] Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning From theelearningcoach.com – Today, 3:06 PM […]

[…] Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning From theelearningcoach.com – Today, 1:59 PM […]

[…] Writing Multiple Choice Questions For Higher Order Thinking … From theelearningcoach.com – Today, 1:38 PM You can also assess critical thinking skills by asking learners to analyze or interpret information from visuals, which are provided as part of the question stem or the answer choices. In many cases, visuals such as job aids, diagrams and graphs … […]

[…] de enseñanza/aprendizaje. Finalmente, de the learning coach, les dejamos este post sobre las actividades de selección múltiple y su uso en actividades de pensamiento elevado, tema relacionado con nuestra última videoconferencia sobre […]

[…] Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning From theelearningcoach.com – Today, 5:11 PM […]

[…] Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning From theelearningcoach.com – Today, 3:12 PM Use multiple-choice questions to measure critical thinking, such as analysis, evaluation and interpretation. Before and After examples. […]

[…] Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning From theelearningcoach.com – Today, 11:58 AM […]

[…] Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning From theelearningcoach.com – Today, 8:49 AM […]

[…] Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning From theelearningcoach.com – Today, 1:34 AM […]

[…] on theelearningcoach.com Like this:LikeBe the first to like this. PermalinkLeave a […]

[…] Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning From theelearningcoach.com – Today, 8:32 PM […]

[…] Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning From theelearningcoach.com – Today, 7:26 PM […]

[…] Writing Multiple Choice Questions For Higher Order Thinking … From theelearningcoach.com – Today, 2:53 AM Use multiple-choice questions to measure critical thinking, such as analysis, evaluation and interpretation. Before and After examples. […]

[…] Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning From theelearningcoach.com – Today, 4:47 AM Use multiple-choice questions to measure critical thinking, such as analysis, evaluation and interpretation. Before and After examples.One of the biggest criticisms of multiple choice questions is that they only test factual knowledge. But it doesn’t have to be that way. We can also use multiple choice questions to assess higher-order thinking. […]

[…] Writing Multiple Choice Questions For Higher Order Thinking: Instructional Design and eLearning From theelearningcoach.com – Today, 3:43 PM Use multiple-choice questions to measure critical thinking, such as analysis, evaluation and interpretation. Before and After examples. […]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed .

Distance Learning

Using technology to develop students’ critical thinking skills.

by Jessica Mansbach

What Is Critical Thinking?

Critical thinking is a higher-order cognitive skill that is indispensable to students, readying them to respond to a variety of complex problems that are sure to arise in their personal and professional lives. The  cognitive skills at the foundation of critical thinking are  analysis, interpretation, evaluation, explanation, inference, and self-regulation.  

When students think critically, they actively engage in these processes:

• Communication
• Problem-solving

To create environments that engage students in these processes, instructors need to ask questions, encourage the expression of diverse opinions, and involve students in a variety of hands-on activities that force them to be involved in their learning.

Types of Critical Thinking Skills

Instructors should select activities based on the level of thinking they want students to do and the learning objectives for the course or assignment. The chart below describes questions to ask in order to show that students can demonstrate different levels of critical thinking.

*Adapted from Brown University’s Harriet W Sheridan Center for Teaching and Learning

Using Online Tools to Teach Critical Thinking Skills

Online instructors can use technology tools to create activities that help students develop both lower-level and higher-level critical thinking skills.

• Example: Use Google Doc, a collaboration feature in Canvas, and tell students to keep a journal in which they reflect on what they are learning, describe the progress they are making in the class, and cite course materials that have been most relevant to their progress. Students can share the Google Doc with you, and instructors can comment on their work.
• Example: Use the peer review assignment feature in Canvas and manually or automatically form peer review groups. These groups can be anonymous or display students’ names. Tell students to give feedback to two of their peers on the first draft of a research paper. Use the rubric feature in Canvas to create a rubric for students to use. Show students the rubric along with the assignment instructions so that students know what they will be evaluated on and how to evaluate their peers.
• Example: Use the discussions feature in Canvas and tell students to have a debate about a video they watched. Pose the debate questions in the discussion forum, and give students instructions to take a side of the debate and cite course readings to support their arguments.  
• Example: Us e goreact , a tool for creating and commenting on online presentations, and tell students to design a presentation that summarizes and raises questions about a reading. Tell students to comment on the strengths and weaknesses of the author’s argument. Students can post the links to their goreact presentations in a discussion forum or an assignment using the insert link feature in Canvas.
• Example:  Use goreact, a narrated Powerpoint, or a Google Doc and instruct students to tell a story that informs readers and listeners about how the course content they are learning is useful in their professional lives. In the story, tell students to offer specific examples of readings and class activities that they are finding most relevant to their professional work. Links to the goreact presentation and Google doc can be submitted via a discussion forum or an assignment in Canvas. The Powerpoint file can be submitted via a discussion or submitted in an assignment.

Pulling it All Together

Critical thinking is an invaluable skill that students need to be successful in their professional and personal lives. Instructors can be thoughtful and purposeful about creating learning objectives that promote lower and higher-level critical thinking skills, and about using technology to implement activities that support these learning objectives. Below are some additional resources about critical thinking.

Additional Resources

Carmichael, E., & Farrell, H. (2012). Evaluation of the Effectiveness of Online Resources in Developing Student Critical Thinking: Review of Literature and Case Study of a Critical Thinking Online Site.  Journal of University Teaching and Learning Practice ,  9 (1), 4.

Lai, E. R. (2011). Critical thinking: A literature review.  Pearson’s Research Reports ,  6 , 40-41.

Landers, H (n.d.). Using Peer Teaching In The Classroom. Retrieved electronically from https://tilt.colostate.edu/TipsAndGuides/Tip/180

Lynch, C. L., & Wolcott, S. K. (2001). Helping your students develop critical thinking skills (IDEA Paper# 37. In  Manhattan, KS: The IDEA Center.

Mandernach, B. J. (2006). Thinking critically about critical thinking: Integrating online tools to Promote Critical Thinking. Insight: A collection of faculty scholarship , 1 , 41-50.

Yang, Y. T. C., & Wu, W. C. I. (2012). Digital storytelling for enhancing student academic achievement, critical thinking, and learning motivation: A year-long experimental study. Computers & Education , 59 (2), 339-352.

Insight Assessment: Measuring Thinking Worldwide

http://www.insightassessment.com/

Michigan State University’s Office of Faculty  & Organizational Development, Critical Thinking: http://fod.msu.edu/oir/critical-thinking

The Critical Thinking Community

http://www.criticalthinking.org/pages/defining-critical-thinking/766

Related Posts

USB Mic Comparison

Copying a Canvas Course with Panopto Videos

Video Strategy: December 2015 Online Learning Webinar

Scripts, Images, Action!: Creating Quality Videos for MUS 370

9 responses to “ Using Technology To Develop Students’ Critical Thinking Skills ”

This is a great site for my students to learn how to develop critical thinking skills, especially in the STEM fields.

Great tools to help all learners at all levels… not everyone learns at the same rate.

Thanks for sharing the article. Is there any way to find tools which help in developing critical thinking skills to students?

Technology needs to be advance to develop the below factors:

Understand the links between ideas. Determine the importance and relevance of arguments and ideas. Recognize, build and appraise arguments.

Excellent share! Can I know few tools which help in developing critical thinking skills to students? Any help will be appreciated. Thanks!

• Pingback: EDTC 6431 – Module 4 – Designing Lessons That Use Critical Thinking | Mr.Reed Teaches Math
• Pingback: Homepage
• Pingback: Magacus | Pearltrees

Brilliant post. Will be sharing this on our Twitter (@refthinking). I would love to chat to you about our tool, the Thinking Kit. It has been specifically designed to help students develop critical thinking skills whilst they also learn about the topics they ‘need’ to.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Insert/edit link

Enter the destination URL

Or link to existing content

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Psychol

Metacognitive Strategies and Development of Critical Thinking in Higher Education

Silvia f. rivas.

1 Departamento de Psicología Básica, Psicobiología y Metodología de CC, Facultad de Psicología, Universidad de Salamanca, Salamanca, Spain

Carlos Saiz

Carlos ossa.

2 Departamento de Ciencias de la Educación, Facultad de Educación y Humanidades, Universidad del Bío-Bío, Sede Chillán, Chile

Associated Data

The original contributions presented in the study are included in the article/supplementary material; further inquiries can be directed to the corresponding author.

More and more often, we hear that higher education should foment critical thinking. The new skills focus for university teaching grants a central role to critical thinking in new study plans; however, using these skills well requires a certain degree of conscientiousness and its regulation. Metacognition therefore plays a crucial role in developing critical thinking and consists of a person being aware of their own thinking processes in order to improve them for better knowledge acquisition. Critical thinking depends on these metacognitive mechanisms functioning well, being conscious of the processes, actions, and emotions in play, and thereby having the chance to understand what has not been done well and correcting it. Even when there is evidence of the relation between metacognitive processes and critical thinking, there are still few initiatives which seek to clarify which process determines which other one, or whether there is interdependence between both. What we present in this study is therefore an intervention proposal to develop critical thinking and meta knowledge skills. In this context, Problem-Based Learning is a useful tool to develop these skills in higher education. The ARDESOS-DIAPROVE program seeks to foment critical thinking via metacognition and Problem-Based Learning methodology. It is known that learning quality improves when students apply metacognition; it is also known that effective problem-solving depends not only on critical thinking, but also on the skill of realization, and of cognitive and non-cognitive regulation. The study presented hereinafter therefore has the fundamental objective of showing whether instruction in critical thinking (ARDESOS-DIAPROVE) influences students’ metacognitive processes. One consequence of this is that critical thinking improves with the use of metacognition. The sample was comprised of first-year psychology students at Public University of the North of Spain who were undergoing the aforementioned program; PENCRISAL was used to evaluate critical thinking skills and the Metacognitive Activities Inventory (MAI) for evaluating metacognition. We expected an increase in critical thinking scores and metacognition following this intervention. As a conclusion, we indicate actions to incentivize metacognitive work among participants, both individually via reflective questions and decision diagrams, and at the interactional level with dialogues and reflective debates which strengthen critical thinking.

Introduction

One of the principal objectives which education must cover is helping our students become autonomous and effective. Students’ ability to use strategies which help them direct their motivation toward action in the direction of the meta-proposal is a central aspect to keep at the front of our minds when considering education. This is where metacognition comes into play—knowledge about knowledge itself, a component which is in charge of directing, monitoring, regulating, organizing, and planning our skills in a helpful way, once these have come into operation. Metacognition helps form autonomous students, increasing consciousness about their own cognitive processes and their self-regulation so that they can regulate their own learning and transfer it to any area of their lives. As we see, it is a conscious activity of high-level thinking which allows us to look into and reflect upon how we learn and to control our own strategies and learning processes. We must therefore approach a problem which is increasing in our time, that of learning and knowledge from the perspective of active participation by students. To achieve these objectives of “learning to learn” we must use adequate cognitive learning strategies, among which we can highlight those oriented toward self-learning, developing metacognitive strategies, and critical thinking.

Metacognition is one of the research areas, which has contributed the most to the formation of the new conceptions of learning and teaching. In this sense, it has advanced within the constructivist conceptions of learning, which have attributed an increasing role to student consciousness and to the regulation which they exercise over their own learning ( Glaser, 1994 ).

Metacognition was initially introduced by John Flavell in the early 1970s. He affirmed that metacognition, on one side, refers to “the knowledge which one has about his own cognitive processes products, or any other matter related with them” and on the other, “to the active supervision and consequent regulation and organization of these processes in relation with the objects or cognitive data upon which they act” ( Flavell, 1976 ; p. 232). Based on this, we can differentiate two components of metacognition: one of a declarative nature, which is metacognitive knowledge, referring to knowledge of the person and the task, and another of a procedural nature, which is metacognitive control or self-regulated learning, which is always directed toward a goal and controlled by the learner.

Different authors have pointed out that metacognition presents these areas of thought or skills, aimed knowledge or toward the regulation of thought and action, mainly proposing a binary organization in which attentional processes are oriented, on occasions, toward an object or subject, and the other hand, toward to interact with objects and/or subjects ( Drigas and Mitsea, 2021 ). However, it is possible to understand metacognition from another approach that establishes more levels of use of metacognitive thinking to promote knowledge, awareness, and intelligence, known as the eight pillars of metacognition model ( Drigas and Mitsea, 2020 ). These pillars allow thought to promote the use of deep knowledge, cognitive processes, self-regulation, functional adaptation to society, pattern recognition and operations, and even meaningful memorization ( Drigas and Mitsea, 2020 ).

In addition to the above, Drigas and Mitsea’s model establishes different levels where metacognition could be used, in a complex sequence from stimuli to transcendental ideas, in which each of the pillars could manifest a different facet of the process metacognitive, thus establishing a dialectical and integrative approach to learning and knowledge, allowing it to be understood as an evolutionary and complex process in stages ( Drigas and Mitsea, 2021 ).

All this clarifies the importance of and need for metacognition, not only in education but also in our modern society, since this need to “teach how to learn” and the capacity to “learn how to learn” in order to achieve autonomous learning and transfer it to any area of our lives will let us face problems more successfully. This becomes a relevant challenge, especially today where it is required to have a broad view regarding reflection and consciousness, and to transcend simplistic and reductionist models that seek to center the problem of knowledge only around the neurobiological or the phenomenological scope ( Sattin et al., 2021 ).

Critical thinking depends largely on these mechanisms functioning well and being conscious of the processes used, since this gives us the opportunity to understand what has not been done well and correct it in the future. Consciousness for critical thinking would imply a continuous process of reuse of thought, in escalations that allow thinking to be oriented both toward the objects of the world and toward the subjective interior, allowing to determine the ideas that give greater security to the person, and in that perspective, the metacognitive process, represents this use of Awareness, also allowing the generation of an identity of knowing being ( Drigas and Mitsea, 2021 ).

We know that thinking critically involves reasoning and deciding to effectively solve a problem or reach goals. However, effective use of these skills requires a certain degree of consciousness and regulation of them. The ARDESOS-DIAPROVE program seeks precisely to foment critical thinking, in part, via metacognition ( Saiz and Rivas, 2011 , 2012 , 2016 ).

However, it is not only centered on developing cognitive components, as this would be an important limitation. Since the 1990s, it has been known that non-cognitive components play a crucial role in developing critical thinking. However, there are few studies focusing on this relation. This intervention therefore considers both dimensions, where metacognitive processes play an essential role by providing evaluation and control mechanisms over the cognitive dimension.

Metacognition and Critical Thinking

Critical Thinking is a concept without a firm consensus, as there have been and still are varying conceptions regarding it. Its nature is so complex that it is hard to synthesize all its aspects in a single definition. While there are numerous conceptions about critical thinking, it is necessary to be precise about which definition we will use. We understand that “ critical thinking is a knowledge-seeking process via reasoning skills to solve problems and make decisions which allows us to more effectively achieve our desired results” ( Saiz and Rivas, 2008 , p. 131). Thinking effectively is desirable in all areas of individual and collective action. Currently, the background of the present field of critical thinking is also based in argumentation. Reasoning is used as the fundamental basis for all activities labeled as thinking. In a way, thinking cannot easily be decoupled from reasoning, at least if our understanding of it is “deriving something from another thing.” Inference or judgment is what we essentially find behind the concept of thinking. The question, though, is whether it can be affirmed that thinking is only reasoning. Some defend this concept ( Johnson, 2008 ), while others believe the opposite, that solving problems and making decisions are activities which also form part of thinking processes ( Halpern, 2003 ; Halpern and Dunn, 2021 , 2022 ). To move forward in this sense, we will return to our previous definition. In that definition, we have specified intellectual activity with a goal intrinsic to all mental processes, namely, seeking knowledge. Achieving our ends depends not only on the intellectual dimension, as we may need our motor or perceptive activities, so it contributes little to affirm that critical thinking allows us to achieve our objectives as we can also achieve them by doing other activities. It is important for us to make an effort to identify the mental processes responsible for thinking and distinguish them from other things.

Normally, we think to solve our problems. This is the second important activity of thought. A problem can be solved by reasoning, but also by planning course of action or selecting the best strategy for the situation. Apart from reasoning, we must therefore also make decisions to resolve difficulties. Choosing is one of the most frequent and important activities which we do. Because of this, we prefer to give it the leading role it deserves in a definition of thinking. Solving problems demands multiple intellectual activities, including reasoning, deciding, planning, etc. The final characteristic goes beyond the mechanisms peculiar to inference. What can be seen at the moment of delineating what it means to think effectively is that concepts are grouped together which go beyond the nuclear ideas of what has to do with inferring or reasoning. The majority of theoreticians in the field ( APA, 1990 ; Ennis, 1996 ; Halpern, 1998 , 2003 ; Paul and Elder, 2001 ; Facione, 2011 ; Halpern and Dunn, 2021 , 2022 ) consider that, in order to carry out this type of thinking effectively, apart from having this skill set, the intervention of other types of components is necessary, such as metacognition and motivation. This is why we consider it necessary to speak about the components of critical thinking, as we can see in Figure 1 :

Components of critical thinking ( Saiz, 2020 ).

In the nature of thinking, there are two types of components: the cognitive and the non-cognitive. The former include perception, learning, and memory processes. Learning is any knowledge acquisition mechanism, the most important of which is thinking. The latter refer to motivation and interests (attitudes tend to be understood as dispositions, inclinations…something close to motives); with metacognition remaining as a process which shares cognitive and non-cognitive aspects as it incorporates aspects of both judgment (evaluation) and disposition (control/efficiency) about thoughts ( Azevedo, 2020 ; Shekhar and Rahnev, 2021 ). Both the cognitive and non-cognitive components are essential to improve critical thinking, as one component is incomplete without the other, that is, neither cognitive skills nor dispositions on their own suffice to train a person to think critically. In general, relations are bidirectional, although for didactic reasons only unidirectional relations appear in Figure 1 ( Rivas et al., 2017 ). This is because learning is a dynamic process which is subject to all types of influence. For instance, if a student is motivated, they will work more and better—or at least, this is what is hoped for. If they can achieve good test scores as well, it can be supposed that motivation is reinforced, so that they will continue existing behaviors in the same direction that is, working hard and well on their studies. This latter point appears to arise at least because of an adjustment between expectations and reality which the student achieves thanks to metacognition, which allows them to effectively attribute their achievements to their efforts ( Ugartetxea, 2001 ).

Metacognition, which is our interest in this paper, should also have bidirectional relations with critical thinking. Metacognition tends to be understood as the degree of consciousness which we have about our own mental processes and similar to the capacity for self-regulation, that is, planning and organization ( Mayor et al., 1993 ). We observe that these two ideas have very different natures. The former is simpler, being the degree of consciousness which we reach about an internal mechanism or process. The latter is a less precise idea, since everything which has to do with self-regulation is hard to differentiate from a way of understanding motivation, such as the entire tradition of intrinsic motivation and self-determination from Deci, his collaborators, and other authors of this focus (see, e.g., Deci and Ryan, 1985 ; Ryan and Deci, 2000 ). The important thing is to emphasize the executive dimension of metacognition, more than the degree of consciousness, for practical reasons. It can be expected that this dimension has a greater influence on the learning process than that of consciousness, although there is little doubt that we have to establish both as necessary and sufficient conditions. However, the data must speak in this regard. Due to all of this, and as we shall see hereinafter, the intervention designed incorporates both components to improve critical thinking skills.

We can observe, though, that the basic core of critical thinking continues to be topics related to skills, in our case, reasoning, problem-solving, and decision-making. The fact that we incorporate concepts of another nature, such as motivation, in a description of critical thinking is justified because it has been proven that, when speaking about critical thinking, the fact of centering solely on skills does not allow for fully gathering its complexity. The purpose of the schematic in Figure 2 is to provide conceptual clarity to the adjective “critical” in the expression critical thinking . If we understand critical to refer to effective , we should also consider that effectiveness is not, as previously mentioned, solely achieved with skills. They must be joined together with other mechanisms during different moments. Intellectual skills alone cannot achieve the effectiveness assumed within the term “critical.” First, for said skills to get underway, we must want to do so. Motivation therefore comes into play before skills and puts them into operation. For its part, metacognition allows us to take advantage of directing, organizing, and planning our skills and act once they have begun to work. Motivation thus activates our abilities, while metacognition lets them be more effective. The final objective should always be to gain proper knowledge of reality to resolve our problems.

Purpose of critical thinking ( Saiz, 2020 , p.27).

We consider that the fact of referring to components of critical thinking while differentiating the skills of motivation and metacognition aids with the conceptual clarification we seek. On one side, we specify the skills which we discuss, and on another, we mention which other components are related to, and even overlap with them. We must be conscious of how difficult it is to find “pure” mental processes. Planning a course of action, an essential trait of metacognition, demands reflection, prediction, choice, comparison, and evaluation… And this, evidently, is thinking. The different levels or dimensions of our mental activity must be related and integrated. Our aim is to be able to identify what is substantial in thinking to know what we are able to improve and evaluate.

It is widely known that for our personal and professional functioning, thinking is necessary and useful. When we want to change a situation or gain something, all our mental mechanisms go into motion. We perceive the situation, identify relevant aspects of the problem, analyze all the available information, and appraise everything we analyze. We make judgments about the most relevant matters, decide about the options or pathways for resolution, execute the plan, obtain results, evaluate the results, estimate whether we have achieved our purpose and, according to the level of satisfaction following this estimation, consider our course of action good, or not.

The topic we must pose now is what things are teachable. It is useful to specify that what is acquired is clearly cognitive and some of the non-cognitive, because motivation can be stimulated or promoted, but not taught. The concepts of knowledge and wisdom are its basis. Mental representation and knowledge only become wisdom when we can apply it to reality, when we take it out of our mind and adequately situate it in the world. For our teaching purposes, we only have to take a position about whether knowledge is what makes critical thinking develop, or vice versa. For us, skills must be directly taught, and dominion is secondary. Up to now, we have established the components of critical thinking, but these elements still have to be interrelated properly. What we normally find are skills or components placed side by side or overlapping, but not the ways in which they influence each other. Lipman (2003) may have developed the most complete theory of critical and creative thinking, along Paul and his group, in second place, with their universal thought structures ( Paul and Elder, 2006 ). However, a proposal for the relation between the elements is lacking.

To try to explain the relation between the components of thought, we will use Figure 2 as an aid.

The ultimate goal of critical thinking is change that is, passing from one state of wellbeing into a better state. This change is only the fruit of results, which must be the best. Effectiveness is simple achieving our goals in the best way possible. There are many possible results, but for our ends, there are always some which are better than others. Our position must be for effectiveness, the best response, the best solution. Reaching a goal is resolving or achieving something, and for this, we have mechanisms available which tell us which are the best course of action. Making decisions and solving problems are fundamental skills which are mutually interrelated. Decision strategies come before a solution. Choosing a course of action always comes before its execution, so it is easy to understand that decisions contribute to solutions.

Decisions must not come before reflection, although this often can and does happen. As we have already mentioned, the fundamental skills of critical thinking, in most cases, have been reduced to reasoning, and to a certain degree, this is justified. There is an entire important epistemological current behind this, within which the theory of argumentation makes no distinction, at least syntactically, between argumentation and explanation. However, for us this distinction is essential, especially in practice ( Saiz, 2020 ). We will only center on an essential difference for our purpose. Argumentation may have to do with values and realities, but explanation only has to do with the latter. We can argue about beliefs, convictions, and facts, but we can only explain realities. Faced with an explanation of reality, any argumentation would be secondary. Thus, explanation will always be the central skill in critical thinking.

The change which is sought is always expressed in reality. Problems always are manifested and resolved with actions, and these are always a reality. An argument about realities aids in explaining them. An argument about values upholds a belief or a conviction. However, beliefs always influence behavior; thus, indirectly, the argument winds up being about realities. One may argue, for example, only for or against the death penalty, and reach the conviction that it is good or bad and ultimately take a position for or against allowing it. This is why we say that deciding always comes before resolving; furthermore, resolution always means deciding about something in a particular direction—it always means choosing and taking an option; furthermore, deciding is often only from two possibilities, the better or that which is not better, or which is not as good. Decisions are made based on the best option possible of all those which can be presented. Resolution is a dichotomy. Since our basic end lies within reality, explanation must be constituted as the basic pillar to produce change. Argumentation must therefore be at the service of causality (explanation), and both must be in the service of solid decisions leading us to the best solution or change of situation. We now believe that the relation established in Figure 2 can be better understood. From this relation, we propose that thinking critically means reaching the best explanation for an event, phenomenon, or problem in order to know how to effectively resolve it ( Saiz, 2017 , p.19). This idea, to our judgment, is the best summary of the nature of critical thinking. It clarifies details and makes explicit the components of critical thinking.

Classroom Activities to Develop Metacognition

We will present a set of strategies to promote metacognitive work in the classroom in this section, aimed at improving critical thinking skills. These strategies can be applied both at the university level and the secondary school level; we will thus focus on these two levels, although metacognitive strategies can be worked on from an earlier age ( Jaramillo and Osses, 2012 ; Tamayo-Alzate et al., 2019 ) and some authors have indicated that psychological maturity has a greater impact on effectively achieving metacognition ( Sastre-Riba, 2012 ; García et al., 2016 ).

At the individual level, metacognition can be worked on via applying questions aimed at the relevant tasks which must be undertaken regarding a task (meta-knowledge questions), for example:

• Do I know how much I know about this subject?
• Do I have clear instructions and know what action is expected from me?
• How much time do I have?
• Am I covering the proper and necessary subjects, or is there anything important left out?
• How do I know that my work is right?
• Have I covered every point of the rubric for the work to gain a good grade or a sufficient level?

These reflective questions facilitate supervising knowledge level, resource use, and the final product achieved, so that the decisions taken for said activities are the best and excellent learning results are achieved.

Graphs or decision diagrams can also be used to aid in organizing these questions during the different phases of executing a task (planning, progress, and final evaluation), which is clearly linked with the knowledge and control processes of metacognition ( Mateos, 2001 ). These diagrams are more complex and elaborate strategies than the questions, but are effective when monitoring the steps considered in the activity ( Ossa et al., 2016 ). Decision diagrams begin from a question or task, detailing the principal steps to take, and associating an alternative (YES or NO) to each step, which leads to the next step whenever the decision is affirmative, or to improve or go further into the step taken if the decision is negative.

Finally, we can work on thinking aloud, a strategy which facilitates making the thoughts explicit and conscious, allowing us to monitor their knowledge, decisions, and actions to promote conscious planning, supervision and evaluation ( Ávila et al., 2017 ; Dahik et al., 2019 ). For example:

• While asking a question, the student thinks aloud: I am having problems with this part of the task, and I may have to ask the teacher to know whether I am right.

Thinking aloud can be done individually or in pairs, allowing for active monitoring of decisions and questions arising from cognitive and procedural work done by the student.

Apart from the preceding strategies, it is also possible to fortify metacognitive development via personal interactions based on dialogue between both the students themselves and between the teacher and individual students. One initial strategy, similar to thinking out loud in pairs, is reflective dialogue between teacher and student, a technique which allows for exchanging deep questions and answers, where the student becomes conscious of their knowledge and practice thanks to dialogical interventions by the teacher ( Urdaneta, 2014 ).

Reflective dialogue can also be done via reflective feedback implemented by the teacher for the students to learn by themselves about the positive and negative aspects of their performance on a task.

Finally, another activity based on dialogue and interaction is related to metacognitive argumentation ( Sánchez-Castaño et al., 2015 ), a strategy which uses argumentative resources to establish a valid argumentative structure to facilitate responding to a question or applying it to a debate. While argumentative analysis is based on logic and the search for solid reasons, these can have higher or lower confidence and reliability as a function of the data which they provide. Thus, if a reflective argumentative process is performed, via questioning reasons or identifying counterarguments, there is more depth and density in the argumentative structure, achieving greater confidence and validity.

We can note that metacognition development strategies are based on reflective capacity, which allow thought to repeatedly review information and decisions to consider, without immediately taking sides or being carried away by superficial or biased ideas or data. Critical thought benefits strongly from applying this reflective process, which guides both data management and cognitive process use. These strategies can also be developed in various formats (written, graphic, oral, individual, and dialogical), providing teachers a wide range of tools to strengthen learning and thinking.

Metacognitive Strategies to Improve Critical Thinking

In this section, we will describe the fundamental metacognitive strategies addressed in our critical thinking skills development program ARDESOS-DIAPROVE.

First, one of the active learning methodologies applied is Problem-Based Learning (PBL). This pedagogical strategy is student-centered and encourages autonomous and participative learning, orienting students toward more active and decisive learning. In PBL each situation must be approached as a problem-solving task, making it necessary to investigate, understand, interpret, reason, decide, and resolve. It is presented as a methodology which facilitates joint knowledge acquisition and skill learning. It is also good for working on daily problems via relevant situations, considerably reducing the distance between learning context and personal/professional life and aiding the connection between theory and practice, which promote the highly desired transference. It favors organization and the capacity to decide about problem-solving, which also improves performance and knowledge about the students’ own learning processes. Because of all this, this methodology aids in reflection and analysis processes, which in turn promotes metacognitive skill development.

The procedure which we carried out in the classroom with all the activities is based on the philosophy of gradual learning control transference ( Mateos, 2001 ). During instruction, the teacher takes on the role of model and guide for students’ cognitive and metacognitive activity, gradually bringing them into participating in an increasing level of competency, and slowly withdrawing support in order to attain control over the students’ learning process. This methodology develops in four phases: (1) explicit instruction, where the teacher directly explains the skills which will be worked on; (2) guided practice, where the teacher acts as a collaborator to guide and aid students in self-regulation; and (3) cooperative practice, where cooperative group work facilitates interaction with a peer group collaborating to resolve the problem. By explaining, elaborating, and justifying their own points of view and alternative solutions, greater consciousness, reflection, and control over their own cognitive processes is promoted. Finally, (4) individual practice is what allows students to place their learning into practice in individual evaluation tasks.

Regarding the tasks, it is important to highlight that the activities must be aimed not only at acquiring declarative knowledge, but also at procedural knowledge. The objective of practical tasks, apart from developing fundamental knowledge, is to develop CT skills among students in both comprehension and expression in order to favor their learning and its transference. The problems used must be common situations, close to our students’ reality. The important thing in our task of teaching critical thinking is its usefulness to our students, which can only be achieved during application since we only know something when we are capable of applying it. We are not interested in students merely developing critical skills; they must also be able to generalize their intellectual skills, for which they must perceive them as useful in order to want to acquire them. Finally, they will have to actively participate to apply them to solving problems. Furthermore, if we study the different ways of reasoning without context, via overly academic problems, their application to the personal sphere becomes impossible, leading them to be considered hardly useful. This makes it important to contextualize skills within everyday problems or situations which help us get students to use them regularly and understand their usefulness.

Reflecting on how one carries things out in practice and analyzing mistakes are ways to encourage success and autonomy in learning. These self-regulation strategies are the properly metacognitive part of our study. The teacher has various resources to increase these strategies, particularly feedback oriented toward task resolution. Similarly, one of the most effective instruments to achieve it is using rubrics, a central tool for our methodology. These guides, used in student performance evaluations, describe the specific characteristics of a task at various performance levels, in order to clarify expectations for students’ work, evaluate their execution, and facilitate feedback. This type of technique also allows students to direct their own activity. We use them with this double goal in mind; on the one hand, they aid students in carrying out tasks, since they help divide the complex tasks they have to do into simpler jobs, and on the other, they help evaluate the task. Rubrics guide students in the skills and knowledge they need to acquire as well as facilitating self-evaluation, thereby favoring responsibility in their learning. Task rubrics are also the guide for evaluation which teachers carry out in classrooms, where they specify, review, and correctly resolve the tasks which students do according to the rubric criteria. Providing complete feedback to students is a crucial aspect for the learning process. Thus, in all sessions time is dedicated to carrying it out. This is what will allow them to move ahead in self-regulated skill learning.

According to what we have seen, there is a wide range of positions when it comes to defining critical thinking. However, there is consensus in the fact that critical thinking involves cognitive, attitudinal, and metacognitive components, which together favor proper performance in critical thinking ( Ennis, 1987 ; Facione, 1990 ). This important relation between metacognition and critical thinking has been widely studied in the literature ( Berardi-Coletta et al., 1995 ; Antonietti et al., 2000 ; Kuhn and Dean, 2004 ; Black, 2005 ; Coutinho et al., 2005 ; Orion and Kali, 2005 ; Schroyens, 2005 ; Akama, 2006 ; Choy and Cheah, 2009 ; Magno, 2010 ; Arslan, 2014 ) although not always in an applied way. Field studies indicate the existence of relations between teaching metacognitive strategies and progress in students’ higher-order thinking processes ( Schraw, 1998 ; Kramarski et al., 2002 ; Van der Stel and Veenman, 2010 ). Metacognition is thus considered one of the most relevant predictors of achieving a complex higher-order thought process.

Along the same lines, different studies show the importance of developing metacognitive skills among students as it is related not only with developing critical thinking, but also with academic achievement and self-regulated learning ( Klimenko and Alvares, 2009 ; Magno, 2010 ; Doganay and Demir, 2011 ; Özsoy, 2011 ). Klimenko and Alvares (2009) indicated that one way for students to acquire necessary tools to encourage autonomous learning is making cognitive and metacognitive strategies explicit and well-used and that teachers’ role is to be mediators and guides. Inspite of this evidence, there is less research about the use of metacognitive strategies in encouraging critical thinking. The principal reason is probably that it is methodologically difficult to gather direct data about active metacognitive processes which are complex by nature. Self-reporting is also still very common in metacognition evaluation, and there are few studies which have included objective measurements aiding in methodological precision for evaluating metacognition.

However, in recent years, greater importance has been assigned to teaching metacognitive skills in the educational system, as they aid students in developing higher-order thinking processes and improving their academic success ( Flavell, 2004 ; Larkin, 2009 ). Because of this, classrooms have seen teaching and learning strategies emphasizing metacognitive knowledge and regulation. Returning to our objective, which is to improve critical thinking via the ARDESOS-DIAPROVE program, we have achieved our goal in an acceptable way ( Saiz and Rivas, 2011 , 2012 , 2016 ).

However, we need to know which specific factors contribute to this improvement. We have covered significant ground through different studies, one of which we present here. In this one, we attempt to find out the role of metacognition in critical thinking. This is the central objective of the study. Our program includes motivational and metacognitive variables. Therefore, we seek to find out whether metacognition improves after this instruction program focused on metacognition. Therefore, our hypothesis is simple: we expect that the lesson will improve our students’ metacognition. The idea is to know whether applying metacognition helps us achieve improved critical thinking and whether after this change metaknowledge itself improves. In other words, improved critical thinking performance will make us think better about thinking processes themselves. If this can be improved, we can expect that in the future it will have a greater influence on critical thinking. The idea is to be able to demonstrate that applying specifically metacognitive techniques, the processes themselves will subsequently improve in quality and therefore contribute better volume and quality to reasoning tasks, decision-making and problem-solving.

Materials and Methods

Participants.

In the present study, we used a sample of 89 students in a first-year psychology course at Public University of the North of Spain. 82% (73) were women, and the other 18% (16) were men. Participants’ median age was 18.93 ( SD 1.744).

Instruments

Critical thinking test.

To measure critical thinking skills, we applied the PENCRISAL test ( Saiz and Rivas, 2008 ; Rivas and Saiz, 2012 ). The PENCRISAL is a battery consisting of 35 production problem situations with an open-answer format, composed of five factors: Deductive Reasoning , Inductive Reasoning , Practical Reasoning , Decision-Making , and Problem-Solving , with seven items per factor. Items for each factor gather the most representative structures of fundamental critical thinking skills.

The items’ format is open, so that the person has to answer a concrete question, adding a justification for the reasons behind their answer. Because of this, there are standardized correction criteria assigning values between 0 and 2 points as a function of answer quality. This test offers us a total score of critical thinking skills and another five scores referring to the five factors. The value range is located between 0 and 72 points as a maximum limit for total test scoring, and between 0 and 14 for each of the five scales. The reliability measures present adequate precision levels according to the scoring procedures, with the lowest Cronbach’s alpha values at 0.632, and the test–retest correlation at 0.786 ( Rivas and Saiz, 2012 ). PENCRISAL administration was done over the Internet via the evaluation platform SelectSurvey.NET V5: http://24.selectsurvey.net/pensamiento-critico/Login.aspx .

Metacognitive Skill Inventory

Metacognitive skill evaluation was done via the metacognitive awareness inventory from Schraw and Dennison (1994) (MAI; Huertas Bustos et al., 2014 ). This questionnaire has 52 Likert scale-type items with five points. The items are distributed in two general dimensions: cognitive knowledge (C) and regulation of cognition (R). This provides ample coverage for the two aforementioned ideas about metaknowledge. There are also eight defined subcategories within each general dimension. For C, these are: declarative knowledge (DK), procedural knowledge (PK), and conditional knowledge (CK). In R, we find: organization (O), monitoring (M), and evaluation (E). This instrument comprehensively, and fairly clearly, brings together essential aspects of metacognition. On one side, there is the level of consciousness, containing types of knowledge—declarative, procedural, and strategic. On the other, it considers everything important in the processes of self-regulation, planning, organization, direction or control (monitoring), adjustment (troubleshooting), and considering the results achieved (evaluation). It provides a very complete vision of everything important in this dimension. Cronbach’s alpha for this instrument is 0.94, showing good internal consistency.

Intervention Program

As previously mentioned, in this study, we applied the third version of the ARDESOS_DIAPROVE program ( Saiz and Rivas, 2016 ; Saiz, 2020 ), with the objective of improving thinking skills. This program is centered on directly teaching the skills which we consider essential to develop critical thinking and for proper performance in our daily affairs. For this, we must use reasoning and good problem-solving and decision-making strategies, with one of the most fundamental parts of our intervention being the use of everyday situations to develop these abilities.

DIAPROVE methodology incorporates three new and essential aspects: developing observation, the combined use of facts and deduction, and effective management of de-confirmation procedures, or discarding hypotheses. These are the foundation of our teaching, which requires specific teaching–learning techniques.

The intervention took place over 16 weeks and is designed to be applied in classrooms over a timeframe of 55–60 h. The program is applied in classes of around 30–35 students divided into groups of four for classwork in collaborative groups, and organized into six activity blocks: (1) nature of critical thinking, (2) problem-solving and effectiveness, (3) explanation and causality, (4) deduction and explanation, (5) argumentation and deduction, and (6) problem-solving and decision-making. These blocks are assembled maintaining homogeneity, facilitating a global integrated skill focus which helps form comprehension and use of the different structures in any situation as well as a greater degree of ability within the domain of each skill.

Our program made an integrated use of problem-based learning (PBL) and cooperative learning (CL) as didactic teaching and learning strategies in the critical thinking program. These methodologies jointly exert a positive influence on the students, allowing them to participate more actively in the learning process, achieve better results in contextualizing content and developing skills and abilities for problem-solving, and improve motivation.

To carry out our methodology in the classrooms, we have designed a teaching system aligned with these directives. Two types of tasks are done: (1) comprehension and (2) production. The materials we used to carry out these activities are the same for all the program blocks. One key element in our aim of teaching how to think critically must be its usefulness to our students, which is only achieved through application. This makes it important to contextualize reasoning types within common situations or problems, aiding students to use them regularly and understand their usefulness. Our intention with the materials we use is to face the problems of transference, usefulness, integrated skills, and how to produce these things. Accordingly, the materials used for the tasks are: (1) common situations and (2) professional/personal problems.

The tasks which the students perform take place over a week. They work in cooperative groups in class, and then review, correct, and clarify together, promoting reflection on their achievements and errors, which fortifies metacognition. Students get the necessary feedback on the work performed which will help them progressively acquire fundamental procedural contents. Our goal here is that students become conscious of their own thought processes in order to improve them. In this way, via the dialogue achieved between teachers and students as well as between the students themselves in their cooperative work, metacognition is developed. For conscious performance of tasks, the students will receive rubrics for each and every task to guide them in their completion.

Application of the ARDESOS-DIAPROVE program was done across a semester in the Psychology Department of the Public University of the North of Spain. One week before teaching began; critical thinking and metacognition evaluations were done. This was also done 1 week after the intervention ended, in order to gather the second measurement for PENCRISAL and MAI. The timelapse between the pre-treatment and post-treatment measurements was 4 months. The intervention was done by instructors with training and good experience in the program.

To test our objective, we used a quasi-experimental pre-post design with repeated measurements.

Statistical Analysis

For statistical analysis, we used the IBM SPSS Statistics 26 statistical packet. The statistical tools and techniques used were: frequency and percentage tables for qualitative variables, exploratory and descriptive analysis of quantitative variables with a goodness of fit test to the normal Gaussian model, habitual descriptive statistics (median, SD, etc.) for numerical variables, and Student’s t -tests for significance of difference.

To begin, a descriptive analysis of the study variables was carried out. Tables 1 , ​ ,2 2 present the summary of descriptions for the scores obtained by students in the sample, as well as the asymmetry and kurtosis coefficients for their distribution.

Description of critical thinking measurement (PENCRISAL).

TOT_PRE, PENCRISAL pre-test; RD_PRE, Deductive reasoning pre-test; RI_PRE, Inductive reasoning pre-test; RP_PRE, Practical reasoning pre-test; TD_PRE, Decision making pre-test; SP_PRE, Problem solving pre-test; TOT_POST, PENCRISAL post-test; RD_ POST, Deductive reasoning post-test; RI_ POST, Inductive reasoning post-test; RP_ POST, Practical reasoning post-test; TD_ POST, Decision making post-test; SP_ POST, Problem solving post-test; Min, minimum, Max, maximum, Asym, asymmetry; and Kurt, kurtosis.

Description of metacognition measurement (MAI).

TOT_MAI_PRE, MAI pre-test; Decla_PRE, Declarative pre-test; Proce_PRE, Procedural pre-test; Condi_PRE, Conditional pre-test; CONO_PRE, Knowledge pre-test; Plani_PRE, Planning pre-test; Orga_PRE, Organization pre-test; Moni_PRE, Monitoring pre-test; Depu_PRE, Troubleshooting pre-test; Eva_PRE, Evaluation pre-test; REGU_PRE, Regulation pre-test; TOT_MAI_POST, MAI post-test; Decla_ POST, Declarative post-test; Proce_ POST, Procedural post-test; Condi_ POST, Conditional post-test; CONO_ POST, Knowledge post-test; Plani_ POST, Planning post-test; Orga_POST, Organization post-test; Moni_ POST, Monitoring post-test; Depu_ POST, Troubleshooting post-test; Eva_ POST, Evaluation post-test; and REGU_ POST, Regulation post-test;

As we see in the description of all study variables, the evidence is that the majority of them adequately fit the normal model, although some present significant deviations which can be explained by sample size.

Next, to verify whether there were significant differences in the metacognition variable based on measurements before and after the intervention, we contrasted medians for samples related with Student’s t -test (see Table 3 ).

Comparison of the METAKNOWLEDGE variable as a function of PRE-POST measurements.

The results show that there are significant differences in the metaknowledge scale total and in most of its dimensions, where all the post medians for both the scale overall and for the three dimensions of the knowledge factor (declarative, procedural, and conditional) are higher than the pre-medians. However, in the cognition regulation dimension, there are only significant differences in the total and in the planning, organization, and monitoring dimensions. The medians are also greater in the post-test than the pre-test. However, the troubleshooting and evaluation dimensions do not differ significantly after intervention.

Finally, for critical thinking skills, the results show significant differences in the scale total and in the five factors regarding the measurement time, where performance medians rise after intervention (see Table 4 ).

Comparison of the CRITICAL THINKING variable as a function of PRE-POST measurements.

These results show how metacognition improves due to CT intervention, as well as how critical thinking also improves with metacognitive intervention and CT skills intervention. Thus, it improves how people think about thinking as well as about the results achieved, since metacognition supports decision-making and final evaluation about proper strategies to solve problems.

Discussion and Conclusions

The general aim of our study was to know whether a critical thinking intervention program can also influence metacognitive processes. We know that our teaching methodology improves cross-sectional skills in argumentation, explanation, decision-making, and problem-solving, but we do not know if this intervention also directly or indirectly influences metacognition. In our study, we sought to shed light on this little-known point. If we bear in mind the centrality of how we think about thinking for our cognitive machinery to function properly and reach the best results possible in the problems we face, it is hard to understand the lack of attention given to this theme in other research. Our study aimed to remedy this deficiency somewhat.

As said in the introduction, metacognition has to do with consciousness, planning, and regulation of our activities. These mechanisms, as understood by many authors, have a blended cognitive and non-cognitive nature, which is a conceptual imprecision; what is known, though, is the enormous influence they exert on fundamental thinking processes. However, there is a large knowledge gap about the factors which make metacognition itself improve. This second research lacuna is what we have partly aimed to shrink here as well with this study. Our guide has been the idea of knowing how to improve metacognition from a teaching initiative and from the improvement of fundamental critical thinking skills.

Our study has shed light in both directions, albeit in a modest way, since its design does not allow us to unequivocally discern some of the results obtained. However, we believe that the data provide relevant information to know more about existing relations between skills and metacognition, something which has seen little contrast. These results allow us to better describe these relations, guiding the design of future studies which can better discern their roles. Our data have shown that this relation is bidirectional, so that metacognition improves thinking skills and vice versa. It remains to establish a sequence of independent factors to avoid this confusion, something which the present study has aided with to be able to design future research in this area.

As the results show, total differences in almost all metaknowledge dimensions are higher after intervention; specifically, we see how in the knowledge factor the declarative, procedural, and conditional dimensions improve in post-measurements. This improvement moves in the direction we predicted. However, the cognitive regulation dimension only shows differences in the total, and in the planning, organization, and regulation dimensions. We can see how the declarative knowledge dimensions are more sensitive than the procedural ones to change, and within the latter, the dimensions over which we have more control are also more sensitive. With troubleshooting and evaluation, no changes are seen after intervention. We may interpret this lack of effects as being due to how everything referring to evaluating results is highly determined by calibration capacity, which is influenced by personality factors not considered in our study. Regarding critical thinking, we found differences in all its dimensions, with higher scores following intervention. We can tentatively state that this improved performance can be influenced not only by interventions, but also by the metacognitive improvement observed, although our study was incapable of separating these two factors, and merely established their relation.

As we know, when people think about thinking they can always increase their critical thinking performance. Being conscious of the mechanisms used in problem-solving and decision-making always contributes to improving their execution. However, we need to go into other topics to identify the specific determinants of these effects. Does performance improve because skills are metacognitively benefited? If so, how? Is it only the levels of consciousness which aid in regulating and planning execution, or do other factors also have to participate? What level of thinking skills can be beneficial for metacognition? At what skill level does this metacognitive change happen? And finally, we know that teaching is always metacognitive to the extent that it helps us know how to proceed with sufficient clarity, but does performance level modify consciousness or regulation level of our action? Do bad results paralyze metacognitive activity while good ones stimulate it? Ultimately, all of these open questions are the future implications which our current study has suggested. We believe them to be exciting and necessary challenges, which must be faced sooner rather than later. Finally, we cannot forget the implications derived from specific metacognitive instruction, as presented at the start of this study. An intervention of this type should also help us partially answer the aforementioned questions, as we cannot obviate what can be modified or changed by direct metacognition instruction.

Data Availability Statement

Ethics statement.

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

SR and CS contributed to the conception and design of the study. SR organized the database, performed the statistical analysis, and wrote the first draft of the manuscript. SR, CS, and CO wrote sections of the manuscript. All authors contributed to the article and approved the submitted version.

This study was partly financed by the Project FONDECYT no. 11220056 ANID-Chile.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

• Akama K. (2006). Relations among self-efficacy, goal setting, and metacognitive experiences in problem solving . Psychol. Rep. 98 , 895–907. doi: 10.2466/pr0.98.3.895-907, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Antonietti A., Ignazi S., Perego P. (2000). Metacognitive knowledge about problem solving methods . Br. J. Educ. Psychol. 70 , 1–16. doi: 10.1348/000709900157921, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• APA (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction. Executive Summary “The Delphi Report.”
• Arslan S. (2014). An investigation of the relationships between metacognition and self-regulation with structural equation . Int. Online J. Educ. Sci. 6 , 603–611. doi: 10.15345/IOJES.2014.03.009 [ CrossRef ] [ Google Scholar ]
• Ávila M., Bianchetti M., González A. (2017). Uso del método “Think Aloud” en la investigación cualitativa . Pistas Educ. 39 , 26–38. [ Google Scholar ]
• Azevedo R. (2020). Reflections on the field of metacognition: issues, challenges, and opportunities . Metacogn. Learn. 15 , 91–98. doi: 10.1007/s11409-020-09231-x [ CrossRef ] [ Google Scholar ]
• Berardi-Coletta B., Buyer L. S., Dominowski R. L., Rellinger E. R. (1995). Metacognition and problem solving: a process-oriented approach . J. Consult. Clin. Psychol. 21 , 205–223. [ Google Scholar ]
• Black S. (2005). Teaching students to think critically . Educ. Digest 70 , 42–47. [ Google Scholar ]
• Choy S. C., Cheah P. K. (2009). Teacher perceptions of critical thinking among students and its influence on higher education . Int. J. Teach. Learn. Higher Educ. 20 , 198–206. [ Google Scholar ]
• Coutinho S., Wiemer-Hastings K., Skowronski J. J., Britt M. A. (2005). Metacognition, need for cognition and use of explanations during ongoing learning and problem solving . Learn. Individ. Differ. 15 , 321–337. doi: 10.1016/j.lindif.2005.06.001 [ CrossRef ] [ Google Scholar ]
• Dahik S., Cáneppa C., Dahik C., Feijoò K. (2019). Estrategias de Think-Aloud para mejorar la habilidad de lectura en estudiantes en el centro de idiomas en la universidad técnica de Babahoyo . Rev. Magaz. Ciencias 4 , 65–83. doi: 10.5281/zenodo.3239552 [ CrossRef ] [ Google Scholar ]
• Deci E. L., Ryan R. M. (1985). The general causality orientations scale: self-determination in personality . J. Res. Pers. 19 , 109–134. doi: 10.1016/0092-6566(85)90023-6, PMID: [ CrossRef ] [ Google Scholar ]
• Doganay A., Demir O. (2011). Comparison of the level of using metacognitive strategies during study between high achieving and low achieving prospective teachers . Educ. Sci. Theor. Pract. 11 , 2036–2043. [ Google Scholar ]
• Drigas A., Mitsea E. (2020). The 8 pillars of metacognition . Int. J. Emerg. Technol. Learn. 15 , 162–178. doi: 10.3991/ijet.v15i21.14907 [ CrossRef ] [ Google Scholar ]
• Drigas A., Mitsea E. (2021). 8 pillars X 8 layers model of metacognition: educational strategies, exercises and trainings . Int. J. Online Biomed. Eng. 17 , 115–134. doi: 10.3991/ijoe.v17i08.23563 [ CrossRef ] [ Google Scholar ]
• Ennis R. H. (1987). “ A taxonomy of critical thinking dispositions and abilities ,” in Teaching Thinking Skills. eds. Baron J. B., Sternberg R. J. (New York: Freeman and Company; ), 9–26. [ Google Scholar ]
• Ennis R. H. (1996). Critical Thinking. Upper Saddle River, NJ: Prentice-Hall [ Google Scholar ]
• Facione P. A. (1990). Critical Thinking: A Statement of expert consensus for Purposes of Educational Assessment and Instruction—Executive Summary of the delphi Report. Millbrae: California Academic Press [ Google Scholar ]
• Facione P. A. (2011). Think Critically. New York: Prentice-Hall. [ Google Scholar ]
• Flavell J. H. (1976). “ Metacognitive aspects of problem solving ,” in The Nature of Intelligence. ed. Resnik L. B. (Hillsdale, N.J: Erlbaum; ), 231–235. [ Google Scholar ]
• Flavell J. H. (2004). Theory of the mind development: retrospect and prospect . Merrill-Palmer Q. 50 , 274–290. doi: 10.1353/mpq.2004.0018 [ CrossRef ] [ Google Scholar ]
• García T., Rodríguez C., González-Castro P., Álvarez-García D., González-Pienda J.-A. (2016). Metacognición y funcionamiento ejecutivo en Educación Primaria [Metacognition and executive functioning in Elementary School] . Ann. Psychol. 32 , 474–483. doi: 10.6018/analesps.32.2.202891 [ CrossRef ] [ Google Scholar ]
• Glaser R. (1994). “ Learning theory and instruction ,” in International Perspectives on Psychological Science. Vol. 2 . eds. D’Ydewalle G., Eelen P., Bertelson B. (NJ: Erlbaum; ) [ Google Scholar ]
• Halpern D. (1998). Teaching critical thinking for transfer across domains—dispositions, skills, structure training, and metacognitive monitoring . Am. Psychol. 53 , 449–455. doi: 10.1037/0003-066X.53.4.449, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Halpern D. (2003). Halpern Critical Thinking Assessment Using Everyday Situations: Background and Scoring Standards. Claremont, CA: Claremont McKenna College. [ Google Scholar ]
• Halpern D. F., Dunn D. S. (2021). Critical thinking: A model of intelligence for solving real-world problems . J. Intellig. 9 :22. doi: 10.3390/jintelligence9020022, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Halpern D. F., Dunn D. S. (2022). Thought and Knowledge. An Introduction to Critical-Thinking . 6th Edn. New York: Taylor and Francis. [ Google Scholar ]
• Huertas Bustos A. P., Vesga Bravo G. J., Gilando León M. (2014). Validación del instrumento “Inventario de Habilidades Metacognitivas (MAI)” con estudiantes colombianos . Praxis Saber 5 , 55–74. doi: 10.19053/22160159.3022 [ CrossRef ] [ Google Scholar ]
• Jaramillo S., Osses S. (2012). Validación de un Instrumento sobre Metacognición para Estudiantes de Segundo Ciclo de Educación General Básica . Estud. Pedag. 38 , 117–131. doi: 10.4067/S0718-07052012000200008 [ CrossRef ] [ Google Scholar ]
• Johnson R. H. (2008). “Critical thinking, logic and argumentation,” in Paper presented at the Conferencia Internacional: Lógica, Argumentación y Pensamiento Crítico. Santiago de Chile. January 8–11. [ Google Scholar ]
• Klimenko O., Alvares J. L. (2009). Aprender cómo aprendo: la enseñanza de estrategias metacognitivas . Educ. Educ. 12 , 11–28. [ Google Scholar ]
• Kramarski B., Mevarceh Z. R., Arami M. (2002). The effect of metacognitive instruction on solving mathematical authentic tasks . Educ. Stud. Math. 49 , 225–250. doi: 10.1023/A:1016282811724 [ CrossRef ] [ Google Scholar ]
• Kuhn D., Dean D. (2004). Metacognition: a bridge between cognitive psychology and educational practice . Theory Pract. 43 , 268–274. doi: 10.1207/s15430421tip4304_4 [ CrossRef ] [ Google Scholar ]
• Larkin S. (2009). Metacognition in Young Children. New York, NY: Routledge [ Google Scholar ]
• Lipman M. (2003). Thinking in Education (2nd Edn.). Cambridge, MA: Cambridge University Press [ Google Scholar ]
• Magno C. (2010). The role of metacognitive skills in developing critical thinking . Metacogn. Learn. 5 , 137–156. doi: 10.1007/s11409-010-9054-4, PMID: [ CrossRef ] [ Google Scholar ]
• Mateos M. (2001). Metacognición y Educación. Buenos Aires: Aique [ Google Scholar ]
• Mayor J., Suengas A., González Marqués J. (1993). Estrategias Metacognitivas. Aprender a Aprendery Aprender a Pensar. Madrid: Síntesis [ Google Scholar ]
• Orion N., Kali Y. (2005). The effect of an earth-science learning program on students’ scientific thinking skills . J. Geosci. Educ. 53 , 387–394. doi: 10.5408/1089-9995-53.4.387 [ CrossRef ] [ Google Scholar ]
• Ossa C., Rivas S.F., Saiz C. (2016). Estrategias metacognitivas en el desarrollo del análisis argumentativo En IV Seminário Internacional Cognição, aprendizagem e desempenho. eds. Casanova J., Bisinoto C., Almeida L. (Braga: Livro de atas; ), 30–47. [ Google Scholar ]
• Özsoy G. (2011). An investigation of the relationship between metacognition and mathematics achievement . Asia Pac. Educ. Rev. 12 , 227–235. doi: 10.1007/s12564-010-9129-6 [ CrossRef ] [ Google Scholar ]
• Paul R., Elder L. (2001). Critical Thinking Handbook: Basic Theory and Instructional Structures. Dillon Beach, CA: Foundation for Critical Thinking [ Google Scholar ]
• Paul R., Elder A. D. (2006). Critical Thinking. Learn the Tools the Best Thinkers Use. Upper Saddle River, NJ: Pearson/Prentice Hall [ Google Scholar ]
• Rivas S. F., Saiz C. (2012). Validación y propiedades psicométricas de la prueba de pensamiento crítico PENCRISAL . Rev. Electrón. Metodol. Aplic. 17 , 18–34. [ Google Scholar ]
• Rivas S. F., Saiz C., Ossa C. (2017). “ Desarrollo de las estrategias metacognitivas mediante el programa de instrucción en pensamiento crítico ARDESOS .” in II Seminario Internacional de 660 Pensamiento Crítico. Manizales (Colombia). de octubre de 11–13, 2017. [ Google Scholar ]
• Ryan R. M., Deci E. L. (2000). Intrinsic and extrinsic motivations: classic definitions and new directions . Contemp. Educ. Psychol. 21 , 54–67. doi: 10.1006/ceps.1999.1020, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Saiz C. (2017). Pensamiento Crítico y Cambio. Madrid: Pirámide. [ Google Scholar ]
• Saiz C. (2020). Pensamiento Crítico y Eficacia. 2ª Edn. Madrid: Pirámide. [ Google Scholar ]
• Saiz C., Rivas S. F. (2008). Evaluación en pensamiento crítico: una propuesta para diferenciar formasde pensar . Ergo. Nueva Época 22-23 , 25–66. [ Google Scholar ]
• Saiz C., Rivas S. F. (2011). Evaluation of the ARDESOS program: an initiative to improve criticalthinking skills . J. Scholar. Teach. Learn. 11 , 34–51. [ Google Scholar ]
• Saiz C., Rivas S. F. (2012). Pensamiento crítico y aprendizaje basado en problemas . Rev. Docenc. Univ. 10 , 325–346. doi: 10.4995/redu.2012.6026 [ CrossRef ] [ Google Scholar ]
• Saiz C., Rivas S. F. (2016). New teaching techniques to improve critical thinking . DIAPROVE Methodol. 40 , 3–36. [ Google Scholar ]
• Sánchez-Castaño J. A., Castaño-Mejía O. Y., Tamayo-Alzate O. E. (2015). La argumentación metacognitiva en el aula de ciencias . Rev. Latin. Cienc. Soc. 13 , 1153–1168. doi: 10.11600/1692715x.13242110214 [ CrossRef ] [ Google Scholar ]
• Sastre-Riba S. (2012). Alta capacidad intelectual: perfeccionismo y regulación metacognitiva . Rev. Neurol. 54 , S21–S29. doi: 10.33588/rn.54S01.2012011, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Sattin D., Magnani F. G., Bartesaghi L., Caputo M., Fittipaldo A. V., Cacciatore M., et al.. (2021). Theoretical models of consciousness: a scoping review . Brain Sci. 11 :535. doi: 10.3390/brainsci11050535, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Schraw G. (1998). Promoting general metacognitive awareness . Instr. Sci. 26 , 113–125. doi: 10.1023/A:1003044231033, PMID: [ CrossRef ] [ Google Scholar ]
• Schraw G., Dennison R. (1994). Assessing metacognitive awareness . Contemp. Educ. Psychol. 19 , 460–475. doi: 10.1006/ceps.1994.1033, PMID: [ CrossRef ] [ Google Scholar ]
• Schroyens W. (2005). Knowledge and thought: an introduction to critical thinking . Exp. Psychol. 52 , 163–164. doi: 10.1027/1618-3169.52.2.163, PMID: [ CrossRef ] [ Google Scholar ]
• Shekhar M., Rahnev D. (2021). Sources of metacognitive inefficiency . Trends Cogn. Sci. 25 , 12–23. doi: 10.1016/j.tics.2020.10.007, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Tamayo-Alzate O., Cadavid-Alzate V., Montoya-Londoño D. (2019). Análisis metacognitivo en estudiantes de básica, durante la resolución de dos situaciones experimentales en la clase de Ciencias Naturales . Rev. Colomb. Educ. 76 , 117–141. doi: 10.17227/rce.num76-4188 [ CrossRef ] [ Google Scholar ]
• Ugartetxea J. (2001). Motivación y metacognición, más que una relación . Relieve 7 :4442. doi: 10.7203/relieve.7.2.4442 [ CrossRef ] [ Google Scholar ]
• Urdaneta M. (2014). Diálogo para la reflexión: compartiendo la experiencia de aula desde el proyecto pedagógico . Innov. Educ. 16 , 43–49. doi: 10.22458/ie.v16i21.902 [ CrossRef ] [ Google Scholar ]
• Van der Stel M., Veenman M. V. J. (2010). Development of metacognitive skillfulness: a longitudinal study . Learn. Individ. Differ. 20 , 220–224. doi: 10.1016/j.lindif.2009.11.005 [ CrossRef ] [ Google Scholar ]