^{a}

^{a}

^{a}

^{a}

Fractions, known to be difficult for both children and adults, are especially prone to misconceptions and erroneous strategy selection. The present study investigated whether a computer tutor improves fraction arithmetic performance in adults and if supplementing problem solving with erroneous examples is more beneficial than problem solving alone. Seventy-five undergraduates solved fraction arithmetic problems using a computer tutoring system we designed. In a between-subjects design, 39 participants worked with a problem-solving tutor that was supplemented with erroneous examples and 36 participants worked with a traditional problem-solving tutor. Both tutors provided hints and feedback. Overall, participants improved after the tutoring interventions, but there were no significant differences in gains made by the two conditions. For students with low prior knowledge about fraction arithmetic, the numerical gains were higher in the erroneous-example group than the problem-solving group, but this effect was not significant. Thus, computer tutors are useful tools for improving fraction knowledge. While erroneous examples may be particularly beneficial for students with low prior knowledge who may hold more misconceptions, more research is needed to make this conclusion.

Fraction skills are integral in science-based professions as well as less math-intensive jobs such as home care (calculating dosages) and retail sales (calculating discounts). Sixty-eight percent of adults report using fractions in their workplace (

Although adults have been exposed to fractions at school, they continue to struggle with this challenging domain (

The present study investigates whether a fraction computer tutor improves fraction performance. A second goal was to compare two types of instructional activities in terms of their impact on fraction performance. A traditional activity corresponds to problem solving, where students are given problems to work on and guidance for that process. However, problem solving alone induces a high cognitive load, which interferes with learning (

A proposed benefit of erroneous examples relates to misconception refutation. Students at the beginning and intermediate stages of cognitive skill acquisition have various misconceptions because their knowledge is not yet complete or refined (

In sum, the goals of the present study were to investigate if a computer tutor for fraction arithmetic improved adult fraction performance and to analyze whether supplementing problem solving with erroneous examples improved learning over problem solving alone for students with low and high prior knowledge. Erroneous examples and traditional problems were presented by a computer tutor we designed and implemented. The tutor provided immediate feedback for correctness on solution entries and interactively prompted users to self-explain and correct example errors. We describe this tutor after describing the target domain, namely fraction arithmetic.

Fraction arithmetic errors are well documented. In the current study we focused on two sources of arithmetic errors: Errors attributed to

One commonly observed source of errors in fraction arithmetic is

Two theoretical views have been proposed to account for natural number bias. The

Case | Operation | Denominator | Error | |
---|---|---|---|---|

1 | Addition | Common | Adding both the numerators and denominators | |

2 | Addition | Different | Adding the numerators, choosing the larger denominator | |

3 | Subtraction | Common & Different | Subtracting both the numerators and denominators | |

4 | Multiplication | Common | Multiplying the numerators, keeping the denominator | |

5 | Division | Common | Dividing the numerators, keeping the denominator | |

6 | Division^{a} |
Common & Different | Inverting the first fraction and then multiplying |

^{a}Error specific to adults.

In addition to errors resulting from natural number bias, fraction errors may be related to gaps in prior knowledge. These gaps are the result of poorly encoded fraction concepts in the first place and/or due to forgetting previously taught content. The degree of forgetting is related to the distribution and amount of practice, with procedures that are not practiced frequently being more likely to be forgotten (

In summary, fraction arithmetic is a suitable domain to assess both the effectiveness of a computer tutor and the effectiveness of erroneous examples for learning and reviewing of mathematical content. Fractions skills are important, but there are well-documented errors in adults’ fraction arithmetic. Some of these errors are based on fraction misconceptions (i.e., natural number bias) and gaps in prior knowledge. However, adults have prior experience with fraction procedures, thus they come with a basic foundation, allowing them to reflect on their mistakes. Erroneous examples are designed to address fraction misconceptions and provide an opportunity for reflection. Fraction arithmetic thus provides an appropriate testing ground for our work.

The format of an erroneous example can vary, but a common approach outlined in

Erroneous examples have been used successfully to refute misconceptions in various areas of mathematics with students in middle and high school (

In a replication of the

Other studies have found more nuanced effects of erroneous examples. In particular, contrary to the findings of

It would, however, be premature to conclude that erroneous examples are only beneficial for high-prior-knowledge-students, as the next study illustrates.

A potential downside of erroneous examples relates to affect. Learning from erroneous examples may be a more confusing and frustrating process, with higher confusion and frustration levels linked to poorer learning outcomes (

In summary, there is some evidence that erroneous examples promote learning, but there is a lack of consensus as to who will benefit from erroneous examples in comparison to traditional examples (i.e., low- vs. high-prior-knowledge students). In general, factors like the target domain and the age of the participants may influence the results and so more work is needed for a full understanding of when and how erroneous examples impact learning. In the present study we explore the utility of erroneous examples to supplement problem-solving with a computer tutor in the domain of fraction arithmetic.

The instructional materials used in this study were administered by two versions of a computer tutor we built using the Cognitive Tutor Authoring Tool (CTAT;

We piloted initial versions of the tutor with both adults and children in order to obtain a range of feedback on aesthetics, wording, and difficulty. Though the tutor was not tested with children in the present study, it was designed so that it could be used by people of all ages who are learning or reviewing fraction concepts and procedures. First, the tutor was piloted with five numerical cognition experts. Based on their feedback, the examples were simplified, and the order of the example solution steps was made more explicit. Once these changes were implemented, the tutor was piloted with three children between the ages of 8 and 10. Based on their feedback, questions were made less wordy and aesthetics, such as the size of pictures, were adjusted.

Using the feedback from the pilots, we implemented two versions of the fraction tutor: An erroneous-example (EE) tutor that supplemented problem solving with erroneous examples, and a problem-solving (PS) tutor that did not include the erroneous example component. Both tutors contained the same six fraction word problems, namely two single-digit addition fraction problems, one single-digit subtraction problem, one single digit multiplication problem, and two single-digit division problems (see

The erroneous example components for the EE tutor were based on the common fraction errors observed in adults (

The EE tutor interface consisted of five sections and a Feedback Centre. As shown in

The PS tutor was populated with the same fraction problems as the EE tutor, but the interface did not include the erroneous example components (i.e., Sections 2 and 3, see

Both the EE tutor and the PS tutor provided immediate feedback on each user response by colouring correct responses green and incorrect responses red. Both tutors included a Feedback Centre that was always visible in the right panel on the screen (see

Both the EE tutor and the PS tutor required that all items in a given exercise be answered correctly before the user could move on to the next exercise (example and subsequent problem in the EE tutor; problem in the PS tutor); all questions were completed in a set order in both tutors and the order was the same for both tutors. To move from one item to the next, participants pressed the “Done” button located in the Feedback Centre (see

The present study compared learning outcomes from the two versions of the computer tutor described above – namely one that supplemented traditional problem solving with erroneous examples (EE tutor) and one that consisted of only traditional problem solving (PS tutor). In both tutor versions participants received hints and feedback. The target population was undergraduate students who had not taken any university-level math courses. A prior study with undergraduate students found that erroneous examples were most effective for students with some prior knowledge of the topic (

Eighty-seven undergraduate students from a Canadian university participated in the study (_{age} = 21.06 years,

Basic demographic information was collected, including participant age, gender, and program of study.

To measure learning, participants completed two fraction operation tests: A pre-test and a post-test. The pre-test and post-test were paper-and-pencil tests designed for the present study. Both the pre- and post-test consisted of 18 unique items: Four addition (e.g.,

As part of a larger study, several other brief questionnaires about math attitudes and beliefs were administered (i.e., Math Confidence Scale (

A between-subjects design was used with the two computer tutors (EE, PS) serving as the two experimental conditions. Participants were assigned to one of the two conditions, alternating between EE and PS assignment to ensure an even number of participants were in each condition. The procedure for both conditions was the same.

The study took place in a laboratory; most experimental sessions included one participant, but several sessions included two (who were seated at opposite ends of a large room). Prior to entering the lab participants were unaware the study involved fractions (the title of the study on the online recruiting site was “Education with AI” and participants were told the study required completing a series of problem-solving tasks). After informed consent was obtained, participants were given 20 minutes to complete the fraction operation pre-test to assess their initial fraction performance; they were not allowed to use any assistive tools (i.e., calculators) and could not ask for help. Following this, participants completed several questionnaires (not reported in this study). They then had 20 minutes to work through the six questions using either the EE or PS tutor. After finishing these questions, participants were given 20 minutes to complete the fraction post-test. Finally, participants completed a post-tutor attitudes and beliefs questionnaire (not reported). The entire study took between one and one and a half hours to complete.

Of the 87 participants recruited for the study, 75 were included in the analyses. Ten participants were excluded because they obtained a perfect score on the pre-test and thus were at ceiling performance. Two other two participants were excluded from the analysis (one had difficulty navigating the computer tutor and was unable to finish the tutor intervention and the other participant’s data was lost). In total, there were 39 participants in the erroneous-example group and 36 participants in the problem-solving group.

The analyses related to our research questions were conducted using ANOVAs and _{01}, is “a ratio that contrasts the likelihood of the data fitting under the null hypothesis with the likelihood of fitting under the alternative hypothesis” (_{10}, puts the ratio in terms of the alternative hypothesis. In the present study, when the Bayes factor is in favour of the null hypothesis, the BF_{01} is reported; when in favour of the alternative hypothesis, the BF_{10} is reported. The interpretation of the strength of the evidence for the null or alternative hypothesis is in accordance with

The descriptive statistics for the pre-test, post-test, and gain scores are in

To measure learning, we used the standard method of calculating gain scores (i.e., post-test score – pre-test score). Collapsing across tutor groups, participants gained significantly from pre-test to post-test, _{10} = 2.19e+08. A one-way ANOVA found no significant difference in gain scores between the EE and PS groups, ^{2} = .01. The estimated Bayes factor, BF_{01} = 2.82 indicates anecdotal support for the null hypothesis that the EE tutor did not provide additional benefits in comparison to the PS tutor. In summary, the computer tutor improved learning (Research Question I) but we did not find support that supplementing problem solving with erroneous examples resulted in higher learning than using a traditional problem-solving tutor (Research Question II).

Previous studies have suggested that a student’s prior knowledge can influence how much they gain from erroneous examples (e.g.,

Measure | Low-Level |
High-Level |
||||||
---|---|---|---|---|---|---|---|---|

EE ( |
PS ( |
EE ( |
PS ( |
|||||

Pre-test | 4.88 | 2.76 | 5.40 | 2.47 | 14.88 | 1.78 | 14.79 | 1.65 |

Post-test | 10.47 | 4.89 | 8.87 | 4.79 | 15.88 | 2.45 | 16.79 | 1.48 |

Gains | 5.59 | 4.45 | 3.47 | 4.16 | 1.00 | 2.10 | 2.00 | 1.80 |

There was no significant difference between the EE and PS groups on pre-test scores for low-level-knowledge participants,

To assess learning, gain scores (post-test – pre-test) were analyzed in a 2(knowledge level: low, high) by 2(tutor: erroneous example, problem solving) between-subjects ANOVA. Of primary interest is the interaction between tutor type and knowledge level, since it informs on whether the effect of erroneous examples depends on knowledge level (see ^{2} = .05. The Bayes factor, BF_{10} = 1.44, indicates anecdotal evidence for the inclusion of the interaction in the model. As shown in

There was also a significant main effect of knowledge level, ^{2} = .18, which indicates that low-level-knowledge participants made greater learning gains than high-level-knowledge participants. The Bayes factor, BF_{10} = 66.60, indicates very strong support for the model that includes the effect of knowledge level. There was no main effect of tutor, ^{2} = .00, and the Bayes factor, BF_{01} = 3.04, indicated substantial support for the null hypothesis that collapsed across knowledge levels, the two tutor groups did not differ in learning gains.

Both versions of the tutor (EE, PS) were designed to provide instruction on procedures for fraction arithmetic. An exploratory analysis was conducted to identify the types of errors participants made, to see if the tutors reduced errors and to analyze if the tutor version reduced specific types of errors.

We first flagged incorrect pre-test and post-test responses and then classified the errors based on the common misconceptions (see

Error | Addition |
Subtraction |
Multiplication |
Division |
||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

Pre-test |
Post-test |
Pre-test |
Post-test |
Pre-test |
Post-test |
Pre-test |
Post-test |
|||||||||

EE | PS | EE | PS | EE | PS | EE | PS | EE | PS | EE | PS | EE | PS | EE | PS | |

Conceptual Error | ||||||||||||||||

Added/subtracted numerator and denominator | 30 | 16 | 12 | 8 | 16 | 13 | 0 | 2 | – | – | – | – | – | – | – | – |

Only multiplied denominator when finding common denominator | 0 | 4 | 3 | 5 | 0 | 3 | 3 | 7 | – | – | – | – | – | – | – | – |

Used bigger denominator as denominator | 1 | 1 | 0 | 0 | 3 | 1 | 2 | 0 | – | – | – | – | – | – | – | – |

Found common denominator then multiplied/added/subtracted/divided numerators | – | – | – | – | – | – | – | – | 15 | 4 | 11 | 11 | 14 | 10 | 6 | 11 |

Cross multiplied | – | – | – | – | – | – | – | – | 7 | 29 | 1 | 5 | – | – | – | – |

Inverted and multiplied | – | – | – | – | – | – | – | – | 6 | 6 | 0 | 12 | – | – | – | – |

Added numerator and multiplied denominator | – | – | – | – | – | – | – | – | 2 | 1 | 0 | 6 | – | – | – | – |

Inverted first fraction | – | – | – | – | – | – | – | – | – | – | – | – | 2 | 4 | 0 | 4 |

Cross divided/ inverted and divided | – | – | – | – | – | – | – | – | – | – | – | – | 7 | 12 | 9 | 0 |

Multiplied instead of divided | – | – | – | – | – | – | – | – | – | – | – | – | 4 | 5 | 0 | 1 |

Arithmetic Error | ||||||||||||||||

Reduction error | 2 | 2 | 2 | 4 | 0 | 4 | 2 | 1 | 1 | 1 | 6 | 2 | 1 | 6 | 1 | 4 |

Arithmetic error | 6 | 5 | 9 | 13 | 14 | 8 | 9 | 6 | 8 | 2 | 8 | 5 | 5 | 1 | 7 | 8 |

Miscellaneous | 8 | 3 | 6 | 2 | 8 | 5 | 14 | 1 | 9 | 3 | 10 | 1 | 16 | 8 | 11 | 6 |

Blank Response | ||||||||||||||||

Blank or “I don’t know” | 0 | 4 | 3 | 0 | 6 | 9 | 6 | 6 | 0 | 28 | 4 | 3 | 54 | 55 | 9 | 9 |

The total number of conceptual errors did not significantly differ between the EE and PS groups on the pre-test (109 vs. 107) or on the post-test (72 vs. 47), χ^{2}(1, ^{2}(1, ^{2}(1,

Importantly, when we totaled errors across both groups from pre-test to post-test, the number of conceptual errors decreased (216 vs. 119), whereas the number of arithmetic errors did not (126 vs. 138), χ^{2}(1, ^{2}(1,

The goal of the present study was to investigate the utility of a computer tutor and erroneous examples in the domain of fraction arithmetic. Fraction arithmetic was chosen because adults frequently require the use of fractions in the workplace (

With the goal of improving fraction understanding, we built two versions of a fraction tutor using CTAT, a platform designed to support the construction of computer tutors. Unlike textbooks or worksheets, CTAT tutors are capable of sophisticated tutoring behaviours (

Although participants learned from their interaction with the fraction tutor, we did not find evidence that supplementing problem solving with erroneous examples improved learning over traditional problem solving. We used Bayesian analyses in addition to frequentist statistics to triangulate results from alternative analysis methods. Bayesian statistics provided definitive evidence that the computer tutor led to significant learning gains. However, neither frequentist nor Bayesian statistics provided strong evidence in favour of erroneous examples. When all participants regardless of prior knowledge were considered, the effect size of erroneous examples was small and the Bayes Factor in favour of the null model was 2.8 (i.e., the null model was 2.8 times as likely as the alternative model that included erroneous examples). A Bayes Factor (BF) of 1-3 is considered anecdotal evidence for a given model (null or alternative), while a BF of 3-10 provides substantial evidence. Our results are close to the boundary between these two thresholds. Even if the Bayesian analysis provided substantial evidence, however, the small sample effect size suggests erroneous examples are not adding much.

While to the best of our knowledge our work is the first to incorporate Bayesian statistics into analyses about erroneous examples, in general our findings are in line with the findings of

Some have reported that erroneous examples increased learning only for students with certain prior knowledge, although conflicting patterns have been reported. Some studies report that erroneous examples are only beneficial for students with high prior knowledge (

The exploratory error analysis focused on identifying fraction misconceptions and whether interacting with the tutor reduced their frequency. Comparing the EE and PS versions of the fraction tutor, there were no significant differences in the types of errors made on either the pre-test or post-test. The latter result was somewhat surprising as the EE tutor explicitly illustrated conceptual errors and gave learners an opportunity to correct them by identifying the proper procedure. For both groups, the number of conceptual errors decreased in the post-test. Thus, while we did not find evidence that erroneous examples reduced errors more than standard problem solving, interacting with the fraction tutor did reduce fraction arithmetic misconceptions. Furthermore, the reduction in blank responses from pre-test to post-test means participants were more willing to try and devise a solution. This is a positive result as in educational settings, a blank response often receives no marks and cannot receive feedback to correct errors since no errors were recorded.

The present study involved the domain of fraction arithmetic, something both children and adults struggle with (

One important consideration is the feasibility of designing a computer tutor. Building a computer tutor traditionally has required technical expertise and many hours of programming. For the present study, we used CTAT (

In the present study, one test was used as a pre-test and a second test was used as a post-test. While the tests were carefully designed so that similar magnitude numerators and denominators were selected, these were not counterbalanced. The study also did not include a delayed post-test. Some previous studies have found that the benefits of erroneous examples only appear on delayed post-tests (

Another limitation relates to the design of the tutor. The erroneous-example group was not required to generate free-form responses to identify and correct the error in the example, instead selecting items from a multiple-choice question. The latter strategy is standard in computer tutors (

Yet another consideration relates to the inclusion of additional conditions. Future studies could compare the computer tutor conditions to paper-and-pencil erroneous example materials as well as paper-and-pencil problem-solving, to separate benefits of a computer tutor from benefits of the type of example. Future studies could also include a “pure” control condition, in which participants would not receive an intervention, instead just completing the pre-test, a non-math related task, and the post-test, to see if any gains occur simply from the pre-test priming participants’ memory about fraction procedures.

The present study investigated the pedagogical value of a computer tutor for fraction arithmetic as well as the utility of erroneous examples to supplement problem solving. Both versions of the fraction tutor were successful as participants improved significantly from pre-test to post-test. In addition to standard frequentist statistics, our results were supplemented with effect sizes and Bayesian statistics. Despite all these methods, we did not find evidence that supplementing problem-solving activities with erroneous examples produced higher learning compared to problem solving without examples. Additionally, although low-prior-knowledge students in the erroneous-example condition had higher learning gains than in the problem-solving condition, this difference was not significant. Given that to date there are relatively few studies on erroneous examples, and even fewer with adults, there is a clear need for more work in this area.

Thank you to Kyle Sale for providing technical assistance with the tutor and Khadijah Brooks for assisting with data collection.

This project was funded by the Social Sciences and Humanities Research Council of Canada and the Natural Sciences and Engineering Research Council of Canada.

^{TM}: Summary research results

The authors have declared that no competing interests exist.