This is an open access article distributed under the terms of the Creative Commons
Attribution License (As students progress beyond arithmetic in middle school, they are challenged to learn to attend to the structure of math notation (Hoch & Dreyfus, 2004; Kieran, 1989; Linchevski & Livneh, 1999). However, there is growing evidence that students struggle with noticing and understanding the structure of math notation (Hoch & Dreyfus, 2004; Linchevski & Livneh, 1999; Papadopoulos & Gunnarsson, 2020). Linchevski and Livneh (1999) first introduced the term “structure sense” to describe students’ ability to grasp arithmetic structures. Examples of having a strong structure sense include being able to identify equivalent forms of a math expression (e.g., 3 (x + 2) + 5 is equivalent to 5 + 3 (x + 2) as well as 3x + 11) and to discriminate expression forms that are relevant to the task from those that are not (Hoch & Dreyfus, 2004). Students often have weak structure sense, which is shown through two widespread misunderstandings: 1) the detachment of a math term from its indicated operation (e.g., mistaking 4 + n – 2 + 5 as equivalent to 4 + n – 7), and 2) incorrectly linking operations to non-adjacent terms (e.g., in the equation 115 − n + 9 = 61, ignoring “n” to subtract 9 and create 106 – n = 61; Linchevski & Livneh, 1999; Papadopoulos & Gunnarsson, 2020). Importantly, developing structure sense is dependent on students understanding the mathematical meaning of notations as well as the appropriate rules and operations to apply within expressions. 1) the detachment of a math term from its indicated operation (e.g., mistaking 4 + n – 2 + 5 as equivalent to 4 + n – 7), and 2) incorrectly linking operations to non-adjacent terms (e.g., in the equation 115 – n + 9 = 61, ignoring “n” to subtract 9 and create 106 – n = 61; Linchevski & Livneh, 1999; Papadopoulos & Gunnarsson, 2020). Importantly, developing structure sense is dependent on students understanding the mathematical meaning of notations as well as the appropriate rules and operations to apply within expressions.
Aside from structure sense, the position of terms within a math expression may play a role in how students reason about math. Students may also struggle to resist the urge to solve math problems from left to right. Early middle school students have a strong tendency to adhere to the left-to-right principle when solving problems, which may lead them to overlook and violate the order of operations (Banerjee & Subramaniam, 2005; Blando et al., 1989; Gunnarsson et al., 2016; Kieran, 1979). For instance, Kieran (1979) found that when solving 5 + 2 * 3, many middle school students added 5 and 2 first, indicating that they overlooked the order of operations and instead evaluated math notations in a left-to-right sequence. By following the left-to-right tendency in the equation 5 + 2 * 3, students reached the wrong answer; however, when those students solved for 4 * 2 – 3, they correctly identified that they had to first multiply 4 and 2 (Kieran, 1979). This observation suggested that the position of higher-order operators (e.g., multiplication and division) within a math expression may influence students’ reasoning and performance in problem solving. To extend this research, our study aims to uncover the isolated effects of higher-order operator position (hereafter referred to as HOO) on students’ performance simplifying order-of-operations problems.
One potential way to support students’ structure sense and help them notice the order of operations when solving math problems is to increase the visual salience of important cues in an expression. Brackets are often used in mathematics to group numbers together, to emphasize, and/or to identify precedence. In some cases, brackets are necessary to indicate precedence (i.e., (a + b) * c + d) and in other times, brackets are used only for emphasis, or are superfluous–meaning they do not change the meaning of the mathematical expression when removed (i.e., a + (b * c) + d). Prior research has found that, even in cases where brackets are not present or necessary, students have a natural inclination to use mental brackets to interpret arithmetic and algebraic expressions (e.g., Papadopoulos & Gunnarsson, 2020; Papadopoulos & Thoma, 2022). Many students are taught the acronyms BEDMAS (Brackets, Exponents, Division, Multiplication, Addition, and Subtraction) or PEMDAS (depending on the country) to remember as a set of rules for applying operations. Due to this emphasis, students may be relying on and using superfluous brackets mentally as a failsafe way to remember their precedence rules. An alternative explanation is that superfluous brackets may be used by students as a perceptual grouping cue that could be used to make crucial mathematical structures like HOO more salient to students and help with encoding during problem solving. Regardless of the mechanism, these studies indicate that the presence of superfluous brackets-—brackets that do not change the meaning of notation—can guide learners’ attention to the correct procedures and improve performance.
A number of previous studies have investigated how superfluous brackets influence mathematical problem solving (e.g., Hoch & Dreyfus, 2004; Marchini & Papadopoulos, 2011). For example, Linchevski and Livneh (1999) found that students who lacked structure sense (and were therefore likely to focus on surface structures over systemic structures of math terms and operations) would struggle with expressions of the type 𝑎 ± 𝑏 × 𝑐, as students would need to understand that b should be connected with the c, not a, due to order of precedence. The authors also suggested that inserting superfluous brackets around 𝑏 × 𝑐 could help students understand that the multiplication operation should be calculated first, followed by addition, making the order of operation rules (i.e., BEDMAS or PEMDAS) more salient and explicit. Other research exploring the role of superfluous brackets as a perceptual cue has found that students are more accurate at algebraic problem solving when equations include superfluous brackets to emphasize structures compared to equations that do not (e.g., Hoch & Dreyfus, 2004; Marchini & Papadopoulos, 2011). In other work, superfluous brackets have been shown to help students achieve higher success rates solving problems (Hoch & Dreyfus, 2004; Marchini & Papadopoulos, 2011; Papadopoulos & Gunnarsson, 2018, 2020). For instance, in a study with elementary school students in Italy and Greece, Marchini and Papadopoulos (2011) found that when students completed math expressions with superfluous brackets, they were more likely to calculate some expressions correctly and were more likely to recognize important expression structures (i.e., the order of operations). This finding suggests that superfluous brackets can highlight the structural elements of an expression for students; in turn, attending to important problem elements may improve students’ structure sense and consequently, support their math problem solving. However, in other work, adding superfluous brackets have been found to have no effect (Gunnarsson et al., 2016) and can lead to misinterpretations and procedural errors (e.g., Ayres, 2006; Hewitt, 2005; Kieran, 1979).
Given these mixed results, it is important for researchers and educators to better understand how adding superfluous brackets to math notations could be used to help support student performance and learning. In addition to understanding superfluous brackets’ influences on outcomes, it is also important to better understand how brackets and HOO position within a math expression influence students’ performance to identify the situations where these brackets may help versus hinder students. The current study aims to advance our understanding of how operator position and superfluous brackets within a math expression can affect students’ problem-solving performance in an online learning environment.
The goals of our study are threefold. First, we aim to examine the isolated impact of HOO position on students’ performance on simple arithmetic problems in an online activity. Second, we aim to extend prior research by testing whether the presence of superfluous brackets can support student performance. Third, we assess if the presence of superfluous brackets moderates the impact of HOO position. Specifically, we examine how the presence of superfluous brackets and the position of HOOs (i.e., multiplication and division) within an expression independently and simultaneously influence student performance.
Theoretical Framework
Perceptual Learning and Mathematical Structure
Learning math is naturally dependent on our perceptual processes; the way that we perceive incoming stimuli informs the way we think about it (Gibson, 1969; Marghetis et al., 2016). Perceptual learning theory suggests that learning and reasoning in math does not only depend on our perceptual processes, but that we actually develop perceptual-motor routines over time to direct our attention towards salient perceptual cues in math notation (Goldstone et al., 2017; Jacob & Hochstein, 2008; Kellman et al., 2010; Kirshner & Awtry, 2004; Patsenko & Altmann, 2010). In particular, the visual features of math notation (e.g., spacing, symbols, color) act as perceptual cues that can highlight information and influence the way we reason and act on notation. Incidentally, intentional changes to the presentation of notation, even very subtle changes, can be used as perceptual scaffolding to impact students’ performance on tasks such as simple arithmetic and equation solving.
Prominent Visual and Perceptual Grouping Mechanisms
Although multiple visual features may have no bearing on the mathematical meaning of notation, they can direct individuals’ attention towards structures within the notation and act as perceptual grouping mechanisms, impacting learners’ performance on problem solving. For example, visual features that are proximal, as opposed to distal, to one another are more likely to be perceptually grouped together by our visual systems according to the Gestalt principles of perceptual grouping (Hartmann, 1935; Wertheimer, 1938). For instance, the spatial proximity between symbols in expressions and equations impacts how individuals solve math problems. Individuals solve problems more quickly and accurately when the spatial proximity in notation supports the order of operations (e.g., 6*3 + 4). However, individuals struggle more to solve problems (i.e., taking more time to solve and making more errors) when the spatial proximity between symbols does not support the order of operations (e.g., 6 * 3+4). This finding is true for grade school and college students alike (e.g., Braithwaite et al., 2016; Gómez, Benavides-Varela, et al., 2014; Gómez, Bossi, & Dartnell, 2014; Harrison et al., 2020; Landy & Goldstone, 2007, 2010; Rivera & Garrigan, 2016). In fact, the effects of spatial proximity on students’ performance have been shown to increase with students’ age from grade two to six (Braithwaite et al., 2016), and do not decline among secondary students (Harrison et al., 2020). These findings suggest that the reliance on spatial proximity as a perceptual grouping mechanism is pervasive across developmental stages as well as ranges in math knowledge.
In addition to the spatial proximity of symbols, several other visual features can also serve as attentional cues in math notation. For example, strategic coloring can be used as a perceptual grouping mechanism to support students’ generation of problem-solving strategies (Alibali et al., 2018). Alibali and colleagues (2018) found that fourth-grade students who received instructional materials that highlighted relevant features to encode with different colored ink than the rest of an equation during a practice session were more likely to generate correct problem-solving strategies and improve from pretest to posttest than their counterparts who received materials without highlighting or with irrelevant equation features highlighted (Alibali et al., 2018). Landy and Goldstone (2007), in a sample of college students, found the same effect by manipulating the alphanumeric proximity of terms within an expression to be congruent or incongruent with the order of operations (i.e., students performed better on problems such as “a + p * q + z = a + q * p + z” and worse on problems such as “r + s * b + c = r + s * b + c”). Together, this research demonstrates the robust effect of perceptual grouping mechanisms on problem solving in math for a wide range of students and ages, suggesting that the ways in which we reason about math notation is highly dependent on our visual perception.
Brackets as a Visual and Perceptual Grouping Mechanism
In arithmetic, brackets can be used in several ways. First, within math expressions, brackets typically signify grouping in which the math content in the group takes precedence over its surroundings based on the order of operations (e.g., in “10 ÷ (2 + 3)”). Second, brackets can be used superfluously to visually highlight the structural elements of an expression without changing the mathematical meaning of the expression (Gunnarsson et al., 2016). For example, when students write , the brackets preserve the structure of the initial rational expression and highlight the relation between the numerator and the denominator of the fraction. In this case, like spatial proximity and color highlighting, superfluous brackets serve as a perceptual grouping mechanism that can prime students to attend to the HOO.
One explanation for why superfluous brackets could work as a visual grouping mechanism is that they create a common visual region within math expressions that draws students’ attention to a specific area of math notation (Landy & Goldstone, 2007). According to the common visual region principle, individuals tend to group together visual elements that are located within the same bounded visual region (Palmer, 1992). These visual regions can be demarcated using color shifts, visual separators, or, in the case of brackets, object boundaries. For instance, when seeing “(6 * 3) + 4”, students tend to quickly group “(6*3)” together because those terms are bound together by brackets. In fact, an eye-tracking study by Schneider and colleagues (2012) showed that adults were drawn to notation presented within a pair of brackets. For example, when presented with two math expressions, “(3 – 2) + 4” and “4 + (3 – 2)”, adults spent more time fixating on “(3 – 2)” than “+ 4”, showing that superfluous brackets serve as perceptual cues to capture attention. Hoch and Dreyfus (2004) also found similar grouping and attention-directing effects of superfluous brackets. Eleventh-grade students solved math problems more efficiently with the presence of superfluous brackets (e.g., than without (e.g., ). This finding suggests that superfluous brackets focus students’ attention and alert them to the possibility of like terms, leading them to more efficient problem-solving strategies.
Research on the use of superfluous brackets as a visual cue has been primarily conducted in classrooms (Gunnarsson et al., 2016; Hoch & Dreyfus, 2004; Marchini & Papadopoulos, 2011; Papadopoulos & Gunnarsson, 2018, 2020). Some of this research provides evidence that superfluous brackets positively impact students’ math performance through more accurate problem solving, higher performance on posttests, and better student understanding of the math expression’s structure (e.g., Hoch & Dreyfus, 2004; Marchini & Papadopoulos, 2011; Papadopoulos & Gunnarsson, 2018, 2020). However, other results have been contradictory. For example, Gunnarsson and colleagues (2016) found that the use of superfluous brackets did not enhance students’ performance as they learned the order of operations and even led to lower performance at posttest. However, that study consisted of a short intervention on the order of operations. It is possible that those findings might be impacted by the quality of the intervention, rather than by the presence of the superfluous brackets itself, motivating further work on the impact of superfluous brackets as a perceptual visual grouping mechanism.
The Present Study
The current study aims to provide additional empirical evidence about how superfluous brackets affect student performance with two important extensions of past work. First, this work explores the differential impacts of higher-order operator position on math performance. Second, much of the prior work has been conducted using paper and pencil tasks. Online learning and use of technology platforms in the math classroom have grown significantly and become more centric to education due to the COVID-19 pandemic, motivating us to explore the impact of superfluous brackets in problem sets implemented in an authentic online educational technology-based learning environment. Thus, our study aims to provide insights on how superfluous brackets may act as visual and perceptual support for students while solving problems in an online learning environment.
In this study, we compare performance on simple arithmetic problems among fifth- to seventh-grade students. Students completed math problems that were presented in one of six different ways varying the presence or absence of brackets and the position of the HOO. Based on previous findings on students’ weak structure sense, we hypothesize that when solving math problems related to the order of operations, students will be more likely to subscribe to the left-to-right tendency in computing. Thus, students will perform better when they see the HOO on the left, compared to when the operator is in the center or on the right side of the expression. In line with perceptual learning theory, we also hypothesize that superfluous brackets will act as a perceptual cue that primes students and draws their attention to the HOO; thus, students will perform better on problems with (as opposed to without) superfluous brackets. Finally, we hypothesize that students will perform the highest on math expressions that contain both, left side HOOs and superfluous brackets. Specifically, we pose the following research questions:
-
Does the position of a HOO (i.e., multiplication or division) impact student performance on simple arithmetic problems in an online homework assignment, as measured by student mastery speed and average response time?
-
Does the presence of superfluous brackets impact student performance, as measured by student mastery speed and average response time?
-
Is there an interaction between the effects of operator position and superfluous brackets on student performance, as measured by student mastery speed and average response time?
Method
We received approval from our university’s ethics committee for this research project.
Additionally, we pre-registered the study design and data analysis plan for this project on Open Science Framework (see Supplementary Materials).
Participants
We recruited students by advertising this study to existing fifth- to seventh-grade teacher-users of ASSISTments (Heffernan & Heffernan, 2014), the educational technology platform in which the study was deployed. Participating teachers assigned a link to the study activity to their class through the ASSISTments platform. Based on the information available in ASSISTments and in order to comply with the ASSISTments IRB, no participant demographic data is recorded or available on the platform. Therefore, we were not able to receive or report participants’ demographics data.
A total of 690 students from 24 middle school classrooms in the U.S. initially opened the assignment. Of those students, 19 students were immediately dropped from the sample because they did not complete the pretest and were therefore not assigned to a condition. An additional 71 students were dropped from the sample because they took an older and longer version of the pretest or had data not logged due to an error. A total of 600 students completed the three-item baseline assessment, were randomly assigned to a condition, and were included in our preliminary analysis examining mastery. Of the 600 students, 46 students quit the assignment before completion, meaning they did not reach content “mastery”. These students were included in preliminary analyses then dropped from the sample for the primary analysis. We then checked the distribution of average response time and mastery speed to identify outliers. Of the 554 students who did reach “mastery”, 17 students had average response times well over five minutes per problem and nine additional students had mastery speeds that exceeded three standard deviations from the sample mean; these 26 students were dropped from the sample. These exclusions resulted in a final sample of 528 students for the primary analysis.
A post hoc power analysis in G*Power showed that a sample size of 528 students would provide 78.59% power to detect a small-to-medium effect size of f = .12 as detected in related previous work on perceptual cues in arithmetic problems (Harrison et al., 2020).
Study Procedures
We created this randomized controlled trial as a problem set in ASSISTments, an online tutoring system with free K-12 content that focuses on math (Heffernan & Heffernan, 2014). Teachers assigned the problem set as a class-wide assignment to their students to be completed individually in students’ web browsers using their own device as a 30-minute in-class or homework activity.
Once students clicked the link to open the problem set, they completed a three-item baseline assessment on simplifying order-of-operations expressions (Figure 1). Each expression consisted of four numbers and three operators; further, the problems varied the position of the HOO (i.e., left: 6 / 3 + 2 – 1; center: 7 + 8 * 4 – 2; right: 7 + 2 + 5 * 3). Students did not receive any accuracy feedback on these problems.
Figure 1
Student View in ASSISTments for the Three-Item Baseline Assessment
After completing the baseline assessment, students were randomly assigned to one of six conditions, described below. Within condition, students simplified order-of-operations expressions that were presented in a randomized order within an ASSISTments’ Skill Builder, where the goal was to “master” the content by answering three questions correctly in a row. In the Skill Builder, once students correctly answered three problems in a row, they were considered to have “mastered” the topic and received a message indicating that they completed the assignment.
Study Design and Conditions
We used a 3 (HOO position: left, center, or right) × 2 (Presence of brackets: superfluous brackets vs. no brackets) between-subjects design consisting of six experimental conditions (Figure 2).
Figure 2
Example Problem Presentations Within the 3×2 Study Design
Each condition varied in the placement of the HOO and the presence or absence of superfluous brackets in math expressions (Table 1). Importantly, the presence of the superfluous brackets did not alter the mathematical meaning of, or answer to, the math expressions in any conditions. Notably, it was mathematically valid to solve expressions in the brackets-left and no brackets-left conditions from left to right, whereas problems in the other four conditions required students to attend to the order of operations in order to correctly solve each problem. Additionally, the problems designed for each condition mirrored one another with the same terms and answers (e.g., in Figure 2, the simplified answer to the example expression is 32 in all six conditions).
Table 1
Problem Structures and Sample Problem by Condition
| Condition Name | Structure | HOO Position | Presence of Brackets | Example |
|---|---|---|---|---|
| Brackets-Left | *++ | Left | Yes | (1 * 6) + 2 + 5 |
| No Brackets-Left | *++ | Left | No | 1 * 6 + 2 + 5 |
| Brackets-Center | +*+ | Center | Yes | 2 + (1 * 6) + 5 |
| No Brackets-Center | +*+ | Center | No | 2 + 1 * 6 + 5 |
| Brackets-Right | ++* | Right | Yes | 2 + 5 + (1 * 6) |
| No Brackets-Right | ++* | Right | No | 2 + 5 + 1 * 6 |
Materials
The problems used in this study were based on the Common Core Standards for fifth grade content on “Operations and Algebraic Thinking” (National Governors Association Center for Best Practices, 2010). Our team designed 49 order-of-operations problems and adapted them for each of the described conditions. All problems consisted of four single-digit numbers (1 - 9) and three operators: one of the three operators was either multiplication or division and the two other operators were either addition or subtraction (e.g., Table 1). Single-digit numbers were evenly used across all problems in the problem set. Approximately half of the problem solutions were magnitudes under 20 (n = 26) and half were equal to, or over, 20 (n = 23).
Measures
Pretest Completion and Performance
ASSISTments recorded whether each student completed the pretest as a binary measure and calculated their performance on the pretest as the number of correct answers across the three items.
Mastery Status
ASSISTments provided a binary measure of whether students reached “mastery” as defined by correctly answering three problems in a row. Students may have dropped out of the assignment before reaching mastery. This measure was used as the outcome in the preliminary analysis to check attrition rates by condition.
Mastery Speed
For each student that achieved mastery, the system recorded their assignment mastery speed, which was measured as the count of problems that a student saw (after the pretest) to successfully complete three problems in a row. For example, Figure 3 shows a problem in ASSISTments after a student answered the first two problems in the Skill Builder correctly. If the student were to answer the third problem correctly, their mastery speed would be three problems. However, if the student were to answer the third problem incorrectly, followed by submitting three correct responses in a row, their mastery speed would be six problems. In the context of this study, a slower mastery speed (i.e., solving more problems to get three problems correct in a row) is an indicator of higher error and lower math performance. Mastery speed has been used as an outcome measure of student performance in previous ASSISTments studies (e.g., Botelho et al., 2015; Harrison et al., 2020; Walkington et al., 2019). Here, we consider students’ mastery speed to be a measure of their problem-solving accuracy.
Figure 3
Example of a Student’s View in ASSISTments in the No Brackets-Right Condition
Average Response Time
For each experimental problem in the ASSISTments Skill Builder, the system recorded the time from which the problem window opened until the student submitted the correct answer to the problem. Students’ response time for each problem was summed and divided by the number of problems that they solved to calculate each student’s average response time per problem. Previous studies have explored response time as an outcome variable of student performance during math problem solving (Kellman et al., 2008; Landy & Goldstone; 2010; Mayer, 1982). In this study, we used average response time per problem as a proxy for efficiency to evaluate if students in one condition simplified math expressions faster than those in other conditions.
Approach to Analysis
Preliminary Analyses – Rate of Mastery by Condition
Prior to conducting primary analyses, we checked for differential mastery rates across conditions to see whether one condition may have been significantly more challenging for students to the point of not completing the assignment. Figure 4 shows the mean mastery rate by condition and Figure 5 shows the mean mastery rate by HOO Position.
Figure 4
Proportion of Students Who Achieved Mastery by Condition
Note. Error bars represent one standard error from the mean.
Figure 5
Proportion of Students Who Achieved Mastery by HOO Position
Note. Error bars represent one standard error from the mean.
We then conducted a logistic regression to examine whether students were more likely to have mastered the assignment when assigned to a condition with superfluous brackets and/or a particular operator position (left, center, or right). The logistic regression model, controlling for pretest, was statistically significant, χ2(5, 593) = 26.29, p < .001 (Table 2). The model explained 10.3% (Nagelkerke R2) of the variance in mastery and correctly classified 93.1% of cases. Presence of brackets was not associated with achieving mastery, OR = 2.49, 95% CI [-0.55, 2.38]; however, students who were in a right position condition were significantly less likely to achieve mastery compared to students who were in a left position condition, OR = 0.27, 95% CI [-2.33, -0.26]. Higher pretest scores did not predict the likelihood of achieving mastery and no interactions were significant.
Table 2
Logistic Regression Predicting Mastery by Condition
| Condition | Estimate | SE | OR | z | Wald Statistic | 95% CI
|
|
|---|---|---|---|---|---|---|---|
| LL | UL | ||||||
| (Intercept) | 2.31** | 0.57 | 10.03 | 4.04 | 16.32 | 1.19 | 3.42 |
| Pretest correct | 0.21 | 0.16 | 1.23 | 1.31 | 1.71 | -0.10 | 0.52 |
| Brackets | 0.91 | 0.75 | 2.49 | 1.23 | 1.50 | -0.55 | 2.38 |
| Position- Center | -0.55 | 0.57 | 0.58 | -0.97 | 0.93 | -1.67 | 0.57 |
| Position- Right | -1.30** | 0.53 | 0.27 | -2.45 | 5.99 | -2.33 | -0.26 |
| Brackets * Position- Center | 0.15 | 0.96 | 1.17 | 0.16 | 0.03 | -1.74 | 2.04 |
| Brackets * Position-Right | 0.57 | 0.91 | 1.76 | 0.62 | 0.39 | -1.12 | 2.36 |
*p < .05. **p < .01. ***p < .001.
Primary Analyses
To answer our first and second research questions, we investigated how the position of the HOO and the presence of superfluous brackets may have separately impacted students’ performance among those who completed the problem set. Using the analytic sample of 528 students, we compared differences across conditions in students’ mastery speed and average response time as two indicators of student performance. Specifically, we conducted a Poisson regression to predict mastery speed and a linear regression to predict average response time. We used students’ pretest scores as a covariate to control for prior knowledge. We chose to conduct a Poisson regression for mastery speed since the variable represents count data. We did not use a multilevel model accounting for the nesting of students in teachers (n = 20) because the intraclass correlation was only 0.02, well below the 0.07 threshold that is recommended for the use of hierarchical linear modeling (Lee, 2000; Niehaus et al., 2014). The HOO position predictor was dummy coded, with the HOO in the center position as the reference group to allow comparisons to both left and right positions. We used R Studio with the lme4 package for all analyses.
For each analysis, we analyzed the main effect of operator position (left and right compared to the center), the main effect of superfluous brackets (superfluous brackets vs. no brackets), and two Operator Position * Presence of Bracket interactions. The main effect of operator position (left and right compared to center) revealed whether and how the position of the HOO (i.e., multiplication or division) in math expressions impacted student performance on simple arithmetic. The main effect for the presence of brackets (superfluous brackets vs. no brackets) informed us whether and how superfluous brackets impacted student performance. Lastly, the interactions indicated whether there was an interaction between the impact of operator position (left and right compared to the center) and the presence of superfluous brackets on student performance.
Results
Descriptive Statistics
All students who were included in the primary analyses achieved “mastery” (i.e., answering three problems correctly in a row) at some point in the study assignment (M = 4.49 problems, SD = 2.66 problems). See Table 3 below for details on numbers of students, average pretest score, average mastery speed, and average response time for the overall sample and by condition. Figure 6 shows average mastery speed by condition.
Table 3
Descriptive Statistics on Student Performance by Condition
| Condition | n | Average Pretest Performance (SD) | Average Mastery Speed (SD) | Average Response Time (SD) |
|---|---|---|---|---|
| Overall | 528 | 2.15 (0.93) | 4.49 (2.66) | 38.57 (34.31) |
| No Brackets-Left | 78 | 2.28 (0.91) | 3.68 (1.55) | 34.29 (26.23) |
| Brackets-Left | 109 | 2.15 (0.93) | 3.92 (2.13) | 36.15 (30.90) |
| No Brackets-Center | 83 | 2.08 (0.90) | 5.17 (3.17) | 35.04 (29.37) |
| Brackets-Center | 99 | 2.18 (0.90) | 3.66 (1.53) | 40.13 (37.95) |
| No Brackets-Right | 77 | 2.20 (1.01) | 5.60 (3.64) | 46.84 (41.29) |
| Brackets-Right | 82 | 2.02 (0.96) | 5.29 (2.82) | 39.77 (37.58) |
Note. Average Pretest Performance and Average Mastery Speed are reported by problem count. Average Response Time is reported in seconds.
Figure 6
Mean Mastery Speed as a Function of Condition
Note. Error bars represent one standard error from the mean.
Main Effects Predicting Mastery Speed
To first examine the effects of brackets and HOO position on students’ mastery speed, we conducted a Poisson regression controlling for students' pretest performance, with the center position as the reference group (Table 4, Model 1).
Table 4
Main and Interaction Effects of Two Predictors on Mastery Speed
| Predictor | Model 1
|
Model 2
|
||||
|---|---|---|---|---|---|---|
| β | SE | t | β | SE | t | |
| Intercept | 1.74*** | 0.06 | 29.02 | 1.83*** | 0.06 | 28.83 |
| Pretest | -0.10*** | 0.02 | -4.54 | -0.09*** | 0.02 | -4.28 |
| Brackets | -0.12** | 0.04 | -2.99 | -0.34*** | 0.07 | -4.77 |
| Position-Left | -0.11* | 0.05 | -2.22 | -0.32*** | 0.08 | -4.29 |
| Position-Right | 0.24*** | 0.05 | 4.90 | 0.11 | 0.07 | 1.66 |
| Left*Brackets | 0.26** | 0.10 | 2.70 | |||
| Right*Brackets | 0.40*** | 0.10 | 3.87 | |||
The results revealed a significant effect of superfluous brackets presence on students' mastery speed, B = -0.12, p = .002** (Figure 7). Specifically, students who did not see superfluous brackets had slower mastery speeds than those who saw superfluous brackets.
Figure 7
Mean Mastery Speed as a Function of Brackets Condition
Note. Error bars represent one standard error from the mean.
Second, the results revealed a significant effect of HOO positions on students' mastery speed. Specifically, the mastery speed of students who solved expressions with the HOO on the left was significantly lower (i.e., quicker) compared to those who solved problems with the operator in the center (B = -0.11, p < .05). Further, students who solved problems with the HOO in the center had significantly quicker mastery speeds than students with the HOO on the right (B = 0.24, p < .001; Figure 8).
Figure 8
Mean Mastery Speed as a Function of HOO Position Condition
Note. Mean number of problems students completed to reach mastery as a function of HOO position condition with error bars reporting one standard error of the mean.
Interaction Effects Predicting Mastery Speed
Next, to examine whether the effects varied by condition, we added two interaction terms with bracket and position to the model (Table 4, Model 2). Results indicate significantly different patterns of effects with and without brackets in the center position, The first interaction of brackets by left position was statistically significant (B = 0.26, p = .01). As shown in Figure 9a, students in the two left conditions performed similarly, regardless of the presence or absence of brackets. While the center position was not related to higher mastery speed compared to the left position in the presence of brackets, the students who were in the center-no brackets condition did perform significantly worse (higher mastery speeds) than students in the left position conditions and the brackets-center position.
Figure 9
Interaction Graphs of Average Mastery Speed by (a) Left and (b) Right Position and Presence of Brackets
Note. Reference group is Center position.
The second interaction comparing brackets and no brackets and the center and right HOO position on mastery speed was also statistically significant (B = 0.40, p = .001). Students who were in the brackets center condition had significantly lower mastery speeds (indicating higher performance) compared to the right positions, but students’ mastery speeds in the no brackets center condition did not significantly differ from students in both of the right conditions (Figure 9b).
Additionally, after including the two interaction terms pretest performance remained a significant predictor of mastery speed (B = -0.09, p < .001), where students who performed higher on the pretest demonstrated quicker mastery speeds. Further, the main effects for brackets and the left compared to center position remained significant (p < .01); however, the main effect for the center vs right position was no longer significant (p = .09).
Main Effects Predicting Average Response Time
Figure 10 shows the average response times per condition.
Figure 10
Descriptive Graph of Average Response Time Per Problem by Condition
Note. Mean number of seconds students took to respond to each problem by condition with error bars reporting one standard error of the mean on each side.
Next, we conducted a regression predicting average response time and controlling for students' pretest performance (Table 5, Model 3).
First, there was no significant effect of HOO position on students' average response time, both p’s > .50. Similarly, there was no significant effect of superfluous brackets on average response time, p = .46. Lastly, there was no significant interaction effect between the position of HOO (both left and right compared to center) and the presence of superfluous brackets on students' average response time, both p’s > .20 (Table 5, Model 4). Pretest performance was not a significant predictor of average response time, p = .87.
Table 5
Main and Interaction Effects of Two Predictors on Average Response Time
| Predictor | Model 3
|
Model 4
|
||||
|---|---|---|---|---|---|---|
| β | SE | t | β | SE | t | |
| Intercept | 238.80 | 789.70 | 0.30 | 601.50 | 860.40 | 0.70 |
| Pretest | -69.60 | 276.20 | -0.25 | -46.20 | 277.10 | -0.17 |
| Brackets | 379.50 | 518.30 | 0.73 | -387.80 | 879.40 | 0.44 |
| Position-Left | 327.80 | 616.80 | 0.53 | -116.00 | 932.90 | 0.12 |
| Position-Right | 398.50 | 641.90 | 0.62 | -440.10 | 934.90 | -0.47 |
| Left*Brackets | 818.00 | 1289.20 | 0.66 | |||
| Right*Brackets | 1590.40 | 1289.20 | 1.23 | |||
Discussion
The goal of this study was to explore whether the position of higher-order operator and the presence of superfluous brackets within math expressions may separately and simultaneously impact student performance on order-of-operations problems in an online tutoring system. Three notable findings emerged from this study. First, students were more likely to not complete (i.e., “not master”) the assignment if they were in a condition that had the HOO on the right, suggesting that this presentation of arithmetic expressions may have posed more challenges to students during problem solving than the other position conditions. Second, main effects show that, on average, students who were assigned conditions where the HOO was in the center of the expressions had slower mastery speeds than when it was on the left, but quicker mastery speeds than students who solved expressions with the HOO on the right. Further, students who saw expressions with brackets tended to have quicker mastery speeds than those who did not see brackets. Third, interaction effects revealed that these main effects were largely driven by the presence of superfluous brackets on the center position which moderated the impacts of HOO position on mastery speed. Among students in the two conditions with the HOO in the center, students who solved expressions with superfluous brackets achieved mastery more quickly (comparable to students who were in the left position conditions) than students who solved expressions without brackets (comparable to the right position conditions).
Right Operator Position Impacted Students’ Assignment Completion
Based on previous work on perceptual cues within ASSISTments (Harrison et al., 2020), we anticipated that operator position and superfluous brackets would impact students’ performance but not their completion rates on an online homework assignment. However, preliminary analyses found that students who were assigned to the right conditions were less likely to reach mastery compared to those in the left conditions. These results show that the position of HOO did impact students’ likelihood of achieving mastery, or generally, completing the assignment. This finding suggests that the impacts of operator position may be greater than anticipated or evidenced in prior work by impacting students’ participation in an assignment and not just their performance.
In our pre-registration, we predicted that the brackets-left condition would be the easiest for students to solve arithmetic problems, and that the no brackets-right condition would be most difficult, as indicated by students’ higher (i.e., slower) mastery speeds. The logistic regression showing that students assigned to the right position condition were significantly less likely to complete the assignment provides support for this hypothesis. We interpret this finding to mean that the position of the HOO may be very influential in how students reason about math; in particular, solving expressions may seem more difficult when the position of the HOO is on the right side of the expression. One plausible explanation for students dropping out more often in this condition may be that they became frustrated by the difficulty of the assignment or getting more problems incorrect.
Since the primary analyses conducted only included students who did achieve mastery in the study activity, the findings on mastery speed and average response time need to be interpreted with the context that there was differential attrition between our six conditions. However, we contend that by dropping students who did not achieve mastery from the analytic sample, the findings may actually present a more conservative estimate of how the position and presence of perceptual cues within arithmetic expressions impact student performance. Future research should explore item-level data to better understand factors such as time on task, accuracy of initial responses, and students’ behaviors while problem solving to unpack the mechanisms behind why students might have dropped out of the assignment before reaching mastery.
Main Effects of Higher-Order Operator Position on Students’ Mastery Speed
We predicted that solving math expressions with the HOO on the left and expressions with superfluous brackets would lead to (a) quicker mastery speeds and (b) quicker response times than solving expressions with the operator in the center or on the right and expressions without superfluous brackets. The main effect result supports the first hypothesis: seeing math expressions with the HOO on the left was, on average, related to quicker mastery speeds (i.e., higher accuracy rates) than the center. The presence of superfluous brackets also independently impacted students’ mastery speed during simple arithmetic. However, these variables did not significantly predict response times. Students had comparable response times across conditions, suggesting that neither operator condition nor superfluous brackets impacted students’ problem-solving speed.
The main effect finding that HOO position impacted student performance aligned with previous research showing that students have a left-to-right tendency during math problem solving (Banerjee & Subramaniam, 2005; Blando et al., 1989; Gunnarsson et al., 2016; Kieran, 1979). For instance, Kieran (1979) found that when solving 5 + 2 * 3, students tend to complete the addition operation first; however, when solving 4 * 2 – 3, students came to the correct answer by following their tendency to compute from left to right (Kieran, 1979). This strong adherence to the left-to-right principle in computing may explain the differences in mastery speed that we saw in our study. In the two HOO-left conditions, the math expressions were presented in ways that benefited students' left-to-right tendency: students did not have to effortfully attend to the problem’s structure or the order of operations of expressions to reach the correct answer, leading to fewer errors and higher accuracy during problem solving. On the other hand, students who solved expressions with the HOO on the right consistently had the slowest mastery speeds, regardless of superfluous brackets (e.g., 4 + 7 + 2 * 5, 3 + 8 + (2 * 7)). In these two HOO-right conditions, the presentation of math expressions required students to notice the problem’s structure and inhibit their left-to-right tendency in order to answer correctly, leading to more errors and lower accuracy. This is especially notable as the students in the no brackets-right condition who were in this analytic sample were likely a higher performing subset of those initially assigned to this condition, as students in the no brackets-right condition were more likely to drop out of the study before achieving mastery or completing the assignment.
This study advances research on the roles of perceptual factors in math notation by isolating the effect of higher-operator position on students’ performance. While the effects of superfluous brackets and HOO position on student performance are much smaller than we anticipated based on prior work (Harrison et al., 2020), this finding suggests that varying the positions of HOOs and presence or absence of superfluous brackets may provide noticeable differences in perceptual structures that affect students’ performance in the context of our study. Although previous research has demonstrated the left-to-right tendency in problem solving (e.g., Banerjee & Subramaniam, 2005; Blando et al., 1989; Gunnarsson et al., 2016; Kieran, 1979), to the best of our knowledge, no previous study had strategically tested whether and how HOO position impacts different aspects of student performance. As we found that students who solved problems with the HOO on the right performed significantly worse than those in the left and center conditions, our study provides empirical evidence that students have a strong tendency to solve math problems from left to right. This finding contributes to the literature on students’ weak structure sense and may have implications for classroom instruction to identify and support students who tend to solve problems from left to right.
Superfluous Brackets in the Center Operator Position Increase Performance
The most notable finding is that the presence of superfluous brackets moderated the effect of HOO position, specifically when students solved math expressions with the operator in the center. While students, on average, demonstrated the quickest mastery speeds in the HOO-left conditions, students in the center position with brackets condition performed comparably well to the two HOO-left conditions, suggesting that superfluous brackets may increase students' accuracy on problems when the HOO is in the center. Conversely, students in the no brackets-center condition had comparable mastery speeds to students in the HOO-right conditions, suggesting that the absence of brackets with the HOO in the center posed challenges to students.
One possible interpretation of these results is that position of HOO might be a type of visual and perceptual feature that works congruously with the left-to-right calculating tendency to impact students’ problem solving: left-sided HOOs facilitate higher accuracy, while center- and right-sided HOO (without brackets) elicit more errors. Students were highest performing when they were able to apply a left-to-right solving strategy. In cases when they could not (i.e., center and right conditions), their performance (mastery speed) dropped. The exception to this trend was for students who saw brackets in the center position, suggesting that in cases when students could not apply a left-to-right solving strategy, the superfluous brackets may have naturally and visually grouped the numbers to prevent a left to right calculation, or shifted their attention and helped them identify the groupings to apply the first steps for problem solving. The brackets could have prevented students from compulsively performing left-to-right calculations by visually breaking up the structure of the math expressions. Specifically, brackets around the center terms may be the most impactful because, in that position, it breaks up the structures into three distinct parts, where the brackets naturally block the flow of left to right computations (i.e., 1+ (7 * 5) – 4).
When solving order-of-operations problems, students seem to rely on both HOO position and superfluous brackets presented in the expressions; however, this work suggests that operator position, particularly when placed in the center, may play a strong attention-guiding role. Aligned with our findings, HOO position seemed to be a salient factor that impacted students’ performance when calculating left to right. Superfluous brackets, while significant, specifically seemed to impact students’ performance when the operator position was in the center. Having brackets on the left or right did not seem to impact performance. Thus, if students relied on left to right calculations, they would use more inaccurate problem solving when the operators were not located on the left side of the expressions. However, when the operators were in the center of the expressions, the presence of the brackets could have helped students attend the HOO first, breaking up the expression into chunks, and helping to facilitate more accurate problem solving.
Harrison and colleagues (2020) suggested that perceptual features may impact student performance in a hierarchical structure. In particular, the operator position may act as a first-order perceptual structure, while cues such as physical spacing between symbols in expressions might act as a second-order perceptual cue to influence how students interpret and act on math notation (Harrison et al., 2020). Aligned with this perspective, we posit that students may have attended to operator positions first, with a compulsion to compute from left to right. The presence of brackets may have helped guide students' attention to the operator in the center, nudging them to notice the order-of-operations structure of the problem, and facilitating more accurate problem solving. Consequently, students may be more likely to notice superfluous brackets when they are used to group symbols that are not on the left-most side of expressions and break up this natural flow. While we cannot be certain of the mechanism of this finding, the helpful effects of superfluous brackets in the center of the expressions is consistent with prior studies indicating the usefulness of brackets to help students see structure and support math performance (e.g., Hoch & Dreyfus, 2004; Marchini & Papadopoulos, 2011).
Another alternative explanation of this finding is that, for students who have not yet conceptually mastered the order of operations, they may have memorized a simpler procedural rule that parentheses must be computed before other operations (i.e., PEMDAS). Therefore, it is plausible that the brackets may not be serving solely as a visual perceptual cue, but rather students could be relying on the PEMDAS (or BEDMAS) rule that parenthesis must be calculated first as a foolproof way for students to perform simple procedures without understanding. However, this explanation is challenged by the finding that, in the case of HOO-left and HOO-right expressions, the presence of brackets did not help students perform more accurately.
Performance vs. Learning: Creating Desirable Difficulties
The findings from this study demonstrate how operator position and superfluous brackets may impact students’ participation in, and performance on, order-of-operations problems in an online homework environment. Importantly, this work provides considerations for designing stand-alone assignments: using problems with left-most HOO may increase students’ performance while using right-most HOOs in expressions may decrease students’ likelihood of finishing the assignment. We posit that variations in perceptual features like HOO position and superfluous brackets (particularly in the center position) may help students quickly infer which operations to address in a given expression, similar to creating semantic alignment in the structure of word problems (Gros et al., 2020). Based on the word structure of problems, students map connections to the underlying mathematical properties: for example, problems stating to “place [objects] in [locations]'' create analogies for using division (Bassok, 2001). This quick encoding of information may lead to increased performance solving problems. However, we did not administer a posttest so we are unable to draw any conclusions about how completing the assignment with perceptual cues like brackets in varying positions may transfer to students’ strategies and performance in subsequent problem solving. We consider this study to be a first step to explore long-term or transfer effects of using perceptual cues in instructional materials. For example, are the perceptual cues that increase performance also those that promote learning, problem-solving flexibility, and efficiency?
Several decades of research on desirable difficulties has shown that learning conditions which are more difficult in the moment and decrease individuals’ performance improve long-term learning and retention (Bjork, 1994; Bjork & Bjork, 2015). For example, perceptually (Diemand-Yauman et al., 2011) or conceptually (in terms of organizational structure, McNamara et al., 1996) creating conditions that decrease students’ in-the-moment fluency encourages them to pause before engaging with the material. Work on algebraic equivalency has also shown that students who pause longer before acting on a problem show higher problem-solving efficiency than their peers who do not pause to think before solving (Chan et al., 2022). More broadly, instructional and study tactics such as varied practice (Smith et al., 1978) and interleaving (Kornell & Bjork, 2008) help learners develop flexibility across time and context to improve problem solving. Interleaved and varied practice train learners to identify distinguishing problem features like the way that perceptual cues can help students see important structures of notation that may influence their decision on which strategy to apply to a problem.
Taken together, we consider the possibility that variations in expression structures and perceptual cues that lend themselves to higher problem-solving performance (i.e., left-most HOO, superfluous brackets) may not be the same features that lead to long-term learning, retention, and flexibility or efficiency. Instead, more difficult mathematical structures or the presence or absence of perceptual cues that challenge students to pause and reflect on procedural or more conceptual rules like the order of operations before acting (i.e., right-most HOO, no superfluous brackets in center, incongruent spacing) may seem counterintuitive but could create desirable difficulties and promising interventions for improved long-term outcomes. This area of inquiry is a focus of our future work.
Limitations and Future Directions
This study had multiple limitations. First, the differential attrition (i.e., students who achieved mastery was different by condition) can be seen as problematic as the attrition was not random. However, although more students were dropped from the analytic sample in the no brackets-right condition, the findings of this study are notable, especially given that the negative effects of no brackets-right were still present, even when dropping those who did not achieve mastery.
Second, given the current data, it is difficult to specify who is impacted by superfluous brackets and HOO position and when. While we intentionally recruited teachers of incoming fifth- to seventh-grade students, the online platform used to deploy the study does not permit collecting any individual participants’ demographic information (e.g., gender, race, age, grade, in-person vs. remote learning status) due to privacy concerns. While we acknowledge that the lack of demographic information is not ideal and limits our ability to understand individual differences, it is a tradeoff for using open educational platforms like ASSISTments for conducting educational research at scale. Further, teachers were aware of the content targeted in the study and may have assigned the content to students in other grades if the content was appropriate for their knowledge levels (e.g., in advanced lower grades or remedial higher grades). Additionally, conducting this study in an online platform provided ecological validity by testing how these experimental manipulations impacted students’ performance on an online class or homework assignment.
Third, the focal dependent variable in this study, mastery speed, is specific to the type of problem set built in ASSISTments. As a result, these findings are not directly generalizable to other online tutoring platforms or contexts. However, estimating treatment effects on students’ mastery speed provides suggestions for another measure for accuracy and a unique analysis of how perceptual cues may impact students’ performance at a granular level.
Looking ahead, future research may consider collecting more demographic information from participants to control for individual differences among students that may affect their susceptibility to perceptual cues. This approach would help uncover when, and for whom, perceptual scaffolding may be the most effective. Further, this approach may be most effective when paired with other theoretical perspectives that explain cognitive and developmental factors of math performance and learning. Additionally, future research may consider replicating this work with students in classrooms or using additional methodologies to tease out the mechanisms and practical significance of presenting and implementing perceptual supports to facilitate enhanced problem solving in everyday math learning. Current work utilizing eye tracking is underway by our team to identify whether students who are presented with brackets do in fact look at the brackets and HOO first or fixate longer within expressions in various positions. This approach can help confirm or challenge more procedural or perceptual/attentional explanations for why students demonstrate improved performance with brackets in the center position.
Implications and Practical Contributions
Broadly, this study adds to the growing body of literature on the importance of perceptual grouping, and variations in structures for developing structure sense and reasoning in mathematics (Kirshner, 1989; Landy & Goldstone, 2007, 2010) and the potential benefits of using superfluous brackets in simplifying expressions (Hoch & Dreyfus, 2004; Marchini & Papadopoulos, 2011). Our study suggests that superfluous brackets may be a helpful perceptual cue for students, especially when the HOO is placed in the center. Superfluous brackets may provide additional support to students when learning order of operations, thus adding more to the literature about how presenting superfluous brackets in different positions of math expressions may serve as an effective intervention to support students during problem solving.
These findings support that adding visual information in notation, such as manipulating the position of HOOs and adding superfluous brackets, may act as perceptual supports that can positively influence students’ reasoning and performance in math. Our findings also highlight the importance of helping students develop structure sense in math learning and identifying students who may be underperforming due to a left-to-right solving tendency. Regarding the implications of this work, instructors and educators may be able to intentionally make changes to the visual presentation of notation, such as adding superfluous brackets, to guide students' attention to relevant features of math problems, lead them to correct solution strategies, and enhance their performance on tasks. Since students tend to solve math problems from left to right, which might lead to inaccurate solutions, it may be useful for teachers to utilize superfluous brackets in the center position to improve students' weak structure sense in arithmetic by helping inhibit their compulsion to compute from left to right. A second possible way to support students' development of structure sense may be to introduce them to superfluous brackets early and slowly in instructional practice. As students first learn how to solve problems with HOOs in an expression (e.g., 2 + (1 * 6) + 3), guiding them to notice the higher order operator by using superfluous brackets or instructing them to put the brackets around the multiplication or division to break up structure and signs might help students gradually build an understanding that these brackets serve as a visual cue to highlight important elements of math problems. From frequent exposure to superfluous brackets as a perceptual cue, students might better understand the importance of finding significant elements within a math expression and comprehending the overall structure of a problem prior to solving. As students become more comfortable with simplifying order-of-operation expressions with different structures, the need for those supports may become less necessary.
Conclusions
The current study applies and integrates work from cognitive science, math education, and educational technology to explore the impact of superfluous brackets and HOO position on students’ math performance simplifying expressions. We found that, generally, students tend to demonstrate the highest performance on expressions with the HOO on the left but performance decreases as the HOO moves to the right. However, we found that adding superfluous brackets in the center position supports learning, while having brackets on the right or left does not provide additional impacts for students. Overall, this study highlights the importance of providing variation in subtle structural and perceptual variations in math notation, showing that the position and presence of perceptual grouping structures, such as superfluous brackets, impacts students’ completion and performance on online assignments.