This is an open access article distributed under the terms of the To read an expression like “1 + 2 * 3”, people typically scan the expression from left to right. However, to evaluate this expression, the correct order is to execute “2 * 3” first and then add “1”. In this scenario, the sequence of evaluation (i.e., from right to left) is different from the sequence of reading (i.e., from left to right) because operational rules have stipulated that high-order operations (i.e., multiplication and division) should precede low-order ones (i.e., addition and subtraction) (Jeon, 2012). Studies recording gaze movements during math evaluation suggest that both operational rules and left-to-right reading habits bias arithmetic problem solving strategies (Egorova et al., 2024; Landy et al., 2008; Schneider et al., 2012). Nevertheless, most eye-tracking arithmetic problem solving studies, to date, have been conducted in a laboratory setting with lab-based eye-trackers, which are relatively expensive, time-consuming, and lack spatial flexibility (Semmelmann & Weigelt, 2018; Van der Cruyssen et al., 2024). To promote the accessibility of eye-tracking technologies in math learning research, the current study investigated the utility of a webcam-based eye-tracker (i.e., WebGazer) in an online arithmetic problem solving experiment. Adapting established paradigms, we examined whether gaze data recorded by WebGazer can replicate the patterns observed in a lab-based eye-tracking study (i.e., Egorova et al., 2024). Additionally, we examined whether the proportion of gazes on the HOO over time, a gaze measure that was not analyzed in Egorova et al. (2024) but commonly used in WebGazer studies (Slim et al., 2024; Slim & Hartsuiker, 2023), can suggest participants’ online problem solving strategies.
HOO Position
High-order operator (HOO) position effects are well-documented in arithmetic problem solving literature (Gunnarsson et al., 2016; Landy & Goldstone, 2010; Landy et al., 2008; Ngo et al., 2023). Specifically, the positions of HOOs (e.g., left: ; center: ; right: ) influence problem solving performances, showing higher accuracy and faster response time (RT) when the HOO is on the left compared to on the right or in the center (Egorova et al., 2024; Ngo et al., 2023). The HOO position effect is due to solvers’ tendency of applying reading habits (i.e., from left to right) when solving arithmetic problems (Blando et al., 1989). When the order of evaluation is mismatched with the order of reading, problem solving efficiency is decreased.
Gaze behaviors during arithmetic problem solving confirm the left-to-right tendency. For example, Egorova et al. (2024) demonstrated that the time to first fixation on the HOO was shorter when the HOO was placed on the left compared to in the center or on the right. Furthermore, participants’ first look and early fixations tended to register on the left rather than the right side of the expression, even when the HOO was placed on the right (Landy et al., 2008; Schneider et al., 2012). These findings confirm the left-to-right tendency by showing that math terms on the left received earlier attentional orientation compared to the ones on the right.
Superfluous Brackets
To suppress left-to-right tendency and facilitate the sense of math structure, researchers have provided superfluous brackets around the high-order operation (Egorova et al., 2024; Gunnarsson et al., 2016; Hoch & Dreyfus, 2004; Marchini & Papadopoulos, 2011; Ngo et al., 2023). According to the operational rules, brackets exhibit higher priority than high- or low-order operators (Gunnarsson et al., 2016; Jeon, 2012). For instance, in the expression , the precedence of brackets supersedes the precedence of multiplication. Superfluous brackets, however, do not change the meaning or precedence of mathematical notation (Gunnarsson et al., 2016; Marchini & Papadopoulos, 2011). For example, brackets in the expression are superfluous brackets because the multiplication “” will have precedence over the addition “” regardless of whether it was in parentheses. Adding superfluous brackets around high-order operations can guide solvers' attention toward high-order operations and thereby scaffold the sense of math structure (Hoch & Dreyfus, 2004; Ngo et al., 2023).
Interactions Between HOO Position and Brackets
The position of HOO impacts the effect of superfluous brackets on problem solving performances and gaze behaviors (Egorova et al., 2024; Ngo et al., 2023). Specifically, Ngo et al. (2023) demonstrated increased problem solving accuracy when superfluous brackets were present compared to absent around the center HOO. The effect of superfluous brackets that occurred for the center HOO is hypothesized to be due to their visual symmetry (i.e., a + (b * c) + d), which perceptually segments the expression into three discrete parts (Todorovic, 2008). Such mental segmentation inhibits left-to-right tendency (Ngo et al., 2023). In an eye-tracking experiment follow-up, with a similar paradigm, time to first fixation was delayed on the center HOO when superfluous brackets were present rather than absent (Egorova et al., 2024). This suggests that brackets themselves in the center may attract attention, leading participants to fixate on the brackets first before directing attention toward the HOO. Moreover, Egorova et al. (2024) observed a reduced RT in the left HOO conditions when superfluous brackets were present rather than absent, suggesting a combined effect of HOO position and brackets presence.
Webcam Eye Tracking
Participants’ eye movement patterns have provided critical insights into how solvers perceive and process mathematical symbols during arithmetic problem solving. To date, most eye-tracking arithmetic problem solving studies record gaze behaviors using laboratory eye trackers (Curtis et al., 2016; Egorova et al., 2024; Landy et al., 2008; Li et al., 2023; Obersteiner & Staudinger, 2018; Schneider et al., 2012). Nevertheless, studies using laboratory eye trackers typically require a specific space to set up eye-tracking devices and a specialist to operate the software system, which prevents researchers from conducting experimental studies in natural learning environments (e.g., schools and classrooms) or with large samples (Van der Cruyssen et al., 2024). To address these restrictions, researchers have developed multiple webcam-based eye trackers (for a review, see Heck et al., 2023), such as WebGazer (Papoutsaki et al., 2016), GazeRecorder (Deja, 2013), and TurkerGaze (Xu et al., 2015), that can record gaze behaviors through web cameras installed on participants’ personal devices (e.g., computers and laptops).
One commonly used webcam-based eye tracker is WebGazer (Papoutsaki et al., 2016), which uses a JavaScript library that can compute on-screen gaze positions based on eye and pupil features. Previous studies have shown that gaze data recorded by WebGazer can replicate the observations in laboratory-based eye-tracking studies across various paradigms, such as the visual-word paradigm (Degen et al., 2021; Prystauka et al., 2024; Slim & Hartsuiker, 2023), face perception (Semmelmann & Weigelt, 2018), and emotion-attention integration (Bogdan et al., 2024). This evidence validates the use of webcam-based eye-tracking technology in online psychological research and motivates us to employ WebGazer in the current study. Nevertheless, certain drawbacks of WebGazer have been reported and discussed, such as 1) a high proportion of excluded participants (i.e., 18%–30%) (Bogdan et al., 2024; Prystauka et al., 2024; Slim & Hartsuiker, 2023), 2) a delayed time course of recorded gaze movements (Slim & Hartsuiker, 2023), and 3) a tendency for gaze noise to be concentrated in the center of the screen (Bogdan et al., 2024). Considering these drawbacks, it remains unknown whether WebGazer can validly record gaze patterns during online arithmetic problem solving, particularly when the structure of mathematical expressions is manipulated.
The Current Study
The current study investigated the feasibility of using WebGazer in online eye-tracking arithmetic problem solving studies. The primary research goal was to examine whether participants’ gaze data can replicate HOO position effects and the combined effects between HOO position and brackets reported in Egorova et al.’s (2024) lab study that used a commercial eye-tracker (i.e., Tobii Pro). In Egorova et al.’s (2024) study, participants’ eye-movement patterns were measured by how long it took participants to fixate on the HOO for the first time. Mirroring Egorova et al.’s (2024) study, the current study examined the time to first gaze on the HOO, as calculated by how long it took participants to register a gaze point on the HOO for the first time. Based on Egorova et al.’s (2024) observations, we predicted the main effect of HOO position such that participants would show reduced time to first gaze on HOO when problems contain a left HOO compared to center and right HOOs; Moreover, we predicted combined effects between HOO position and brackets. Specifically, the time to first gaze would be longer when brackets were present compared to absent around the center HOO.
The secondary research goal was to validate the use of a specific gaze measure, i.e., the proportion of gazes over time, in online eye-tracking arithmetic problem solving contexts. The inclusion of the proportion of gazes over time was driven by two reasons. First, arithmetic problem solving involves multiple steps. Extracting the proportion of gazes over time enables researchers to continuously track participants’ gaze behaviors throughout the evaluation, which advances the understanding of problem solving strategies. Second, as a conventional WebGazer metric, the proportion of gaze over time can complement measures based on a single gaze point, such as the time to first gaze. One of the technological limitations for WebGazer currently is the difficulty in detecting fixations using velocity-based methods, which are commonly used in eye-tracking literature (Mahanama et al., 2022). A fixation can be operationalized as a cluster of gazes remaining on a specific area over a period of time. To estimate fixations, previous WebGazer studies have extracted the proportion of gazes in a series of time bins (e.g., 100 ms) (Slim & Hartsuiker, 2023; Slim et al., 2024). If this conventional WebGazer metric (i.e., the proportion of gazes over time) can suggest problem solving strategies validly, the feasibility of using WebGazer in online eye-tracking arithmetic problem solving research would be further supported.
In a previous in-lab eye-tracking arithmetic problem solving study, Schneider et al. (2012) analyzed the proportion of fixations over time and demonstrated that when the HOO was placed on the right side of expressions, participants started viewing the expression from the left side and then shifted their attention toward the right. Referring to Schneider et al.’s (2012) observations, we examined whether this left-to-right viewing trajectory can be indicated by the proportion of gazes on the HOO over time in online environments. Specifically, we hypothesized that the position of HOO would influence the trajectories of gazes on the HOO. The proportion of gazes on the HOO would increase in the later stage when the HOO was on the center/right compared to on the left. Table 1 depicts research goals, corresponding hypotheses, and conclusions.
Table 1
Hypothesis and Results in the Current Study
| Primary research goal | |||
|---|---|---|---|
| Gaze measure | Effect | Hypothesis according to Egorova et al. (2024) | Replicated or not |
| Time to first gaze at the HOO | HOO position | The time to first gaze on the HOO would be slower when the HOO was in the center compared to on the left. | No |
| The time to first gaze on the HOO would be slower when the HOO was on the right compared to on the left. | Yes | ||
| HOO position * Brackets | When the HOO was in the center, the time to first gaze on the HOO would be slower when brackets were present compared to absent. | Yes | |
| Secondary research goal | |||
| Gaze measure | Effect | Hypothesis according to Schneider et al. (2012) | Confirmed or not |
| Proportion of gazes on the HOO over time | HOO position | The proportion of gazes on the HOO would increase in the later stage when the HOO was in the center compared to on the left. | No |
| The proportion of gazes on the HOO would increase in the later stage when the HOO was on the right compared to on the left. | Yes | ||
Note. RT = response time; HOO position * Brackets = the combined effect between HOO position and brackets.
Method
Participants
Two hundred and eleven undergraduate students (Mage = 19.90, SD = 1.92) at a university in the northeastern part of the United States participated in the experiment. Participants were enrolled in Psychology classes and received 0.5 research course credits as compensation for completing the study online. Online consent approved by the IRB committee was obtained before the experiment.
The final sample size in the gaze data analysis was 119 participants (Mage = 19.79, SD = 1.68). Five participants were removed due to duplicated participation (n = 1), extremely low (i.e., 3 SD below the mean) accuracy (n = 2), severe (i.e., 3 SD above the mean) math anxiety (n = 1) or vision problems (n = 1). Furthermore, 32 participants without any valid gaze samples on the target area-of-interest (AOI) and 55 participants who did not have sufficient problems for regression modeling were further removed (see data exclusion for details).
Materials and Design
24 mathematics problems were designed to manipulate the presence of brackets and the position of the high-order operand. Identical to previous studies (Egorova et al., 2024; Ngo et al., 2023), HOO was located on the left, in the center, or on the right of expressions, with or without superfluous brackets. Each of the six conditions, i.e., three levels of HOO position (left, center, and right) two levels of brackets presence (present and absent), contained four math expressions, resulting in 24 problems in total.
Expressions were identical to the ones in Egorova et al.’s (2024) study since this study aimed to examine the replicability of previous findings with new eye-tracking technology. For each problem, four one-digit numbers (i.e., 1 to 8) and three operators, all in Arial font, formed a mathematical expression. Among these three operators, one was a high-order operator (HOO), i.e., multiplication (*) or division (/), while the other two were low-order operators (LOO), i.e., addition (+) or subtraction (-). Among the 24 problems, multiplication and division were evenly distributed as the HOO. We included multiplication and division as HOOs because both have been taught before college and appeared in previous studies (Landy & Goldstone, 2010; Pappanastos et al., 2002). Half of the problems contained two additions as LOOs, while the other half contained one addition and one subtraction as LOOs. No problem contained two subtractions as LOOs to prevent negative solutions. Digits were chosen from 1 to 8 to ensure that the numbers of expressions whose solution magnitudes were above ten (n = 11) and below ten (n = 13) were relatively balanced.
Apparatus
The online experiment was programmed using Pavlovia and conducted through a link accessed on participants' personal devices (25.59% desktop computer, 73.46% laptop, and 1.42% tablet). Once participants opened the link and consented to the study, we employed a webcam-based eye tracker, i.e., WebGazer, to calculate participants’ gaze locations based on their facial and eye features. No videos were recorded.
Noticeably, due to the lack of information on physical sizes (inches) of individual screens and difficulties in precisely controlling eye–screen distances, it is hard to convert the distance between consecutive gaze points into visual angles. Thus, the present online eye-tracking study does not apply velocity-based algorithms to identify fixations but instead focuses on gaze movements. The sampling rate varied by participants’ personal devices and was on average 70.98 Hz (min = 8.54 Hz, max = 239.75 Hz). Previous WebGazer studies in adults have applied a < 5 Hz sampling rate as the criterion for participant exclusion (Prystauka et al., 2024; Yang & Krajbich, 2021). Given that the lowest sampling rate in the current study was 8.54 Hz. We did not exclude any participants based on the sampling rate.
Procedure
The experiment started with a head-adjustment and calibration session requiring participants to adjust the distance and angle of their heads so that the WebGazer could detect sufficient eye and facial features to compute gaze locations (see Figure 1a). The webcam video and box appeared on the top left of the screen, whose color indicated whether the head position was appropriate (i.e., green) or not (i.e., red). Participants were asked to position themselves so that their heads were in the box. Once the color turned green, participants proceeded to the next step by themselves.
Figure 1
Illustrations of Procedure and Stimuli
Note. (a) The procedure of the experiment. The parameters on the “Problem Solving” panel indicate the size and position of problem images as proportions relative to the screen height. h = screen height. (b) Two areas of interest (AOIs) were defined, corresponding to the high-order operator (blue outline, HOO) and the low-order operators (red outline, LOO). (c) To decide whether a gaze point was located in a particular AOI, the proportion-based AOI boundaries, e.g., (x1, y1), were translated into pixel-based coordinates, e.g., (x1’, y1’), based on the real size of individual screens (e.g., 1440 x 900 pixels). h = screen height.
Next, participants completed a 13-point calibration, during which a red dot appeared and moved around the screen. Participants were instructed to follow the red dot with their eyes. After the calibration, another red dot appeared whose movements were mediated by WebGazer to reflect participants’ onscreen gaze coordinates. Once participants confirmed that the red dot synchronized with their eye movements, they started the experiment by pressing “Enter.” If not, they could redo the calibration by pressing “R.” On average, participants underwent the calibration 1.46 times (SD = 0.74, Min = 1, Max = 5). Of the 211 participants, 135 (63.98%) completed the calibration only once, 61 (28.91%) completed it twice, 10 (4.74%) three times, 3 (1.42%) four times, and 2 (0.95%) five times. After calibration was completed, the color box was no longer visible but would appear as an indicator if WebGazer detected inappropriate head positions during the main experiment.
Participants then completed the mathematical problem solving portion (see Figure 1a). Participants calculated arithmetic expressions mentally and typed their answers on the screen using their keyboards. All problems appeared as images on the upper part of the screen, with a center-to-center distance of 0.2*screen height. The width (1.25*screen height) and height (0.35*screen height) of all problem images were identical across six different conditions. Therefore, the size of each mathematical term was relatively smaller if brackets were present compared to absent (see Figure 1b). Each problem stayed on the screen until participants submitted their answers. No feedback was provided during the experiment. Participants first completed three practice problems to familiarize themselves with the procedure. Afterward, participants completed 24 main problems in a random order varying across individuals.
During the mathematical problem solving portion, participants were probed for task-unrelated thought (TUT) and valence after the 11th and 22nd problems. Probes were spaced to be at least 1 minute apart, as is common in the literature (Weinstein, 2018). For TUT, participants were asked if they were thinking about anything other than the task (1 = yes, 0 = no). For valence, participants were asked to rate the valence of their feelings on a 7-point scale (1 = very negative, 7 = very positive). The results of TUT or valence were not reported in this paper, as they were not related to the main research questions.
After completing all math problems, participants answered questionnaires assessing their math anxiety, vision problems, and demographic information. The experiment lasted about 20 minutes.
Data Preprocessing
AOI Definition
Two areas of interest (AOI) were identified: HOO and LOO. Identical to Egorova et al. (2024), gaze behaviors on the HOO AOI were analyzed. To decide whether a gaze point was in an AOI, we compared the coordinates of individual gaze points and the coordinates of AOI boundaries (i.e., top-left, top-right, bottom-left, bottom-right). The gaze coordinates generated by WebGazer were represented in pixels with the top-left corner as the reference. The coordinates of AOI boundaries were represented as proportions of screen height with the screen center as the reference (i.e., a Cartesian system). To implement the comparison, we translated the proportion-based AOI boundaries into pixel-based boundaries, taking the screen size (in pixels) of individual devices into consideration (see Figure 1c).
Gaze Measures
We extracted one global gaze measure (i.e., proportion of onscreen gazes) and two local gaze measures (i.e., time to first gaze and proportion of gazes over time) on the HOO AOI.
The proportion of onscreen gazes was calculated as the ratio of onscreen gaze samples to all recorded gaze samples, which indicated the extent to which participants were looking at the screen during problem solving.
Time to first gaze referred to the duration between stimulus onset and the initial gaze registering on the HOO AOI, reflecting the speed of attentional orientation toward the HOO. Moreover, to examine the viewing trajectory during evaluation, we computed the proportion of gazes over time as the ratio of gaze samples on the HOO to gaze samples on all operators (i.e., one HOO and two LOOs) for each standardized time bin. Following the procedure reported in Schneider et al. (2012), we divided the entire response window (i.e., from stimulus onset until the first keypress) into 100 standardized time bins. Each standardized time bin represented one percent of the period from problem onset until participants responded.
Data Exclusion
Five participants were removed from behavioral and gaze datasets because they solved problems with extremely low (i.e., 3 SD below the mean) accuracy (n = 2), exhibited severe (i.e., 3 SD above the mean) math anxiety (n = 1) or vision problems (n = 1), or conducted the experiment twice (n = 1).
The remaining 206 participants were submitted to problem-level data exclusion. First, problems (0.65%) with extreme solving RTs (i.e., over 3 SD deviant from the mean) were excluded from behavioral and gaze datasets since participants may not fully concentrate on the experiment when answering these problems. Second, problems answered incorrectly were excluded from the RT and gaze datasets, following the same procedure as previous arithmetic studies (e.g., Curtis et al., 2016; Obersteiner & Staudinger, 2018). Third, trials with less than 70% onscreen samples and without any gaze samples registered on the HOO AOI were excluded from the gaze dataset. Problem-level data cleaning resulted in two participants being excluded from the RT dataset and 32 participants being excluded from the gaze dataset. These participants were removed because they did not retain any valid trials after problem-level data exclusion.
Following problem-level data cleaning, we calculated the number of valid problems under each of the six conditions (i.e., three HOO position two brackets presence). Participants (n = 55) who failed to retain at least one problem in each of the six conditions were excluded from the gaze dataset.
The final sample sizes for analyzing problem solving accuracy, RT, and gaze behaviors were 206, 204, and 119, respectively. On average, each participant retained 17.42 questions (SD = 3.78, Min = 7, Max = 24) for final gaze data analysis. The Chi-square analysis revealed that participants’ inclusion in the final gaze analysis was not significantly associated with their experimental devices, χ2(2) = 1.58, p = .454 or whether they completed the calibration procedure more than once, χ2(1) = 0.58, p = .446.
Statistical Analysis
We conducted mixed-effects regression model analyses in R using the lme4 package, which is identical to the data analysis approach used in Egorova et al.’s (2024) study. Mixed-effects regression models estimate the relationships between an outcome measure and its predictor(s). Based on the coefficient (B) of a predictor (e.g., brackets), one can interpret the extent to which the variance of that factor (e.g., present versus absent) accounts for the variance in the outcome measure (e.g., problem solving accuracy). A mixed-effects regression model contains both fixed effects and random effects as predictors. Fixed effects are associated with experimental manipulations (e.g., brackets), while random effects are associated with experimental units (e.g., participants) that are randomly selected from a population. By including random effects, mixed-effects regression models can estimate the influence of our manipulations on outcome measures while accounting for the variance attributable to participants’ individual differences.
We estimated mixed-effects linear regression models for time to first gaze and proportion of gazes over time, separately1. Referring to Egorova et al.’s (2024) methods, the model for time to first gaze contained the main effect of HOO position (left as reference, dummy coding) and the main effect of brackets (present as reference, dummy coding) as fixed effects. The model for proportion of gazes additionally included the main effect of quarters of time bins (first, second, third, and fourth, first as reference) and its interactions with HOO position and brackets as fixed effects. The cut-off points of the first, second, third, and fourth quarters of time bins were 0–25%, 25–50%, 50–75%, and 75–100%, respectively. Referring to Egorova et al. (2024), we assumed that outcome measures may vary across participants. Thus, the intercept for participants was included as a random effect in all models. To verify this random effect structure, we estimated the intraclass correlation coefficients (ICCs) in unconditional models for all behavioral and gaze measures.
Furthermore, to examine combined effects of HOO position and brackets, we additionally added interactions between HOO position and brackets as fixed effects in the model for time to first gaze. In the model for proportion of gazes, we added three-way interactions between HOO position, brackets, and quarters of time bins as fixed effects. Post-hoc analysis was conducted by emmeans, which estimates and compares marginal means between different levels of factors (e.g., present versus absent brackets). Bonferroni adjustments were applied to correct multiple comparisons.
We reported the impacts of HOO position and brackets on problem solving performance (i.e., accuracy and response time) in Supplementary Materials (Appendix A), given that behavioral outcomes are not the focus of the current study.
Results
Intraclass Correlation Coefficient (ICC)
The ICC was 0.095 for the time to first gaze model, indicating that variance among participants was sufficiently large (> 0.05) to warrant inclusion in the modeling analysis. Consistent with the time to first gaze model, the model for proportion of gazes also included the participant intercept as a random effect, although its ICC was 0.012.
Proportion of Onscreen Gazes
Before data cleaning, 211 participants showed 77.20% of gaze points located on the screen. WebGazer achieved over 70% onscreen sampling in 71.78% of problems. After data cleaning, the 119 participants whose data were included in gaze data analyses showed that 87.75% of their gaze points were located on the screen.
Primary Research Goal
HOO Position Effect: Egorova et al. (2024) vs. Current Study
In Egorova et al.’s (2024) lab study, participants fixated on the HOO more quickly when the HOO was placed on the left compared to in the center or on the right. They took on average 420 ms, 674 ms, and 1220 ms to fixate on the HOO when the HOO was placed on the left, in the center, and on the right, respectively.
Figures 2a and 2b present the gaze data patterns in Egorova et al.’s (2024) lab study and the current online study. Partially consistent with Egorova et al.’s (2024) findings, results of the time to first gaze model (see Table 2) showed that when the HOO was placed on the left (M = 811 ms, SD = 784 ms), participants fixated on the HOO more quickly compared to when the HOO was placed on the right (M = 1153 ms, SD = 1143 ms), .
Nevertheless, the time to first fixation on the HOO did not significantly differ between the left HOO condition versus the center HOO condition (M = 788 ms, SD = 809 ms), . Descriptively, the mean time to first gaze on the left HOO in the current study was particularly longer (approximately 400 ms difference) compared to the results reported in Egorova et al. (2024). The beta coefficient of HOO position (right versus left, B = 325.81, t = 5.44) in the current online study was 59% less compared to that reported by Egorova et al. (2024) (B = 797.92, t = 16.65)2. The effect size of HOO position (right versus left, Cohen’s d = 0.35) on time to first gaze was 72% less compared to the one in Egorova et al. (2024) (Cohen’s d = 1.27).
Table 2
Mixed-Effects Regression Models for the Time to First Gaze on the HOO
| Variables | Time to First Gaze_1 | Time to First Gaze_2 | ||||
|---|---|---|---|---|---|---|
| B | t | p | B | t | p | |
| (Intercept) | -0.56 | -11.34 | < .001*** | -0.63 | -10.90 | < .001*** |
| Brackets[A] | -0.01 | -0.13 | .895 | 0.13 | 1.78 | .075 |
| HOO Position[C] | -0.09 | -1.91 | .057 | 0.08 | 1.15 | .249 |
| HOO Position[R] | 0.30 | 5.89 | < .001*** | 0.31 | 4.28 | < .001*** |
| Brackets[A] * HOO Position[C] | -0.34 | -3.50 | < .001*** | |||
| Brackets[A] * HOO Position[R] | -0.02 | -0.22 | .827 | |||
Note. HOO = high-order operator; A = absent; C = center; R = right.
*p < .05. **p < .01. ***p < .001.
Combined Effect of Brackets and HOO Position: Egorova et al. (2024) vs. Current Study
In Egorova et al.’s (2024) lab study, the effect of brackets on time to first fixation varied depending on the position of the HOO. Specifically, presenting brackets around the center HOO delayed the time to first fixation on the HOO by approximately 200 ms, while no such brackets effect was observed when the HOO was placed on the left or right.
Identical to the pattern observed in Egorova et al.’s (2024) lab study, the time to first gaze in the current online study was longer when brackets were present compared to absent around the center HOO, mean difference = 159 ms, B = 0.21, p = .001 (see Figure 2b). Brackets effects were not significant when the HOO was on the left (mean difference = −199 ms, B = −0.13, p = .075) or on the right (mean difference = −117 ms, B = −0.10, p = .153). The beta coefficient of brackets × HOO position (center versus left) in the current online study (B = −294.82, t = −2.58) was nearly identical to that reported by Egorova et al. (2024) (B = −229.87, t = −2.45). However, when the HOO was placed in the center, the effect size of brackets (present versus absent, Cohen’s d = 0.17) on time to first gaze was 57% less compared to the one in Egorova et al. (2024) (Cohen’s d = 0.39).
Figure 2
Results From Egorova et al.’s (2024) Lab Study and the Current Online Eye-Tracking Study
Note. Box plots depict the time to first fixation (Panel a) in Egorova et al. (2024) and time to first gaze (Panel b) in the current study when brackets were present or absent around the HOO in the left, center, or right positions. Segments inside rectangles show means. Upper and lower bounds of rectangles indicate one standard error above or below the mean, respectively. Upper and lower vertical lines indicate the first and third quartiles, respectively. HOO = the high-order operator.
Secondary Research Goal
HOO Position Effect on the Proportion of Gazes Over Time
To validate the use of the proportion of gazes over time in studying online arithmetic problem solving strategies, the current study examined whether gazes on the HOO showed different trajectories when the HOO was placed on the left, in the center, and on the right. The proportion of gazes for each participant, each condition, and at each quarter of time bins, was submitted to mixed-effects linear regression models with brackets, HOO position, and quarters of time bins as fixed effects, and participant intercepts as a random effect. Full model results and post-hoc analyses can be found in Supplementary Materials (Appendix B). Below, we summarize the key observations.
We hypothesized that the proportion of gazes on the HOO would increase in the later stage when the HOO was in the center or on the right compared to on the left. Partially consistent with the hypothesis, the proportion of gazes reached the peak at a later time course when the HOO was placed on the right compared to on the left (see Figure 3a). Specifically, when the HOO was on the left, the proportion of gazes exhibited the highest value at the first quarter of time bins. In comparison, when the HOO was on the right, the proportion of gazes peaked at the second quarter of time bins, with an average increase of 2.16% when time proceeded from the first to the second quarter () and an average decrease of 8.15% when time proceeded from the second to the third quarter (). Nevertheless, when the HOO was in the center, gazes tended to consistently concentrate on the HOO, which was different from our hypothesis. This center-distribution pattern was evidenced by lower variability in gaze proportions across time bins when the HOO was in the center compared to when it was on the left (Figure 3a).
Figure 3
The Proportion of Gazes on the HOO in Different Conditions of Brackets (Present and Absent), HOO Position (Left, Center, and Right), and Quarters of Time Bins (First, Second, Third, and Fourth)
Note. Box plots depict the mean proportion of gazes on the HOO when (a) the HOO was placed in the left, center, or right positions over the first, second, third, and fourth quarters of time bins, (b) brackets were present or absent over four quarters of time bins, and (c) the HOO was placed in the left, center, or right positions with and without brackets over four quarters of time bins. Boxes within the same HOO position or brackets presence conditions are connected with lines to visualize the changing trajectory, respectively. Segments inside rectangles show means. Upper and lower bounds of rectangles indicate one standard error above or below the mean, respectively. Upper and lower vertical lines indicate the first and third quartiles, respectively. HOO = the high-order operator.
Additional Findings
Beyond effects of HOO position, the presence of brackets significantly impacted gaze trajectories. As shown in Figure 3b, when brackets were present, the proportion of gazes increased 3.49% when time proceeded from the third to the fourth quarter of time bins (), while when brackets were absent, the proportion of gazes was not significantly different between the third and the fourth quarter of time bins (mean difference = 1.52, p = .112). This pattern indicated that gazes tended to revisit the HOO at the end of evaluations when the brackets were present. Moreover, a combined effect between brackets and HOO position (right versus center) was observed. As shown in Figure 3c, when brackets were absent, the proportion of gazes on the right HOO increased 7.99% as time proceeded from the first quarter to the second quarter . However, when brackets were present, the proportion of gazes on the right HOO was not increased but quantitatively decreased 3.74% as time proceeded from the first quarter to the second quarter (). This pattern indicated that, when the brackets were present around the right HOO, gaze tended to concentrate on the HOO earlier.
The Impact of Sampling Rate
In this session, we explored the impact of sampling rate on recorded gaze patterns. We calculated participants’ mean time to first gaze as well as the mean proportion of gazes in the first, second, third, and fourth quarters of time bins across all conditions. Then, we examined the correlation between participants’ WebGazer sampling rate and these gaze metrics. The WebGazer sampling rate was negatively correlated with the mean time to first gaze (r = −0.29, p = .001) but was not significantly correlated with the mean proportion of gazes in any quarter of time bins (pfirst = .250, psecond = .054, pthird = .769, pfourth = .557). These results suggest that the sampling rate of WebGazer may influence how fast the initial gaze onset can be detected.
Discussion
The current study investigated the feasibility of moving eye-tracking arithmetic problem solving studies online with webcam-based eye-tracking technology (i.e., WebGazer). The primary research aim was to examine whether participants’ gaze behaviors recorded through WebGazer replicated the findings reported in Egorova et al.’s (2024) lab study. Consistent with Egorova et al. (2024), the current study demonstrated a delayed time to first gaze on the HOO when the HOO was placed on the left compared to on the right, as well as when brackets were present compared to absent around the center HOO. Nevertheless, the WebGazer data did not replicate the HOO position effect (left versus center) on time to first gaze, which may be attributed to the delayed gaze onset recorded by WebGazer when the HOO was on the left.
The secondary research aim was to examine whether a conventional WebGazer metric, i.e., the proportion of gazes over time, can be used in online eye-tracking arithmetic problem solving studies. Supporting the use of this gaze measure, results of the proportion of gazes on the HOO over time not only confirmed the left-to-right viewing trajectory reported in Schneider et al. (2012) but also provided new insights into the effect of superfluous brackets. Extending the results for time to first gaze, our results on the proportion of gaze over time suggest that superfluous brackets can influence gaze trajectories. Specifically, the presence of brackets prompted participants to revisit the HOO toward the end of the evaluation period. Moreover, when the HOO was placed on the right with superfluous brackets, the highest proportion of gazes occurred in the first rather than second quarter of time bins, suggesting that participants may look at the right HOO earlier.
The Validity and Cautions of Using WebGazer
Our findings suggest the validity of using WebGazer in online eye-tracking arithmetic problem solving studies but also raise certain concerns. Supporting the use of WebGazer, the gaze data collected online successfully replicated two eye-movement patterns observed in Egorova et al.’s (2024) lab study, i.e., the effect of HOO position (left vs. right) and the effect of center brackets on time to first fixation. Moreover, the proportion of gazes over time extracted from the WebGazer data readily indicates when participants concentrated on the HOO, as well as how the gaze trajectories on the HOO varied depending on the HOO position and the presence of superfluous brackets. These findings indicate that gaze data recorded by WebGazer can validly reflect students’ problem solving strategies and the effect of perceptual cues, with time to first gaze and proportion of gazes over time as outcome measures. Therefore, our study supports the use of WebGazer in online eye-tracking arithmetic problem solving research to investigate students’ problem solving strategies and perceptual processing.
Nevertheless, the current online eye-tracking study showed a reduced effect size for the HOO position effect (right versus left) on time to first gaze and did not replicate the time to first fixation difference between the center and left HOO conditions. These results may be due to increased time to first gaze when the HOO was placed on the left. In Egorova et al.’s (2024) lab study, participants tended to fixate on the left, center, and right HOO at approximately 420 ms, 674 ms, and 1220 ms after stimulus onset, respectively. Nevertheless, the current online eye-tracking study found that the mean time to first gaze was 811 ms, 788 ms, and 1153 ms for the left, center, and right HOO, respectively. Previous WebGazer studies have found delayed gaze onsets when the eye-tracking experiment was conducted online compared to in laboratories (Slim & Hartsuiker, 2023). Building upon this view, this study observed that the delayed gaze onset was particularly pronounced (i.e., around 400 ms) when the HOO was on the left. Moreover, individuals’ mean time to first gaze on the HOO was negatively related to their WebGazer sampling rate, indicating that low sampling rates may limit the detection of the initial gaze onset. Considering that participants tended to initiate a fixation on the left HOO at an early stage of evaluation (< 500 ms) in Egorova et al.’s (2024) lab study, our findings suggest that WebGazer may exhibit certain limitations in capturing gaze movements within early time windows.
In addition, when the HOO was placed in the center, the proportion of gazes on the HOO remained generally high throughout the evaluation period. This pattern may, on one hand, reflect participants’ problem solving strategies when the HOO was in the center, such as viewing the HOO frequently during the task. On the other hand, this pattern may be due to a high density of gaze points in the center of the screen. We propose this alternative hypothesis based on a previous WebGazer study, which reported that noisy gaze points were more densely distributed at the center of the screen than at the periphery (Bogdan et al., 2024). In future online eye-tracking arithmetic problem solving studies, researchers should be cautious of this center bias and consider placing critical mathematical elements (e.g., the HOO) in more appropriate screen locations. For example, in follow-up arithmetic problem-solving studies using WebGazer, researchers can consider designing two-step arithmetic expressions (e.g., 5 + 3 * 4 or 5 * 3 + 4) to avoid presenting the HOO in the center of the screen.
The proportion of excluded participants (i.e., 43.60%) in the current study was relatively high compared to the rate reported in WebGazer literature (e.g., 20% in Prystauka et al., 2024). We attribute this high proportion of data loss to the limited number of trials in each condition. In the current study, data exclusion was conducted in two major steps. The first step aimed to ensure the quality of gaze data submitted to the analysis. Data was excluded based on participants’ task performance, visual and anxiety status, off-screen gaze proportions, and gaze registration on the target AOI. This step resulted in 17.53% (= 37 / 211) excluding samples, which falls within the typical range of data loss reported in previous WebGazer research. Nevertheless, the second step of data exclusion eliminated a substantial number of participants and increased the overall exclusion rate. Specifically, to ensure a valid comparison across conditions when submitting participants into the regression model, 26.07% (= 55 / 211) participants who did not retain at least one item in each condition were excluded. The high proportion of data loss in the second step is primarily attributable to the limited number of problems (n = 4) within each condition. Accordingly, our findings underscore the importance of assigning a sufficient number of trials to each condition in future online eye-tracking studies of arithmetic problem solving.
Implications
Previous studies have demonstrated that adding perceptual cues (e.g., proximity, color, and superfluous brackets) in math expressions can affect arithmetic problem solving, suggesting the involvement of perceptual processing in math reasoning (Braithwaite et al., 2016; Closser et al., 2023; Egorova et al., 2024; Goldstone et al., 2010; Hornburg et al., 2025; Jiang et al., 2014). The current study, as a first attempt, incorporates webcam-based eye-tracking technology into arithmetic problem solving and perceptual cue research. This new methodology enables researchers to collect large-scale, eye-tracking data outside laboratory settings (e.g., classrooms or schools), which can leverage research regarding perceptual processing and math reasoning in the long term. Taking both validity and caution into account, below we outline four suggestions regarding the application of WebGazer in future online arithmetic problem solving research. First, researchers can employ WebGazer to examine students’ problem solving strategies in an online environment using two outcome measures, i.e., time to first gaze and proportion of gazes over time. Second, the temporal resolution of WebGazer is relatively low. Researchers should carefully consider this when examining rapid eye movements at the early stage of problem solving. To mitigate this drawback, a buffer period can be inserted prior to stimulus onset, allowing WebGazer to refresh and calibrate. Third, WebGazer may record more gaze points near the center of the screen so that the placement of mathematical elements is crucial. Fourth, given the large proportion of excluded data, it is important to assign a sufficient number of trials to each condition.
Beyond methodological contributions, the current study offers insights into how superfluous brackets scaffold problem solving. According to the results of the proportion of gazes on the HOO over time, the presence of brackets may prompt participants to revisit the high-order operations before submitting the answer. Moreover, when the HOO is placed on the right, superfluous brackets may encourage participants to look at the high-order operator earlier. Extending previous literature on superfluous brackets (Egorova et al., 2024; Gunnarsson et al., 2016; Marchini & Papadopoulos, 2011; Ngo et al., 2023), the current study emphasizes that superfluous brackets guide solvers’ attention in both early and late stages of evaluation, which underscores a continuous interplay between perceptual processing and mathematical problem solving.
Limitations and Future Studies
Future studies can improve methodologies in terms of procedure and stimuli. For example, the current study asked participants to subjectively decide whether WebGazer predicted their gaze positions accurately but did not objectively evaluate calibration performance. Future studies can incorporate a validation stage following calibration to compute the distance (i.e., prediction error) between the reference location and the gaze coordinates predicted by WebGazer. The prediction errors obtained during the validation stage can serve as an objective indicator of calibration performance and can be used as an exclusion criterion in subsequent data analyses. Moreover, based on the gaze patterns observed during the validation stage, researchers can account for the systematic errors in WebGazer when extracting gaze measures.
In addition, the current studies used the same stimuli as Egorova et al. (2024) for replication purposes, while future studies can improve the design of arithmetic expressions in several ways. For example, the font of numbers and operators can be standardized by equation editors (Chan et al., 2023). Moreover, numbers used to form expressions across six conditions can be controlled. For example, expressions applied to left- and right- HOO conditions can be symmetric, such as “1 * 2 + 3 + 4” and “4 + 3 + 2 * 1”. In addition, LOOs should be selected with caution, as addition and subtraction have been found to shift spatial attention toward left and right, respectively (Li et al., 2018). With refined stimuli, future eye-tracking studies can further assess whether the effect of HOO position (left versus center) on time to first fixation/gaze is robust.
Conclusions
Gaze data recorded by WebGazer during arithmetic problem solving successfully indicated a left-to-right tendency and brackets scaffolding effects, validating the usage of this webcam-based eye-tracker in online eye-tracking arithmetic research. Nevertheless, a delayed gaze onset was observed in the left HOO condition, indicating the need for caution when using WebGazer to investigate early gaze movements (< 500 ms) during arithmetic problem solving.