Assessing in-class participation for EFL: considerations of effectiveness and fairness for different learning styles

Abstract

Background

This study investigates the effectiveness and fairness of teacher-led assessment of students’ in-class participation and its effect on actual participation levels and language test scores, taking into account the diversity of second language learners’ learning styles.

Methods

The level of participation was measured across ten criteria over a one-semester period in four classes of beginner/intermediate level adult Korean students of English as a foreign language (EFL). The classes were divided into two test groups who had their level of participation assessed as part of their overall grade (n = 76) and two control groups whose participation was measured covertly according to the same criteria (n = 65), alongside a pre- and post-course general English proficiency test (the Oxford Quick Placement Test), and a questionnaire designed to ascertain a learner’s general learning style orientation.

Results

The results suggest a broad range of learning styles may be found even in mono-cultural language learning groups, dispelling the stereotype of the ‘quiet’, ‘rote-learning’ Asian student. There were only minor differences between test and control groups in terms of proficiency test scores and participation levels, suggesting that including participation as a measure of course achievement has little impact on performance. Learners with individualistic learning styles generally achieved lower proficiency test and participation scores than those with styles suited to in-class interaction. However, we also report partial evidence of improved proficiency test scores for learners with group-oriented learning styles at the expense of learners with individualistic learning styles in the test group (and vice-versa in the control group), an effect of pedagogy known as the ‘meshing hypothesis’ - a hypothesis that has often been criticised in the learning styles literature.

Conclusion

The results suggest that including in-class participation as part of a measure of achievement for EFL courses may be both ineffective and unfair for those with certain learning styles, and greater care must be afforded to promote inclusivity of assessment practices given the diversity of learning styles that might be present within a given cohort.

Introduction

The inclusion of a measure of classroom participation in EFL assessment rubrics has become increasingly popular. In a recent (2014) straw poll conducted by the authors of this paper on members of the Facebook group ‘University English Teachers in Korea’ (n = 40),¹ 92 % of teachers surveyed included some measure of classroom participation as part of their overall assessment criteria, typically worth an average 15 % of the total grade for the course, separate to grades for attendance, midterms and finals. Rogers (2011) sampled 521 college instructors in the US, also finding that the majority of instructors there incorporated participation into the overall grade for their courses. However, the present study asks whether including a measure of participation in the assessment of EFL performance is fair, given the claim by Richards (2014) that even in mono-cultural classes, individual L2 learners may approach the learning of a foreign language in different ways - a feature of the language learner known as their learning style:

‘in any one class, there may be students […] with very different learning experiences and approaches to learning. In these circumstances, teaching approaches and activities that work well with one group of learners, or with some learners in a class, may not be as effective or successful with others.’ (2014: e-book, page numbers variable).

Despite the wealth of literature on learning style typologies and their relative effectiveness (or ineffectiveness) for tailored pedagogy, there is less research on the relationship between a learner’s learning style, their level and type of classroom participation in an EFL context, and whether the teacher-led assessment of classroom participation is fair to holders of various learning styles - a gap the present study aims to fill. The study begins with a review of the literature on classroom participation and learning styles in EFL contexts, focusing particularly on the East Asian EFL context.

Literature review

Classroom participation

There have been a number of studies attempting to define classroom participation. Vandrick (2000) claims that most teachers’ ideas of participation are limited, requiring students to speak in class, answer questions, make comments and join in discussions. Fritschner (2000) defines participation in terms of ‘talkers’ who prefer ‘speaking out in class’, and ‘non-talkers’ who participate through ‘attendance, active listening, sitting in their seats, doing the assignments, and being prepared for class’ (2000:352). More recently, Heyman and Sailors (2011) describe classroom participation as ‘a form of active learning in which students publicly discuss the course material’ (2011:605). However participation is defined, making participation part of assessment requirements is claimed to ‘encourage students to develop their oral communication skills and to demonstrate other skills such as interacting and cooperating with their peers and with their tutor’ (Dancer and Kamvounias 2005:446) or lead to increased performance on, and retention of, course content (Dallimore et al. 2010).

Classroom participation is defined in the present study as playing an active role in all in-class activities. In a typical EFL context, such activities might include brainstorming activities, games, quizzes, surveys (in-class or within the institution during lesson time), group debates, role plays, simulations, collaborative creative writing, presentations, and speeches. We highlight the term ‘active role’ in our definition of classroom participation following Weaver and Qi (2005), who claim that ‘students who actively participate in the learning process learn more than those who do not’(2005:570), as well as Dallimore et al. (2010) who claim that students’ comfort when participating in classroom discussion is positively related to learning. In terms of how an ‘active role’ might be defined, one teacher surveyed from the Facebook group states:

‘I started at zero [participation points] and you can lose [points] for sleeping, phone games or Kakaoing [Korean instant messenger], speaking Korean, not participating in group activities, not having textbooks and workbooks and not completing the prep work for classes. Positive points are awarded for asking questions, actively volunteering for things like reading out loud or dialogues, completing prep work and having textbooks and workbooks.’

Thus, students may be rewarded for taking a pro-active approach to in-class activity, but may also be penalised for inattentiveness or perceived non-participation. Another teacher suggests the rationale for an ‘active role’ as follows:

‘I am teaching Basic English and Business English Conversation Classes. One needs to talk. They need to take a chance to open their mouths. Many students have to learn that they have to talk and answer questions, participate in discussions, etc.

Taking into account the different definitions of participation in the literature mentioned above, and considering EFL contexts in particular, we adopt ten criteria (in no particular order) determining classroom participation, listed in Table 1 below:

Table 1 Criteria determining level of in-class participation

In a recent (albeit non-EFL related) study on in-class participation, Dancer and Kamvounias (2005) adopted five measurement criteria, namely ‘preparation’, ‘contribution’, ‘group skills’, ‘communication skills’ and ‘attendance’. With the exception of attendance, these criteria are largely reflected in our criteria 1, 2, 3, 4, 5, 7 (for L1), 8, 9 and 10. However, given the EFL focus of the present study, we adopt the addition of criteria 6 and the L2 provisions in criteria 2, 4 and 7 as suitable for EFL contexts. Particularly, criteria 6 (‘using English at all times’) is perhaps unique to EFL or second/foreign language classroom contexts, given that any oral interaction in the target language is technically part of the expectations of the course. This is unless the L1 is used for helping others through scaffolding (criterion 7), following Kim and Lee (2012) who note that sometimes explaining a difficult word or concept with other difficult terms in the L2 ‘has limitations’ and that certain ‘funds of knowledge’ can be brought to bear in the L1 as long as they lead to meaningful L2 production (2012:129). However, even though the L1 can be used to aid L2 acquisition in a classroom setting, we conditionally include criterion 6 among the total criteria because many institutions, particularly in East Asia, still have an ‘English-only’ policy in place for L2 classroom settings. It is important to note from this list that participation (whether in an EFL context or otherwise) can be measured outside of activities centred around course content, as shown in criteria 4, 5, 6, 8, and 9. A number of these criteria may be considered as general classroom management practice (outside of any intended educational value), yet we feel that such criteria may fall under the general umbrella of classroom participation. This also means that ‘willingness to communicate’ in the L2 (e.g. Peng, 2012; Hsieh, 2014) - while a sufficient condition for receiving a high score for participation - is not a necessary condition.

Alongside the ten criteria adopted in Table 1, previous research suggests participation rates increase when a participation score is included in the assessment rubric (Dallimore et al. 2004, 2013; Carstens et al. 2012). Dallimore et al. (2004) found students’ level of participation increased when participation was overtly included as part of the grading criteria, while Dallimore et al. (2013) found an increase in the participation of those who were randomly called upon to participate in in-class discussions, constituting an element of ‘surprise’. We also note that in certain EFL curricula types (such as task-based curricula),participation in the tasks themselves is the focus of the total course assessment, yet these curricula lay outside of the scope of this paper. In the present study, to ascertain whether the inclusion of participation in the assessment rubric has a positive or negative effect on student performance, we include a control group whose participation is measured according to the criteria in table 1, but for whom such participation is not included as part of the overall grade for the course.

The majority of the criteria in Table 1 do not entail any measure of L2 proficiency ability. Rather, participation is typically measured holistically based on other, non-linguistic, criteria. This can make the tracking of participation over a term difficult (Bean and Peterson, 1998) and such tracking might not hold up to student scrutiny if challenged (Heyman and Sailors, 2011). Additionally, a lack of training for teachers in assessment results in issues of fairness, because it may be difficult to accurately and systematically assign participation scores to individual students over time, especially with large class sizes. If as much as 35 % of a total grade is awarded for participation (not entirely uncommon in EFL contexts, as suggested by a number of teachers in our initial survey and from the authors’ own experiences), then this becomes the primary measure of learning achievement. However, it is incredibly rare to find examples of EFL participation grading where either intra-individual or inter-rater reliability estimates are accounted for. Further training is needed to increase the validity of such practice, but we consider that it is most often left to the teachers’ discretion. This is potentially problematic on two fronts, firstly with Mello (2010) noting that participation grading is ‘value-based’ and that teachers ‘need to be aware of their inherent biases’ (2010:93) and with Rogers’ (2011) claim that instructors are likely to grade participation based on their own beliefs and attitudes of what constituted participation, rather than take the student’s learning styles into account. Secondly, from the student’s point of view, those who know that ‘participation’ counts towards their grades ‘may tend to contribute in superficial ways that actually detract from the learning environment’ (Rogers, 2011:5), or wish to ‘appear attentive without risking too much involvement’ (Weaver and Qi, 2005:571). However, given that teachers do commonly measure participation along such holistic lines, we feel that by holding such practice to scrutiny, we can provide empirical evidence of any advantages or shortcomings that result.

Given the above, Weaver and Qi (2005) ask ‘if student participation is so central to the learning process, why is participation in the college classroom frequently so low?’ (2005:571). Outside of considering the effect of L2 proficiency or ‘willingness to communicate’ on the strategies used to participate in-class (e.g. Hsieh, 2014), the present study attempts to further answer this question with reference to the learning style orientations of our L2 participants.

Learning styles

The present study adopts Reid (1995) general definition of learning styles as ‘an individual’s natural, habitual and preferred way(s) of absorbing, processing, and retaining new information and skills’ (Reid 1995: viii). However, there has been a consistent lack of agreement on the terminology and number of categories of learning styles so far in the literature (Wintergerst and DeCapua, 2001; Coffield et al. 2004; Coffield, 2012; Griffiths, 2012). For example. Coffield (2012) argues that the educational community should move away from conventional models of learning styles research due to endless overlapping and poorly defined dichotomies (e.g. verbal vs. auditory, globalists vs. analysts and left brainers vs. right brainers). In the present study, we adopt a simplified three-category approach following Wintergerst et al. (2003)) and DeCapua and Wintergerst (2005), where a broad range of learning styles can be divided between ‘project’, ‘group’ or ‘individual’ learning style orientations. Membership of a particular group is not exclusive of membership with another group, although the three groups are ‘statistically mutually exclusive’ (Wintergerst et al. 2003:93). These groups are summarised in Table 2 below.

Table 2 Description of Learning Style Orientations (DeCapua and Wintergerst, 2005)

Wintergerst et al. (2003) define participants holding a project learning style orientation (henceforth ‘PO’) as ‘learning best when […] involved in “hands-on” activities or when working with materials in a learning situation’, while participants with a group learning style orientation (henceforth ‘GAO’) learn best when ‘working with one or more students in a learning situation’. Those with individual learning style orientations (henceforth ‘IAO’) learn best when ‘working alone in a learning situation’ (2003:93). We consider that choosing to focus on learning style orientations (henceforth ‘LSO’s) rather than individual learning styles is preferable for both researchers and teachers, as tracking the effect of numerous individualised learning styles on classroom participation is likely difficult, particularly with large class sizes.

The main drive of learning styles research was to show that certain learning styles correlate with certain pedagogical preferences during the language learning process, such as classroom activities, styles of teaching, classroom arrangements, or studying particular aspects of language (Richards, 2014). For example, learners whose learning style takes grammar as central to the language learning process will place more emphasis on certain in-class activities and styles of teaching than learners for whom fluency may be more important (Richards, 2014; see also Dörnyei, 2005; Griffiths, 2012), and some learners might be more predisposed to a certain teaching methodology (e.g. Massa and Mayer, 2006). This view implies that it would be beneficial for instructors to both identify and adapt to students’ learning styles.

The contrast between methodology and expectation is particularly sharp in East Asian cultures, where the educational environment has been considered particularly sensitive to issues of power and politeness in the form of ‘face’ (Brown and Levinson, 1987). In Asian contexts, teachers are supposedly models of correct behaviour and students are allowed to speak only when asked to do so (Scollon and Scollon, 1995), or are discouraged to question either their teachers or other classmates who might be older, or of a different gender (Kennedy, 2002). Interpersonal communication in such contexts could be considered risky (Richards, 2014); and learners believe that certain activities such as ‘role play’ are considered more as ‘play’ than appropriate for learning (Yu, 2001), which could potentially clash with typically Western communicative forms of pedagogy. All of the above can influence willingness to participate in language courses (Park, 1997). Is it then the case that there is an ‘Asian’ learning style preference? Reid (1987), in a study including Arabic and Spanish EFL learners alongside South-East Asian students and East Asian (including Korean) students, found that while ESL students generally preferred kinaesthetic learning styles, Korean and Japanese students held a visual learning style preference. This finding is similar to Littlewood and Liu (1997) who found that Mexican-American language learners tended to participate better in verbal exercises than students from East Asian countries, who tended to do better with visually engaging exercises. Lee (2009) observed that Korean graduate students felt challenged to participate in whole class discussions, regardless of their length of study in the US. Wintergerst et al. (2003) go so far as to suggest Asian students ‘expect’ modes of learning where all of the information comes from the teacher and where ‘little’ to ‘no’ classroom participation is required (2003:99). However, the apparent inability or unwillingness of Asian students to participate is not irreversible, with Kang (2014) reporting a direct correlation between the level of in-class participation and the amount of time Korean students had spent studying abroad. Moreover, Tani (2005) reports that the stereotypically quiet behaviour of Asian students may only be restricted to the classroom itself, noting that students who are quiet in-class may well be boisterous in an ‘English Cafe’ setting where an informal student/teacher set-up is found. Wintergerst et al. (2003) also acknowledge that due to in-group variation, a cautious and judicious interpretation of results is required when linking culture to learning styles’ (2003:100).

While criticism of learning style stereotypes is becoming increasingly valid, it has also proven very difficult to prove the interaction or ‘meshing’ (e.g. Pashler et al. 2008) of a particular teaching methodology and any perceived benefit of that methodology for learners of a particular learning style, at the expense of learners with different learning styles. i.e. to prove the ‘learning styles hypothesis’ or the ‘meshing’ hypothesis (2008:108). Under this hypothesis, students who are predisposed to receptive input would get greater marks on tests than students who preferred productive exercises, if the teaching methodology focused on receptive listening skills. Early evidence of such an effect was suggested in Domino (1979), who claimed that understanding students’ learning styles, and teaching to those styles, was shown to increase student test scores. More recently, however, Griffiths (2012), found no positive correlations between learners’ learning styles (n = 31) and test scores, with 11 out of 17 possible learning styles (e.g. styles that incorporated rote memorisation) actually negatively correlated with test score. Thus, modifying pedagogy to suit learning styles may provide an inverse (and, we suggest, an unintended and unwanted) example of the meshing hypothesis for proponents of the learning styles approach to pedagogy. In particular, given the participation-heavy focus of Western, communicative language teaching-based EFL methodology, the development and sequencing of in-class activities that require a high degree of in-class participation may not benefit all types of certain learning styles, or at least might benefit some at the expense of others. This may then have an effect on the fairness and validity of any subsequent assessment of participation, when it is considered as part of the total grade for a given EFL course.

Research questions and methodology

Research questions

From the literature review, the present study proposes the following research questions:

• 1) What is the distribution of LSOs within a given Korean EFL cohort? Is there a particular ‘Korean’ LSO?

• 2) What is the relationship between participants’ learning style orientations, their general English ability pre- and post- semester, and their level of classroom participation?

• 3) Does the inclusion or non-inclusion of participation in the assessment rubric have any effect on learners of a particular learning style orientation; i.e., is there data that might support the ‘meshing hypothesis’ of Pashler et. al. (2008)?

Participants

The participants were South Korean freshman or sophomore university students attending courses at two universities in Korea. These participants were a mixture of men and women between 18 and 22 years of age, all monolingual Korean native speakers learning EFL as adults. To assess the learner’s L2 proficiency for recruitment and participation purposes, the Oxford Quick Placement Test (henceforth QPT), a standardised 30-min multiple choice general English proficiency test) was administered at the beginning of the semester. This test was preferred as it is freely available, can be administered in a short time frame, and the results can easily be cross-referenced with other standardised tests such as IELTS® or proficiency criteria such as the Common European Framework (Council of Europe, 2001). Originally, 196 students took part in the study across both universities. Students who scored within the range specified for A2-B1 (‘Basic User’ to ‘Independent User’) levels of the Common European Framework were selected for inclusion in this study, as these levels are broadly representative of the majority of Korean high school graduates at the two universities surveyed. The data of students whose proficiency was above or below A2-B1 levels was not included in any analyses (n = 55), although they were free to continue with their classes and with the study. The participants were split between two ‘instructor’ groups (one for each university), who were further divided into a ‘participation’ group whose participation in-class was assessed overtly as part of their overall grade according to the ten criteria in Table 1, and a ‘control’ group (taught separately) whose participation was measured covertly using the same criteria, and not included as part of their overall grade. Participants were organised into these groups with the aim of creating an approximately equal mix of genders and L2 proficiencies (as measured by the QPT) for all groupings. Table 3 describes the grouping and numbers of participants, while the differences between the assessment criteria between groups is described in Table 4 below.

Table 3 Groupings and numbers of participants

Table 4 Assessment criteria between ‘participation’ and ‘control’ groups (% of final grade)

There are three key points to note regarding the breakdown of the total grade. First, the control groups have a greater percentage of their grade apportioned to the midterm and final exams, while having no grade for participation. Second, the participation groups have 35 % of their grade apportioned to participation, which is the largest proportion of the total grade among the other criteria.² Third, while it is common for attendance to be included as part of the participation grade criteria in some contexts, in the present study attendance is graded separately from participation, as was the norm in the two universities where the study was performed.

Other instrumentation (Learning styles indicator, post-semester questionnaires)

To ascertain the distribution of learning styles, a modified and translated version of the ‘Learning Styles Indicator’ (LSI) (Wintergerst and DeCapua, 1999, 2001; Wintergerst et al. 2003; DeCapua and Wintergerst, 2005) was used on all groups. The original LSI is a 23-item questionnaire with a 4-point Likert scale (with responses ranging from always, often, sometimes and never). However, when examining the linguistic issues of the LSI, DeCapua and Wintergerst (2005) reported that their M.A. TESOL students preferred adding a 5th point to the Likert scale and possibly changing the responses of always and never to usually and hardly ever respectively, as the former were considered too extreme by some L2 participants in that study. In addition, the questions on the instrument should be appropriately contextualised (i.e. the questions should focus on the Korean tertiary EFL context) and the participants themselves should be given an appropriate amount of time to complete the instrument, avoiding a ‘just get it done’ feeling that might occur when presented with a lengthy questionnaire (DeCapua and Wintergerst, 2005:8). The document was translated from English to Korean by a native Korean research assistant with a high L2 English proficiency. The translation was necessary as DeCapua and Wintergerst (2005) suggest that difficulties validating learning style instruments are ‘compounded’ when the instrument is not in the L1 of the participants (2005:2). We also modified some of the responses where the word better was used in the original LSI (e.g. Q3, Q5) to more easily upon the recommendation of DeCapua and Wintergerst (2005). The complete English/Korean version of the questionnaire is included in the online supporting information. The responses to the questions on the LSI allow a participant’s learning style to follow one or more of three main modalities or Learning Style Orientation(s) described in section 2.2. Participants were labelled as being of a particular learning style orientation in the present study if they gave enough responses at the usually or often levels for items on the questionnaire. For the project (PO) orientation, having 7 usually or often responses out of the 11 items for this orientation was enough for a participant to be included as having a PO orientation, with 3 out of 5 similar responses for the group (GAO) orientation and 4 out of the 7 responses for the individual (IAO) orientation. We allow some participants to be included in some or all groups, and some participants were not included in any orientation.

Procedure

Initial recruitment began using the QPT described in section 3.2 in the March semester of the 2014 academic year. The modified LSI included in Additional file 1 was also completed by all participants at this time. The teachers were not informed of the distribution of learning styles within their classes to avoid any potential changes in pedagogy or behaviour towards certain students. Participants then took their EFL credit courses over the duration of one semester. The curriculum for both instructor groups was one of a general textbook-led program with exercises focusing on presentation and conversation skills to improve grammar, pronunciation and vocabulary (although different textbooks and materials were used in each instructor group). From experience, we believe such a curriculum is typical of the majority of university EFL courses held both in Korea and internationally. The teachers assigned scores for participation holistically over the semester for both participation and control groups, according to the 10 criteria previously listed in Table 1. The teacher assigned a tally of these scores to each participant at the end of each class, rather than during each class, to minimise disruption to classroom activities. The level of participation was also assessed during the ‘project’ tasks (listed in Table 2 above). Post-semester, the participants repeated the QPT, and the difference in scores between the first and second sittings of the QPT were compared between participation and control groups. Participant data was anonymised during the analysis procedure and any subsequent reporting of the data. All statistical analyses were performed using SPSS.

Results and findings

Learning style orientations

Table 5 shows the distribution of learning style orientations between all groupings.

Table 5 Distribution of learning style orientations

These results suggest that in any given cohort of participants, the distribution of learning style orientations (LSOs) is diverse. A Multivariate Analysis of Variance (MANOVA) was performed with the three LSOs as dependant variables and each type of grouping variable (Participation/Control/Instructor) as fixed factors to ascertain any significant differences of LSO between groupings. The results suggested the distribution of LSOs between all groupings was largely equal (even though the project [PO] orientation was the more frequently occurring in all groups) with significance values all greater than p = 0.050 for PO, individual (IO), and group (GAO) orientations regardless of grouping type, alongside an insignificant Box’s M test (sig = .718), suggesting that the data has equal observed covariance matrices of the dependant variables across groupings. Post-hoc Pearson correlation found that GAO members were also likely to be PO members, although the r and p values were not significant, and that IAO members were not likely to be PO or GAO members in instructor 1’s participation group (henceforth ‘I1P’, r = −.371, sig < .050 for PO and r = −371, sig < .050 for GAO), and IAO members were not likely to be PO members in instructor 2’s control group (henceforth ‘I2C’, r = −.433, sig < .050) and instructor 2’s participation group (henceforth ‘I2P’, r = −.373, sig <0.50). Thus, learners who were either PO and/or GAO orientated were generally not IAO oriented, while despite a weak positive association between PO and GAO orientations, membership of PO and GAO orientations was not so inter-correlated as to be indistinguishable.

LSO and Quick Placement Test (QPT) scores

Next, data analysis focused on the relationship between LSO and the difference in scores between the two QPT sittings. I1P had a mean difference in raw scores of −0.26, I1C had +1.06, I2P had +0.28 while I2C had +1.233. MANOVA was performed with instructor group and participation/control groupings as fixed factors and QPT scores as the dependant variable, but neither ‘instructor’ nor ‘group’ nor a combination of ‘instructor’/‘group’ were responsible for the variation in QPT scores, i.e. the distribution of score variation was uniform across all groupings (all sig > 0.050). Linear regression analyses were performed on both I1P and I2P groupings with PO, GAO and IAO LSOs as predictors of QPT performance, and found that the IAO orientation was a negative predictor of test performance between test sittings 1 and 2 in the I1P group (r ² = .166, b = −2.892, StdE 1.307, sig < .050) and I2P group (r ² = .167, b = −2.711, StdE = 1.305, sig < .050). Thus, the data appear to suggest that in classes where participation was assessed, participants with an IAO LSO performed poorly on the second sitting of the QPT compared to participants with PO and/or GAO LSOs. This result was not found in I1C or I2C groups, and so this finding is restricted to groups where participation is assessed. This result is summarised in Fig. 1 below.

Fig. 1

Effect of learning style orientation on QPT performance pre- and post-semester

Another significant finding is reported for instructor 2, where evidence of the ‘meshing’ hypothesis appears to be present in the difference between QPT scores from sittings 1 and 2 for GAO learners compared with those of PO and IAO learners, dependant on whether participation as assessed or not. This result is illustrated in Fig. 2 below.

Fig. 2

Evidence of ‘meshing’ in terms of QPT performance, learning style orientation, and participation/control for instructor 2

From the chart, it is apparent that for instructor 2, GAO learners performed significantly less well than PO and IAO learners on the QPT between sitting 1 and sitting 2 when participation was not included as part of the overall assessment, compared to the test score for GAO learners if participation was included. As the QPT score improved under the control condition in PO and IAO groups, it can be said there is a direct effect of assessing participation on QPT test scores for learners with PO and IAO orientations at the expense of learners with GAO orientations. We suggest that this result constitutes validation of the ‘meshing’ hypothesis with this particular instructor group. Surprisingly, the same result was not found in instructor group 1, with learners from all three LSOs each improving their QPT scores when participation was not included as part of the assessment criteria, a result shown in Fig. 3 below.

Fig. 3

No evidence of meshing in terms of QPT performance, learning style orientation, and participation/control groupings for instructor 1

LSO and classroom participation

The analysis then focused on the relationship between LSO and the participation scores given to participation and control groups respectively at the end of the semester. Participants were assigned a raw score (starting from 0), with teachers adding points to this score in-class when they observed evidence of participation according to the ten criteria for participation outlined in Table 1. To ensure parity between instructor group scoring, the raw total was converted into a percentage where the highest scoring participant would receive a total participation score of 100 %. This would be reduced by 5 % for every 5 point difference in the raw score between participants (e.g. if the highest raw score was 65, then other participant’s raw scores from 60–65 were also assigned 100 %, participants with raw scores between 55–60 gained a participation score of 95 %, etc.). These scores are summarised in Table 6 below.

Table 6 Participation scores assigned to participants with particular learning style orientations

There are two main points to consider from the data. Firstly, participants in the control groups for both instructor 1 and 2 gained higher participation grades than those in the participation groups. A linear regression analysis was performed to ascertain if the participation/control grouping was a predictor of participation grade, with significant results for instructor group 1 (r ² = .146, b = −9.224, StdE 2.687, sig < .001) and instructor group 2 (r ² = .221, b = −11.924, StdE 2.715, sig < .001). Thus, those participants in contexts where participation was included as part of the overall assessment rubric were significantly less likely to get the highest scores for participation, compared to participants in the control group, for whom participation was not included in the assessment criteria, but was measured covertly.

The second point is that students with IAO LSOs appeared significantly less likely than those with PO or GAO orientations to get high scores for participation in both participation and control groups. A linear regression was performed with learning style orientation (PO, GAO and IAO) as predictors of participation score, with the results showing that the PO orientation was a significant positive predictor of participation score in instructor group 1 (r ² = .166, b = 6.873, StdE 2.933, sig < .050), and in instructor group 2 (r ² = .322, b = 8.911, StdE 2.798, sig < .002). Moreover, IAO was a significant negative predictor of participation in instructor group 2 (r ² = .322, b = −7.858, StdE 3.334, sig < .050). IAO was also a negative predictor of participation in instructor group 1 but the reported result was not significant (sig = .131). In summary, those with PO LSOs generally achieved higher participation scores, while those with IAO LSOs generally did not get high scores for participation. However, as this result was found in both participation and control groups, there was no evidence to support the ‘meshing’ hypothesis in terms of the overt assessment of participation and the success/failure of learners with a particular LSO.

The finding that participation scores were generally lower in the participation groups, and that IAO learners received lower participation scores generally, appears to be linked. The difference between the PO and IAO scores in the participation groups averaged 11.6 % in instructor group 1 and 20 % in instructor group 2 respectively, while in the control groups this difference was only 7.9 % and 9.3 % respectively. It thus appears from the data as though the participation requirement pushed PO learners to participate more at the expense of IAO learners, which had the subsequent effect of reducing the overall participation score assigned to the participation groups. In the control group, PO learners were more likely to gain higher participation scores than IAO learners, but not nearly to the same extent as they were in the participation groups.

Discussion

In terms of RQ1, there are a broad range of LSOs within each grouping, which does not suggest any particular ‘Korean’ learning style. As for RQ2, the data suggest that there is a generally positive relationship between general English proficiency (at least, as measured by the QPT), the level of classroom participation observed, and L2 learners with PO or GAO learning style orientations. This comes at the expense of those with IAO orientations, whose QPT scores and levels of observed participation were lower than their PO and GAO counterparts. This finding was true of both participation and control groups, and was replicated between instructor groups 1 and 2. While the finding that IAO learners generally receive fewer marks for classroom participation is perhaps not surprising at face value, we feel that for at least 8 of the criteria for participation listed in Table 1, these criteria should not cause any undue difficulty for IAO learners. The exceptions are criteria 1 (‘Participating in course content activities appropriately and pro-actively’) and 7 (‘Helping others who are having trouble with course content, either in their L1 or in the L2’). The majority of learning tasks labelled under the umbrella of classroom participation for criteria 1 are generally pair, project or group based (e.g. role plays, presentations), and IAO learners may have been less likely to take an active role in such situations. Criteria 7 also entails some form of pair/group communication, and as such those with an IAO orientation may feel unwilling to speak out to help others. Another possibility is that they may be unwilling to help those who are older or of a different gender than themselves, in line with Korean social etiquette, but we feel that if this were the case, then this behaviour should apply to GAO and PO orientations equally. One important possibility is that participation under certain criteria (such as 1 or 7) may be more salient for the teachers involved in holistically assessing the participation grade. The teachers in the present study assigned participation scores to students at the end of class, yet we entertain the possibility that criteria befitting an individual approach (e.g. ‘active listening’, ‘coming prepared’, ‘taking notes’ etc.) may often go unnoticed by teachers, who are involved in the cognitively demanding task of delivering their classes while at the same time assessing the level of participation of those involved in that class.

The issue of ‘fairness’ regarding the holistic grading of participation as outlined in section 2.1 applies equally to the discussion of these results, in that it is precisely because it may be difficult to accurately and systematically assign scores to each student for classroom participation. Due to the lack of intra-individual or inter-rater reliability estimates and lack of training on assigning scores for participation, such an assessment procedure is difficult, if not impossible. Nonetheless, from the authors’ experiences, the procedure is widely adopted in many EFL contexts, and so the implication of these results is that IAO learners may unfairly be penalized in such contexts, and, thus, that the grading of participation using current practice may no longer be advisable.

An unexpected finding was that IAO learners’ test scores actually regressed over the semester compared to their GAO and PO counterparts, with this result only found in the participation groups of instructors 1 and 2, but not the control groups. Thus, it appears as though the pressure to participate over the semester may have contributed to the poor performance of IAO learners on the test, who may have felt unable to study using their preferred learning style preferences under such conditions. Another unexpected consequence was that for both instructors 1 and 2, the participation group received lower marks for participation than the control group, a finding confirmed in the linear regression analysis. This, we assume, is the exact opposite finding to what one might expect by having participation measured as part of the total grade for the course in the first place, given that the intention in doing so is to force the quieter students to join in more. We suggest that what might actually happen is that those students who were already well-suited to classroom participation (i.e. PO and GAO learners) take an even more active role than they would have otherwise done due to the level of competition enforced upon them, making it more difficult for others to be noticed. It is certainly possible that when participation is not included as part of the overall grade, there is less pressure on those who are unwilling to or find it difficult to participate, and the overall level of participation is improved for the cohort. We also acknowledge comments from an earlier review of this paper that suggests an alternative explanation, where instructors assign higher participation scores when they know those scores don’t actually “count” for the students’ grades. In either case, the consequences are unsatisfactory, with the former predicting a negative washback effect on the part of the students, and with the latter having instructors artificially boosting scores under the criteria, bringing into to question the validity of such a process.

In terms of RQ3, namely the validation of the ‘meshing’ hypotheses (Pashler et al., 2008), the data does not indicate that assessing participation in EFL courses leads to increased classroom participation for learners of a particular LSO at the expense of another, given that PO, GAO and IAO learners all received higher scores for participation in the control groups. In terms of performance on the QPT, evidence is provided in favour of the ‘meshing’ hypothesis in instructor group 2 between GAO/IAO/PO groupings. Unfortunately this result was not replicated in the data for instructor 1, and so there is still insufficient evidence to suggest that pedagogy can be tailored towards those of a particular orientation at the expense of another. Pashler et al. (2008) also suggest that ‘if classification of students’ learning styles has practical utility [for pedagogy], it has yet to be demonstrated’ (2008:117). Moreover, if the purpose of including assessing participation is to get IAO learners to participate more, our data suggests the opposite effect, given that GAO learners performed significantly better than IAO (and PO) learners when participation was included as part of the assessment. However, in a country such as Korea where test scores (even in EFL credit courses) are seen as an important measure of achievement, it is difficult to ignore the struggles of IAO learners on the QPT test (alongside the marks awarded for participation), even if a decision to include (or not include) participation as part of the assessment could be validated via the ‘meshing’ hypothesis. The poor performance of these IAO learners compared to their PO and GAO counterparts is presumably an unintended consequence for the majority of teachers who do decide to include participation as part of the overall assessment for an EFL course.

In terms of future direction, Mello (2010) argues that ‘giving students some “voice”’ (2010:93) in how their participation is assessed may go some way towards alleviating the pressure felt by IAO learners. Heyman and Sailors (2011) suggest alternatives to teacher-led participation assessment, including the use of peer nominations (or ranking) of other peers’ participation in class discussion, while Tani (2005) proposes using written communication outside of class when measuring participation, an approach claimed to be particularly suitable for Asian students. Perhaps the greatest potential solution to measuring (and encouraging) participation is through the use of the online technologies, allowing for ‘blended’ learning or the so-called ‘flipped’ classroom approach. The introduction of learning management systems such as Moodle® can be used to facilitate classroom participation with exercises tailored to students’ individual preferences, with accompanying tailored participation rubrics. Such platforms allow the level and quality of participation to be quantified statistically rather than assessed holistically, potentially removing the element of teacher bias/cognition from the equation. In addition, given the potential challenges to Asian students in terms of ‘face’ (e.g. Brown and Levinson, 1987) in classroom settings, shifting certain collaborative activities to an online format potentially relieves the pressure on students when producing L2 content or challenging the stance of others in student-student or student-teacher interactions. We see opportunities for inclusivity rather than the exclusion of learners with different learning styles in EFL contexts and beyond, and consider the increasing importance of online or ‘flipped’ classroom activity as the key to such inclusivity.

Limitations and potential for further study

Due to limitations of resources, we acknowledge that the sample sizes for each grouping are relatively small, and so our findings may be less generalizable than would be the case with a larger sample. However, as certain findings have been replicated in both groups (while some have not, such as the evidence in favour of the meshing hypothesis), this is an additional measure of validity for our findings, despite the small sample size. A potential criticism of choosing to group participants by LSO rather than individual learning styles is that certain learning styles may not be well represented within a particular LSO, although the LSI has been tested and validated in numerous previous studies. Moreover, we acknowledge that LSO is only one of a multitude of within-learner affective factors than can influence a learner’s willingness to participate. These include proficiency levels, interests, motivation, culture, etc. In other words, classroom participation is affected by a multi-dimensional interaction of individual characteristics, making it difficult to account for participation with reference to learning styles alone. We do not feel that this distracts too heavily from our findings, however, which have clear statistical support from the data. We are also interested in further studies on classroom participation with participants at lower (e.g. A1) and higher proficiency levels (e.g. B2-C2) to assess whetherL2 proficiency is a better predictor of classroom participation than a learners’ LSOs, as well as comparing classroom participation by East Asian and Western participants, to isolate cultural background as a predictor of classroom participation.

Conclusion

The evidence offered here suggests that assessing classroom participation as part of the overall grade for tertiary EFL courses does not have a universally positive effect. The data suggest that those with individualistic LSOs participate less than those with group or project LSOs, at least from their teacher’s point of view. The test performance of those with individualistic ISOs may be hampered if their level of participation is assessed. The authors maintain that great care be afforded to those with individualistic LSOs when assessing their level of participation. If possible, assessing participation as part of the overall grade for EFL courses should be either be avoided or given a seriously reduced prominence in EFL assessment rubrics, given the diverse distribution of LSOs within even culturally homogenous classes. Teachers should be aware of such diversity when developing assessment criteria, and work to provide an inclusive environment where every student has the opportunity to participate in a manner that plays to their strengths rather than highlights their weaknesses.