Attitudes on Skillful 2 and teacher-created
ancillary material
A Costa Rican case study
Actitudes hacia Skillful 2 y material auxiliar creado por docentes
Un estudio de caso en Costa Rica
Kelsey Peterson
Programa Cursos de Conversación
Escuela de Lenguas Modernas
Universidad de Costa Rica, Costa Rica
Vladimir Pereira Brenes
Accenture, Costa Rica
Abstract
Selecting course materials is a crucial decision for many English as a Foreign Language (EFL) departments, as the textbook is often the guide for any course. This case study aims to determine student and teacher attitudes toward a textbook and online platform used for the first time in a first-year English course at a public university in Costa Rica. This study also addresses whether students and teachers consider a teacher-created ancillary grammar booklet necessary for the course. A questionnaire and interviews were the sources of data collection that assessed specific impressions about the course materials. The results of the research show that perspectives regarding the books, from both students and teachers, are not as positive as one might expect. Preference for a supplemental grammar booklet was shown, but questions remain as to whether this material is truly necessary for the course. Positive attitudes toward some aspects of the new books were indicated, yet various issues with the material were voiced throughout the entire study. Following a thorough analysis and discussion of results, we provide recommendations for the First-year English department and other EFL institutions considering
using Skillful 2 or changing textbooks.
Keywords: textbook evaluation, materials evaluation, EFL, student perspectives,
Skillful 2
Resumen
Seleccionar los materiales de un curso es una decisión esencial para muchos departamentos de inglés como lengua extranjera (ILE). El libro de texto suele ser la guía de muchos cursos. Este estudio de caso busca determinar las actitudes de estudiantes y docentes hacia un libro de texto y una plataforma en línea utilizados por primera vez en un curso de primer año de la carrera de inglés en una universidad pública en Costa Rica. Además, esta investigación aborda si los estudiantes y profesoras consideran necesario un folleto suplementario de gramática. Un cuestionario y entrevistas fueron las fuentes de recopilación de datos que evaluaron impresiones específicas con respecto a los materiales. Los resultados del estudio muestran que las perspectivas de estudiantes y docentes sobre los libros no son tan positivas como se esperaba. Se mostró una preferencia hacia tener un material suplementario de gramática, pero aún queda la duda de si es realmente necesario. Se detallan actitudes positivas hacia ciertos aspectos de los nuevos libros, pero también problemas con los materiales a lo largo de la investigación. Tras un análisis exhaustivo y una discusión sobre los resultados, ofrecemos recomendaciones para el departamento de Primer Año de Inglés y otras instituciones ILE que estén considerando usar Skillful 2 o cambiar sus libros de texto.
Palabras clave: evaluación de libros de texto, evaluación de materiales, ILE, opinión de estudiantes, Skillful 2
Introduction
Textbooks are considered key components of most language programs since they provide classroom lessons with different activities, readings and explanations (Richards & Tomlinson as cited in Rahimi & Hassani, 2012). Deciding which textbook to use to guide the course is a
significant task.
In March 2019, the First-year English department of the School of Modern Languages at the University of Costa Rica (UCR) discontinued using the books Q: Skills for Success levels 2 & 3 from Oxford (used for approximately six years) and started to use Macmillan’s Skillful 2, for the Integrated English II course (LM-1002). LM-1002 is a required first-year English course designed for intermediate students; it integrates the macro skills of listening, speaking, reading, and writing throughout the duration of the semester (typically 14 to 16 weeks). Both Q: Skills for Success and Skillful 2 include a Listening & Speaking book, Reading & Writing book, and online platform. In addition to changing textbooks, the department stopped using a grammar booklet produced by professors of the First-year English department; this ancillary grammar booklet had been designed to supplement the components of the previous Q: Skills for Success books with additional grammar explanations and exercises. After hearing some negative comments from the students of this course about the new book and the lack of teacher-created ancillary material, the researchers decided to carry out this project, which presents the opinions of the main users of this book to the First-year
English department.
The goal of this study is to determine students’ and teachers’ attitudes toward the new textbooks and to address whether students and teachers need the supplemental grammar booklet that is no longer in use. To this purpose, we evaluated perspectives on the books, online platform, and ancillary teacher-created grammar booklet in depth. We believe this study may aid the First-year English department in forming conclusions on this recent change in materials. It is also our hope that EFL departments considering material-change implementation, such as textbook substitutions or the addition and/or elimination of ancillary material, benefit from the results provided in this case study. With these objectives in mind, 20 students from two LM-1002 groups and three instructors provided data via an online survey and face-to-face interviews.
After an overview of relevant previous studies about textbook evaluation and students’ involvement in the field, the methodology and results of this study are presented; these are followed by the discussion of the findings and the presentation of conclusions and recommendations for EFL departments and researchers interested in material evaluation and material-change implementation.
This case study is unique in that it evaluates the 2nd edition of Macmillan’s Skillful 2 textbook and online platform. At the time of this study, no other textbook evaluations of Skillful 2 (2nd edition) were found. Similarly, this study also examines the necessity of teacher-created ancillary material, an aspect that was not identified in any of the textbook evaluation case studies researched for this case. Given the lack of published evaluations of this newest edition and considerations in the literature of teacher-created ancillary materials, the data here could be especially useful for departments considering the use of Skillful 2 or the addition/elimination of materials
produced by teachers.
Literature Review
EFL textbooks have experienced significant changes over the years, particularly in terms of their content; this evolution can be better understood when considering previous research on textbook evaluation. As Quirós (2018) explains, EFL materials have been heavily influenced by the English teaching methods and approaches of the age. She cites the Grammar-Translation Method as one example: due to the emphasis on direct translations and little need for oral communication, materials during this period (roughly 1840s-1940s) consisted of sentences to translate, bilingual vocabulary lists, and abstract grammar explanations (pp. 2-3). Today, many EFL textbooks follow a communicative approach and contain exercises dedicated to building communicative skills and critical thinking, often with real-life
applications and cultural aspects (Mohammadi, 2013). According to Quirós (2018), the need for communicative skills and cultural competence may be due to the needs of the world: a more globalized world requires EFL learners to develop new competencies. This is supported by Korpela (2007) in the context of Finnish comprehensive schools: not just national factors and institutional factors influence textbooks; international factors have contributed to changes in the content of EFL materials as well (pp. 8-9). In addition to the influence of EFL methods/approaches and globalization, advancements in technology have contributed to EFL textbook evolution, resulting in textbooks available in digital formats with ancillary digital platforms for supplementary practice (for an in-depth explanation of the advantages of digital textbooks, see Woollerton, 2016).
Research on the topic of textbook evaluation shows varied approaches to this means through the past decades. These range from private, ad hoc checklists that employ numerous methods to assess the value of a particular textbook (Ansary & Babaii, 2002, p. 3), to professional reviews published in specialized journals that for a time went virtually undetected by most teachers (Sheldon, 1988
p. 240). Textbook evaluation has also occurred through nonprofit initiatives such as one by the Educational Products Information Exchange (EPIE), implemented to “counsel [the public] on availability, use and effectiveness of educational materials” (Lehr, 1979,
p. 888). The common ground in all these examples is seeing textbook evaluation as a process of matching the characteristics of an educational material to an established conception of what the material should contain.
Identifying the characteristics of a good foreign language (FL) textbook is a process that has generated different published studies through the decades. In their research, Ansary & Babaii (2002) analyzed 10 different checklists (from 1971 to 1996) and 10 different Textbook Reviews (from 1981 to 1994) and gathered what they considered “a set of universal features of EFL/ESL textbooks” (p. 6). They grouped those characteristics in four main areas: Theory or approach (language and learning), Content presentation (objectives, rationale, and syllabus), Physical makeup (layout, quality, editing, size, and weight), and Administrative concerns (state policies, local situations, and price). Other researchers on the topic, like Garinger (2002), Litz (2005), and El-Dakhs (2011) also provide an analysis of the characteristics of textbooks and checklists from different decades to determine concepts of ideal FL textbooks. However, all conclude that there is no definite system and that the result is subjective and guided by experience or practice rather than theory. Similar to Ansary & Babaii (2002), these authors propose varied areas with sub-topics to consider when searching for a good FL textbook, including diverse aspects from the textbook’s content, pedagogy, and practicality. Some of the features from the studies above are part of what the participants on the current case study would need to analyze and comment on to report positive or negative attitudes towards them.
When choosing a population for materials evaluation research, teachers are not the only option available. Several researchers in the field of textbook and curriculum evaluation support the inclusion of students into the materials evaluation process (Kalmus, 2004; Knecht & Najvarová, 2010; Roberts, 1996; Wright, 1990). As Kalmus (2004) argues, when researchers ignore the capacities of learners to critically discuss curriculum, they “underestimate pupils' roles as active, resistant, and sometimes cynical readers” (p. 2). Consistent with this advocacy for student involvement, Knecht & Najvarová argue that involving students in materials evaluation could lead to textbook improvement (2010). While students may have been overlooked in the past as sources of valuable feedback (Wright, 1990), in recent decades, numerous studies have been conducted that take students’ opinions of EFL materials into consideration (Alshehri, 2018; Daneshfar & Abdollahi, 2018; Kalmus, 2004; Litz, 2005). Given the capacity of students for active interaction with materials, as well as the benefits that student perspectives provide in terms of data collection, this materials evaluation case study considers both teacher and student perspectives.
One additional aspect to consider when discussing textbook evaluation is the fact that, frequently, class materials involve more than just published textbooks. It was of particular relevance for this study to include the concept of extra or ancillary materials (commonly created at UCR by the language instructor) to the official textbook from a publishing house since their use is part of this research’s background. Unfortunately, only a brief reference to this type of material was found in the literature reviewed. Sheldon (1988) mentions in his investigation that “it is a cruel paradox that for students, teacher-generated
material (which potentially has a dynamic and maximal relevance to local needs) often has less credibility than a published textbook, no matter how inadequate that may be” (p. 238). The topic of how differently published textbooks vs. teacher-created ancillary materials are seen will bring an interesting discussion to some sections of this paper.
With this study, it is expected that the First-year English department consider reviewing our findings and contrast them to what current literature states regarding the textbook evaluation process. With this expectation in mind, the following research questions were generated:
Methodology
Qualitative research does not seek to prove a causal relationship; rather, it explores attitudes, process, or experiences through “what” and “how” research questions (Riazi, 2017, p. 112). One subset of qualitative research is the case study, defined by Dörnyei (2011) as “a method of collecting and organizing data so as to maximize [...] understanding of [...] the ‘Particular One’” (p. 152). Consistent with the
previous definitions, the present research is a qualitative case study that seeks to obtain information on attitudes of a specific population at a
particular institution.
The researchers decided to work with two LM-1002 groups (i.e. students from two different class sets, taught by different instructors) and three LM-1002 teachers. This was a convenience sampling.
Due to class attendance and students’ willingness to participate, 20 students participated in the student survey. The student survey (see Appendix A) consisted of 12 sections, seeking specific details on the participants’ opinions about micro and macro skills presented in the materials; it was created using Google Forms and was sent to the participants via email. Participants had up to 50 minutes to complete and submit the form; the vast majority of participants finished in under 20 minutes. Out of the 20 participants, researchers then selected six individuals for follow-up interviews (see Appendix B); five of these six respondents collaborated.
As an additional source of data for triangulation, the instructors of LM-1002 participated in semi-structured interviews (see Appendix C). Due to busy schedules and academic responsibilities, responses were obtained in three different ways: instructor #1 provided her input during an interview with one of the co-researchers; instructor #2 completed the interview questions digitally, and instructor #3, as co-researcher of this project, used introspective analysis to write her answers to the interview questions. This information is highly valued, coming from the professionals using the teaching materials examined in this study.
In order to receive approval for the study, a letter was sent to the coordinator of the First-year English department to explain the intention of the study and formally request permission to carry out the research. Permission was granted under the condition that the instruments administered to students be written in their mother tongue. For this reason, the student survey and interviews were created in Spanish and translated to English later for this study.
Before administering the instruments, the researchers informed the population of the purpose, process, and steps of the study and obtained written consent. In accordance with international and professional regulations, participants were made aware that their involvement in this study would have no repercussions on their grades (or job in the case of the instructors), that they could stop participating whenever they decided to, and that all information would be registered as anonymous. Code names were used to guarantee anonymity and call specific participants for a follow-up interview.
Results
The following is a review of our main findings, organized into Student and Teacher opinion sections. Five students participated in a follow-up interview (information presented here as bulleted lists), during which they provided additional details (AD) to their initial responses.
Student Survey and Interviews
Book topics: Over 60% of participants selected positive adjectives to describe the topics (useful, up to date, and relevant). The Reading and Writing book always scored higher than the Listening and Speaking. 35% mentioned the topics of both books are boring. Unit 8, Stories, was popular among responses of interesting topics (3 participants). One participant described Skillful 2’s topics as more up to date than the previous books’. Among the suggestions provided regarding book topics, respondents included: topics do not generate further conversation, topics should deal with the country’s reality, students do not feel identified with the topics, and this does not motivate them to discuss further. 65% of participants reported the topics are “adequate for the level of English intended,” highlighting varied and detailed vocabulary. Within the 35% who did not agree to this claim, six respondents commented that content was similar to LM-1001, one person said grammar was basic, and another person said readings were simple.
Exercises: 100% of respondents agreed that the instructions for exercises and activities were clear and easy to follow. More than 50% reported grammar and listening exercises were easy. 70% marked reading exercises as adequate. No participant marked these exercises as difficult.
Speaking: More than 75% of respondents agreed that the books contain enough exercises on conversation and group discussion. 80% reported there are not enough exercises on pronunciation. All respondents marked that there are not enough exercises of individual presentations. More than 50% reported the Critical Thinking section is not challenging but also mentioned this section motivates them to learn. From 30% to 50% selected negative adjectives (irrelevant and boring) to describe this section. Among the comments provided for this section, participants listed: simple, basic, repetitive, and that it can improve. Regarding the Pronunciation for Speaking section, 75% of participants reported that it helps improve pronunciation. Other comments indicated that the course should emphasize pronunciation more (2 participants), there is not enough pronunciation practice
(1 participant), and the phonetic alphabet should be covered in the course (1 participant). One respondent indicated that he/she searches for the pronunciation of words in other sources
like dictionaries.
Grammar: 95% of participants agreed that the grammar explanations are clear; however, 35% provided comments saying that explanations are short and that the professors’ help is needed; one respondent described the grammar explanations as similar to Q: Skills’. 65% reported there are not enough exercises on grammar.
Writing: Less than 50% of respondents reported that the sections writing skill and writing model have helped them improve their writing. 60% agreed the section writing task is helpful, and 20% reported none of these sections have helped them improve their writing. Among the positive comments, participants reported these sections provide practice, guide, and recommendations on how to do the tasks. As negative comments, they listed: not challenging, always the same, not enough writing practice.
Listening: Less than 25% of the participants reported that close exercises and global exercises have helped them develop their listening skills. 45% said the unit video section is helpful. 60% agreed the pronunciation for speaking sections have helped them develop their listening skills. 10% said none of those units are helpful. With respect to the book’s audios, participants selected the following descriptors for them: boring (45%), motivates learning (35%), irrelevant (25%), motivate participation in class (10%), and challenging (5%). As part of the comments provided on this section, participants listed: audios are easy and people speak slowly. A shared complaint among most participants is that the level of difficulty of the audios in class is much lower than the audios used in the exam.
Reading: 50% of participants reported that the readings motivate them to learn. 25% said readings are challenging and motivate them to participate. 20% labeled readings as boring, while 10% did as outdated and irrelevant. As part of the comments for additional details, participants agreed that readings are easy; one respondent reported that readings did not add much for the exam, and another participant said readings are informal. When asked if there was something they liked about the readings, participants listed: unknown facts and topics that are interesting, help to write professionally, applicable to daily life, and one participant reported readings provide ideas to talk about during the oral exam.
Vocabulary: 90% of participants reported that the vocabulary sections have helped them improve their English, highlighting new words to learn and their usage. Within the remaining 10%, one person reported words were known and the other that words were covered in LM-1001.
Videos: 75% of participants agreed that the videos are interesting. 50% said videos are useful to improve their English. 10% labeled videos as outdated. A common positive comment in additional details is that participants value that the videos show different accents. When asked if there was something they liked about the videos, participants reported: use of native speakers of English, varied accents and nationalities, and topics unknown to the student.
Study Skills: 55% of participants reported that the Study Skills section has not helped them improve their study skills. As part of the comments for additional details, respondents mentioned that the section included boring tips and known topics. A shared response was that students ignored or forgot the section when studying. One person said that s/he already had a study technique that worked well for him/her. Half of the participants reported that this section has not helped them learn English; three of them highlighted that these are not new techniques and were covered in LM-1001, and five respondents confirmed they do not pay attention to this section.
Online Platform: 65% of the participants reported that the online platform has helped them learn English. Within the remaining 35%, four participants complained that the online exercises are simple and boring, two people reported some system bugs when answering incorrectly, and one person mentioned problems accessing scripts. When asked about the frequency with which participants use the online platform, 30% of participants almost never use it, 20% use it during class time, and the remaining participants use it once a week (10%), every two weeks (10%), once a month (5%), at the start of each unit (5%), when I need to practice (5%), or N/A (5%). When asked to describe the online platform, common participants’ responses were “easy” (4 people) and “boring” (6 people). Other responses included: professors provide better materials, it has technical bugs, a good way to rehearse the class topics, and useful to re-play videos
and audios.
Personal considerations: When asked what participants would add to the current textbooks, common responses included: more grammar exercises (9 people), more challenging audios—similar to the ones used in the exam (12 people), and suggestions regarding topics (more interesting, related to our reality, and leading to more challenging discussions). Two respondents mentioned discussion topics could be more challenging for university students and related to the reality of the country. Regarding suggestions that participants would offer about LM-1002 materials, 35% of participants reported that books are fine as they are, 20% indicated that the materials should be more challenging, 10% suggested a grammar booklet, 10% mentioned that the materials have typos, and 5% suggested “something to make the class less monotonous.”
Instructor Interviews
For this section, the three instructors of LM-1002 will be referred to as Instructor 1 (I1), Instructor 2 (I2), and Instructor 3 (I3).
Book Topics: For the two out of three instructors who commented on the book topics, the topics of both books are relevant to their students’ lives, and they facilitate critical thinking. I1 indicated that the topics from Skillful 2 were “more academic” than the topics of Q: Skills.
Speaking: In terms of speaking tasks and pronunciation exercises, I1 and I2 gave their opinions, and the results were mixed. I1 indicated that the speaking tasks at the end of each unit do not vary and are not motivating for her students. I2 maintained that the speaking tasks are varied and are more appropriate for the level of the course than the speaking tasks from the previous Q: Skills textbooks. For both I1 and I2, the pronunciation exercises are useful, but I1 indicated that they are not sufficient, while I2 sustained that the pronunciation skills prepare students for some skills they will see later in the major such as linking and
minimal pairs.
Listening: I1 and I2 commented on the listening exercises of Skillful 2. For both, neither the audios nor the exercises in the Listening and Speaking books prepare students for the listening exam. The difficulty level of the exam audios and exercises is much higher compared to the main audios and exercises of the Skillful 2 Listening and Speaking book. Additionally, for both I1 and I2, the Skillful audios are inauthentic, while the exam audios are taken from authentic sources like U.S. podcasts. As I1 said, “The audios are a bit more challenging [in Skillful 2], but the test audios are less guided and more authentic.” I2 appreciated that the audios of Skillful 2 are much longer than those of Q: Skills (five to seven minutes compared to two to four, respectively) but emphasized that Skillful 2 contains fewer exercises:
I did not like the Q: Skills audios at all, but the book did contain a higher number and more varied format of exercises. Skillful 2 has one global listening exercise and maybe two close listening exercises per audio, whereas on the exam, each audio comes with sometimes ten items to complete, so they end up doing a lot for the exam but not a lot of practice.
Reading: All three instructors found the unit readings more challenging than Q: Skills readings. I1 reported that they increase in difficulty throughout the book. I2 mentioned that the readings are more academically-focused than Q: Skills readings, albeit slightly less authentic and less extensive in number of exercises than the articles used in course exams.
I2 also commented that students would benefit from reading “real-life articles” in addition to those in Skillful 2.
Writing: All three instructors
observed that some aspect of the writing sections of each unit do not always accurately reflect the type of writing expected and evaluated in the course [a one-paragraph composition]. As I1 stated, “I like the models, but we ask them to write a specific type of paragraph; it’s...hard to find a book that reflects what the department wants the students to learn and produce.”
I3 commented that the writing model and task are too advanced for the course at times: “Students are asked to write an essay and do research for their writing, which is not part of the program of first year [English].” I2 insisted that the writing models are helpful even when the model does not match the type of writing evaluated: “Although sometimes the writing task is different than what we expect in the department, the models can be used for identifying the parts of a paragraph, finding transition words, etc., so they are always useful in some way.”
Grammar: All instructors noted that the textbook grammar exercises are not sufficient and that more practice needs to be provided. I3 concluded that further explanation of grammar concepts is not necessary; I2 reported that further explanation is necessary. I1 made no reference to further explanation in the interview.
Vocabulary: Results regarding the vocabulary section were mixed. Negative aspects mentioned included the type of vocabulary exercises (I1: the format “gets repetitive”; I2: it is “almost always matching and fill-in-the-blanks”), the difficulty level
(I3: the vocabulary is “easy for Spanish speakers”), and the relevance of the vocabulary words for the unit being
studied (I1: “currency” as a vocabulary word and key factor in discussion was not appropriate within a unit about storytelling). Positive aspects mentioned by the instructors included an increase in more challenging vocabulary compared to Q: Skills (I2’s examples: “follow-up,” “impair, “insurance,” and others) and the fact that each vocabulary section relates to the same unit topic, making the sections a good complement for the other unit
skills (I1).
Videos: All three instructors approve of the videos but believe that the accompanying video exercises are simplistic and could be extended. As I2 described, “the exercises are almost always discussion questions for the pre- and post-listening and either True/False or multiple choice for the while-listening section.” Positive adjectives used to describe the videos included authentic (I1 and I2), interesting (I1 and I2), and up to date (I2).
Online Platform: Opinions regarding the online platform were mostly negative. I2 and I3 supposed that the online platform exercises may be too easy for students, explaining that their students finish the entire set of unit exercises quickly (in 10-20 minutes maximum). I2 commented that the platform contains the same writing task as the one in the book, which is “basically a lost opportunity for extra writing practice.” Two teachers noted that the students do not have access to the videos in Skillful 2, though they could access them with Q: Skills. I2 concluded that students seem to prefer doing other types of online exercises than those provided by the platform:
“I think students prefer doing online practice with the documents I send them, which have links to games, online quizzes, and explanations of concepts.” I2 stated that in comparison to the Q: Skills platform, Skillful 2 is not as complete given the lack of some items (unit videos, for example). For I3, the Q: Skills online platform was preferable for instructors, as it was easy to use and had a variety of extra resources for teaching.
Personal Considerations: I1 and I2 mentioned and approve of the implementation of a supplemental listening booklet in the course. I1, I2, and I3 supported the implementation of a supplemental grammar booklet. When asked about the change of books in general, all three instructors preferred Skillful 2 over Q: Skills.
Discussion
A first glance at the results confirms Knecht & Najvarová’s (2010) claim that students are capable of evaluating a textbook “responsibly and validly” (p. 12). Valuable feedback was provided, including most useful and troublesome sections, examples of what really works for the majority and what does not, and ideas for improvement that only users of the materials are able to generate.
The researchers categorized the aspects analyzed here based on percentages of responses and our analysis of whether the answers provided show support or disapproval from the participants. Following this logic, five levels were identified to report the attitudes of LM-1002 students and instructors:
This discussion is organized based on these levels of positive or negative attitudes for each of the instruments applied. The opinions of the researchers are presented after listing aspects on each level, including contrasting results and cross-references between answers or instruments.
Student Survey: Clear dominance of positive attitudes toward Skillful 2 was present for the following opinions: the instructions for the exercises are clear and easy to follow, the grammar explanations are clear, at least one of the three writing sections helps improve their writing, the vocabulary sections help improve their English, readings are interesting, up to date, and relevant, and the videos contain up-to-date content. The researchers think it is satisfactory that the majority of students indicated that the instructions are clear and easy to follow since they are essential to the completion of tasks and, as such, must be clear (Viales Angulo & Carmona Miranda, 2014); the clarity of instructions is also a common item described on several textbook evaluation checklists found in published studies (Ansary & Babaii, 2002; Knecht & Najvarová, 2010; Skierso, 1991). No doubt it is great that a robust majority of the participants think the grammar explanations are clear; notwithstanding, there were contrasting comments where respondents maintained that grammar explanations were understood thanks to the further explanations of the instructors. This dichotomy will be addressed later in this paper. Initially, it seemed positive that students said that at least one of the writing sections helps with their writing; however, on a deeper analysis, what was expected is that all three writing sections would help students in some way. This is not a major detriment for the book evaluation because the percentage of the grade dedicated to writing is small. In terms of the vocabulary sections, the most common comment is that respondents are learning new words here; this seems a good indication for instructors to dedicate ample time to these sections. The researchers have contrasting feelings with the fact that students ranked positively three aspects of the readings because the study included six: only 50% were rated as useful. There is a reference to this and an additional contrast about readings later in this paper. Finally, videos being labeled as up to date matched the researchers’ predictions since this edition of Skillful 2 was recently released (2018 edition).
At the other end of the spectrum, clear dominance of negative attitudes toward the books occurred for the following areas: books show little focus on pronunciation, low difficulty level of the audios—especially when compared to the audios used in listening exams, audios do not promote participation in class, the Close Exercises and Global Exercises sections do not help develop listening skills, insufficient exercises about individual presentations, and the general opinion that the Skillful 2 books need some type of improvement. Regarding pronunciation, the researchers wondered whether students are interested in learning the International Phonetic Alphabet (IPA) because they want to know the proper pronunciation of words or because they know it would be part of future courses and want to have some exposure to this topic. It could also be that some students studied IPA during high school and wish to continue using it in LM-1002. Additionally, given that English is not a phonetically-spelled language, students may struggle to pronounce words correctly and, consequently, might wish to study pronunciation more deeply to feel more comfortable pronouncing words in class and in exams. Almost all students insisted that the audios are too easy; some students and the instructors (as will be seen in Instructor Interviews) emphasized that there is a clear difference between the difficulty level of the audios used in class vs. the ones used in evaluations. In addition to these major issues, students say that the difficulty level of the audios is low, and they also say that the listening comprehension exercises are not helping them improve this macro skill. The researchers do not think it is strictly necessary to have more practice on individual presentation due to the types of evaluation on which LM-1002 focuses. It is not clear for this study whether students want more practice on this type of
presentations or that they were just noting that there are not enough
opportunities for it. In the end, the reasons why students would want more practice on this type of presentation remain unclear. Lastly, it is important to mention that students signaled three main areas of improvement for the Skillful 2 books: listening activities, grammar exercises, and topics. This has pedagogical implications as it indicates that instructors may want to consider adding and/or adapting material in these areas to respond to students’ needs and wants.
For most of the aspects covered in the survey, there was a moderate dominance of opinions (60% to 79%) from the respondents, either positive or negative. The following aspects of the books to be discussed reflect
those particularities.
Aspects within a moderate dominance of positive attitudes include the following perspectives: the books’ topics are useful, up to date, relevant, and adequate or the level of language intended; the books contain enough exercises on conversation and group discussion; the Pronunciation for Speaking section helps improve their pronunciation and develop listening skills; the Writing Task section is helpful; the audios are relevant and videos are interesting; and the opinion that the online platform is useful. The researchers thought students would agree that topics are relevant for multiple aspects of their lives: professional, academic, societal, and personal; therefore, it was surprising that only 65% of participants agreed with this prediction. It is curious that three students highlighted Unit 8 (Storytelling) as interesting; however, during that unit, the instructor integrated extra activities that were not included in the Skillful 2 books. The researchers cannot know if Unit 8 was relevant to these students because of the book material or because of the supplemental activities. It is somewhat beneficial to have a majority of students claim that there are enough exercises on conversation and group discussion and that the Pronunciation for Speaking section has helped them improve listening skills because 60% of the evaluation of LM-1002 is dedicated to listening and speaking; however, a higher percentage of students indicating approval here was expected given the greater weight of these macro skills on students’ final grades. Regarding the positive opinion about the Writing Task, the researchers believe it derives from the fact that it is production-based, as will be discussed later in combination with other writing activities. There was a little confusion with the fact that students reported audios as easy and boring but also relevant. Notwithstanding, it is understandable why students like the videos, saying videos feature accents and native speakers, combined with the fact that they show real-life situations. Then, it was surprising that a moderate majority of students reported that the online platform is useful for language learning; based on critiques of the platform heard in class during the first weeks of the semester by one of the researchers, no dominance of positive opinions was expected. The researchers think this result could be related to time, as students had spent more time working with the platform when they completed the survey. Another possibility is that the students could have formed their initial impressions upon general aspects such as whether they liked the platform overall; perhaps they had not considered the usefulness of the platform for language learning specifically until asked that particular question in the student survey.
In terms of negative attitudes toward the new books, the researchers found moderate dominance of opinions including the comment that readings are not challenging and do not motivate students to participate in class, that there are insufficient exercises about grammar, and that audios do not motivate students to learn. Motivation to participate in class could be affected by the readings not being challenging. Thus, the two responses are interrelated. It is curious that although a moderate majority of students reported that more grammar exercises are needed, only two respondents specifically indicated that they want an additional grammar booklet, as will be seen in later analysis. As previously mentioned, students emphasized that the audios used in class are easier than the ones in the evaluations; given this and other negative adjectives used to describe the audios (slow, boring), the idea that audios do not motivate students to learn is understandable.
So far, items for which the results showed clear or moderate dominance of positive or negative opinions have been discussed. However, the study also revealed aspects for which the participants’ opinions were divided at almost 50 - 50 percent, revealing contrastive, unclear attitudes. These aspects included the perceived importance and usefulness of the Study Skills section, the usefulness of the Writing Skill and Writing Model sections, how useful the videos are, whether participants perceive that the audios are boring or that the readings motivate them to learn, and the suggestions provided for the materials of the course. The researchers were shocked to find seven respondents (35%) claiming that they did not remember or pay attention to the Study Skills section; since this is not evaluated, students might think they do not need to dedicate time to it, and therefore, they ignore it. Students may consider that they already know how to study or have a study technique that is enough, and that might be why part of them do not pay attention to it. Since Writing Skill provides something to focus on and Writing Model just presents an example of what is expected, students may think these sections are not as helpful as actually writing and producing (which is done in Writing Task—evaluated positively). As mentioned earlier, this idea of production may be what is seen as useful. In the same sense, regarding the participants’ opinions about videos, the researchers discussed the possibility that students may have understood “your English” (on the question’s wording) as production; this may be why half of them do not see videos as useful: because they do not associate watching videos to production. During the interviews, students commented that they think videos are pedagogically-designed and slow; they also think audios should contain natural speech and not dialogues that seem learning-oriented. The comments about readings being interesting and boring can be related to motivation: since 50-50 of the respondents said reading are interesting/not interesting, it was not surprising that also 50-50 said readings motivate them to learn (or do not). Based on the results from the grammar section of the student survey, the researchers expected that the majority of students would suggest the addition of a grammar booklet, but only two mentioned that. This last result was intriguing; there will be an additional comment about it in the recommendations section.
Interestingly, results also brought to light at least three aspects of Skillful 2 where participants provided contrasting ideas; in other words, there were inconsistencies or dichotomies in participants’ responses. For example, when asked about the quality of grammar explanations, some of the participants who indicated that explanations are clear also mentioned that they would not understand the grammar explanations without the further clarifications from the instructors. Since the course instructors have been giving further explanations, students may have not had a true sense for whether the grammar explanations of the book are clear enough. They may have indicated that the concept was clear because they understood, but they could have been taking into account their teachers’ supplementary explanations. Another point of contrast is in reference to participants suggestions about the course materials, where seven respondents said the books are just fine and would not add anything in addition; however, as mentioned before in the section about clear dominance of negative opinions, all students provided areas of improvement and commented on what could be added to the books per se. Since there were seven participants who said that books were just fine as they were, the researchers expected to find these seven people with no comments on improvements for the book. Surprisingly, that was not the case. There was a clear dominance on the negative opinion that the books have room for improvement —100%. Finally, there was a clear dominance of opinions that readings are interesting, up to date, and relevant; however, half of the participants mentioned that readings do not motivate them to learn. It was surprising that half of the students are not motivated to learn by the readings when the majority (80-90%) described readings with the positive adjectives mentioned. Those seem to be
characteristics enough for readings to motivate learning.
Student Interviews: When able to express their ideas face to face with the co-researcher, participants seemed free to show their discontent toward two aspects in particular: low difficulty level of the listening exercises compared to the audios used in the exams and little grammar practice. Statements like “it is unfair,” and “that’s probably why I didn’t do well in the exam” showed discontent from their part and also validated past findings. The researchers agree that triangulating the survey results with this follow-up interview helped validate the attitudes perceived on some aspects and gather extra details where these five participants had not been clear enough. Unfortunately, interviewing every respondent was out of scope due to timing and academic limitations. However, the researchers see the benefit of giving participants the opportunity to develop their ideas further, sometimes limited by the survey form or the requirement of typing.
Instructor Interviews: Overall, instructor perspectives were not as supportive of the new books as one might expect. Although the instructors explicitly chose Skillful 2 over Q: Skills when asked which they prefer, all three indicated that some issues from Q: Skills still occur in Skillful 2 (clear negative dominance): inauthentic audios, writing tasks that do not match the composition expectations of the course, and a simplistic online platform. Other drawbacks of the new books included repetitive vocabulary exercises and simple video listening comprehension exercises. Skillful 2 did present some improvements from Q: Skills, such as relevant topics, challenging readings, and authentic videos. When asked about the implementation of a supplemental grammar booklet, all instructors supported a booklet containing extra exercises, but no dominance of opinion occurred as far as the necessity of a booklet for extra explanations. Two instructors also mentioned the benefits of a supplemental listening booklet for the course.
Instructors’ opinions coincided with their students’ on the following specific aspects: inauthentic audios, insufficient listening comprehension exercises, and a need for more grammar exercises. It is curious to note that instructors perceived readings as challenging, but students did not share the same opinion. In the end, the researchers wonder whether instructors would agree to a third option beyond Skillful 2 if given the choice. In other words, if asked to choose between Skillful 2 and a third, to-be-determined book, would instructors choose the latter? Further investigation would need to be conducted.
Conclusions and Recommendations
We believe that the findings of this study indicate mixed results as to whether LM-1002 students have positive or negative attitudes towards the Skillful 2 books as a whole. There are aspects of the books for which a solid majority of the respondents expressed positive attitudes, and there were other sections for which negative attitudes were reported by more than 80% of the participants. However, for several aspects, due to the percentages that moderate dominance represents, we cannot say they are significant enough as to claim that these are the general attitudes of the participants.
In general, teacher attitudes towards Skillful 2 are moderately positive, and they approve of the change of books implemented by the First-year English department; however, teachers are aware that more challenging audios and additional grammar exercises are needed.
Following the opinion of Kalmus (2004), Knecht & Najvarová (2010), Roberts (1996), and Wright (1990), we believe that educators, researchers, and publishers should take into account users’ perspectives toward teaching materials. Consistent with previous research, the findings from this case study reflect that students are capable of critically analyzing a textbook. Additionally, their input provided valuable insights from a main user perspective.
We encourage the First-year English department to review the results of this study and follow up on the aspects that do not align with the expectations set when they decided to acquire Skillful 2 as a substitute for Q: Skills 3 for LM-1002. The numbers and percentages, opinions, examples, and researchers’ discussion provided in this study can definitely useful data for the department to contrast against the initial evaluation and expectations set for the materials selected for the pilot project, as well as for the decision to discontinue the use of an additional grammar booklet.
Problems that arise in light of these findings include the actions needed by the language instructors (or by the department as decision makers for the LM-1002 material) to palliate the deficient aspects of Skillful 2 that
participants identified.
Based on major findings in this study, we provide the following suggestions to the First-year English department and any EFL institutions that might use Skillful 2:
Instructors support the use of teacher-created ancillary grammar booklets, like the ones used in past semesters. We cannot conclude if students fully support the use of a supplemental grammar booklet with the data obtained. While more than half of the student participants indicated a lack of grammar exercises in the new materials, very few explicitly suggested the addition of a grammar booklet when given the opportunity to recommend any ancillary materials. Although an overwhelming majority of students did not suggest the use of a grammar booklet, it would be interesting to see if further research provides data on whether or not this type of material improves the students’ performance. We propose further experimental research that compares written exam scores and grammatical performance in oral exams between a control group (no additional grammar exercises), experimental group #1 (additional exercises provided when requested), and experimental group
#2 (additional exercises provided within a teacher-created ancillary grammar booklet). The data gathered from such research may provide further guidance in terms of the effectiveness and necessity of a teacher-created ancillary
grammar booklet.
Bibliography
Alshehri, H.K. (2018). Evaluating Q: Skills for Success from students’ and teachers’ perspectives at the English Language Institute Yanbu (YELI). Journal of Studies in Education, 8(4). http://doi:10.5296/jse.v8i4.13737
Ansary, H., & Babaii, E. (2002). Universal characteristics of EFL/ESL textbooks: A step towards
systematic textbook evaluation. The Internet TESL Journal, 8(2), 1-9.
Daneshfar, S., Abdollahi, J. (2018). Textbook evaluation: A case study of Iranian teacher and student perspectives. International Journal of English Literature and Social Sciences, 3(3), 450-456.
Dörnyei, Z. (2011). Research Methods in Applied Linguistics. Oxford
University Press.
El-Dakhs, D. (2011). How to choose your EFL textbook? Some useful guidelines. International Journal of the Book, 8(1), 1-10.
Garinger, D. (2002). Textbook selection for the ESL classroom. Center for Applied Linguistics Digest, 2(10).
Kalmus, V. (2004). What do pupils and textbooks do with each other? Methodological problems of research on socialization through educational media. Journal of Curriculum Studies, 4(36), 469-485.
Lehr, F. (1979). ERIC/RCS: Textbook evaluation. The Reading Teacher, 32(7), 886-890.
Litz, D. R. (2005). Textbook evaluation and ELT management: A South Korean case study. Asian EFL journal, 48(1), 1-53.
Mohammadi, S.M. (2013). A chronological comparison of unit frameworks of EFL textbooks; how have units changed over time? Iranian EFL Journal, 9(6), 405-415.
Knecht, P., & Najvarová, V. (2010). How do students rate textbooks? A review of research and ongoing challenges for textbook research and textbook production. Journal of Educational Media, Memory & Society, 2(1), 1-16.
Korpela, N. (2007). If I were a textbook writer: Views of EFL textbooks held by Finnish comprehensive school
pupils. Retrieved from https://jyx.jyu.fi/bitstream/handle/123456789/7321/1/URN_NBN_fi_jyu-2007440.pdf
Quirós, M.R. (2018). A comparative evaluation of two English textbooks: An analysis of teaching evolution through time. Retrieved from https://repositori.upf.edu/
Rahimi, M., & Hassani, M. (2012). Attitude towards EFL textbooks as a predictor of attitude towards learning English as a foreign language. Procedia-Social and Behavioral Sciences, 31, 66-72. Retrieved from https://www.sciencedirect.com/science/article/pii/S1877042811029478
Riazi, A.M. (2017). Mixed Methods Research in Language Teaching and Learning. Sheffield: Equinox Publishing Ltd.
Roberts, J.T. (1996). Demystifying materials evaluation. System, 24(3), 375-389.
Sheldon, L. E. (1988). Evaluating ELT textbooks and materials. ELT Journal, 42(4),237-246. http://doi=10.1.1.473.7638
Skierso, A. (1991). Textbook selection and evaluation. Teaching English as a second or foreign language, 2, 432-453.
Viales Angulo, J.E. & Carmona Miranda, A. G. (2014). Evaluating a textbook’s efficiency to enhance students’ English proficiency in a Costa Rica’s [sic.] school. Revista de Lenguas Modernas, 20, 415-416.
Woollerton, M. (2016). Taking EFL textbooks digital – what, why and how? Chuo University Bulletin of Humanities, 83, 313-351.
Wright, D. (1990). The role of pupils in textbook evaluation. Internationale Schulbuchfor-schung, 12(4), 445-454.
Appendix A
Note for readers: This survey was administered digitally through Google Forms. When asked “Why? Please give examples,” and “Why? Please explain,” the students could type long responses into a box provided in the digital survey. Here, said space is indicated with one line. Likewise, for all open-ended questions, students could type the “paragraph response” length in Google Forms.]
Note for participants: Whenever possible, please add specific examples from the books to support your answers, including the name of a specific section of the book, page numbers, etc.
Code Name: __________________________
A. Book Topics
1. Which of the following adjectives accurately describe the book topics?
Please mark an X in the parentheses ( ) of all that apply.
Listening & Speaking book topics:
( ) interesting ( ) boring
( ) up to date ( ) out of date
( ) varied ( ) monotonous
( ) useful ( ) not useful
( ) relevant ( ) irrelevant
( ) other(s) ____________________________________
Reading & Writing book topics:
( ) interesting ( ) boring
( ) up to date ( ) out of date
( ) varied ( ) monotonous
( ) useful ( ) not useful
( ) relevant ( ) irrelevant
( ) other(s) ____________________________________
Why? Please give examples.
______________________________________________________________________
2. Are the topics adequate considering the level of English that is expected upon finishing the course? Please mark one answer: - Yes ( ) - No ( )
Why? Please give examples.
______________________________________________________________________
B. Exercises
1. Are the instructions for the exercises and activities clear and easy to understand? Please mark one answer for each book.
Listening & Speaking: - Yes ( ) - No ( )
Reading & Writing: - Yes ( ) - No ( )
2. Please rate the following exercises by marking one of the three difficulty levels with an X.
Easy Adequat Difficult
Grammar exercises ( ) ( ) ( )
Listening exercises ( ) ( ) ( )
Reading exercises ( ) ( ) ( )
C. Speaking
1. Complete the statement by marking all the adjectives that apply: “The discussion questions in the section Critical Thinking…”
Listening & Speaking:
( ) are challenging
( ) make me want to participate
( ) motivate me to learn more
( ) are relevant
( ) other(s): ____________________________________
Reading & Writing:
( ) are challenging
( ) make me want to participate
( ) motivate me to learn more
( ) are relevant
( ) other(s): ____________________________________
Why? Please give examples.
______________________________________________________________________
2. Complete the statement by marking all that apply: “There are enough _________ exercises.”
( ) Conversation
( ) Pronunciation
( ) Group discussion
( ) Individual presentation
3. Has the section Pronunciation for Speaking helped you improve your pronunciation? Please mark one answer: - Yes ( ) - No ( )
Why? Please explain.
______________________________________________________________________
D. Grammar
1. Are the grammar explanations clear? Please mark one answer: - Yes ( ) - No ( )
2. Are there enough grammar exercises? Please mark one answer: - Yes ( ) - No ( )
E. Writing
Which of the following sections has/have helped you improve your writing? Please mark all that apply.
( ) Writing Skill
( ) Writing Model
( ) Writing Task
( ) None
Why? Please explain.
______________________________________________________________________
F. Listening
1. Which of the following exercises have helped you develop your listening abilities? Please mark all that apply.
( ) Close exercises
( ) Global exercises
( ) Unit Video
( ) Pronunciation for speaking
( ) None
2. Complete the statement by marking all that apply: “The audios…”
( ) are boring
( ) are challenging
( ) make me want to participate
( ) motivate me to learn more
( ) are relevant
( ) other(s): ____________________________________
Why? Please give examples.
______________________________________________________________________
G. Reading
1. Complete the statement by marking all that apply: “The readings…”
( ) are boring
( ) are challenging
( ) are out of date
( ) make me want to participate
( ) motivate me to learn more
( ) are irrelevant
( ) other(s): ____________________________________
Why? Please explain.
______________________________________________________________________
2. If you have liked the readings, what do you like the most about them?
______________________________________________________________________
H. Vocabulary
Have the Vocabulary sections helped you improve your English? Please mark one answer: - Yes ( ) - No ( )
Why? Please explain.
______________________________________________________________________
I. Videos
1. Complete the statement by marking all that apply: “The videos from the books...”
( ) are interesting
( ) have out-of-date content
( ) are useful for improving your English
Why? Please explain.
______________________________________________________________________
2. If you have liked the videos, what do you like the most about them?
______________________________________________________________________
J. Study Skills
1. Does the Study Skills section help you improve your study abilities? Yes ( ) No ( )
Why? Please explain.
______________________________________________________________________
2. Does this section help you learn English? Yes ( ) No ( )
Why? Please explain.
______________________________________________________________________
K. Online Platform
1. Does the online platform help you learn English? Yes ( ) No ( )
Why? Please explain.
______________________________________________________________________
2. How often do you use the online platform?
______________________________________________________________________
3. Please complete the sentence with your opinions: “Using the online platform is _____.”
______________________________________________________________________
L. Personal Considerations
1. What do you like about the Skillful 2 textbook?
______________________________________________________________________
2. What suggestions would you give regarding the materials used in LM-1002?
______________________________________________________________________
Appendix B
University of Costa Rica
Master’s Program in Teaching English as a Foreign Language
PF-0312 Classroom Research
Semi-Structured Interview for Students
Pereira, Vladimir – Peterson, Kelsey
The purpose of this interview is to gather more information from two groups of students (1) students who gave some information but who the researchers want to hear more from, and (2)students taking the LM1002 course again, who could provide a particularly unique perspective having used both QSkills 3 and Skillful 2 books).
Possible questions for students whom the researchers are interested in hearing more from:
The interviewer can use additional questions not presented here in order to extract more specific information from the interviewee and clarify that the
interviewer’s understanding is accurate.
Appendix C
University of Costa Rica
Master’s Program in Teaching English as a Foreign Language
PF-0312 Classroom Research
Semi-Structured Interview for Professors
Pereira, Vladimir – Peterson, Kelsey
Regarding the QSkills books (level 2 for LM1001 and level 3 for LM1002), which have you used in the past?
In general, what did you like about the QSkills Listening & Speaking books?
What did you dislike about these books?
What did you like about the QSkills Reading & Writing books?
What did you dislike about the Reading & Writing books?
What did you like about the QSkills online platform?
What did you dislike?
In terms of the Skillful 2 books being used currently, what do you like about the Listening & Speaking book?
What do you dislike about it?
What do you like about the Skillful 2 Reading & Writing book?
What do you dislike about it?
What do you think of the online platform?
Would you support or not support using a supplemental grammar booklet in LM1002 to use with Skillful 2? Why?
What’s your opinion about the change of books for this semester based on your teaching experience?
Recepción: 27-03-20 Aceptación: 24-08-20