16_zuniga,_vargas

Assessing the Quality of Paraphrases Written by Third Year Costa Rican EFL Students

Evaluación de la calidad de las paráfrasis escritas por estudiantes costarricenses de tercer año de EFL

Randolph Zúñiga Coudin

Escuela de Lenguas Modernas

Universidad de Costa Rica

José Miguel Vargas Vásquez

Sede de Occidente

Universidad de Costa Rica

Abstract

The aim of this study was to assess factors affecting the quality of paraphrases written by college level EFL learners. A three-dimensional paraphrasing competence model by McCarthy, Guess, and McNamara (2009) was followed to assess the texts for semantic completeness, lexical difference, and syntactic difference. The selected method was a mixed method approach. The quantitative analysis was used to measure the rate of success of each paraphrasing competence whereas the qualitative analysis aimed to describe the perceived difficulty of the competences and the reasons behind it. The subjects were 17 EFL students enrolled in a third year English composition course at the University of Costa Rica. The corpus of the study consisted of 51 paraphrases. Following three rounds of examinations and the analysis of students’ perceptions, the lexical dimension emerged as the most challenging area interfering with paraphrase quality, with semantic completeness also emerging as difficult. Other factors affecting paraphrase quality were inappropriate use of passive voice, style and register, and inaccurate lexical substitutions. The study concluded with recommendations to improve instructional cycles by reinforcing reading comprehension and lexical development, and by assessing the different dimensions of paraphrasing separately.

Keywords: paraphrasing, academic writing, english as a foreign language, higher education

Resumen

El objetivo del estudio fue evaluar los factores que afectan la calidad de textos parafraseados por estudiantes universitarios de Inglés como Lengua Extranjera. Se utilizó el modelo de competencias tridimensional de McCarthy, Guess y McNamara (2009) para evaluar los textos según la integridad semántica y la diferencia léxica y sintáctica. Se eligió un enfoque de método mixto. El análisis cuantitativo fue utilizado para medir la tasa de éxito de cada competencia de parafraseo mientras que el cualitativo buscó describir la percepción de dificultad de las competencias y las razones subyacentes. Los sujetos fueron 17 estudiantes de ILE en un curso de composición inglesa de tercer año en la UCR. 51 textos parafraseados conformaron el corpus. Después de revisiones y del análisis de las percepciones del estudiantado, la dimensión léxica mostró mayor interferencia con la calidad del parafraseo. La integridad semántica también resultó ser difícil. Otros factores que afectaron la calidad de los textos fueron el uso inapropiado de la voz pasiva, el estilo y el registro, y las sustituciones léxicas inadecuadas. El estudio concluyó con recomendaciones para mejorar la enseñanza por medio del refuerzo de la comprensión de lectura, del desarrollo léxico y de la evaluación separada de las diferentes dimensiones.

Palabras clave: parafraseo, escritura académica, inglés como lengua extranjera,
educación superior

Introduction

Paraphrasing is a common requirement in a variety of written academic assignments; however, much of it receives less attention than it deserves. Students often approach the process as mere reproduction, making a few changes to the text and approaching the task as if acknowledging the author by itself were sufficient to comply with citation requirements and to avoid plagiarism. Likewise, instructors may not recognize the multiplicity of cognitive and linguistic operations involved in paraphrasing and carry out the assessment based on an overall aesthetic impression without using a clearly defined list of criteria that accurately reflects the scope of the operations performed (Uemlianin, 2000, p. 357). This problem can reinforce the students’ lack of confidence for performing paraphrasing tasks independently, especially at university-level language courses where the stakes are high.

Several authors have provided criteria to guide the production and the assessment of paraphrases. For example, McCarthy, Guess, and McNamara (2009), have proposed a basic framework for assessing paraphrases recognizing the need for significant lexical and syntactic changes that still result in a text with equivalent semantic content. Additionally, Yamata (as cited in Shi, 2012) asserts that it is important for the resulting text to show evidence of higher-order thinking reflecting the premises in the original passage and how they are related, all the while carefully integrating them with the writer’s own voice and line of thought (p.135). While this combination of cognitive
processes seems challenging enough, the researchers’ experience suggests that successful paraphrases also call for discourse-based considerations such as the use of neutral vs. biased expressions, appropriate style and register, and preferably, the use of concise and concrete language.

In the foreign language learning context, there is the added difficulty of the language gap, where the source material may seem too accomplished for students to attempt changes that, to them, would only yield a poorer version of the original text. In this light, studies in other foreign language contexts such as those of Iran, Malaysia, and Saudi Arabia have looked into problems in academic writing and have identified paraphrasing as a common denominator in students’ perceived challenges (Abdulkareem, 2013; Al Fadda, 2012; Setoodeh, 2015; Singh, 2015).

In the Costa Rican English as a foreign language context, no studies have looked into this problem so far, nor has there been an examination of paraphrase assessment using a framework of clearly defined guidelines corresponding to the cognitive and linguistic dimensions of the task. At the University of Costa Rica in particular, the skill of paraphrasing is formally taught in the B.A. in English and English Teaching programs in the course LM-1352, English Rhetoric I, in the first semester of the third year. The researchers undertook this study with the aim of developing a pedagogical strategy for paraphrase assessment based on the analysis of a corpus of student-generated paraphrases and of the students’ perceptions of difficulties involved in the process.

Review of literature

Academic writing and the strategy of paraphrasing. In second language learning, academic writing is both one of the most necessary and the most demanding competences that students, particularly at the college level or in graduate school, need to acquire. The expertise gained in this skill and the students’ ability to produce their own texts according to the prevalent standards will influence their ability to complete their degree programs successfully and to gain acceptance by the larger scientific community into which they will emerge.

Institutions of higher learning typically offer their students websites with guidelines for academic writing including characterizations of what academic writing means. The University of Manchester (2016) in the U.K., for example, lists in its Skills Website, aimed at helping students cope with their academic assignments, a series of characteristics and requirements of the skill. These include consistency, formality, precision, conciseness, objectivity, and appropriate referencing of source material. Another important aspect is that audience and purpose determine the specific style required of a text. These considerations point to the need for teaching students the characteristics of the particular genres that will allow them to participate in their respective communities and the necessary accompanying strategies to achieve such a goal. This idea is paralleled by Abdulkareem (2013), who asserts that familiarity with the different types of academic texts and with the strategies necessary to create them is expected of university students. Among these strategies, critical thinking and particularly, paraphrasing, emerge as key abilities for students to develop in their formative years (p. 1553).

Widely accepted definitions of paraphrasing have in common the notion of rewording a sentence so that its lexical and syntactic components vary from the original, but so that its semantic quality remains the same or very close to the original, in effect transmitting the same information intended by the author
(McCarthy, Guess, and McNamara, 2009, p. 682). It follows that the making of a paraphrase requires strategies oriented to achieving appropriate lexical substitutions and grammatical changes that respect the original meaning.

This definition is complemented by authors who claim that there is more to appropriate paraphrasing than lexical and structural changes. For example, Yamata (as cited by Shi, 2012) has called for inferential thinking to be considered an element of quality paraphrasing, oriented towards deducing, which uses given information to reach a conclusion, or towards establishing analogies, which identifies similarities between ideas. This additional requirement makes the production of quality paraphrases a degree more difficult than the more basic definition. It would imply instruction that went beyond merely linguistic command of the text and also demanded training in critical thinking.

The level of difficulty involved in the skill is reflected in the large number of students who reportedly fail to paraphrase appropriately, showing poor processing of the information in the source material and engaging in levels of copying exceeding what instructors consider acceptable. In addition to this, differences in the expected
discourse practices of academic communities across cultures and the varying expectations of instructors can make the achievement of successful paraphrasing a more complex task (Angelova and Riazantsewa, as cited in Singh, 2015).

Other problems, identified by Al Fadda (2012) are that students in an English as a foreign or second language context tend to have insufficient knowledge of the practices associated with academic writing and insufficient experience with them, while at the same time, they still struggle with the learning of the language (p. 125). Similarly, Setootdeh (2015) reports that poor language proficiency makes students more strongly dependent on the original wording of a text. This may be linked to their concern of not being able to achieve a similar level of sophistication in their expression, which may result in loss of face before their instructors and before the larger academic community.

The previous discussion points to the need for a separate examination of the components of a quality paraphrase and of the difficulty they pose to students, particularly to non-native speakers of English in a second or foreign language context.

Elements of paraphrase evaluation

As previously discussed, paraphrasing goes beyond the mere redistribution and substitution of a sentence’s elements to make it appear different from the original text in order to avoid plagiarism, as students of writing may initially tend to regard the skill. Vila, Bertran, Martí, and Rodríguez (2015) claim that paraphrases ínvolve multiple and diverse linguistic manifestations. A great variety of linguistic operations give rise to paraphrases, and a single paraphrase may include multiple combined paraphrase phenomena; hence, determining the scope of each phenomenon is not an easy task. These linguistic manifestations can be grouped into three clearly established categories. According to McCarthy, Guess, and McNamara (2009), the three elements are semantic completeness, lexical difference, and syntactic difference. In the following paragraphs, a brief description of these elements will be given.

Semantic completeness. This refers to the degree to which a student’s paraphrase has the same meaning as the sentence targeted for paraphrasing. Semantic completeness is evaluated without regard to word or structural overlap between sentences. The completeness aspect refers to the possibility that the meaning of the paraphrase only targets a part of the original sentence. In this regard, Uemlianin (2000) claims that “failure to paraphrase is not random, but seems to be systematic. […] A high proportion of paraphrases can omit or misrepresent a central point of the source text, and can weakly represent the passage’s central theme” (p. 357). For evaluation purposes, this means that instructors should look closely at the degree of representation of the original material in the paraphrase.

Lexical similarity. This aspect refers to the degree to which the same words have been employed in the response, regardless of syntax and semantics. An exception to the requirement for lexical substitutions can be made in the case of what the University of Wisconsin’s Writer’s Handbook (2014) describes as the “shared language” of a discipline. This consists of phrases commonly used by professional communities to designate the specialized jobs, tools, and tasks of their discipline, among others, in an efficient and very precise manner whose meaning is understood by all their members. Paraphrasing these would result in wordy and less precise alternatives. According to the site, shared language extends beyond technical terms to include bias-free language and conventional designations. The exception implies that the correct identification and inclusion of shared language in a paraphrased text is another element of its quality.

Syntactic similarity. This refers to the degree to which similar syntax, i.e. parts of speech and phrase structures, is employed in the user response, regardless of the words used. The quality of paraphrases will depend on the students’ capacity to integrate all these aspects in a paraphrase of a given text. This implies that grammar accuracy is an important element for the evaluation of a paraphrase that has introduced significant structural modifications; its success will be closely linked to the accuracy of the grammar employed to construct the new sentence.

There needs to be an integration of the four elements previously discussed at a deep level so that the product can be considered an advanced (high-quality) paraphrase rendering the original meaning with a novel structure and displaying a thorough understanding of the logical relationships present therein. According to Setoodeh (2015), failure to do so would lead to the result being classified on a scale ranging from “near copy” to “moderate revision”.

Strategies for writing paraphrases. The previous sections have made evident that the creation of quality paraphrases requires the careful integration of structural and lexical modifications resulting in the preservation of the original meaning of a text. Authors such as Leki (1998) and Swales and Feak (2012) have studied these modifications in order to devise practical strategies for application in writing classrooms. The strategies presented below are a compilation of the authors’ work in the field of paraphrasing.

Changing the word class of content words: The reformulation of the original sentence often includes changes in the part of speech of key words. This implies that nouns, verbs, adjectives, and adverbs adopt a new word class in the paraphrased version of a text. Changes in the structure of the sentence necessarily accompany these changes as different word classes will require different positions. Additionally, the changes often include the combination or division of sentences, resulting in longer or shorter sentences respectively.

Finding appropriate synonyms or equivalent expressions: The wording of a paraphrase should modify the original in such a way that there is only a minimal trace or no trace at all of the voice in the source material. The original wording of the author should not be adopted, for a very similar representation of a text will be taken as plagiarism. Paraphrases should reflect a thorough processing of the source material in the voice of a new author since the use of pieces of the original wording mixed with the new voice may lead to discordances in the new author’s flow of ideas. Thorough processing should allow a smooth integration of the paraphrase into the new author’s discourse.

Identifying technical terms that should not be substituted: An expression that is viewed as particularly creative, original, or technical, and for which there does not seem to be a more efficient way to say it, should be kept the same, but enclosed in quotation marks. According to the University of Wisconsin (2014), if the expression is so common in a field that any member would commonly use it, then the choice of words does not reflect the particular style of an author, nor does it require the use of quotation marks.

Changing the order of the information in the sentence: The order of the clauses in a sentence is often rearranged and the grammar adjusted to prevent high similarity between the source material and the paraphrased version. The writer of the paraphrase should pay attention to important linking phrases and connectors when rearranging the clauses to ensure that the reformulation accurately reflects the relationship between ideas.

Changing the structure of the sentence: Further syntactic changes involve changing the voice from active to passive or vice-versa between the source material and the paraphrase. Another strategy, proposed by Zemach and Stafford-Yilmaz, (2009) and not aimed specifically at paraphrasing, but generally useful for achieving variety in structures and thus, also applicable to writing paraphrases, consists of substituting modal expressions with adjectives for the corresponding modal verbs or vice-versa. An example of this strategy is changing adjectives such as crucial, important, or possible, to modal verbs such as must, should, and could, respectively.

Maintaining the content of the original: The meaning of the source material should correspond to the meaning of the paraphrased version. A common problem in paraphrasing is that students fail to understand important information from the original text and write an incomplete version of the source material. Lack of understanding of the source material leads to stressing unimportant information from the original in the paraphrased version instead of simply omitting it.

Final considerations deal with the need to be careful when selecting synonyms given that not all substituted words mean exactly the same, and also when changing the order of words given that this can change the meaning of the sentence. As for style, it should be different from the original without risking an alteration in the meaning of the text. Lastly, the authors of this article consider precision, the use of clear and concise sentences, a determining factor for writing quality paraphrases. Achieving a better understanding of how the individual modifications interact and contribute to overall paraphrase quality would facilitate the teaching of paraphrasing in academic writing courses.

Methodology

The method selected for this study was a mixed method approach. A sequential-explanatory design was used to conduct the study. According to Cresswell (2014), this design entails an initial quantitative stage followed by a qualitative stage with the aim of attempting to explain the numerical results obtained. In this case, the quantitative analysis focused on measuring the rate of success of each paraphrasing competence whereas the qualitative analysis aimed to describe the participants’ perceived difficulty of the competences as well as the reasons behind it.

Setting and Subjects

This study was carried out in a Rhetoric I course at the School of Modern Languages at the University of Costa Rica. The main objective of this course is to produce well organized and coherent academic writing samples ranging from summaries to short expository and response essays, often requiring the use of source material. Therefore, paraphrasing is an essential skill for success in this course and as such is included in the lists of topics taught to students. The course was offered three hours per week for third year students in the B.A. in English or Teaching English. The data collected consisted of 51 paraphrases produced by 17 students in the course. The students were 5 men and 12 women. Their ages ranged between 19 and 47 years old.

Instruments

Rubric for assessing paraphrase quality (Appendix 1). The goal of this instrument was to assess the degree of achievement in each dimension of the paraphrases. It consisted of an analytic scale with four dimensions of assessment: lexical substitutions, syntactic
transformations, semantic completeness,
and direct, concise style. The scale was developed for the purpose of the study based on McCarthy, Guess, and McNamara’s (2009) three-dimensional model of paraphrase evaluation and including paraphrasing strategies proposed by a variety of authors (Leki, 1998; Swales and Feak, 2012; University of Wisconsin, 2014; Zemach and Stafford-Yilmaz, 2009). The instrument was used by the researchers to rate the quality of each of the texts in the corpus after the paraphrasing assignment was completed.

Survey for rating the difficulty of achieving paraphrase requirements
(Appendix 2) This instrument aimed at collecting the students’ perceptions of the difficulty of applying paraphrasing strategies and fulfilling paraphrase requirements. Like the assessment rubric, the survey was created for the purpose of the study. It included a list of competences for students to rate from not difficult at all to very difficult using a Likert scale from one to five respectively. The competences were the same as those included in the assessment rubric and based on the same authors. The Likert scale was followed by an open ended section in which students could explain their ratings for strategies and paraphrase requirements with scores of three or higher, corresponding to difficult or very difficult.

Procedures

The students paraphrased short excerpts from a variety of sources provided by the researchers. This assignment was part of a cycle of explicit instruction
in paraphrasing skills. In the first part of the cycle, the three-dimensional model used for the instruments and the corresponding paraphrasing strategies were taught to the students. The rubric for assessing paraphrase quality was introduced as part of the instruction cycle; the students completed a first round of practice paraphrases with the purpose of receiving formative feedback on their production. This served to validate the scale as well. In the second part of the cycle, the group of students was formally evaluated on their production. To this end, each student wrote three paraphrases, which were examined for lexical and syntactic similarity to the original as well as for semantic completeness by using the previously introduced assessment rubric.

To ensure reliability in the rating process, the texts were examined separately by the researchers and given a rating, then compared to achieve a consensus where necessary. To be rated as passing in each dimension, a text had to achieve a minimum score of seven according to the rubric. In the first stage of the evaluation sequence, texts yielding a high degree of lexical similarity were rated as not-paraphrases, or plagiarism-quality responses, and discarded. In the second stage, the remaining texts were examined for syntactic similarity. Once again, texts not meeting the standards for a paraphrase were separated from the corpus. At this point, the remaining texts were rated for semantic completeness, and those passing this criterion were further examined for the presence of other features interfering with quality.

In the qualitative stage of the study, the researcher in charge of the group of students asked them about their
perceptions of the skills involved in paraphrasing and their perceived difficulties after the first practice round. The feedback from the students led to the need for an open ended section in the survey in order to understand the challenges faced when writing the paraphrases in the assignment. This corresponded to the explanatory component of the research design. For the study, the results of the quantitative analysis were compared with the students’ responses to this survey about their perceived difficulties in producing appropriate paraphrases.

Results

Two aspects were examined to address the research inquiry into the quality of paraphrases written by 3rd-year Costa Rican EFL university students: the rate of success in each of the dimensions of the examination and the students’ perceptions of the difficulty of paraphrasing strategies and paraphrase requirements. The results for these two aspects were compared with the purpose of identifying similarities or differences between performance and perception.

Figure 1. Number of paraphrases passing each of the three examinations. Source: Assessment of texts written by students in response to course assignment according to type of examination, September 2016.

Figure one shows the number of passing texts obtained out of the total number of texts after each examination, leading to the final number of 19 texts rated as successful paraphrases out of the original 51. As can be seen, the lexical transformation yielded the highest number of failing texts out of the total. The most common reason for failing was the reproduction of too many words from the original, followed by inaccurate lexical substitutions. Texts that achieved successful substitutions were found to have a higher rate of success with the syntactic transformation. The majority of cases failing in the syntactic dimension did so because they copied three or more structures from the original, notably patterns of subordination. As for the last dimension, problems with semantic completeness, that is, missing information from the original, affected nearly one third of the texts with adequate lexis and syntax. Overall, successful lexical substitutions seem to be a good indicator of syntactic command, but syntactic command does not seem to guarantee semantic completeness, which is dependent on the understanding of all the content in the original text. This suggests that attention must be given to comprehension of all the parts of the original as the starting point of the process before engaging in the exploration of alternative lexis and grammatical forms.

A follow-up analysis of the 19 texts rated as passing paraphrases identified a series of features affecting their quality beyond the achievement of semantic completeness. The most frequent was overuse of the passive voice. While the active-to-passive transformation is one of the strategies suggested by instructional materials for paraphrasing, the use of the passive voice is not always deemed appropriate as it can cause wordiness or indirectness by deemphasizing a central piece of information in the original, and thus, altering the intended meaning of a passage.

Another frequent feature was inconsistent register, leading to paraphrases reproducing the complete original content, but evidencing a gap in the level of formality. This was often linked to faulty lexical parallelism, where avoidance of repetition was manifested in intended synonyms with different levels of formality and appropriateness such as “lady” and “girl” for “woman”, for example. These particular choices have different connotations, are not fully accurate equivalents of the word “woman”, and can affect the meaning of the paraphrase to some degree. Such examples point to the need for including strategies for cohesion and coherence that value the careful repetition of keywords and stress the importance of avoiding confusing or inappropriate synonyms.

One implication of these results is that a formula such as the transformation of active sentences to passive should not be applied indiscriminately, but only after careful consideration of the strength of the resulting passive sentence. Another implication is that strategies for achieving cohesion and coherence must be closely integrated into the instruction cycle of paraphrasing with the purpose of preventing practices that detract from the clarity of the end result in comparison to the original. Such integration can, in turn, contribute to the achievement of a level of formality that closely mirrors that of the source.

Figure 2. Students’ perception of difficulty with dimensions of paraphrasing. Source: Analysis of the students’ responses in the survey about paraphrase requirements, September 2016.

Figure two shows the students’ perception of the difficulty of the different paraphrasing strategies taught in the course. The answers were classified into two main groups: not difficult at all or very difficult and moderately to very difficult. The answers are organized from the easiest to the most challenging. As shown in the figure, finding appropriate synonyms was rated as the most challenging strategy. Among students’ comments, we find: “I need to expand my vocabulary”, “The words in the text seem very accurate and it is difficult to think of others”, or “It is difficult to know if synonyms are accurate”, or “some words were difficult to understand in some contexts”. The ratings and the students’ comments suggest that at their current level, lexical development is not yet sufficient to allow for the retrieval of similar terms to those in the original from their active vocabulary, nor does it allow them to judge the appropriateness of many of the alternatives that they find in dictionaries.

The other two dimensions rated as challenging were the inclusion of complete content and the preservation of original meaning. Students’ comments include: “when rephrasing and changing the order of words, I may miss something that changes the meaning of the author”, “changing the sentence structure can affect meaning”, “working with specific terms makes me change the meaning without noticing”, and “keeping the same meaning is one of the most difficult aspects of paraphrasing”. The students’ comments for these dimensions can still be linked to lexical gaps as the source of difficulties with the reproduction of complete content, even though they refer to competences related to changes in the structure of the text. The fact that lexical gaps and problems with full comprehension appear together in the students’ perceptions as the most difficult aspects of paraphrasing suggests that comprehension is at the heart of paraphrasing tasks and should be the starting point of instruction in the skill and of classroom activities centered on working with source material.

Discussion

Across foreign-language contexts, the skill of paraphrasing is a common denominator of difficulties reported by students of academic writing (Abdulkareem, 2013; Al Fadda, 2012; Setoodeh, 2015; Singh, 2015). The skill poses difficulties to students of academic writing in general (Yamata, as cited in Shi, 2012), in part because of the need for inferential thinking, necessary to make the multiple transformations required for the process. This type of thinking represents a cognitive demand which is demonstrated through higher command of the language. If, in addition, the task is attempted in a foreign language context such as in the present study, two more factors emerge: First, the relative lack of knowledge of the standards of academic writing in English (Al Fadda, 2012), and second, insufficient language proficiency (Setootdeh, 2015).

The standards of academic writing in English include an appropriate formal register for the various associated genres as well as the expectation of clarity and conciseness (University of Wisconsin, 2014). The students in the study have taken two previous courses on academic writing, one focused on the production of paragraphs, and one on the transition from paragraphs to academic essays. This relatively short exposure seems to match Al Fadda’s (2012) scenario of insufficient experience with academic writing in the target language and can also be reflected in the students’ difficulty to achieve a consistent register in their paraphrases. At their level, the students’ still developing language proficiency makes for a much more challenging task as it implies that optimal lexical options may not yet be readily available.

The results of the present study seem to match the conditions described by Setoodeh (2015), namely, that poor language proficiency results in students’ adopting the vocabulary in the original passage because their repertoire still does not include appropriate alternatives to achieve the expected register. For the students in the study, the fact that they often failed at finding synonyms strongly suggests that their productive linguistic skills did not match the level of the texts that they were expected to paraphrase. This supports the proposal for a pedagogical strategy that begins the work of paraphrasing with extensive reading comprehension activities ensuring full understanding of both the content of the text and the nature of the lexical features that establish the level of formality of the text.

The researchers found that the successful lexical transformations required for a quality paraphrase appear to be largely dependent on thorough comprehension of the original text, which ultimately allows for semantic completeness. Although this may seem evident, the researchers believe that through the separate examination of the students’ performance in the three dimensions, it can be concluded with more certainty that the main weakness in paraphrasing lies in poor understanding of the original passage and that this is linked to the students’ level of language proficiency.

A further important result is that proficiency strongly affects the ability to make lexical choices matching the expected level of the original passage. This was evidenced in the students’ own assessment of their difficulties. As they reported, they often felt that the words in the text were the most appropriate alternatives and that making changes was difficult. This can be paralleled by Setoodeh’s (2015) finding that the students’ fear of worsening the quality of the text frequently results in their abandoning the search for other appropriate expressions. To alleviate this lack, the researchers propose that the pedagogical approach to paraphrasing also include vocabulary buildup activities where the levels of formality and the connotations of likely synonyms are made clear.

The higher rate of success of syntactic transformations among texts with successful lexical substitutions is understandable as both dimensions can be seen as components of higher language proficiency. As for the weaknesses identified in this dimension, the majority consisted of the reproduction of subordinate structures. This can be explained through existing deficiencies in the skill of transforming sentences with a variety of transitional devices while maintaining the original logical relationship, an aspect that should also be addressed in the pedagogical approach to paraphrasing.

Conclusions

The final rate of success of the texts in the corpus at achieving paraphrase quality after the three rounds of examinations was rather low, with lexical substitution emerging as the most challenging dimension. The analysis of students’ answers in the qualitative stage also pointed to the lexical dimension posing the highest degree of difficulty in the process.

It appears that the students at their current level do not yet possess a lexical repertoire similar to that of the texts they are expected to paraphrase, and this clearly impacts their ability to produce comparable pieces. Lexical limitations appear to have a ripple effect on the other dimensions of paraphrasing and result, for example, in the difficulty to understand the content of the source completely and to produce equivalent texts, both from a content and from a linguistic point of view. This means that even if students know a word, they may not be aware of other possible uses in specific contexts pertaining to the needs of a particular passage. Consequently, their ability to paraphrase the text is affected.

A relevant conclusion of this study is that the separate analysis of the three dimensions of paraphrasing has proven useful for assessment purposes as they do not necessarily overlap. For the same reason, the three-dimensional framework can also be used as the basis of an approach for the teaching of paraphrasing skills. The results lead the researchers to propose a pedagogical cycle that expands the paraphrasing strategies commonly taught to account for the weaknesses identified.

The proposed model for paraphrase instruction contains five steps. The first step consists of extensive reading comprehension activities aimed at thoroughly understanding the original passages that are to be paraphrased. In the second step, the students use a color code to highlight all the key terms in the original. This not only sets the stage for the subsequent lexical exploration, but also for the students’ having a tool to confirm semantic completeness at the end of the process, assuring that all of the elements pertaining to complete meaning have been covered. In the third step, the students examine the key terms in the original and search for appropriate synonyms matching them both in connotation and in the level of formality. To prepare the students for syntactic transformations, the fourth step focuses on the use of a variety of strategies like exploring word form and changing sentences from active to passive, but with special attention to the use of logical connectors and to alternatives to subordinate structures. Once the students write their paraphrase drafts, they check for semantic completeness in the fifth and final step by following the color coding system that was used in the second step. Given the link between paraphrasing and developing language proficiency, this pedagogical cycle should not be limited to a few weeks of instruction in a single course, but should be revisited in all subsequent composition courses given that the full development of the skill can likely not be achieved without further formal training over a longer period of time.

The study was not carried out without limitations. Due to the structure of the course in which paraphrasing skills are first taught to students, the researchers had limited time to apply the proposed instruction cycle and to collect data. In this sense, a longitudinal study would yield more robust information about the students’ improvement in the skill after a higher number of attempts. In addition, because the study was part of a tightly-scheduled instruction cycle, the conditions for production were very controlled and unlike those of a more authentic paraphrasing task carried out as part of an authentic writing assignment. Lastly, the study was conducted with a group of seventeen students, which limited the size of the corpus. A larger corpus would be required to confirm if the findings are likely trends in a population of the same level. Finally, the study did not look into the impact of different text genres and their associated lexis on paraphrasing success.

For future studies, the researchers recommend an inquiry into the use of the proposed three-dimensional assessment model and the associated instruction cycle with a larger corpus of paraphrases written for authentic tasks, such as reviews of literature or essays of various genres. In addition, future research should be conducted on the evolution of paraphrasing success with increasing proficiency of a group of students, or with different proficiency groups across different years in the English program, with the goal of identifying possible changes in the dimension (lexical, syntactic, or semantic) posing the highest difficulty to students.

Bibliography

Abdulkareem, M. (2013). An investigation study of academic writing problems faced by Arab postgraduate students at Universitu Teknologi Malaysi (UTM). Theory and Practice in Language Studies, 9(3), 1552-1557.

Al Fadda, H. (2012). Difficulties in academic writing: From the perspective of King Saud University postgraduate students. English Language Teaching, 5(3), 123-130.

Creswell, J. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Thousand Oaks, CA: Sage Publications.

Leki, L. (1998). Academic writing: Exploring processes and strategies. England: Cambridge University Press.

McCarthy, P.; Guess, R.; & McNamara, D. (2009). The components of paraphrase evaluations. Behavior Research Methods, 41(3), 682-690.

Setoodeh, K. (2015). The effect of proficiency and task type on the use of paraphrase type in writing among Iranian EFL university students. Practice in Language Studies, 12(5), 2480-2489.

Shi, L. (2012). Rewriting and paraphrasing source texts in language writing. Journal of Second Language Writing, 21, 134-138.

Singh, M. K. (2015). International graduate students’ academic writing practices in Malaysia: Challenges and solutions. Journal of International Students, 5(1), 12-22.

Swales, J.; & Feak, C. (2012). Academic writing for graduate students: Essential tasks and skills. University of Michigan Press: USA.

Uemlianin, I. (2000). Engaging text: Assessing paraphrase and understanding. Studies in Higher Education, 25(3), 347-358.

University of Manchester. (2016). Using English for academic purposes. Retrieved from: http://www.humanities.manchester.ac.uk/studyskills/essentials/writing/academic_english.html

University of Winsconsin. (2014). Successful vs. unsuccessful paraphrases. Retrieved from: http://writing.wisc.edu/Handbook/QPA_paraphrase.html#shared

Vila, M., Bertran, M., Martí, M. A., & Rodríguez, H. (2015). Corpus annotation with paraphrase types: New annotation scheme and inter-annotator agreement measures. Language Resources and Evaluation, 49(1), 77-105.

Zemach, D.; & Stafford-Yilmaz, L. (2008). Writers at work: The essay. New York, USA: Cambridge University Press.

Appendix I. Scoring rubric for paraphrases

UCR-ELM LM-1352 English Rhetoric I

Criteria	Excellent-Very good	Good to average	Fair to poor	Very poor
	1 – 0.9	0.89 – 0.7	0.69 – 0.5	0.49 – 0
Successful lexical substitutions -appropriate synonyms -intact technical terms 1 pt.	All synonyms are appropriate; technical terms are left intact for clarity.	One or two synonyms are inappropriate; technical terms are left intact for clarity.	Several synonyms are inappropriate; technical terms are replaced by less clear expressions.	Errors in lexical changes lead to inaccurate / unclear content. Technical terms are replaced by unclear expressions.
Successful change in sentence structure(s). -successful changes in word class -successful change of the order of information 1 pt.	1 – 0.9	0.89 – 0.7	0.69 – 0.5	0.49 – 0
	Successfully constructs a new grammatical structure reflecting the original content. All changes in word class are correctly executed. Contains only minimal errors, if any, that do not interfere with meaning.	New grammatical structure reflects the original content. Changes in word class are appropriate but not always correctly executed. Contains a few errors, but these do not interfere with meaning.	New grammatical structure does not fully reflect the original content. Changes in word class are inaccurate. Several errors interfere with meaning somewhat.	New grammatical structure does not reflect the original content. Many errors make comprehension difficult.
Successful reporting of complete original content 1 pt.	1 – 0.9	0.89 – 0.7	0.69 – 0.5	0.49 – 0
Successful reporting of complete original content 1 pt.	Clearly includes all of the key content in the original	Includes most of the key content in the original; some may be unclear or missing.	Leaves out key information in the original or adds content not in the original.	Leaves out most of the original content; demonstrates misreading.
Style 1pt	1 – 0.9	0.89 – 0.7	0.69 – 0.5	0.49 – 0
Style 1pt	Writing style is direct, concise, and convincing.	Writing style is mostly direct, concise, and convincing.	Writing style tends towards wordiness and indirectness, and it is only sometimes convincing.	Writing style is wordy and indirect throughout the essay, and fails to convince.

Appendix II. Student survey on paraphrasing

UCR LM-1352 English Rhetoric I

Name: _____________________________________ Age: ______________

Reflect on the paraphrase writing process required for your recent assignment. Rate your own perceived degree of difficulty for the application of paraphrasing strategies and for the fulfillment of paraphrase requirements. 1 means not difficult at all and 5 means very difficult.

Paraphrasing strategy (1-5) or paraphrase requirement (6-8)		1 not difficult at all	2	3	4	5 very difficult
1	Changing the word class of content words
2	Finding appropriate synonyms or equivalent expressions
3	Identifying technical terms that should not be substituted
4	Changing the order of the information in the sentence
5	Changing the structure of the sentence (ex.: active to passive voice, using modal verbs corresponding to adjectives, etc...)
6	Writing clear, concise sentences / expressions
7	Maintaining the meaning of the original
8	Including the complete original content

For all aspects rated 3 or more on the scale, please explain why they were challenging for you. Write your explanation in the space provided.

Aspect	Reason why it was difficult. Please be as specific as possible.
1
2
3
4
5
6
7
8

Are there any other aspects of writing paraphrases that you would like to address? Your answers will be very valuable to help improve the teaching of this skill.