Evaluation of improvement in grammatical and listening comprehension skills
of first year language students taking English as a “minor”
after following two experimental modules
during their first semester in the RomaTre language program.

(A talk given for a group of language teachers, IAL, Rome, 25.5.2006)

Patrick Boylan – University of Rome III –

All incoming students at RomaTre University are given a Foreign Language Placement Exam (run by the Facoltà, not by the language department) to determine their entry level for the language program.  Students who choose to take an English minor, in order to major in some other language, take three “English minor” exams during the three years of the degree course. Since these students are generally certified by the Placement Test at between CEF* Level A1 and A2 upon arriving, language instruction is geared to that level for the first year course, and raised by half a CEF level in each of the following two years. (Half a level is all that can be taught since the official yearly language courses do not last a year, but rather only one semester.)  
*CEF = Common European Framework of reference for language competence.

In 2002 and 2003 I called in an outside language school to run their computerized “TOEFL”-style English Test (called the “L-Test^®”) on the students of my first-year English modules. This test is traditional in nature, focusing on grammar and discrete-point reading+listening comprehension.

I chose this test for 3 reasons:

1. I wanted a test similar to the Facoltà Placement Exam in order to verify that exam's findings.

2. I was curious to see if my students made improvement according to “traditional criteria” (i.e., according to whether their grammar and discrete-point knowledge improved), since my experimental modules do not focus on such things and I do not evaluate them in the final exam;

3. I wanted an exam that did not cost anything, since I had no research funds at the time. I knew the L-Test^® creator and I knew that he wanted to try his new computerized version out on a large number of students before using it at his language school.   Manus manum lavat.

My hypothesis was that, since all my students are regularly classified by the entry test as false beginners (A1 to A2, some B1s), they must have latent grammatical and discrete-point knowledge that a holistic style of teaching, of the kind I do in my experimental modules, would most likely revitalize.

In other words, according to my hypothesis, the students' grammatical and discrete-point knowledge of English ought to improve noticeably in only one semester, not because specific explanations or exercises were given, but only by being “dynamized” in carrying out the required research tasks.

This proved to be the case – and resoundingly so.

As for the answer to the third question – whether the L-Test^® was valid (for the areas it was designed to measure) and reliable (over a large number of students) – it also was affirmative, and resoundingly so. 

As for the answer to the first question (whether the Facoltà Placement Test was accurate), the issue will be discussed in another report.

DATA

for academic years 2002-03,
                          and  2003-04,
(partial data for 2001-2002)

Academic year 2002-03

Population: 201 first year “English minor” students students (M=37 F=164) taking the first semester modules “Seeing and saying things in English” (October to November, 3 credits) and “Accommodation in intercultural conversations in English” (December to January, 2 credits).

(NOTE: The class was exceptionally large and required an amphitheatre to permit the 201 students to form the work groups required to carry out the various research projects. The number of students assigned per class was thankfully reduced in the following years.)

Total class contact hours: 40

NOTE: In addition to my teaching, students were also supposed to take “practical language classes” taught by lettori at level A2/B1.  Because of conflicting time schedules, however, less than one fourth attended the lettori lessons regularly. Another reason for non attendance was the highly demotivating University policy of certifying only the attainment of level B1 (for speaking/writing) for English minors upon their graduation. This means that even if students brought their level up to B2 or C1, this fact would not be certified on their graduation certificate. Since many English minor students were already at level A2 upon entering the University, they had no incentive to take higher level lettori courses voluntarily. They (wrongly, in my opinion) chose to abandon the lettori courses until their third year, which is the only course needed to bring their level up to the minimum required (B1 – speaking/writing).

(First exam: 29 October 2002, Second Exam: 28 February 2003)
 

Average results:

In addition, during the Academic year 2002-03 I asked students from my 2001-02 first-year (first semester) class to retake the single L-Test^® they had taken a year before, at the end of their semester with me. (I had not yet arranged for an L-Test^® at the beginning of the semester, as I would do in the following years.) 

I wanted to see if their English had continued to improve after their semester in the experimental modules and if progress was as noticeable.

Fifteen students (M=0, F=15) volunteered to take the test. (They were compensated with the bestowal of a certificate by the outside Language School, useful in applying for a job. One of the students' certificates, with her marks, can be seen here.).

At the time of the exam, the students were completing their second semester of their second year.

“One year after” Exam: 30 April 2003 (Exam results from 27 Feb. 2002 also appear for comparison)

Average results:

Academic year 2003-04

Population: 76 first year “English minor” students students (M=11 F=65) taking the first semester modules “Seeing and saying things in English” (October to November, 3 credits) and “Accommodation in intercultural conversations in English” (December to January, 2 credits).

Total class contact hours: 40

NOTE: In addition to my teaching, students were also supposed to take “practical language classes” taught by lettori at level A2/B1.  Three quarters did not. See the note in Academic year 2002-03 for the reasons.

(First exam: 30 October 2003, Second Exam: 26 February 2004)
 

Average results:

Update to the talk (June, 2007)

I said at the beginning that my exams do not focus on grammar or discrete-point skills. They attempt to measure holistic knowledge, in particular (1.) the capacity to grasp and verify what a speaker of English means with a given tone, expression or gesture and to appropriate that expressive mode authentically for personal use; (2.) the capacity to see beyond the common myths and understand what “English” really is and what it means to learn it (students will have to be lifelong learners in our globalized world).