8.1 Procedure
8.2 Administration
8.3 Describing reading performance
8.4 Rationale for the questions
8.5 Example of read aloud transcript
8.6 Findings of the read aloud testing
8.7 Reading aloud: Assessment sheet, School P
In each of classes 3, 4 and 6, the class teachers were asked to select four pupils to read, two girls (one good, one average) and two boys (one good, one average). Only average and above average pupils were tested, since it seemed destructive to risk subjecting weak pupils to what would certainly have been a demoralising experience.
Since the pupils had also done the Word Find test, it was possible to check the teacher's choice against the Word Find results. In over 80% of cases one of the pupils selected occupied one of the three highest scoring ranks for their sex group. It therefore appears that in most cases teachers know who are the most able pupils, whether male or female. However in 68% of cases the weaker pupil in the pair (presumably the "average" one), was also in the top half of the rankings according to Word Find. It therefore seems likely that the Read Aloud test does not adequately feature "average" performance, and this is to be borne in mind in considering the results.
The Read Aloud test consists of five passages based upon the course book "English in Malawi" and graded as follows:
Passage 1, 2 and 3: for year 3
Passage 4: for year 4
Passage 5: for year 6
The type face and size of the test corresponded to their text books. Each passage is followed by questions which are asked immediately after that passage has been read (see Appendix D). Pupils could look at the passages while they answered the questions.
Pupils were tested by the researcher individually. The sessions generally took place in an empty classroom, store room, or office. The session opened with a few simple questions about the pupil's name and age, and continued with some questions about the picture accompanying the first reading text. Pupils began with the first text, and continued to the fifth.
If at any point it appeared that pupils were experiencing severe problems, then the session would terminate at that point. All readings were audio-recorded.
The first passage and questions were intended to be very easy so as to give the pupils confidence, and for the most part proved to be so.
The reader's performance on this test is judged on the basis of accuracy of reading, plus comprehension as judged by the response to the questions.
As far as accuracy of reading is concerned, the reader's accent is disregarded, since the immediate aim is to see whether the pupils can convert the written words into a spoken form accurate enough to be understood. Clearly in the case of second language readers it is dangerous to infer comprehension from reading aloud. Indeed one of the problems in judging the reading is that there are some readers who appear to read reasonably accurately, but with little evidence of understanding as judged by answers to questions, while others may make mistakes in accuracy, but have clearly understood.
Thus a child who reads "Muno is playing with a ball" as "Muno is playing a ball" and answers the question "What is Muno doing?" in the same way has almost certainly understood. What such a testee is doing, however, is "transforming" the text and the answer in terms of the stage of language development that he or she has reached. Although such utterances do not represent standard English, our concern in these tests is not to assess the testees' speaking ability in terms of standard English, but rather their comprehension of written text.
Such deviancies from the text (and from standard English) indicate that the child has almost certainly understood the text, but is demonstrating comprehension through a system which is still at a developmental stage. These deviancies may be compared to an English child who reads "John isn't going" as "John ain't going" where the non-standard pronunciation is in itself evidence that the child has understood, but "translated" as it were, the written sentence into his own variety of English. This type of transformation in reading (which can only result from comprehension), may be contrasted with the case of pupils who read accurately word by word, but do not comprehend what they read because their language development is inadequate. The latter rely entirely on the text for what they say; the former "transform" the text in terms of their own language system. Thus, certain meaning preserving slips and systematic errors may, paradoxically, constitute evidence that the reader has understood the text. Since the aim of this testing was to investigate reading comprehension and not accuracy in terms of standard English, errors which are judged to be developmental are not penalised.
On the other hand a child who reads passage 2 accurately, but answers the question "What does Mary like to eat?" by "They are red" may well not have understood the text. (Of course it is possible that they have not understood the question.) Again, a child who answers the question "Why did Robert go to the hospital?" by reading out "When Robert was a little boy he was very sick, so he went with his mother to the hospital" may or may not have understood the text. Certainly the answer is not appropriate to the discourse. Such answers make clear decisions about the pupils' comprehension rather difficult.
For the above reasons precise objective quantification of comprehension on the basis of reading aloud is difficult. Standard miscue analysis (eg Goodman, 1973) or modified versions (Vincent & de la Mare, 1990) have been developed for native speaker readers. They are inappropriate in the context of Malawi in that they do not readily allow for language learners in a developmental stage.
The questions which follow each passage practice various operations which are considered relevant at this level, and which are fundamental to reading with comprehension. They are:
(i) obtaining information through understanding direct reference made in the text. Thus the first question in passage 3 (Who is Miss Moyo?) requires the student to make direct reference to the relevant sentence in the text (Miss Moyo is a teacher.) and produce the answer. Clearly such direct reference questions may result in testees providing the correct answer through "matching" without necessarily understanding what they are saying.(ii) obtaining information through recovery of an element in the text which is referred to indirectly, typically by means a pronoun, although other parts of speech may be used. An example occurs in question 2 of passage 3 (Where did Miss Moyo put the flowers?). In this case the reader must realise that "them" in "... she put them on her table" refers to "flowers".
(iii) obtaining information through making appropriate inference. Here the reader infers a relationship (that is not explicitly stated) between two parts of a text. Question 3 of passage 3 (Why was Miss Moyo happy?) requires the reader to infer from the text the most likely reason why Miss Moyo was happy (...they gave her some flowers. She was very happy and...). It may be that in some cases appropriate knowledge of the world facilitates the appropriate inference. Indeed there was a suggestion that pupils in Malawi would be unlikely to bring flowers to the teacher.
The three operations (direct reference, indirect reference and inference) were distributed in the 18 questions as follows:
|
Passage 1: |
1, direct reference. |
2, indirect reference. |
|
Passage 2: |
1, direct reference. |
2, indirect reference. |
|
|
3, indirect reference. |
|
|
Passage 3: |
1, direct reference. |
2, indirect reference. |
|
|
3, inference |
|
|
Passage 4: |
1, indirect reference. |
2, indirect reference. |
|
|
3, indirect reference. |
4, indirect reference. |
|
Passage 5: |
1, direct reference. |
2, indirect reference. |
|
|
3, inference. |
4, inference. |
|
|
5, indirect reference. |
6, direct reference. |
Pupils who are unable to obtain the relevant information in answer to a question that involves one of the above operations are unlikely to be reading with comprehension. However, there is a possibility that pupils could indeed understand the text, but were not able to demonstrate comprehension either because:
(i) they could not understand the question. (Here it should be noted that the question was spoken by me, and pupils were also shown the written form of the question.) or because:(ii) they were not able to produce an answer in English (although they understood both text and question).
However, for all 18 questions all the language necessary for the answers is in, the text. It is therefore probable that pupils who cannot answer these questions are not failing simply because of insufficient productive proficiency in English, but rather because they have not understood the text or the question. Nevertheless, the possibility always remains that asking and answering questions in the child's mother tongue might have yielded different results.
To illustrate the issues discussed above, an example of a read aloud transcript is provided below. The transcription is restricted to the reading and to the questions and answers; the introductory, linking and concluding remarks are omitted. The transcript is of a Standard 3 boy in School P who scored 7 out of 60 in Word Find and was ranked 16 out of 34 (ie an "average" pupil. He appears as 16B in the Read Aloud result table in 8.7).
Notes on Transcription Conventions
Miscues which consist of misreadings in the pupils' readings have been underlined.
Miscues which consist of repetions in the pupils' readings are transcribed but not underlined.
R = Researcher
Numbers in brackets { } indicate the mark awarded (either 2, 1, or 0 for the reading or question: see next section, 8.6)
Numbers in round brackets ( ) preceded by Q indicate the question number for the passage
|
School: P |
Standard: 3 |
Pupil: 16B |
Passage 1 |
16B: (reads) Muno is playing with a ball. He is playing under the tree. {2}
R (Q I): What is Muno doing?
16B: Muno is playing a ball. {2}
R: Good. Good boy. (Q 2) Where is Muno playing?
16B: Muno is playing under the tree. {2}
Comment: "with" omitted in Q 2 answer, but comprehension is assumed.
Passage 2
16B: (reads) Mary is holding some tomatoes. They are red. She likes to eat them. {2}
R: (Q 1) What is Mary holding?... What is Mary holding?
16B: Mary is holding a tomato. {2}
R: Good. (Q 2) What does Mary like to eat?
16B: Mary like to eat tomatoes. {2}
R: Good boy. Well done. (Q 3) What colour are the tomatoes?
16B: Tomato colour - tomatoes is colour - a red. (2)
Comment: "a tomato" instead of "some tomatoes" for Q 1; "like" for "likes" for Q 3; difficulty in phrasing the answer to Q3, as he wants to "mirror" the vocabulary of the question and use the word "colour". In all cases comprehension is assumed and full marks are awarded.
Passage 3
16B: (reads) Mrs Moyo is a teacher. Sara and Miriam - Miriam - Miriam - Miriam - Miriam are in Mrs Moyo class. One day they give her some following (?). She was very happy and she put them on his table. "Thank you, girls," she said. {1}
R: (Q 1) Who is Miss Moyo?
16B: Mrs Moyo is teacher. {2}
R: (Q 2) Where did Miss Moyo put the flowers?
16B: Mrs Moyo is putting them on her table. Thank you girls - {1}
R: Good, good. That's it. She put them on her table. Well done. Good. (Q 3) Why was Miss Moyo happy? Why was Miss Moyo happy?
16B: Mrs Moyo... Mrs Moyo is teaching the children. {0}
Comment: "Mrs" for "Miss"; "Moyo class" for "Moyo's class"; "give" for "gave"; "his" for "her". None of the foregoing are considered to evince lack of comprehension. Indeed those miscues suggest that the reader is comprehending, but has "transformed" the text according to his own interlanguage (see 8.3). Such a process can only take place if some degree of comprehension has taken place. However "following (?)" for "flower" is so deviant that it suggests lack of comprehension.
Passage 4
16B: When Robert was a little boys boy he was very sick, so he went with his mother to the hospital. A nurse took Robert and his mother to a big room and she put him into a bed.
"You - oh you you oh soon soon be train (?)." he say. {1}
R: (Q 1) Where did Robert go?
16B: Robert is sick. {0}
R: OK. (Q 2) Where did the nurse put Robert?
16B: A nurse took took the took Robert and he (?) mother. {0}
R: OK. (Q 3) What did the doctor say?... What did the doctor say?
16B:.......... The nurse..... {0}
Comment: The penultimate line of the passage ("Later the doctor came and looked at Robert") was omitted. The pupil does not appear to have understood the last line ("You'll soon be better!" he said.") There is no evidence of comprehension in the answers to the questions (though one cannot tell whether the problem is due to difficulty with the questions, the text, or both). The session terminated at this point.
Although only the better pupils were tested the general pattern which emerges (see 8.7 below, and Appendix L) suggests that there is not a great deal of difference between year 3 and year 4 pupils. Year 3 experience difficulty with Passage 3, which is meant to be at their level. Year 4 also experience difficulty with Passage 3, and 13 out of 15 (the other 5 having terminated on passage 3) have problems with Passage 4 which is meant to be at their level. (But most standard 3 and 4 pupils cope reasonably with the first passage, intended to be well within their range.) Only 5 out of 20 of the Standard 6 children cope reasonably with the questions of Passage 5 (intended for their level). However, 18 out of 20 cope with the reading aloud of that passage, which again indicates the problem of relying on reading aloud as a guide to comprehension in second language reading. As with the Word Find test, the majority of Standard 6 pupils seem to be able to cope with the passages intended for Standards 3 and 4.
In terms of question types the direct reference questions are handled fairly easily. However, the indirect reference questions were frequently answered by the Year 3 and Year 4 pupils by repeating the pronoun. Thus the question "What does Mary like to eat?" (passage 2) was answered by "She likes to eat them". It is not clear whether this answer stems from lack of comprehension of the text, or is simply an inadequate piece of communication by readers who are excessively text-bound.
Among the question types which proved the most difficult are the inference questions (notably passage 3, q. 3), particularly for Standards 3 and 4. Passage 3 question 3 was answered satisfactorily by only 2 out of 40 pupils from those standards, and only 8 out of 20 from standard 6. It was suggested that this was due to cultural differences - that pupils in Malawi would be unlikely to bring flowers to their teacher. It is also possible that they did not know the word "happy" although it is taught in standard 3. A further possible reason is that the pupils are not accustomed to being asked inferential questions, typically beginning "Why?". Mchazime (1989: 40) notes that 76% of questions in books 2, 4, 6 and 8 of "English for Malawi" are "plain sense" questions (ie do not require inference) and that all the questions in books 2 and 3 are "plain sense".
The general conclusions drawn from the read aloud exercise are:
(i) That the findings confirm the performance of pupils in the Word Find written test, namely that most pupils have difficulties with passages at their level, but most standard 6 pupils can read standard 3 and 4 texts with comprehension.(ii) That pupils' ability to read aloud outstrips their demonstrable comprehension and that reading aloud alone is not a reliable indicator of comprehension.
(iii) That developmental errors in reading aloud are best treated as evidence of comprehension rather than as misreadings.
(iv) That pupils need more practice in answering inferential questions.
The twelve pupils (4 from standards 3, 4 and 6) from School P gave a representative performance. The assessment scores for that school are provided in 8.8 below. A clear pattern emerges, as one would expect, of higher marks in the top right hand section (results of standard 6 pupils reading easy passages) shading into lower marks, or none, in the bottom left hand corner (results of standard 3 children reading difficult passages). For the complete set of results see Appendix L.
Key to abbreviations:
15B: boy number 15; 15G: girl number 15
Pass 1: Passage 1; Ques 1: Question 1
T: test terminated at this point
na: question not asked
Assessment of Overall Reading of Passages: key to numbers:
2 = clearly comprehensible; has read aloud adequately
1 = incomprehensible in places; could not read aloud everything
0 = experienced severe problems
Assessment of Answers to Comprehension Questions: key to numbers:
2 = clear evidence of comprehension
1 = partial evidence of comprehension
0 = no evidence of comprehension
|
|
Standard 3 |
Standard 4 |
Standard 6 |
||||||||||
|
15B |
16B |
10G |
15G |
12B |
2B |
5G |
13G |
1B |
10B |
4G |
7G |
||
|
Pass 1 |
2 |
2 |
1 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
|
|
Ques 1 |
2 |
2 |
0 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
|
|
Ques 2 |
2 |
2 |
0 |
2 |
2 |
0 |
2 |
1 |
2 |
2 |
2 |
2 |
|
|
Pass 2 |
2 |
2 |
1 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
|
|
Ques 1 |
2 |
2 |
1 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
|
|
Ques 2 |
1 |
2 |
0 |
0 |
2 |
0 |
1 |
2 |
2 |
2 |
2 |
2 |
|
|
Ques 3 |
0 |
2 |
0 |
0 |
2 |
0 |
2 |
2 |
2 |
2 |
2 |
2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
Pass 3 |
2 |
1 |
|
1 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
|
|
Ques 1 |
2 |
2 |
|
2 |
2 |
0 |
2 |
2 |
2 |
2 |
2 |
2 |
|
|
Ques 2 |
0 |
1 |
|
0 |
2 |
0 |
0 |
0 |
2 |
0 |
0 |
2 |
|
|
Ques 3 |
0 |
0 |
|
0 |
1 |
0 |
0 |
0 |
2 |
2 |
0 |
2 |
|
|
T |
|
|
|
|
T |
|
|
|
|
|
|
||
|
Pass 4 |
|
1 |
|
1 |
1 |
|
2 |
1 |
2 |
2 |
2 |
2 |
|
|
Ques 1 |
|
0 |
|
0 |
1 |
|
1 |
2 |
2 |
2 |
0 |
1 |
|
|
Ques 2 |
|
0 |
|
na |
1 |
|
2 |
0 |
2 |
2 |
0 |
2 |
|
|
Ques 3 |
|
0 |
|
na |
2 |
|
1 |
0 |
2 |
2 |
2 |
2 |
|
|
Ques 4 |
|
0 |
|
0 |
0 |
|
0 |
0 |
2 |
2 |
0 |
2 |
|
|
|
T |
|
T |
T |
|
|
|
|
|
|
|
||
|
Pass 5 |
|
|
|
|
|
|
1 |
|
2 |
2 |
2 |
2 |
|
|
Ques 1 |
|
|
|
|
|
|
2 |
|
2 |
0 |
1 |
0 |
|
|
Ques 2 |
|
|
|
|
|
|
0 |
|
2 |
na |
1 |
0 |
|
|
Ques 3 |
|
|
|
|
|
|
0 |
|
2 |
2 |
0 |
2 |
|
|
Ques 4 |
|
|
|
|
|
|
1 |
|
na |
2 |
0 |
na |
|
|
Ques 5 |
|
|
|
|
|
|
1 |
|
2 |
2 |
1 |
2 |
|
|
Ques 6 |
|
|
|
|
|
|
1 |
|
2 |
2 |
0 |
1 |
|
