8.1. Procedure
8.2. Administration
8.3. Describing reading performance
8.4. Rationale for the questions
8.5. Example of read aloud transcript
8.6. Findings of the read aloud testing
8.7. Reading aloud: Assessment sheet for School D
8.9. Conclusions on read aloud testing
For the data structure of this test see 4.6.2. It will be recalled that 2 girls and 2 boys (1 average, 1 above average) were requested from each class that had done Word Find test, but that time constraints meant this was not possible in all cases. Below average pupils were not requested for this particular test, since it would have been destructive to subject them to what would have been a demoralising experience. The selection mode means that the Read Aloud test does not adequately feature "below average" performance, and this is to be borne in mind in considering the results. (Even so, there were some very weak performances.)
The Read Aloud test consists of five graded passages based upon language from the Zambia Primary Course as follows:
Passage 1, 2 and 3: for year 3
Passage 4: for year 4
Passage 5: for year 6
Each passage is followed by questions which are asked immediately after that passage has been read (see Appendix D). Pupils could look at the passages while they answered the questions, and could also look at the questions if they wished.
Pupils were tested by the researcher individually (however on occasion an inspector and/or teacher was present). The sessions generally took place in an empty classroom, store room, or office. I opened the session with a few simple questions about the pupil's name and age, and continued with some questions about the picture accompanying the first reading text.
If at any point it appeared from their reading that pupils were experiencing severe problems, then the session would terminate at that point. In the case of clearly competent children, some passages and questions were omitted. Some questions were also omitted if the reader's previous answer made it redundant. All readings were audio-recorded.
The first two passages and questions were intended to be very easy so as to give the pupils confidence, but proved to be difficult for some Grade 3 pupils (eg School D; see 8.7 below).
8.3.1. The reader's performance on this test is judged on the basis of accuracy of reading, plus comprehension as judged by the response to the questions. As far as accuracy of reading is concerned, the reader's accent is disregarded, since the immediate aim is to see whether the pupils can convert the written words into spoken form accurately enough to be understood. Clearly in the case of second language readers it is dangerous to infer comprehension from reading aloud. Indeed one of the problems in judging the reading is that there are some readers who appear to read reasonably accurately, but with little evidence of understanding as judged by answers to questions, while others may make mistakes in accuracy, but have clearly understood.
8.3.2. Thus a child who reads "Muno is playing with a ball" as "Muno is playing a ball" and answers the question "What is Muno doing?" in the same way has almost certainly understood. What such a testee is doing, however, is "transforming" the text and the answer in terms of the stage of language development that he or she has reached. Although such utterances do not represent standard English, our concern in these tests is not to assess the testees' speaking ability in terms of standard English, but rather their reading comprehension of written text.
8.3.4. Such deviancies from the text (and from standard English) indicate that the child has almost certainly understood the text, but is demonstrating comprehension through a system which is still at a developmental stage. These deviancies may be compared to English children who read "John isn't going" as "John ain't going" where the non-standard pronunciation is in itself evidence that the child has understood, but "translated" as it were, the written sentence into their own variety of English. This type of transformation in reading (which can only result from comprehension), may be contrasted with the case of pupils who read accurately word by word, but do not comprehend what they read because their language proficiency is inadequate. The latter rely entirely on the text for what they say; the former "transform" the text in terms of their own language system.
8.3.5. Thus, certain meaning preserving slips and systematic errors may, paradoxically, constitute evidence that the reader has understood the text. Since the aim of this testing was to investigate reading comprehension and not accuracy of reproducing the text, errors which are judged to be developmental are not penalised.
8.3.6. On the other hand a child who reads passage 2 accurately, but answers the question "What does Mary like to eat?" by "They are red" may well not have understood the text. (Of course it is possible that they have not understood the question.) Again, a child who answers the question "Why did Robert go to the hospital?" by reading out "When Robert was a little boy he was very sick, so he went with his mother to the hospital" may or may not have understood the text. Certainly the answer is not appropriate to the discourse. Such answers make clear decisions about the pupil's comprehension rather difficult.
8.3.7. For the above reasons precise objective quantification of comprehension on the basis of reading aloud is problematic. Standard miscue analysis (eg Goodman, 1973) or modified versions (Vincent & de la Mare, 1990) have been developed for English native speaker readers. They are inappropriate for most Zambian children in that they do not allow for language learners in a developmental stage.
The questions which follow each passage practice various operations which are considered relevant at this level, and which are fundamental to reading with comprehension. They are:
(i) obtaining information through understanding direct reference made in the text. Thus the first question in passage 3 (Who is Miss Moyo?) requires the student to make direct reference to the relevant sentence in the text (Miss Moyo is a teacher.) and produce the answer. Clearly such direct reference questions may result in testees providing the correct answer through "matching" without necessarily understanding what they are saying.(ii) obtaining information through recovery of an element in the text which is referred to indirectly, typically by means a pronoun, although other parts of speech may be used. An example is question 2 of passage 3 (Where did Miss Moyo put the flowers?) which requires the reader to realise that "them" in "... she put them on her table" refers to "flowers".
(iii) obtaining information through making the appropriate inference. Here the reader infers a relationship between two parts of a text that is not explicitly stated. Question 3 of passage 3 (Why was Miss Moyo happy?) requires the reader to infer from the text the most likely reason why Miss Moyo was happy (...they gave her some flowers. She was very happy and...). It may be that in some cases appropriate knowledge of the world facilitates the appropriate inference.
The three operations (direct reference, indirect reference and inference) were distributed in the 18 questions as follows:
|
Passage 1: |
1, direct reference |
2, indirect reference. |
|
Passage 2: |
1, direct reference |
2 indirect reference |
|
3, indirect reference. |
|
|
|
Passage 3: |
1, direct reference |
2, indirect reference. |
|
3, inference |
|
|
|
Passage 4: |
1, direct reference |
2, indirect reference. |
|
3, direct reference |
4, indirect reference. |
|
|
Passage 5: |
1, direct reference |
2, indirect reference. |
|
3, inference |
4, inference |
|
|
5, indirect reference. |
6, direct reference. |
Pupils who are unable to obtain the relevant information in answer to a question that involves one of the above operations are unlikely to be comprehending the relevant section of the text. There is, nonetheless, a possibility that pupils could indeed understand the text, but were not able to demonstrate comprehension either because:
(i) they could not understand the question. (Here it should be noted that the question was spoken by me, and pupils were also shown the written form of the question.) or because:(ii) they were not able to produce an answer in English (although they understood both text and question).
However, all the questions can be answered acceptably by using the language in the texts. It is therefore likely that pupils who cannot answer these questions are failing not because of insufficient productive proficiency in English, but rather because they have not understood the text or the question. Nevertheless, the possibility always remains that asking and answering questions in the child's mother tongue might have yielded different results.
To illustrate the issues discussed above, an example of a read aloud transcript is provided below. The transcription is restricted to the reading and to the questions and answers; the introductory, linking and concluding remarks are omitted. The transcript is of a Grade 6 girl from School D. She obtained 39 on the Word Find test, well above the mean of 28 for all Grade 6 pupils.
Notes on Transcription Conventions
Miscues which consist of misreadings in the pupils' readings have been underlined.
Miscues which consist of repetitions in the pupils' readings are transcribed but not underlined.
R = Researcher
Numbers in brackets {} indicate the mark awarded (2, 1, or 0: see 8.6)
Numbers in round brackets () preceded by Q indicate the question number for the passage
|
School: D |
Grade: 6 |
Pupil: 3G |
Passage 1
P: (reads) Muno is playing with a ball. He is playing under the tree. (2)
R: Very good. (Q 1): What is Muno doing?
16B: He is playing - he is playing a ball. {2} R: Good. (Q 2) Where is Muno playing?
P: He is playing under the tree. (2)
Passage 2
P: (reads) Mary is holding some tomatoes. They are red. She likes to eat them. (2)
R: Good. (Q 1) What is Mary holding?
P: Mary is hold some tomatoes. {2}
R: Good. (Q 2) What does Mary like to eat?
P: Mary likes to eat them.
R: Mm. To eat what?
P: Tomatoes. (2)
R: That's it. Good. (Q 3) What colour are the tomatoes?
P: Red. {2}
R: Yes. Very good.
Comments: No problems. She has understood the texts, although responses to Q 1 in both texts indicate developmental error. Again, although her first answer to Passage 2, Q 2 is text bound, she amplifies easily.
Passage 3
P: (reads) Miss Moyo is a teacher. Sara and Miriam are - are in Mrs Moyo class. One day they - one day they gave her some flowers. She was very happy and she putting them on her table. "Thank you - thank you - thank you girls," she said. {2}
R: OK, very good. (Q 1) Who is Miss Moyo?
P: Mrs Moyo is a teacher. {2}
R: (Q 2) Where did Miss Moyo put the flowers?
P: Miss Moyo put the flowers on the table. {2}
R: Very good. (Q 3) Why was Miss Moyo happy? Why was Miss Moyo happy? (MOE Inspector translates into Nyanja. Tells her to answer in Nyanja)
P: One day they gave - (MOE Inspector interrupts. P answers in Nyanja after some help {0})
Comments: "Putting" for "put" is regarded as a developmental slip. She has clearly understood, and indicates by the mistake that she has recognised it as a verb. She has problems with Q 3, and is only able to answer after considerable coaxing.
Passage 4
P: (reads) When Robert was a little boy he was very sick, so he went with - with his mother to the hospital. A nurse took Robert and his mother to a big room and she put him into a bed. Later the doctor come came and looked at Robert. "You will soon be better" he said. {2}
R: Good. (Q 1) Where did Robert go?
P: Robert go to the hospital. {2}
R: Good. (Q 2) Why did Robert go to the hospital?
P: She - he is sick. {2}
R: Good. OK. (Q 3) Where did the nurse put Robert?
P: He put him into the bed. (2)
R: Good. (Q 4) What did the doctor say?
P: You will - you will soon be better. {2}
Comments: She seems to have no problem here, although there are developmental errors with "come" for "came" and "go" for "went".
Passage 5
P: (reads) Simon and his sister Ruth went by bus to visit their uncle. As the bus was going over a bridge - a bridge they heard a loud noise. The driver stopped the bus and got out. A large sack of - a large sack of - of tomato had fallen from the roof. It had burst open and there were tomato everywhere on the road. Simon and Ruth helped to pick them up and put - and put them back in the sack. The sack was so heavy that two men had to help the driver to lift it into the roof of the bus. {2}
R: OK. Very good. Thank you. (Q 1) Did Simon and Ruth go in a car?... Did they go in a car?... Or in a bus?
P: In a bus. {2}
R: In a bus. OK (Q 3) Why did the driver stop the bus?
P: (no response. MOE inspector translates into Nyanja)
P: (replies in Nyanja: Because there was a bridge {0})
R: OK. (Q 5) How did Simon and Ruth help? How did Simon and Ruth help? (MOE inspector translates into Nyanja)
P: (Replies in Nyanja: She helped them to carry the sack.{0})
Comments: She seemed to be struggling with this passage. "Potatoes" was twice misread as "tomato", but this is a fairly trivial slip. She was given help with the answer to Q 1, and was again unable to answer the inference question with "Why" (Q 3). As she had considerable difficulty with Q 5 the session terminated at that point. Although she had read this text reasonably her stress and intonation suggested that in places she did not fully comprehend, and this was borne out by her response to the questions. Here we have an example of a reasonable reading (awarded a mark of 2) but incomplete comprehension of the text as a whole, which illustrates the risk of relying on reading alone as an indicator of comprehension.
Although only the better pupils were tested, the general pattern which emerges (see 8.7 below) supports the findings of the group reading test Word Find. The Grade 3 rural pupils have great difficulty with Passage 1, which is meant to be well below their level. Indeed it seems that that for practical purposes 3 readers in School D, Grade 3 (see 8.7 below) cannot read in English. Rural Grade 4 likewise have difficulty with Passage 1, and with Passage 2 (meant to be well within their level). Only urban school J Grade 3 pupils coped well with the first 3 passages.
Around half the Grade 6 rural children have difficulties with the questions of Passage 5 (intended for their level). However, the majority cope with the reading aloud of that passage, which again indicates the problem of relying on reading aloud as a guide to comprehension in second language reading. As with the Word Find test, the majority of Grade 6 pupils seem to experience relatively little difficulty with the passages intended for Grade 3 and 4, although 7 out of 10 rural Grade 6 pupils have not produced good answers to the Passage 3 inference question.
The mark sheets for all read aloud tests are provided in Appendix L. The assessment scores for rural school D are provided below. It should be remembered that the marking of this test is more impressionistic than Word Find.
Key to abbreviations:
B: boy; G: girl
Pass 1: Passage 1; Ques 1: Question 1
T: test terminated at this point
na: question not asked
Assessment of Overall Reading of Passages: key to numbers:
2= clearly comprehensible
1= incomprehensible in places
0= mostly incomprehensible; no evidence of understanding
Assessment of Answers to Comprehension
Questions: key to numbers:
2= clear evidence of comprehension
1= partial evidence of comprehension
0= no evidence of comprehension
|
|
Grade 3 |
Grade 4 |
Grade 6 |
||||||||||
|
4B |
3B |
G |
G |
5B |
1B |
8G |
5G |
8B |
3B |
3G |
12G |
||
|
Pass 1 |
1 |
1 |
1 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
|
|
|
Ques 1 |
1 |
1 |
1 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
|
|
Ques 2 |
0 |
0 |
0 |
2 |
2 |
2 |
0 |
0 |
2 |
2 |
2 |
2 |
|
|
T |
T |
T |
|
|
|
|
|
|
|
|
|
|
|
Pass 2 |
|
|
|
1 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
|
|
|
Ques 1 |
|
|
|
2 |
2 |
2 |
2 |
1 |
2 |
2 |
2 |
2 |
|
|
Ques 2 |
|
|
|
2 |
1 |
2 |
2 |
1 |
1 |
2 |
2 |
2 |
|
|
Ques 3 |
|
|
|
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
2 |
|
|
|
|
|
|
|
|
|
T |
|
|
|
|
|
|
Pass 3 |
|
|
|
1 |
2 |
2 |
1 |
|
2 |
2 |
2 |
2 |
|
|
|
Ques 1 |
|
|
|
2 |
2 |
2 |
0 |
|
2 |
2 |
2 |
2 |
|
|
Ques 2 |
|
|
|
2 |
0 |
2 |
2 |
|
2 |
2 |
2 |
2 |
|
|
Ques 3 |
|
|
|
1 |
0 |
0 |
0 |
|
1 |
1 |
0 |
0 |
|
|
|
|
|
T |
T |
T |
T |
|
|
|
|
|
|
|
Pass 4 |
|
|
|
|
|
|
|
|
2 |
2 |
2 |
2 |
|
|
|
Ques 1 |
|
|
|
|
|
|
|
|
2 |
2 |
2 |
2 |
|
|
Ques 2 |
|
|
|
|
|
|
|
|
2 |
2 |
2 |
2 |
|
|
Ques 3 |
|
|
|
|
|
|
|
|
2 |
2 |
2 |
2 |
|
|
Ques 4 |
|
|
|
|
|
|
|
|
0 |
1 |
2 |
2 |
|
Pass 5 |
|
|
|
|
|
|
|
|
2 |
2 |
2 |
2 |
|
|
|
Ques 1 |
|
|
|
|
|
|
|
|
na |
na |
2 |
2 |
|
|
Ques 2 |
|
|
|
|
|
|
|
|
na |
na |
na |
na |
|
|
Ques 3 |
|
|
|
|
|
|
|
|
2 |
2 |
0 |
2 |
|
|
Ques 4 |
|
|
|
|
|
|
|
|
na |
na |
na |
2 |
|
|
Ques 5 |
|
|
|
|
|
|
|
|
2 |
0 |
0 |
2 |
|
|
Ques 6 |
|
|
|
|
|
|
|
|
2 |
0 |
T |
2 |
A clear pattern emerges, as one would expect, of higher marks in the top right hand section (results of Grade 6 pupils reading easy passages) with low or no marks, in the bottom left hand section (results of Grade 3 children reading difficult passages).
In terms of question types the direct reference questions are handled fairly easily. However, the indirect reference questions were frequently answered by the Grade 3 and Grade 4 pupils by repeating the pronoun. Thus the question "What does Mary like to eat?" (passage 2) was answered by "She likes to eat them". It is not clear whether this answer stems from lack of comprehension of the text, or is simply an inadequate piece of communication by readers who are excessively text-bound.
The questions which prove difficult are the inference questions (passage 3, q. 3 and passage 5 q. 3 and q. 4). Grade 6 pupils who did well on all questions in passage 4 had problems with passage 3 question 3. The probable reason is that the pupils are not accustomed to being asked inferential questions, typically beginning "Why?". In this connection we may note that Chikalanga (in press) concludes from tests on 158 grade 8 (ie secondary) pupils in Zambia that their performance on inference questions was "amazingly low". He attributes this possibly to decoding problems, but also to the fact that although the ZPC course cites inferencing as a skill to be taught, the teachers' handbooks pay little attention to it, "and consequently little is done about it", since training colleges and teachers themselves are largely directed by the handbooks.
The general conclusions drawn from the read aloud testing are:
(i) That the findings confirm the performance of pupils in the Word Find written test, namely that most Grade 3 and 4 pupils have difficulties with passages which are aimed at their level. Grade 6 pupils can read standard 3 and 4 texts with comprehension but many seem to be struggling with passage 5 which is intended to be at their level.(ii) That pupils' ability to read aloud sometimes outstrips their demonstrable comprehension and that reading aloud alone is not a reliable indicator of comprehension.
(iii) That developmental errors in reading aloud can be treated as evidence of comprehension rather than lack of comprehension.
(iv) That pupils need more practice in answering inferential questions.
