7 Results of English reading test "Word Find" and Chichewa test

7.1 The interpretation of results of Word Find
7.2 Results by school and year
7.3 Subtest scores
7.4 Overall comments on word find results
7.6 Results of Chichewa reading test, Chewa V.01
7.7 Correlation of English test results with Chichewa test results
7.8 Significance of Sex, Urban/rural, School Year, And Home Language
7.9 Relationship between factors in educational provision and scores

7.1 The interpretation of results of Word Find

The results of Word Find are difficult to interpret in an absolute evaluative sense. One cannot say whether the results indicate that the pupils concerned are "good" readers or "bad" readers relative to any external standard since there are no standardised tests of reading available for the population. It would, obviously, be inappropriate to assess pupils in Malawi on tests standardised in Britain.

One approach to establishing guide lines for the interpretation of results is to ask the pupils' class teachers to examine the tests and to estimate what they feel the average pupil will get (not should get). This was done, with the following results:

Pooled Teacher Judgements on Word Find (maximum score: 60)

Year	Urban Number of Teacher Judgements: 9	Rural Number of Teacher Judgements: 10
	Mean Judgement	Mean Judgement
3	23	19
4	31	26
6	48	44

While such a procedure indicates the teachers' judgement of the pupils' ability it does not establish any relationship between score and degree of comprehension. Indeed any attempt to establish such a relationship would be subjective. However, while it would be wrong to arbitrarily fix a test score below which there is no comprehension, and above which there is comprehension, it is permissable to identify a score below which there is inadequate comprehension. (Such a procedure does not necessarily imply that scores immediately above the point selected constitute adequate comprehension.)

Since the three 20 item subtests in Word Find are aimed at three different years (3, 4 and 6) and each subtest is based on the language in the relevant course book (and accepted as such by the teachers), then I would suggest that to score less than 40% on a subtest (ie less than 8 out of 20 items) indicates inadequate reading on that subtest. Obviously setting criterion measures in a subjective manner may be criticised, but an "inadequate comprehension" point of 7 or less out of 20 for these subtests appears to be a generous judgement (and one supported by all British colleagues and Malawi teachers who saw the tests). It would seem difficult to maintain that a person who scores 7 or less out of 20 in these subtests has adequately comprehended them.

In examining the results then, we need to bear in mind the different sources of the scores (in other words which of the three subtests, namely items 1-20, 21-40, 41-60 the correct items come from). The reason for this is that random completion of the items would yield by chance a score of two and two thirds on each of the three 20 item subtests, making a total of 8 for the test as a whole (Department of Applied Statistics, Reading University). For pupils scoring close to 8 then, one should look at where those correct items come from. If they are concentrated in the first twenty items, then we might imagine a slow reader, but one who is giving some evidence of comprehension. If the correct answers are randomly scattered throughout the 60 items, then it is likely that the pupil is guessing. A full breakdown of results appears in Appendix I.

7.2 Results by school and year

Results by School and Year: Word Find

YEAR	SCHOOL	N	MEAN	SD	MAX	MIN	MEDIAN
3	M	39	10.3846	5.9321	25	2	9.0
3	N	34	5.1176	2.7935	12	1	5.0
3	P	34	7.7059	3.3713	15	2	7.0
3	S	40	6.3250	4.3817	21	0	5.0
3	W	22	8.7727	2.7591	14	3	9.5
4	M	39	20.8718	7.7330	37	7	20.0
4	N	35	7.2571	3.1747	18	2	7.0
4	P	29	16.8966	7.3307	39	7	17.0
4	S	30	14.6333	5.9043	27	5	14.0
4	W	20	14.6000	6.3693	28	6	13.5
6	M	42	32.6667	8.9243	53	17	30.5
6	N	25	21.0000	7.8156	42	11	21.0
6	P	35	27.1429	9.1656	54	12	25.0
6	S	40	29.9500	7.1034	47	17	29.0
6	W	16	35.2500	12.4177	53	8	36.0

Comparing the actual means with the pooled teacher judgements reveals considerable overestimating by the teachers (results to the nearest whole number):

	ESTIMATES		ACTUAL MEAN SCORES
	Urban	Rural	Urban	Urban	Rural	Rural	Rural
	Tchrs	Tchrs	Sch S	Sch M	Sch N	Sch P	Sch W
Year
3	23	19	6	10	5	8	9
4	31	26	15	21	7	17	15
6	48	44	30	33	21	27	35

From this one may conclude either that teachers are not aware of their pupils' limitations, or that, despite the instructions, they estimated what the pupils should, rather than would, score.

7.3 Subtest scores

Rather than dwelling on overall means, where it will be recalled, random completion may result in a score of 8, it is more useful to look at the scores subtest by subtest, on the assumption that within each subtest of 20 items a score of 7 or less indicates inadequate comprehension.

7.3.1 Items 1-20

YEAR	SCHOOL	N	MEAN	SD	MAX	MIN	MEDIAN
3	M	39	8.3590	4.40984	17	1	7.0
3	N	34	4.6765	2.70488	11	1	4.0
3	P	34	7.0588	3.40179	14	1	7.0
3	S	40	5.1250	3.63873	16	0	5.0
3	W	22	6.5455	2.73822	12	2	7.0
4	M	39	12.6923	3.94804	19	5	13.0
4	N	35	6.3429	2.89972	14	1	6.0
4	P	29	9.4828	4.53259	18	1	9.0
4	S	30	9.2667	3.23700	16	3	9.0
4	W	20	8.8000	4.88392	17	2	8.0
6	M	42	16.5476	2.46143	20	10	17.0
6	N	25	12.0400	3.06159	17	7	13.0
6	P	35	14.3429	2.57754	18	9	15.0
6	S	40	15.5750	2.37387	19	9	16.0
6	W	16	15.4375	3.55844	19	5	16.5

At year 3, inspection of the individual scripts reveals that 60 pupils out of 169 have scored 8 or over in this subtest. This suggests that some 65% of the pupils in Year 3 are not comprehending adequately at this level.

In year 4, 94 pupils out of 153 have scored 8 or over, suggesting that approximately 40% are not comprehending adequately at this level.

All the Year 6 classes have means well over 8 on this subtest. Notice that 3 schools have minimum scores of more than 8. In fact, at this level, only 2 pupils out of 158 have scores of 7 or less, suggesting that fewer than 2% had comprehension problems at this level.

7.3.2 Subtest 2: Items 21-40

YEAR	SCHOOL	N	MEAN	SD	MAX	MIN	MEDIAN
3	M	39	1.8205	2.10102	8	0	1.0
3	N	34	0.4412	0.85957	3	0	0.0
3	P	34	0.6471	0.84861	4	0	0.5
3	S	40	1.1750	1.69293	6	0	0.0
3	W	22	1.7273	1.24142	5	0	1.0
4	M	39	5.7179	3.77630	17	0	5.0
4	N	35	0.8857	1.36708	5	0	0.0
4	P	29	4.8966	2.78189	15	1	5.0
4	S	30	3.3667	2.56614	8	0	3.0
4	W	20	4.0500	2.43818	9	0	4.0
6	M	42	10.8333	4.28222	20	0	10.5
6	N	25	5.8800	3.53930	15	1	6.0
6	P	35	8.7714	4.94729	19	1	8.0
6	S	40	9.7500	3.51371	18	3	9,5
6	W	16	11.1250	5.51211	19	0	12.0

Year 3 fall away suddenly here. Only 1 pupil has achieved a score of 8 in this section. From observing the pupils taking the test, I feel that one reason (other than lack of English) is that the Year 3 pupils seem to read very slowly in English, and many of them simply did not have time to get on to this section. Year 4 pupils too seem to have had more difficulty here, with all means well below 8, and only 17 out of 153 obtaining a score of 8 or over (9 in School M), suggesting that about 89% do not have adequate comprehension of this section.

Year 6 classes in all schools with the exception of school N have means of over 8. 100 pupils out of 158 have scores of 8 or over on this subtest suggesting that about 36% were not comprehending adequately.

7.3.3 Subtest 3: Items 41-60

YEAR	SCHOOL	N	MEAN	SD	MAX	MIN	MEDIAN
3	M	39	1.8205	2.10102	8	0	1.0
3	N	34	0.4412	0.85957	3	0	0.0
3	P	34	0.6471	0.84861	4	0	0.5
3	S	40	1.1750	1.69293	6	0	0.0
3	W	22	1.7273	1.24142	5	0	1.0
4	M	39	5.7179	3.77630	17	0	5.0
4	N	35	0.8857	1.36708	5	0	0.0
4	P	29	4.8966	2.78189	15	1	5.0
4	S	30	3.3667	2.56614	8	0	3.0
4	W	20	4.0500	2.43818	9	0	4.0
6	M	42	10.8333	4.28222	20	0	10.5
6	N	25	5.8800	3.53930	15	1	6.0
6	P	35	8.7714	4.94729	19	1	8.0
6	S	40	9.7500	3.51371	18	3	9.5
6	W	16	11.1250	5.51211	19	0	12.0

Years 3 and 4, as expected, have given no evidence of comprehending this. The vast majority probably did not have time to attempt it. Only 3 pupils in Year 4 scored 8 or above.

In year 6, 34 pupils out of 158 at this level achieved scores of 8 or over, suggesting that some 78% did not achieve adequate comprehension with material deemed to be at their level.

7.4 Overall comments on word find results

Overall the results suggest that well over half the pupils at Year 3 have problems of comprehending English reading material supposedly at their level. Only a minority of Year 4 pupils have problems with reading material that is below them, but the vast majority (89%) appear to have difficulty when faced with text at their level. Year 6 pupils, again, seem for the most part to be able to cope well with material aimed at Year 3 and 4, although over three quarters seem to have difficulty with material aimed at their own level.

It is difficult to see how such low English reading proficiency permits the pupils in standard 6 to use reading in English to learn in other subject areas, as they are meant to be doing. It is also highly likely that most pupils will not be able to improve sufficiently over the next two years to perform reasonably on the Primary School Leaving Certificate which appears to be much more difficult than the Word Find test (see Appendix H for extracts from the 1990 certificate examination).

7.6 Results of Chichewa reading test, Chewa V.01

As Chichewa is the mother tongue of most pupils who did the test and the medium of instruction until Year 5, one would expect the pupils to perform reasonably well. The pooled teacher judgements, together with actual means for the five schools, to the nearest whole number, are as follows:

	ESTIMATES	ACTUAL MEAN SCORES
	Urban	Rural	Urban	Urban	Rural	Rural	Rural
	Tchrs	Tchrs	Sch S	Sch M	Sch N	Sch P	Sch W
Year
3	33	36	22	14	9	20	15
4	45	44	32	35	20	28	31
6	55	53	47	48	40	44	49

The actual scores are again below those predicted by the teachers, but do not display such great differences as for English score estimates. There are no great differences between rural and urban, and the main feature of the mean scores is the consistently weaker performance of School N, which was also the weakest overall in English. (See Appendix J for results.)

Examination of individual scores reveals a number of strong performances from Year 3 upwards, although again years 3 and 4 fall away in successive subtests. However, the results generally indicate good comprehension by most pupils in Years 3 and 4 on the first two subtests, while in Year 6 a considerable majority have scored over 66% on the test as a whole. It seems safe to suggest that most Year 6 pupils can read adequately in Chichewa.

One may go on from this to suggest that in the case of Year 6 any weakness in the English results is likely to arise from a weakness in English language rather than a weakness in reading per se.

7.7 Correlation of English test results with Chichewa test results

Product moment correlations for all years are generally positive but not high. Those for Year 6 are as follows:

School M (urban):	0.56	(p<0.0001)
School S (urban):	0.40	(p<0.0110)
School N (rural):	0.42	(p<0.0368)
School P (rural):	0.40	(p<0.0191)
School W (rural):	0.73	(p<0.0022)

This suggests a slight tendency for relative performance in English to correlate with that in Chichewa. Certainly it does not suggest that competence in English is acquired at the cost of competence in Chichewa, or vice versa. These results indicate that those who are good at Chichewa are also good at English, although of course no causal relationship can be inferred.

7.8 Significance of Sex, Urban/rural, School Year, And Home Language

The statistical analysis of sex differences, urban/rural differences, home language effect (for the Chichewa test) and year effect (for the different standards) appear in Appendix K. An interpretative account of the statistics is provided in the following paragraphs.

7.8.1 Sex differences

Overall there is a slight difference with boys estimated to score 1.95 points more than girls in English and 1.25 points more in Chichewa. Only the difference in the English score is statistically significant however. There was no immediately apparent reason from classroom observation as to why this should be the case. However personal communications suggested that girls are less likely to have books bought for them than boys, and girls may also miss school more to help at home. Further investigation would be needed to investigate causes.

7.8.2 Urban/rural differences

The estimated differences in score between urban and rural schools are large (3.3 points for English and 4.7 for Chichewa), but they are not statistically significant because of the small numbers of schools involved in this study. It is possible that real differences between urban and rural schools do exist but a larger sample of schools (not pupils) would be needed to confirm this.

7.8.3 Year differences

Differences between years 3, 4 and 6 are large and strongly significant, indicating a progressive improvement in ability with age both for English and Chichewa. This is reassuring in that it clearly indicates that pupils are actually learning more the longer they stay in school.

7.8.4 Language differences

In the Chichewa reading test, there is a difference of 1.4 points in favour of Chichewa home language speakers as compared to non-Chichewa home language speakers. However, this difference is not statistically significant. The small difference suggests that in the schools tested the minority of children from a non-Chichewa background have in fact learned Chichewa (probably from their peers as well as their teachers).

7.9 Relationship between factors in educational provision and scores

The factors that are considered here are books per pupil, class size, and teacher qualifications. The results are illustrated through scattergrams which relate the factors to the mean score. Results generally are not very conclusive, partly because of the small number of schools, partly for factor-specific issues.

7.9.1 Relationship between Books per Pupil (Standard 1 to 6) and mean scores in Standard 6

Sch	St 6 Mean Word Find Score	Books per Pupil
W	35.25	0.24
M	32.67	0.58
S	29.95	0.37
P	27.14	0.56
N	21.00	0.15

Schools N, S and M demonstrate a positive relationship with higher scores corresponding to more books per pupil. Per pupil however School W has a higher score but fewer books per pupil than schools P, M and S. Greater variation in book provision would be needed for clearer differences to emerge.

The issue is complicated by the fact that even if the books are in the schools, they may not always be used, and that some pupils own their own books which are not recorded. In addition, other factors may compensate for lack of books. School W judging from the observation had at least one very good teacher (see 6.6.2), while School M had many pupils from a relatively socially favoured background.

7.9.2 Relationship between Mean Class Sizes and Mean Score at Standard 6

Schl	St 6 Mean Word Find Score	Mean Class Sizes (yrs 1-6)
W	35.25	52.83
M	32.67	84.00
S	29.95	120.67
P	27.14	148.77
N	21.00	74.17

Obviously, it is not simply the size of the current Standard 6 class which is relevant, but also the size of the classes that the children have previously been in. An approximate idea of relative size may be obtained by calculating the current mean class size from standards 1 to 6 for each school. This was done for the above calculations, and a remarkably consistent relationship emerges where, (taking school W as the starting point) an increase of class size by roughly 12 pupils correlates with a decrease in mean score of roughly 1 mark. The exception is School N which again appears to be underperforming. Larger numbers of schools would be needed to confirm this tendency.

7.9.3 Relationship between Teacher Qualifications and Mean Score

Visual inspection of mean scores and the teachers' qualifications suggests no positive relationship between teacher qualifications from Standards 1 to 6, and mean score at Standard 6. For example, School W with untrained teachers at Standards 2 and 3, and all the other teachers at T3 (T4 being the lowest, T1 the highest qualification) has the best mean score at Standard 6.

In order to attempt a more objective basis for correlation, one may convert teacher qualification to points as follows:

Untrained =2

T4 = 4

T3 = 5

T2 = 6

T1 = 7

A mean teacher qualification score for teachers currently in standards 1 to 6 may then be computed, and compared with the mean score for pupils in standard 6 on Word Find. Rather surprisingly this indicates a tendency towards a negative correlation, with School W having poorly qualified staff but achieving the highest standard 6 Word Find mean, and school N with overall best qualified staff achieving the lowest standard 6 mean.

Schl	St 6 Mean Word Find Score	Teacher Mean Qualification Score
W	35.25	3.88
M	32.67	5.25
S	29.95	5.45
P	27.14	5.37
N	21.00	5.5

Again, one would not wish to make much of this finding, since, apart from factors already mentioned, the differences between the computed mean scores on teacher qualification are very small; furthermore one class in school N has (temporarily) no teacher; this did not enter into the computation. Again it is not known how long all these teachers had been in the schools.