4 Reading research: 1992

4.1 Description
4.2 Results of English test
4.3 Results of local language tests

4.1 Description

4.1.1 Aims
4.1.2 Test methods
4.1.3 Schools and year levels
4.1.4 The testees

4.1.1 Aims

The aims of the 1992 reading tests were to investigate:

(i) reading proficiency in English in Zambian and Malawian primary schools at years 3, 4 and 6

(ii) reading proficiency in a local language (Chinyanja in Zambia; Chichewa in Malawi) at the same years.

Particular attention was paid to the effect of location (urban and rural) and gender differences. Over and above the investigation of reading competencies at national level, carrying out parallel testing in two countries also allows comparison of contrasting language policies upon reading proficiency at international level.

4.1.2 Test methods

Following the initial piloting of various test formats, it was decided that the most appropriate group reading testing methods would be:

(i) a modified cloze test in English for testing English reading proficiency
(ii) a modified cloze test in local languages for testing local language reading proficiency

(i) Modified English Cloze Test: here the reader has a short passage with 4 or 6 gaps, and is asked to choose a word from a list provided in a box above the passage to fill each gap. A 60 item test was produced, referred to in this report as Word Find (see Appendix D for extracts). The 1992 English courses in the two countries (English for Malawi and the New Zambia Primary Course) were sufficiently similar for language common to both courses to be used in testing. The Word Find test was divided into three sections of 20 items, graded in terms of language, and aimed at the years as follows:

Section 1, items 1 - 20: Year 3
Section 2, items 21-40: Year 4
Section 3, items 41-60: Year 6

The whole 60 item test was administered to all testees, since pilot testing had indicated there was considerable overlap between years, with good year 3 pupils scoring considerably higher than weak year 6 pupils. In topic and style the test passages imitate the text types used in the course books. While far from "authentic", such texts are among the few text types which can safely be assumed to be within the experience of all the children tested. Most rural villages do not have shops, and the children of subsistence farmers rarely see cereal packets, sweet wrappers, tins or other "authentic" texts of the consumer society.

(ii) Local Language Modified Cloze Test: Chichewa and Nyanja versions of Word Find, also with 60 items, were prepared with the help of language experts from Malawi and Zambia.

Since there are minor variations in the standard written forms of the two languages, slightly different versions of the two tests had to be prepared (see Appendix E for extracts). Both versions were adaptations of the English tests. Obviously, it cannot be proved that the two local language tests are of equivalent difficulty. However, as standard Chichewa and Nyanja are "the same language" (Kashoki, 1978: 45) with a few orthographic differences, then it was possible for those who prepared the tests to cooperate so as to ensure close equivalence on a subjective basis.

4.1.3 Schools and year levels

Tests were administered in two urban schools and three rural schools in both countries. All schools were selected by the respective Ministries of Education for Malawi and Zambia. Time constraints meant that there was insufficient time to cover more than three rural schools per country, so the sample is not representative. Testing was carried out in only two of the three Malawian provinces (Central and Southern), and only two of the nine Zambian provinces (Lusaka and North Eastern), since schools had to be in areas where Chichewa/ChiNyanja were known to be local languages, and were also, in the case of Zambia, designated as the "official" language to be taught in primary schools. The lack of representative sampling in terms of provinces and school categories is to be kept in mind in the interpretation of the results.

Pilot work suggested that testing should be carried out at years 3, 4 and 6. Testing at years 1 or 2 would have been inappropriate since in many schools very little achievement would be registered through a group test in those years (this does not, of course, mean that no learning is going on in those years). Year 6 was selected as a point by which pupils should be able to demonstrate reasonable progress.

4.1.4 The testees

The group reading tests were administered to intact classes in cases where classes had fewer than 34 pupils. In cases of larger classes a representative sample was selected. The English and local language reading tests were administered to the following numbers of pupils:

Table 5: Testees for reading tests in Malawi (1992)

Year	Sex	Urban	Rural	Total
3	Boy	38	45	83
3	Girls	41	45	86
4	Boys	34	42	76
4	Girls	35	42	77
6	Boys	43	49	92
6	Girls	39	27	66
Total		230	250	480

Table 6: Testees for reading tests in Zambia (1992)

Year	Sex	Urban	Rural	Total
3	Boys	27	40	67
3	Girls	41	43	84
4	Boys	29	49	78
4	Girls	39	31	70
6	Boys	33	46	79
6	Girls	27	47	74
Total		196	256	452

The principal carers - both male and female - of the majority of rural children were subsistence farmers and/or casual labourers, while those of urban children had a range of occupations, including security guards, drivers, market traders and a variety of clerical occupations.

The mean ages of testees - bearing in mind that many rural children are unsure of their date of birth - were calculated, in years and months, to be:

Table 7: Mean Ages of Testees: 1992

Year	Malawi	Zambia
3	12,3	11,4
4	13,4	12,5
6	14,10	13,9

The mean ages are higher than would be predicted from the official starting ages (6 in Malawi, 7 in Zambia) as many children, especially in rural areas start school later than the official age. The mean age in Malawi is higher than that in Zambia, despite the fact that officially Malawian children start aged 6, rather than 7. The reasons are that more Malawian pupils repeat years, and that many pupils start school even later than Zambians. Only 24% of year 1 Malawian pupils are in fact aged 6 (Ministry of Education [Malawi], 1991: 16). A further anomaly is that in both countries there is less than two years difference between the mean ages in years 4 and 6, possibly because year 4 is the year at which many older pupils who are not intending to attempt secondary school entrance leave. Although these schools are referred to as "primary" schools, the ages of most pupils from year 4 onwards are equivalent to those of secondary school pupils in the UK.

None of the children spoke English at home; all claimed to be able to speak Chichewa (Malawi) or Nyanja (Zambia), and this was supported by the teachers. Observation of children outside class revealed the languages were also the medium of child-to-child communication. Not all children spoke the languages at home; in Malawi 13.1% spoke a language other than Chichewa, while in Zambia 25% spoke a language other than Nyanja at home. This was particularly the case in urban areas. However, the investigation is concerned with providing a description of reading performance in the local language, not in assessing "first language" reading.

4.2 Results of English test

4.2.1 General
4.2.2 Comments on general English test results
4.2.3 School year
4.2.4 Gender effects
4.2.5 Location effects
4.2.6 Gender and location effects

4.2.1 General

The results are considered here in terms of countries, year, gender and location. Statistical significance is set at the conventional 0.05 which means that in cases where differences between groups (male/female, rural/urban etc.) are said to be statistically significant, we may be 95% confident that they did not occur by chance. The chance score for this 60 item test is 8 i.e. filling in the gaps randomly is likely to achieve 8 items correct (Department of Applied Statistics, Reading University). The overall reliability estimate for the English test was 0.86 (Kuder Richardson 21). Statistics were calculated using the MIXED programme in the SAS package.

Table 8: Malawi, 1992 Results for English Reading Test, Items 1-60¹

YEAR	N	MEAN	SD	MAX	MIN	MEDIAN
3	169	7.62	4.56	25	0	7
4	153	14.96	7.89	39	2	14
6	158	29.17	9.71	54	8	29

¹ Key: N: number of testees; MEAN: average; SD: standard deviation; MAX: highest score; MIN: lowest score; MEDIAN: score of the middle testee.

Table 9: Zambia, 1992 Results for English Reading Test, Items 1-60

YEAR	N	MEAN	SD	MAX	MIN	MEDIAN
3	151	4.97	7.43	43	0	2
4	148	11.43	10.60	53	0	8
6	153	28.10	13.82	60	1	29

4.2.2 Comments on general English test results

There is an overall inter-country difference on the English reading test of 2.63 points in favour of Malawi. This difference, however, is far from significant (P>0.4) and there is thus no evidence of real difference between the two countries on the overall comparison. Likewise comparing the countries on the basis of school years reveals consistent differences in favour of Malawi, but again not to a statistically significant extent. Again this suggests that there is no real difference between pupils at years 3, 4 and 6 in the two countries. This finding is not what one would predict, and it would appear that the official Zambian policy of English as a medium of instruction for the first 4 years has not led to a superior performance in English reading.

4.2.3 School year

The mean scores reveal large differences between years 3, 4 and 6, as one would expect: in Malawi year 6 score approximately 14 points higher than year 4, who in turn score some 7 points higher than year 3; in Zambia year 6 score approximately 17 points higher than year 4, who in turn score nearly 7 points higher than year 3. These differences are statistically strongly significant, and indicate that pupils do improve with time - i.e. they are learning more the longer they stay in school. (Although this tendency is probably emphasised by the fact that weak pupils tend to drop out of school, and feature less prominently in the higher years.)

Despite these large mean differences it is apparent from the maximum (MAX) and minimum (MIN) scores that in both countries there is a large range of scores within each year, with an overlap between each year, and even an overlap between scores in year 3 and year 6. The detailed results reveal that this is true for all schools, although urban schools have higher maximum scores.

4.2.4 Gender effects

Table 10: Mean 1992 English Scores by Year and Sex in Malawi and Zambia

Year	Sex	N	Malawi	N	Zambia
3	Boys	82	8.56	67	4.00
3	Girls	86	6.78	84	5.75
4	Boys	76	14.42	78	11.05
4	Girls	77	14.50	70	11.84
6	Boys	92	30.69	79	27.14
6	Girls	66	27.06	74	29.12

There are small and consistent differences in favour of boys in Malawi, and in favour of girls in Zambia. However, the only statistically significant differences are at years 3 and 6 in Malawi, which accounts for the overall statistically significant gender difference in Malawi, with the boys scoring nearly 2 points more than girls. In Zambia girls scored overall just over 1 point more than boys (non-significant). However, the combined gender and location effect (see 4.2.6 below) suggests that difference may not be attributable to gender alone.

4.2.5 Location effects

Table 11: Mean English Scores by Year and Location in Malawi and Zambia

Year	Location	N	Malawi	N	Zambia
3	Rural	89	7.05	83	2.86
3	Urban	79	8.33	68	7.56
4	Rural	84	12.33	80	8.63
4	Urban	69	18.16	68	14.70
6	Rural	76	26.83	93	23.77
6	Urban	82	31.34	60	34.80

Urban-rural differences in both Malawi and Zambia are considerable at all years. However, they are not statistically significant, due to the small number of schools (not pupils) in the sample. Overall the differences are 3.3 (Malawi) and 7.2 (Zambia) in favour of urban pupils but again non-significant, for the same reason. It is highly likely, however, that real differences do exist between urban and rural schools, but a larger sample of schools would have been needed to confirm this statistically.

4.2.6 Gender and location effects

Combining the effects of sex and location yields the following table:

Table 12: Mean English Scores by Year, Sex and Location in Malawi and Zambia

Year	Sex	Location	N	Malawi	N	Zambia
3	Boy	Rural	44	8.43	40	2.75
3	Girl	Rural	45	5.69	43	2.95
3	Boy	Urban	38	8.71	27	5.85
3	Girl	Urban	41	7.97	41	8.68
4	Boy	Rural	42	14.17	49	10.04
4	Girl	Rural	42	10.50	31	6.42
4	Boy	Urban	34	16.97	29	12.76
4	Girl	Urban	35	19.31	39	16.15
6	Boy	Rural	49	29.02	46	23.21
6	Girl	Rural	27	22.85	47	24.32
6	Boy	Urban	43	32.58	33	32.60
6	Girl	Urban	39	29.97	27	37.48

Analysis of this table illuminates the gender issue in that it reveals there are no statistically significant differences between boys and girls in urban schools in Malawi. However, there are significant differences between the sexes in rural schools in favour of boys, at years 3 and 4, and a near significant difference in favour of boys at year 6 (p = 0.051). It would appear that it is particularly in rural Malawi that girls are at a disadvantage, possibly because they are expected to do more domestic work (e.g. head-loading water and maize, invariably done by girls, is very likely to take up more time in rural areas than urban ones).

In Zambia, on the other hand, girls consistently have higher means than boys, but not to a statistically significant extent. The exception is in year 4 rural schools, where Zambian boys have significantly higher scores than girls - this exception has no ready explanation.

The only inter-country difference that attains statistical significance is that of rural year 3, where Malawian boys have considerably higher means than Zambian boys. Indeed the latter's mean score is well below the chance score of 8, as is that of the rural Zambian girls. Observation of the Zambian rural pupils during test administration suggests this is due to the fact that they wrote very slowly and laboriously, and so were unable to complete 60 items which would have given them a better statistical chance of getting some items randomly correct (their scores in Nyanja are comparable, for the same reason).

4.3 Results of local language tests

4.3.1 Mean scores by year
4.3.2 Pupils from different language backgrounds
4.3.3 Correlations of English and local language results

4.3.1 Mean scores by year

There are striking inter-country differences here. The Malawian pupils, as one would expect, have achieved considerably higher means than Zambian pupils at all years in local language reading tests, and the differences in all cases attain clear statistical significance.

Malawian pupils have scored approximately 5 times more at year 3, 3 times more at year 4, and 2 times more at year 6 than their Zambian counterparts. The Zambian mean scores are below the chance score of 8 in years 3 and 4. These results suggest that most Zambian children have problems in reading their local language. While the direction of this finding is one that one would predict from the differing language policies of the two countries (Zambia going "straight for English" as a medium of instruction, and Malawi using Chichewa for the first 4 years), the extent of the difference is surprising.

Table 13:1992 Results of Chichewa Reading Test (Malawi) Items 1-60

YEAR	N	MEAN	SD	MAX	MIN	MEDIAN
3	169	16.11	8.48	43	1	15
4	153	29.14	10.36	52	7	29
6	158	45.66	8.29	59	22	47

Table 14:1992 Results of Nyanja Reading Test (Zambia) Items 1-60

YEAR	N	MEAN	SD	MAX	MIN	MEDIAN
3	151	3.31	4.44	37	0	2
4	148	7.50	5.06	27	0	7
6	153	21.63	12.17	58	3	20

4.3.2 Pupils from different language backgrounds

Pupils who claimed to use a language other than Chichewa or Nyanja at home did not appear to be penalized in the local language tests. There are no statistically significant differences in terms of home language in combination with gender or location in either country.

Table 15: Mean scores on Chichewa test of Chichewa and non-Chichewa home language speakers

Year	Chichewa home lang.	Other home lang.
3	16.08	16.26
4	28.88	30.33
6	45.63	46.11

Table 16: Mean scores on Nyanja test of Nyanja and non-Nyanja home language speakers

Year	Nyanja	Other
3	2.92	4.47
4	7.12	8.45
6	23.26	15.91

4.3.3 Correlations of English and local language results

For Malawi, correlations between the English results and the Chichewa results for all years are generally positive but not high. Those for Year 6 (by school) are as follows:

School M (urban):	0.56 (p<0.0001)
School S (urban):	0.40 (p<0.0110)
School N (rural):	0.42 (p<0.0368)
School P (rural):	0.40 (p<0.0191)
School W (rural):	0.73 (p<0.0022)

This suggests a slight tendency for relative performance in English to correlate with that in Chichewa. Certainly it does not suggest that competence in English is acquired at the cost of competence in Chichewa. In Zambia correlations of results in English and Nyanja for year 6 yielded the following:

School C (rural)	0.82 (p<0.0001)
School D (rural)	0.73 (p<0.0001)
School K (rural)	0.78 (p<0.0001)
School J (urban)	0.57 (p<0.0011)
School T (urban)	0.67 (p<0.0001)

These higher correlations likewise suggest that pupils who score relatively well in English also score relatively well in Nyanja. Again it does not support the view that reading proficiency in one language is gained at the expense of reading proficiency in another.