6. Findings on student learning

The effects of the new science curriculum materials on student learning have been considered in two ways:

(a) the effects on learning through these new materials compared with the SWISP approach in terms of science conceptual understanding;

(b) the difference in the effect of SWISP and the new approach on students' attitude to science learning.

(a) the learning effects of the Matsapha approach compared with the SWISP approach in terms of science conceptual understanding

An experimental group of just over 300 students from 8 classes at 4 schools used the Matsapha materials for the units 'Air and Life' (AL) and 'Electricity' (EL). A comparison group of almost the same number of students from 8 classes from a different set of 4 schools used the SWISP materials which covered the same science concepts. Each group represented rural, peri-urban and urban schools. No attempt was made to stratify the sample of schools on the quality of their examination results.

For each unit, test instruments each with three components were administered as an end-of-unit test by the class teachers of both groups. Firstly, each test consists of around ten standard, short-answer, examination-type questions designed to measure concept achievement. The questions required recall (e.g. what is the name of a negatively charged particle; a test for carbon dioxide), understanding (given a parallel circuit with bulbs and switches; which bulb will light if a specific switch is opened) and application (explain if one can use a low resistance wire in an electric kettle; explain how to discriminate between two colourless gases). A standard marking scheme was applied. As an alternative to pre-testing, a benchmark of performance of both groups was established by testing both on a different topic. This topic ('Detecting the Environment' (DE)) was taught to both groups with the SWISP materials.

In addition to the standard attainment questions each test paper contained questions to collect data on students' abilities in problem solving. For example, one question provided pupils with a set of related statements containing true science ideas and asked them to select the statements needed to solve the given practical problem and then to propose a solution. A third form of question asked pupils to write a plan of an experiment to test a hypothesis. Answers to these questions were coded and provided evidence of students' investigative ability.

In order to compare student responses for the various units, only the respondents with a complete set of answers to all three tests were considered in the following analysis. Thus the experimental group comprises 104 students from 3 schools, and the comparison group includes 184 students from 3 schools.

Table 5 Test scores of experimental and comparison groups for Air and Life, Electricity and Detecting the Environment teaching units.

		Experimental group (n=104)	comparison group (n=184)
Air and Life (maximum 15 points)
	mean	6.06	6.66
	standard deviation	2.36	2.29
	standard error of mean	0.23	0.17
			t=-2.09
			p=0.04
Electricity (maximum 11 points)
	mean	4.10	4.43
	standard deviation	2.35	2.27
	standard error of mean	0.23	0.17
			t=-1.17
			p=0.24
Detecting the Environment (maximum 11 points)
	mean	5.66	4.90
	standard deviation	1.89	1.80
	standard error of mean	1.80	0.13
			t=3.34
			p<0.01

Table 5 presents an overview of the data on pupil attainment. A striking point is that the mean level of pupil attainment is low and often very low. It also shows that there is no significant difference at the 5% level between the performance of both groups on the Air and Life unit or on the Electricity unit. For the SWISP unit on Detecting the Environment. however, the experimental group significantly out-performs the comparison group though the mean is just above the 50% mark. This seems to suggest that the experimental group, learning through the Matsapha materials, achieve less than might be expected.

When comparing the performance of students in each of the experimental schools with the performance of the whole comparison group (not shown here), the achievement on the baseline (DE) test is very similar: achievement in each experimental school is significantly better than for the comparison group. However, the achievement for the AL and EL units falls in particular in those schools, where the teacher only partly adopted the Matsapha teaching approach.

The attainment of the students, using the Matsapha materials, is similar to a comparison group, using the SWISP materials. However, the experimental students are under-performing as they show significantly higher attainment in the benchmark test. This under-performance is less pronounced for students of teachers who have internalised the new teaching approach more fully. This suggests that student performance may improve when teachers are given more time to gain confidence in using the new materials.

In designing the Matsapha materials one assumption was that they might appeal more to girls than the more traditional SWISP workbooks, and so aid motivation and hence learning. With this in mind the attainment was looked at again in relation to gender differences. Table 6 presents data on attainment of boys and girls in the experimental and comparison groups for the AL, EL and DE units.

Table 6 Test scores of boys and girls in the experimental and comparison groups for Air and Life, Electricity and Detecting the Environment teaching units.

		experimental group (n=104)		comparison group (n=184)
		boys (n=46)	girls (n=58)	boys (n=95)	girls (n=89)
Air and Life (maximum 15 points)
	mean	5.91	6.17	6.51	6.80
	standard deviation	2.40	2.35	2.22	2.37
	standard of error of mean	0.35	0.31	0.23	0.24
			t=-0.55		t=-0.87
			p=0.58		p=0.38
Electricity (maximum 11 points)
	mean	4.43	3.83	4.56	4.29
	standard deviation	2.32	2.37	2.19	2.36
	standard error of mean	0.34	0.31	0.22	0.25
			t=1.31		t=0.79
			p=0.19		p=0.43
Detecting the Environment (maximum 11 points)
	mean	6.35	5.12	5.28	4.49
	standard deviation	1.90	1.72	1.74	1.78
	standard error of mean	0.28	0.23	0.18	0.19
			t=3.41		t=3.04
			p<0.01		p<0.01

Table 6 shows clearly that while the boys do better than the girls in both samples in the Detecting the Environment (DE) test, their performance in the other tests is indistinguishable. This indicates that, indeed, the girls' performance improves relatively to the boys' when using the Matsapha materials, but so does their performance when using the SWISP materials.

The notion that achievement by boys and girls is affected differently by the teaching approach does not find support. Similarly there is no indication that any variation in the level of teacher acceptance of the Matsapha approach is reflected in different outcomes for boys and girls.

But what of the ability of pupils to apply their science knowledge by selecting appropriate science ideas to solve problems or their competence to design valid experimental tests? This is reported in detail elsewhere for the EL test questions (Lubben et al, 1995). Suffice it to state here that no difference between groups is apparent in pupil competence in identifying and using science ideas relevant to a every day circuit, despite this being an aspect of the Matsapha materials but not of the SWISP materials. Essentially, this competence is randomly distributed across the ability range for both the experimental and comparison groups. More strikingly, when students were asked to sketch an electrical circuit, the data show that the circuit diagrams bear no relation to the science statements selected as relevant to the problem.

The Matsapha materials have not improved students' competence in applying science ideas to solve everyday problems. The assumption that the application of relevant science ideas to solve a practical problem is facilitated by guiding students to consciously select relevant ones from a set of true science statements, is not supported by the data. Students do not see any relationship between the two tasks.

With regard to performance in designing practical investigations, the results from pupils' attempts to plan an experiment to see if black covered wire is a better conductor of electricity then white covered wire were analysed. The outcomes, generally, are disappointingly poor and equally so for boys and girls. They are not differentiated in favour of pupils who had used the Matsapha materials. Just over half of the students in each group offer an (invalid or valid) experimental design, but a quarter find it adequate to merely justify an expected outcome without any experimental verification. Similarly, Knamiller et al. (1995) found for a sample of Tanzanian science teachers that around two-thirds responded to an experimental design task as if it were a knowledge question.

However, the very small percentage of students indicating the need to control extraneous variables, and paying attention to increased accuracy of measurements was higher in the experimental group than in the comparison group. This can be considered a modest success of the Matsapha materials. Whereas almost half of both the low-achieving and high-achieving students in the comparison group suggest a valid investigative design, this percentage is slightly higher in the high-achievers in the experimental group but dramatically lower in the low achievers in this group.

The Matsapha materials have helped the abler students in formulating valid investigative designs, but confused the less able ones. The technological approach also encouraged some greater attention to procedural aspects of practical work. One in every four students in the experimental and comparison groups respond to an experimental design task as if it were a knowledge question.

(b) students' attitude to learning activities for science with a technological approach

Almost 300 students from 8 classes from urban, peri-urban and rural schools were taught through the Matsapha materials, and returned a questionnaire after completing each of the units. The first part of the questionnaire listed thirteen types of learning activities included in the Matsapha materials, and asked students to identify the three activities they most liked, with an explanation of their choices. Some of these activities are specific to the Matsapha materials, others are common to many other materials. The second part of the questionnaire asked them to select the three least liked activities from the same list. This section was completed by only about half of the sample for each unit, but these numbers are still considered large enough to provide a representative sample.

The frequencies with which various activities were ranked amongst the three most, or three least, liked learning activities were counted. The reasons provided by students were allocated to one of a small number of coding categories for each activity. The completeness and distinctiveness of these coding categories, and their application, were validated by peer review.

Detailed findings are reported in Dlamini et al. (1995). For each of the 13 activities, the percentages of students who included the activity amongst their most liked ones have been transformed into grades. The first four ranked activities are graded as high, the bottom four ranked activities as low, and the remainder as medium. The same has been done for the percentage of students including each activity amongst their three least liked ones. Combining these grades and taking cognisance of the fact that a low grade may mean indifference towards the activity concerned, a classification of student perceptions of each learning activity can be made. This is reflected in Table 7 below.

Table 7 Classification of student perceptions of learning activities.

activity	most liked grade	least liked grade	student perception
c. group discussions	high (53%)	high (28%)	contentious
k. reading stories	high (45%)	medium (22%)	mainly liked
g. listening to the teacher explain	high (43%)	medium (21%)	mainly liked
a. doing a play	high (29%)	medium (20%)	mainly liked
1. recording experimental results	medium (15%)	low (18%)	slightly liked
m. solving practical problems	medium (15%)	medium (21%)	mixed
d. group practical work with instructions	low (11%)	low (12%)	indifferent
b. explaining what happens around us	low (11%)	low (14%)	indifferent
j. observing practical demonstration by teacher	low (6%)	low (17%)	indifferent
e. identifying science in everyday life	low (4%)	low (18%)	indifferent
h. planning an experiment	medium (17%)	high (26%)	mainly disliked
f. labelling diagrams	medium (17%)	high (33%)	mainly disliked
n. writing reports	low (10%)	high (33%)	disliked

Table 7 shows that group discussion is a contentious activity since it ranks highly amongst the most popular and amongst the least popular activities. Positive views of group discussion are based on its perceived benefit in terms of gains of understanding from sharing and refining ideas. Those who do not favour this activity do so because of problems with social interaction in the groups. Group discussion is encouraged in the Matsapha materials, as it is in the SWISP. However, in SWISP little teacher guidance is provided and these activities can be more easily avoided than in the Matsapha materials.

Group discussion as a learning activity is contentious as strong positive and negative opinions are expressed by a large proportion of the sample. In order to optimise the beneficial effects of group discussions, teachers need to provide clear guidelines on group interaction, i.e. that all members should have a chance to bring in their suggestions, and that a group does not necessarily have to reach a consensus but may report two or more opinions.

Students mainly liked listening to the teacher explain (they attract only a medium percentage of respondents who include them as least popular) with a very strong emphasis on the perceived knowledge gains. A large proportion of the students suggested that the teacher provided focus. Apparently, both boys and girls in this cluster are more concerned with identifying what needs to be learnt rather than gaining understanding of the concepts to be learnt. The explanation of the teacher provides confidence to students that what they learn is the 'right thing', both in terms of the correct science and the required science for the examination. By contrast, students who identified listening to the teacher explain amongst the least liked activities mostly reasoned that they gained a better understanding of the science by 'doing' rather than 'listening'.

Listening to the teacher explain is mainly liked as a learning activity. Teacher exposition is played down in the Matsapha materials. It seems, however, imperative that at the end of contextualised lessons a summary of the scientific concepts to be learnt is provided which the teacher can reiterate and which will provide the students with confidence that the 'right thing' to learn has been identified for them. The final application phase of most Matsapha lessons when the initial every day incidence is re-visited provides an opportunity for consolidating the understanding of this science.

On balance, reading stories and doing plays are mainly liked. Reasons for the popularity of both activities focus primarily on their perceived support of increased understanding of the science ideas, and less so because the interesting contexts they provide. A considerably smaller proportion classifies these learning activities amongst the least liked activities. Their reasons usually also focus on knowledge acquisition and indicate that the contextualised stories or plays do not provide anything students need to know, or provide only knowledge students already know!

The stories and plays as lesson introductions are specific to the Matsapha materials and represent ways of contextualising the science content which are apparently well appreciated. Reading the stories and doing plays as learning activities are, however, primarily appreciated because they help the understanding of science concepts, rather than link these concepts to everyday life.

A separate observational classroom study into student interest and lesson participation related to the contextualised aspects of the Matsapha lessons (Lubben et al., forthcoming) shows several students indicating surprise that everyday instances may be combined with classroom science. Student interest in learning about the science behind the common applications was raised by three types of contexts. The classroom observations show that the choice of interest-generating contexts does not need to be limited to those from students' own experiences, but includes also historical applications, such as traditional ways of storing maize to prevent germination, and covers advanced technological applications such as the use of oxygen masks used during aeroplane emergencies are appropriate. Students need to be able to relate to the context but familiarity is not a requirement. Interest increases for contexts in which students are perceived to be able to offer expertise themselves, such as the workings of electric hot combs, or welding procedures. Contexts which are considered contentious, such as the causes of lightning, or traditional eating or sexual habits also create interest.

Observational studies have shown that student interest in learning about the science behind the common applications was raised by the following three types of contexts: (i) contexts students can relate to including familiar experiences but also historical and high-tech industrial settings students have heard about (ii) contexts where students perceive themselves (or their peers) to have expert knowledge; (iii) contexts students perceive as contentious

The findings show that contextualisation has a potential for encouraging student participation in science lessons, although students frequently remain focused on suggesting solutions for the incidence, rather than providing a (scientific or non-scientific) explanation for their suggestions. However, a high pupil commitment to positions taken in contextualised discussion has been noted.

Several instances are reported where increased participation provides the opportunity for teachers to identify misconceptions amongst their students. If teachers are able to make use of these diagnostic possibilities they can more easily build new learning on existing understanding. However, the findings indicate that, in order to make optimal use of the potential of contextualised lessons, teaching styles need to move away from the traditional teacher-centred approach.

Data show that contextualised materials stimulate student participation and provide the opportunity of identifying student misconceptions. If the full range of benefits of contextualised lesson material is to be realised, in-service activities to introduce such materials must not only aim at familiarising teachers with the curriculum but also at promoting acceptance and adoption of a learner-centred approach to teaching.

Contextualised teaching approaches such as in the Matsapha materials depend on the provision of large amounts of information, either as a story, a play, a role play, tabular data, or otherwise. In each case, limited proficiency of second language learners provides additional hurdles, and requires extra time to digest the information given, and to respond to it. Although many teachers blamed the slow pace of their teaching on the requirement of student participation in many of the learning activities, the lack of language mastery equally caused delays.

Teaching with contextualised materials is slow, because of its novelty, the requirement of student participation and the high demand on processing verbal information. The latter is likely to remain a lasting reason for a slow teaching pace with such resources.

Returning to the appreciation of the various learning activities (Table 7 above), labelling diagrams and writing reports are two activities which are clearly disliked. A high percentage of students include them amongst their least liked activities, and only a medium or low percentage lists them as their most liked activities. The reasons provided confirm in both cases that students are uneasy about committing themselves to definite answers such as are required in both of these activities. This type of reaction is in line with the student demand for authoritative teacher exposition of 'truths'. Neither of these activities is specific to the Matsapha materials, although they both feature.

Learning activities such as labelling diagrams and writing reports are clearly disliked as both activities are perceived to be closely related with assessment.

There are four activities about which students do not have firm opinions. These are listed as popular or unpopular by a low percentage of students. Amongst these are activities such as explaining what happens around us and identifying science in everyday life, both expressions of the application nature of the Matsapha materials. Although the research instrument was field-tested, there is some anecdotal evidence that a number of respondents were unclear about these descriptors, which may explain, in part, why very few firm opinions about these activities have been submitted. Further research is needed.

The responses to the investigative nature of the Matsapha materials may be found in the appreciation, or lack of it, of activities such as solving practical problems and planning an experiment. A discussion of student perception of these activities follows below after the consideration of the gender aspects.

Although students value learning activities representing the contextualised nature of the materials, they are indifferent to those activities emphasising the application aspects, and, by and large, dislike the investigative activities. In the latter case gender differences seem to have an influence.

Thus far we have looked at perceptions on the various learning activities for the whole sample. However, it is of interest to identify the activities which are particularly attractive to girls. The responses have been separated according to gender and are presented in Table 8 for most liked and least liked activities.

Table 8 Frequency of learning activities listed amongst three most liked and least liked by gender, (percentages in brackets)

activity	amongst 3 most liked				amongst 3 least liked
	after AL unit		after EL unit		after AL unit		after EL unit
	boys (n=127)	girls (n=144)	boys (n=134)	girls (n=157)	boys (n=55)	girls (n=82)	boys (n=59)	girls (n=87)
a. doing a play	32 (25%)	47 (33%)	29 (22%)	55 (35%)	16 (29%)	14 (17%)	14 (24%)	12 (14%)
b explaining what happens around us	16 (13%)	16 (11%)	14 (10%)	16 (10%)	8 (15%)	9 (11%)	8 (14%)	16 (18%)
c. group discussions	63 (50%)	78 (54%)	76 (57%)	80 (51%)	15 (27%)	23 (28%)	17 (29%)	25 (29%)
d group practical work with instructions	10 (8%)	17 (12%)	17 (13%)	20 (13%)	7 (13%)	10 (12%)	5 (8%)	12 (14%)
e. identifying science in everyday life	7 (6%)	7 (5%)	7 (5%)	3 (2%)	12 (22%)	14 (17%)	12 (20%)	14 (16%)
f. labelling diagrams	33 (26%)	22 (15%)	17 (13%)	23 (15%)	18 (33%)	26 (32%)	20 (34%)	29 (33%)
g. listening to the teacher explain	50 (39%)	58 (40%)	61 (46%)	70 (45%)	10 (18%)	19 (23%)	15 (25%)	15 (17%)
h planning an experiment	25 (20%)	23 (16%)	31 (23%)	19 (12%)	15 (27%)	16 (20%)	13 (22%)	29 (33%)
j. observing practical demonstration by teacher	10 (8%)	4 (3%)	11 (8%)	9 (6%)	4 (7%)	20 (24%)	10 (17%)	13 (15%)
k. reading stories	65 (51%)	75 (52%)	39 (29%)	72 (46%)	13 (24%)	17 (21%)	15 (25%)	19 (22%)
l. recording experimental results	21 (17%)	19 (13%)	24 (18%)	21 (13%)	12 (22%)	16 (20%)	10 (17%)	13 (15%)
m. solving practical problems	9 (7%)	19 (13%)	33 (25%)	23 (15%)	16 (29%)	18 (22%)	12 (20%)	13 (15%)
n. writing reports	8 (6%)	12 (8%)	12 (9%)	26 (17%)	17 (31%)	30 (37%)	16 (27%)	30 (34%)
TOTAL:	127	144	134	157	55	82	59	87

The first five columns of Table 8 present the most liked activities for boys and girls. This shows that girls were strikingly consistent in their preferences across the two units. Girls seem to like learning activities regardless of the topic being taught. The few cases of significant differences in the popularity of the various activities in the different units are due mainly to the boys. Labelling diagrams and reading stories were significantly less popular in the Electricity unit than in the Air and Life unit because they were less liked by the boys. Planning an experiment is liked more in the Electricity unit than in the Air and Life unit because of the greater enthusiasm of the boys. Only the significantly higher level of popularity of writing reports in the Electricity unit is due to its greater support from girls.

The responses for each unit may now be compared for boys and girls separately. In the responses to the activities in the Air and Life unit, only one gender difference was significant. More boys than girls listed labelling diagrams amongst their favoured activities (chi-squared value 4.78). However, for the activities in the Electricity unit, the responses showed four significant gender differences. The high support of girls for doing a play and reading stories was common to both units, but the popularity of these activities amongst the boys was significantly lower for the Electricity unit (chi-squared values 6.31 and 8.60 respectively) than the Air and Life unit. Instead, significantly more boys listed planning an experiment and solving practical problems amongst their most favoured activities for this unit (chi-squared values 6.18 and 4.63 respectively).

In the results for the least liked activities, only one gender difference is significant. In the responses about the Air and Life unit, a significantly lower percentage of boys than girls included practical demonstration by the teacher amongst their least liked activities (chi-squared value 6.68).

Girls were very consistent in their preferred learning activities across the two units. Any differences in the appreciation of learning activities between the two units are due to a difference in popularity with the boys. Consequently, learning activities highly favoured by girls, such as doing a play and reading stories, may be considered as topic independent. These activities could be emphasised specifically to make science learning more attractive to girls and, in particular, offer an avenue to maintain girls' interest in 'boys' topics such as circuit electricity.

The investigation aspect is represented by activities such as planning an experiment and solving practical problems. There, a significant gender difference is found in the preferred activities in the Electricity unit. Both planning an experiment and solving practical problems are favoured significantly more by boys than by girls. Overall, we may conclude that the investigative nature of the Matsapha materials obtains a mixed response. While mainly disliked, it is most liked by boys studying Electricity. This seems to indicate that, in general, students are unwilling to take responsibility for independent thinking and acting unless the context is challenging enough to overcome this hesitance.

A significant gender difference is found in the popularity of investigative work, but only in one unit (Electricity). Boys respond more positively than girls to such learning activities as planning an experiment and solving practical problems. This is a fundamental change, away from the normal request for 'spoon feeding'. Additional research is needed to identify investigative contexts which are appealing to Swazi girls.