8. Overall Conclusions

8.1 Strategy and Methods As mentioned at the outset, the approach to this study was broad rather than deep and whilst this has enabled us to evaluate the PCCAL materials and approach using a number of case studies, the results represent a set of indicators rather than a comprehensive and exhaustive study. Nevertheless the approach has been able to maximise the time and resources available and involve a large number of students (854 cases) at five universities within the UK, incorporating their views and opinions in addition to more quantitative data. Conducting evaluation programmes across whole year groups has in itself been a valuable exercise and a larger study has helped to consolidate and add weight to the findings. Some aspects of the strategy could be improved for future studies. The focus has been almost exclusively on the students and the classroom setting. More attention could have been paid to eliciting the specific views of the tutors for each group. This was done in a very general way, working largely with the PCCAL representatives who were often themselves the tutors for the chosen packages. In some cases however, where the tutor was not the PCCAL representative, contact between the tutor and the evaluator was not established before the timetabled event and although the aims and objectives for each package had been recorded at the outset, we should have determined more precisely the aims and objectives for each teaching episode in order to be able to assess the effectiveness of the different teaching methods against their own criteria. There was also an intention to interview tutors after analysing the data and to elicit their opinions on the teaching episodes and to formulate specific questions raised by the results. However, this was not possible due to the large amount of data and subsequent analysis required and the resultant lack of time available on such a short project. The evaluation measures used appear in general to have served their purposes. In particular asking students to indicate their levels of confidence before and after has been very useful in assessing learning effectiveness from the students' viewpoint and has acted as a good counterbalance to the pre and post test quantitative measures. Some of the early questions on the pre-test questionnaire concerning students' technological awareness and confidence could have been eliminated as these showed in every case study that the overwhelming majority of students have sufficient computer experience to undertake the task and are therefore probably unnecessary questions to ask. Open-ended answers on questionnaires, observation of students and comments made in discussion groups have all proved invaluable in building up a picture of how students have approached the task and where the strengths and weaknesses of the packages and teaching methods lie. There is much further analysis of the data that could be undertaken, some of which will be outlined at the end of this section. The limitations imposed upon the design of the pre and post test content questions, as already mentioned, have ensured that the evaluation could be time-tabled into a realistic time frame and have maintained the students' willing co-operation and continuing motivation throughout. They are a very basic set of indicators of pre and post knowledge and skills in the subject and their limitations may have been more pronounced in the studies using the packages for revision purposes where learning gains may be smaller and more subtle. Whilst the length of each test appears to have been appropriate, the content questions should in future be more finely graded, especially if evaluating the package in use for revision. This leads to the conclusion that there is great skill required in setting such tests and that the evaluator needs to give very specific guidance to those academics setting the tests and reviewing them.