4. Methods

4.3 Limitations

A number of limitations or constraints have already been mentioned, such as available resources, size of the PCCAL portfolio and the need to represent the whole consortium. One further constraint on our endeavours was undoubtedly the need to ensure the goodwill of the students involved. This had an influence on the length and nature of all evaluation measures, but in particular the content questions. It was important to avoid students losing patience with the study and the demands we were making on them, thereby failing to put the required effort into each of the tasks. In order to ensure this, content questions and questionnaires were kept deliberately short and discussion groups were limited to twenty minutes. The tests were devised by the academic subject specialist who has been involved with the development of the package, and based on the learning objectives. These test items were then reviewed by a panel of experts, representative of the institutions involved in the study in order to assess their comparability and accuracy. Appropriate adjustments were made before the tests were finalised. However, it must be recognised that these tests represent a very limited measure of the learning taking place and should be seen as an indicator of learning gains rather than a definitive measurement. Again with the goodwill of students in mind, observation of groups was kept as unobtrusive as possible, with observers sitting at the side, monitoring general activities such as requests for help or note taking. More detailed observation of one or two students, cataloguing their progress through the program would have been illuminating, however it would also have intruded more into classroom dynamics and possibly have acted as an impediment to those under such close scrutiny. Finally, it is appropriate to mention the issue of control groups. When evaluating a CAL package, the notion of setting up a control group is often considered as it was in deciding on the methodology for this project. However, most of the PCCAL materials are used to complement "traditional" teaching such as lectures and therefore cannot be seen as offering an alternative approach to the same teaching. If a control group had been engineered, then certain groups of students would have been denied access to the whole range of resources available within the undergraduate programme. Furthermore, the evaluation is taking place in a classroom situation, where there are many other variables that will affect learning, for example, the teacher's approach and development of the subject, teacher's interaction with different groups, use of other aids, level of discussion and note taking. Therefore, it is probably of little value to attempt to compare learning achieved by CAL with that achieved by lectures or other means, as the learning situation to which each group is exposed is unlikely to be comparable.