8. Overall Conclusions

8.2 Package Design and Content The consistent design of the PCCAL packages has proved very popular with the students and they repeatedly report that the packages are easy to use, interactive and flexible. Although the packages are self sufficient, when used with additional materials such as screen shots or structured exercises, this is almost always perceived to be beneficial and helps to improve effectiveness. The observation of students was not focused upon individual students and therefore the separate components of the interface have not been assessed in great depth. Overall impressions suggest that students do not use many of the features, such as map, aims and objectives and hot text as much as they might but this has not been quantified. Nevertheless, the vast majority of the remarks made in discussions and on questionnaires bear out the earlier evaluation findings, and show that the PCCAL package design and style is well received and intuitive for students to use, presenting no major barriers to learning. The report contains many quotations from students showing the generally favourable reception that each of the three packages received. Most students clearly enjoyed using them and found them very helpful. The findings show that the packages are well designed and appropriate for the task. However, each of them have been shown to have some minor deficiencies or areas in need of clarification. On Nomenclature of Organic Compounds students do not begin with sufficient awareness of the importance of punctuation in compound names and this needs more emphasis at the beginning of the package. Also, some guidance in the feedback to text input questions would assist with this and help to avoid some of the much reported frustrations students feel when attempting to get these questions correct. This need not involve a detailed analysis of every input but could take the form of suggestions or indicators as to where the mistake might have occurred or simply a reminder to check punctuation. Simulations of Experiments on the Isolated Guinea Pig Ileum may need some more clarification on the actual experiment which is not always clear enough for the students and the fact that the description of the experimental set up is in a separate section to the results generated adds to this lack of clarity. There was also some concern with the amount of repetition of dose response curves and very long activities which tended to de-motivate. In Problem Solving in Community Practice the students reported that on prescriptions and labels with information missing, sometimes they knew the correct answer but could not establish precisely where to click on the prescription or label to demonstrate this and this led to some frustrations. Those new to the subject also pointed out that they were asked to enter warning numbers for labels which they did not know, without looking them up in the BNF book. They had not been told the BNF would be required. Some teaching in the package or pointers to the BNF would alleviate this problem which was cited very widely.   8.3 Learning Outcomes Learning outcomes have been assessed using several measures - pre and post test content questions, self assessed levels of confidence and student views and responses to several questions on questionnaires and in discussion groups. In most cases these levels of confidence, pre and post test scores and student responses have all indicated the same positive trends. Students have recorded an increase in confidence, their scores have shown a significant increase in learning and responses have indicated that students feel themselves to have benefited from the teaching episode. This effectiveness is evident in most cases studies, regardless of the teaching methods used. Two case studies stand out as showing a different view, namely the two studies where students were asked to do the package in their own time. In one of these case studies, where first year students studied the Guinea Pig Ileum package, results showed very little increase in confidence and very little increase in learning. Comments were critical of the approach and the timing of the teaching episode. In the second study, where second year students did the Problem Solving in Community Practice package, increased levels of confidence were recorded, comments were generally favourable but students appeared to have deteriorated rather than improved their scores after the teaching episode. This may be just an unexplainable anomaly, but another factor singles these two case studies out from the rest. In time-tabled sessions, there were generally only one or two people absent from any group and the tutor was able to ensure that students undertook the required task(s). In the two studies mentioned here, very few from each group were shown to have completed the task as required. For the group studying Guinea Pig Ileum - 25% and the group studying Problem Solving - 39%. This in itself is a good indicator of how much less effective this approach to using CAL packages with students appears to be. Furthermore, it has been shown that in asking students to work on the material in their own time, the responsibility for ensuring the accessibility and viability of the technology is transferred to the student and the inevitable problems that will arise with this increase the likelihood of students failing to complete the task. One further issue should be mentioned here. Students expressed anxiety and a lack of motivation if the teaching and learning context was not clear to them. The more scene setting and guidance students had at the beginning, the more confident they felt. In her book "Rethinking University Teaching", Diana Laurillard lists five pieces of information students need to get the most out of a learning session - Why the topic is important and interesting, how it relates to other topics on their course, what they should already know, the learning objectives and how to approach the learning session (Laurillard, 1993, p212). This information is mostly included in the packages themselves but is not context specific and should not replace the integration required of the tutor to ensure that the material is directly relevant to the specific teaching and learning situation. In addition then, it is suggested that information such as the following be included - Why they are using CAL for this topic, why it is introduced at this point in the course, the objectives of this learning session, which sections (activities) are essential/not required. In addition to this, the anxiety expressed by a large number of students, in several studies, about the amount of time available also shows that students cannot usually assess this for themselves and again need some very specific guidelines on how much time each section as well as the overall package is likely to take. The information contained in this report on the time students have spent on the package could provide a basis for such guidance on the three specific packages in the future.   8.4 Learning Strategies On all three packages involved in this evaluation, we have seen examples of student approaches to the learning task which have challenged the view that students have existing study mechanisms which can be utilised and are appropriate. Students often appear to see the CAL experience as a highly individualised activity (unless guided to work in groups, students were rarely observed to do this spontaneously) and to see the CAL as a prescribed "auto tutor" rather than a tool which they can control. Unless the guidance given to students was very specific (for example a structured exercise), then most students adopted a linear approach to the material. The flexibility within each activity has been understood and utilised, as many quotations included in this report have shown, but within a package as a whole students have, in the main, failed to adopt a flexible approach or to assess for themselves the areas that they require to study. The high degree of suggestibility shown by students, mentioned earlier in the study on Nomenclature, implies that we cannot assume students will know how to approach these materials and that teachers need to talk about the different possible strategies students could adopt or tutors would like them to adopt in studying them. Strategies for improving student study methods need not rest with the tutor. Some modifications to the packages themselves could be considered. The PCCAL packages studied in this report have not contained any overviews of the material beyond the main menu structure. An overview might be an appropriate way of assisting students to select suitable areas for study and rejecting those that they are already familiar with. Another approach built-in to the packages, might be to begin packages (this may not be suitable for all, but perhaps for revision purposes) with a pre-test, to act as a diagnostic tool to enable students to be more targeted. Whilst it is accepted that in some cases, tutors may wish students to go through all the material, by encouraging more adventurous study habits, students will be more likely to explore other packages and gain some benefit from this increasingly large bank of CAL resources that students have access to.   8.5 Future Work Thelarge amount of data collected for these studies could be examined in more detail than has been possible to date. The link between individual learning objectives and corresponding test scores could be examined , especially for groups using the packages for revision purposes as this may help to identify specific areas of the package where students improved and any weak spots in the packages which have not yet come to light. Some examination of the effects of a low or high pre-score has been undertaken but this could be developed further and again linked to specific parts of the package .There is also much further work that could be undertaken on the qualitative data. Further analysis could be carried out to quantify specific problem areas cited and the students' understanding of the context from the responses to: "What did you expect to learn from the package?". In general the PCCAL house style is well received but, as mentioned previously, there is some evidence that some elements are rarely used and during discussion groups students admitted that they often ignored the aims and objectives sections of the package. Some detailed observations of two or three students working through a number of PCCAL packages (not necessarily the same students) would provide a good basis for evaluating the house style in more depth and ensuring that all components are useful and appropriate. The PCCAL portfolio is large and continues to grow. The need for evaluation of these products grows alongside it. The prototyping approach adopted by the PCCAL development team has been very productive and is an effective way of ensuring that materials are developed quickly and easily. Whilst the content of the package is evaluated by academic specialists and the products are reviewed for technical and operational errors, no educational evaluation has been carried out until after the products have been released. Formative educational evaluation, using small groups of students should be established to assess the effectiveness of the materials from an early stage. This would build upon the work begun during this evaluation project and ensure that PCCAL products continue to improve and develop.