6. Discussion
6.1 Methodology
Section 3 of this report describes in some detail the full process followed in carrying out this research. In particular it reported on the processes of selecting languages and encyclopaedias for comparison with Wikipedia articles, the sampling strategies for student and academic expert reviewers, the selection of articles and the review process.
We have nothing to add in terms of the decisions made for which languages to study, beyond saying that nothing which occurred subsequently in the process suggested that these were inappropriate choices. Considerable care was spent in trying to select the online encyclopaedias that were most appropriate for comparison with Wikipedia, in terms of the nature of content, style and readership. We have no doubt, in retrospect, that even given the difficulties in finding articles of equivalent focus and length to Wikipedia articles on a number of occasions, we could not have made better choices with respect to the three languages chosen.
The processes of establishing the samples of students and academic experts proved to be largely appropriate and productive. Achieving an initial pool of 116 students provided an excellent foundation for the selection of 24 students (12 as the main cohort, and a further 12 as back-up). Given the time pressures and commitments of such high level students, we were pleased that we were able to select this number of committed and capable people who had such a key role to play in the research, both in terms of identifying academic experts and in carrying out their own reviews of articles. The identification of experts was carried out rapidly and productively and resulted in a generally satisfactory outcome in terms of numbers and quality of reviews. However, we were not always able to meet our target of at least two academic experts for each article (as against one student and one academic expert, a minimal requirement that was met on every occasion). Given the considerable efforts and enthusiasm of all involved, especially the students, this does raise serious questions about the viability of a significantly large-scale study in the future.
Similar questions arise from the difficulties encountered in selecting and preparing pairs of articles for comparison. In the event, it proved extremely difficult to locate encyclopaedias which provided articles that could be compared with Wikipedia in a number of the specialist areas of experts. We had no alternative but to select topics that were broader and less specialist than we would have preferred and which did not match the expertise of academic reviewers as closely as originally intended. This, once again, has significant implications for any future scaling up of the research, although we do believe that the actual process of comparison proved to be extremely valuable (discussed further below).
Another potential problem in preparing articles concerns the inclusion of additional material such as photographs, charts and tables. Images, as was explained in Section 3, had been removed from articles and presented separately as part of the anonymisation process, so that although reviewers were not able to see images in context, they were able to comment on them. A few reviewers commented on the issue of imagery as a concern especially for
50
