Straining Qualities: evolving metadata for Web-based learners

Jon Dron, Phil Siviter, Richard Mitchell, Chris Boyne

University of Brighton, UK

Abstract

This paper reports on the ongoing development of CoFIND (Collaborative Filter in N Dimensions), a Web-based system designed to form the basis of a self-organising learning environment, where individual acts of learners combine to organise a coherent system of relevant learning resources and experiences. The paper considers how a group of learners whose learning environment is mediated through existing web technologies and through CoFIND can manage their learning and become their own teachers. CoFIND is based around explicit exposure of metadata, making the learning needs of the learners visible through a process of establishing the qualities that they seek in a resource, rather than simply the facts that they seek to learn. The system organises itself through a combination of speciation, ordering and extinction that is in many ways akin to the process of evolution.

Keywords

web-based learning, self-organisation, education, collaborative filter, evolution, learning, metadata

Context

Educational systems, web-based or otherwise are (on the whole) complex systems and thus evolve. As Brad Cox writes, "what actually governs complex systems is rarely the industrial age’s notion of design at all. Rather, they evolve, shaped by an interaction in which system and environment minutely adjust to each other as biological organisms evolve within ecologies " (Cox 1997). Unfortunately, the forces that drive this evolution are not always the needs of the learners, but may be swayed by everything in their evolutionary environment from government policies to a University’s traditions. The Web has the potential to free us from many of these forces and discover new ways of lifelong learning, liberated from the traditional ties of lectures, classrooms and institutions that evolved in an age where they were the fittest solutions. We are seeking ways of encouraging the evolution of virtual learning environments through a process that actively promotes competition and selection pressure so that a better educational experience may be self-generated by its users.

Replacing the teacher

Teachers have a variety of roles in a conventional educational system. Amongst other things they provide:

• expert help
• a guided path through new materials and resources
• inspiration and motivation
• feedback and assessment
• the boundaries of the subject

We would like a web-based learning environment to duplicate and extend these areas of functionality. If not, then we must critically consider those areas of functionality, reflecting carefully upon their use in helping the learner to learn.

Self-directed management of learning using web-based and other Internet technologies

There are many resources available on the Internet that may either explicitly or implicitly help learners address almost any topic. We start by looking at ways that are already available to match the roles we have identified for a teacher.

Expert help

There is a great deal of freely and not-so-freely available help that is accessible through the Web or other Internet-based mechanisms such as mailing lists and newsgroups. In seeking the answers to any question a search through Deja.com (Deja, 1999) or some such search interface to discussion groups will often reveal the answer and a source of expertise for further help. Similarly, mailing lists, newsgroups and related discussion mechanisms can provide answers to almost any problem as well as a means of exploring a topic through discourse. However, users of such a system must know how to frame the question and understand the answer given. Such a level of understanding may be too great for beginners in a subject, and much time can be wasted posing irrelevant questions or questions at the wrong level. Even after overcoming these hurdles the correct question may lead to the wrong answer. We seldom know much about the credentials of the expert whose assistance we seek. Solutions such as the collaborative filtering reputation brokering system proposed by Chernenko (Chernenko 1997) might be of some assistance here, as would a vetted list of trusted experts as used by AGORA21 (Zellouf et al, 1999) but they are by no means commonplace.

An associated issue with most threaded discussion mechanisms is the problem of losing the thread. Experience suggests that a large bundle of messages displayed hierarchically becomes difficult to navigate and that learners tend to give up. Although sites such as Deja.com can help through providing a search mechanism it can still be difficult to follow appropriate threads at an appropriate level. Some of the most promising work in this area comes from researchers in Computer Supported Collaborative Argumentation as exemplified in the D3E system used in the Journal of Interactive Media in Education (JIME, 1999), where metadata is provided with the message indicating where it lies in relation to other messages (agree, disagree or neutral). Similar results are achieved by systems exemplified by GroupLens (Resnick 1997) that apply collaborative filtering technology to help identify interesting threads. Promising though such technologies are as a means of sorting threads, they do not fully address the changing needs of learners as they assume a slow-moving or static range of needs. Learning is about change and our needs of yesterday do not necessarily match those of today. An alternative approach is applied by Ackerman in his use of Answer Gardens (Ackerman & McDonald, 1996), where possible answers to questions are suggested by the computer and, should these answers be deemed unsuitable, the user presses an "I’m not happy" button to elicit further help from a human expert, whose reply is then added to the database of answers. This is an effective self-organising technique, although it relies upon the presence of a known group of experts within a subject area. If one of the things that we are seeking is such a known group of experts, or should those experts be unable to effectively communicate their expertise to us, then the system might not help us.

Requirement: to overcome difficulties finding appropriate and reliable help.

A guided path

The Web is a big place with massive amounts of information and data on almost any topic.  Finding information on any given topic is usually not too difficult, but identifying what is relevant or appropriate is much harder. Systems such as PICS may help in the future but are not yet widely implemented and rely on authors', not users' classifications.

Knowing where to begin and where to seek information at an appropriate level as our needs develop is central to beginning to learn about a subject. A key concern is the fact that there are many different approaches to learning the same thing, some of which will be more or less suitable for a given learner. It is assumed that we do not always learn best in the same ways as each other. It is also assumed, after Vygotsky, that as a learner learns, so the learner changes and so the learner's needs change (Vygotsky , 1962). Consequently, our needs are dynamic, and the path on which we started may not be the same as the path towards our final goal.

Requirement: to find a path through a subject or topic that matches our needs as a learner, not just the level but also the kind of resources that will help us.

Inspiration and motivation

There are many intrinsic and extrinsic reasons that we are inspired or motivated to learn. Knowing about a subject is not enough. Knowing why we should know about a subject is central to learning it. Understanding the relationships between what we are learning now and what we already know, placing it in a suitable context that emphasises its relevance, seeing where we could be if we learn this topic, how it might change us, all of these things are key roles for a teacher. Teachers help encourage intrinsic motivation and also manage some aspects of extrinsic motivation such as applying pressure, setting tests and so on. For some, surfing the Web is inspiration enough but it is often helpful to be guided by those who have gone before and who know why we must pass through grey and dull areas before reaching the goals we seek. Although the Web provides hyperlinks, these tend to embody a particular perspective which may or may not match our own. We may easily become lost in hyperspace, following links that may not be relevant to our needs. Adaptive hypertext (Brusilovsky 1996) takes us some of the way there, but only looks at the single qualitative dimension of prior knowledge to decide on the content provided. If we seek inspiration then we seek more qualities in a resource than that we have enough knowledge to understand it.

Requirement: to provide motivation and find links that match understanding and perceptions at a given time.

Feedback and assessment

In a world of established teaching methods where examinations and assessments become ends in themselves, it is easy to overlook the importance of assessment and feedback in forming the understanding of the learner. A critical part of learning is knowing what you have learnt and that you have learnt, assisting in the reflective process that forms an integral part of the Kolb cycle (Kolb 1984). It is not essential to understanding a subject to have received a grade, but it is useful to know where you are now. There are many mechanisms available through the Web that may help us to achieve this. Most notable of these are systems such as CASTLE (CASTLE, 1999), Hot Potatoes (Hot Potatoes, 1999) and any number of commercial systems best exemplified by Question Mark (Question Mark, 1999) that provide a mechanism for producing multiple choice and other fixed-answer questions. Sometimes we may take advantage of discussion mechanisms (mailing lists, newsgroups and so on) to play with ideas and get feedback. If we are willing to pay for it, we may even go for a full distance-learning course and get tutor feedback. There are some mechanisms that provide feedback to non-formalised questions based on lexical semantic techniques and even utilising word counts of reports, some of which give fairly accurate grades but none of which give particularly meaningful feedback (Whittington & Hunt, 1999). Systems such as SAFT Self-Assessed Free Text (Kjöllerström and Mårtensson 1999)   provide a model answer for comparison with the student’s own answer and show a possible route forwards for a limited range of assessable outcomes.

Requirement: to provide a mechanism for providing effective feedback that does not rely on tutor assessment and yet which is not undermined by the uncertain understanding of individuals in self-help groups or the limitations of machines .

Establishing the boundaries

As well as establishing a path, it is important to know the boundaries of that path and when you have left it. On the Web, the problem is seldom the paucity of information. Instead, the information must be filtered to fit it to our current learning needs. When we learn to read using a book by Dr Seuss, we do not necessarily need to know about iambic pentameters. There are many courses and modules available on the Web that provide topics and subject references, providing clear guidance on what lies inside and outside of a subject. Unfortunately, despite broad consensus in most subject areas there are usually variations between one expert and another as to the boundaries of a given subject, and it is often difficult to choose an appropriate course from the many that are available. There is very little help provided with this on the Web apart from personal seals of approval or the reputation of a known authority.

Requirement: To provide help with finding an appropriate path through a topic, ideally by combining the best of what is already provided, or at least by identifying a consensus of what is valuable and what is not.

A description of the CoFIND system

Figure 1: CoFIND's main screen showing some of its main features

CoFIND (Collaborative Filter in N Dimensions) is a Web-based system to support learners. It is aimed at relatively small focussed groups of motivated adult users with common learning goals. It has been developed to demonstrate methods of self-organising the combined individual actions of its users into a coherent and meaningful learning environment, bringing about collaboration without really trying. It works on the premise that the Web provides very effective means of communication and publication, as well as an extremely rich source of learning resources, but that it must be tamed to be of use to learners.

CoFIND is essentially a collaborative bookmarking system where bookmarks added by its users not only point to Web-based resources but also to anything that may be of use to learners, from books to films to people and places.  Metadata (known as "qualities") are supplied by its users to share opinions about resources with others. CoFIND is also able to originate resources, both through a threaded discussion forum and through a form-based option to publish simple Web pages. Resources and pointers to resources are:

1. added to CoFIND by its users,
2. categorised into one or more topics and
3. rated according to one or more qualities.

Once a resource has been added to the system it becomes available to other users, who may also rate it according to one or more qualities and categorise it into one or more topics. New topics and qualities may be added at will and applied to existing resources.

To assist with filtering resources, CoFIND incorporates a simple search engine.

Creating new resources

Although CoFIND makes a useful repository for categorised and rated shared bookmarks, it is also designed to support the creation of those resources:

1. users may add comments on any resource or topic, providing a mechanism for seals of approval, be they raw recommendations or expressions of disapproval
2. users may also use CoFIND to create resources, as plain text or HTML. These resources are stored in the CoFIND database and may be referred to by a URL. They are thus accessible to other users in exactly the same way as any other resource. This closes the feedback loop, allowing the student to change or modify the resource in response to ratings and comments from other users of the system
3. CoFIND incorporates a threaded discussion mechanism which allows a user to discuss any topic with other users of the system. Messages of particular interest can be added to the CoFIND rating system and rated/categorised like any other resource

The underlying technology on which CoFIND is based is Microsoft's Active Server Pages linked to an Access database, although it is planned to also port it to Lotus Notes.

Qualities

Qualities are the things that users value in a resource. Qualities are metadata. Typical qualities might include "useful", "good for beginners", "beautiful", "overview", "detailed", "accurate" and so on. Ratings for qualities may always be on a continuous scale of more to less. For instance, a resource may be more or less detailed, beautiful, good for beginners and so on. Ratings may be given to resources according to how well they match the selected qualities. Resources move up and down the list of returned results according to their accumulated ratings for the selected qualities. Thus, a given resource may be at the top of the list for a quality of "detailed" but at the bottom for a quality of "simple to understand". As more than one quality can be selected at a time, there is a vast range of possibilities for the order of the returned resources, creating a kind of variegated evolutionary landscape. The role of a quality is to provide a mechanism to recommend resources, rated according to a student's perceived needs. It provides a method of fuzzy categorisation.

There are no significant limits placed on the quantity and type of qualities that users may add to the system. However, were qualities allowed to proliferate without check they would soon become unmanageable and it would be hard to find those that are useful, a kind of "self-disorganisation" if you like. CoFIND is therefore designed with evolutionary principles in mind. Qualities have to compete for a space on a limited list with other qualities (currently set to twenty items, although this is still subject to experiment) and those that are not used fall to the bottom and eventually die. Early iterations of CoFIND used a timeout period of non-use to retire qualities, but it has since been found to be more useful to base this function on the ratio of logins to uses. Further selection pressure is applied by limiting the number of qualities that may be selected at once (currently set to three, although this is adjustable and may yet evolve), thus forcing the users to make choices about which are valuable to them. There is a feedback loop established between qualities and the resources to which they relate. Successful qualities are generally those which provide useful lists of resources. It should not take long to discover which qualities produce the desired results. This becomes self-reinforcing through two mechanisms:

1. successful qualities are more accessible, appearing at the top of the list for selection
2. apart from when adding new resources (when anything goes) it is only possible to rate resources according to the currently selected qualities

Thus we generate what starts as a positive feedback system, where success breeds more success, much as we see in other evolutionary systems. This leads to a system that fluctuates dynamically around a stable point and that has the capacity to evolve. This feedback mechanism is required as it is equally important that too much change is discouraged as too little. Kauffman and others have identified as a characteristic of complex evolving systems that they lie on the ‘edge of chaos’ (Kauffman, 1996). An excess of change leads to chaos, where no species can develop as the rules and environment change too fast (Kauffman calls this the 'Red-queen' regime, always running to stay in the same place), whereas too much stability results in unremitting stasis, without movement or improvement (Kauffman refers to this as the 'Stalinist regime'). One of the important areas of research in the development of CoFIND is to discover algorithms for ordering qualities that allow the edge of chaos to be reached without slipping into the Red-queen or Stalinist regimes.

Topics

Topics are treated like qualities in that similar evolutionary pressures are applied to them. The main difference is that topics are binary rather than scalar: a resource either belongs to a topic or not. Topics are to do with subject matter rather than what is valuable about a resource. A resource may be categorised as belonging to more than one topic, although a pragmatic choice for the purpose of simplifying queries has been made to only allow one to be selected at a time. The role of the topic in CoFIND is much the same as the role of a topic within a conventional course, providing an indication of the subject matter. It is intended that CoFIND will eventually incorporate mechanisms for relating topics to each other, allowing self-organised hierarchies, webs and sequences of topics to develop.

CoFIND as an enabler of self-organised learning

CoFIND is a system designed to help learners to learn. In some ways it is therefore taking the place of a teacher, although it does not necessarily exclude the teacher from the proceedings. Therefore, we will revisit our minimal requirements for a teacher to see how CoFIND provides an alternative:

Expert help

Requirement: to overcome difficulties finding appropriate and reliable help.

CoFIND's main function is to provide a mechanism to find the help we need. CoFIND supports learners helping each other. Pask observes that one of the best ways of learning is through teachback (Pask 1977). Equally, this fits well with Laurillard’s model of conversations (Laurillard 1994). CoFIND supports teachback both through its discussion mechanism and through collaborative bookmarking. Qualities help establish the level and kind of help that is sought whilst topics provide the areas of interest. Through sharing, rating and categorising resources learners communicate and construct knowledge and understanding.

CoFIND’s discussion mechanism provides a means for direct questions to be asked and answered by other users of the system, to the benefit of the questioner and the replier. The mechanism differs from conventional threaded discussion groups as the relevance of such conversations can be discovered through the process of quality rating and classification into topics. Appropriate and reliable help is easily identified through qualities and their ratings.

As learners find resources that help them to solve problems these are added to CoFIND. As any resource that can be referred to by an URL may be entered onto the system, not only are static and dynamic Web pages made available, so too are references to experts via such things as email addresses, newsgroups and even Web-based expert systems. The ability to create free text resources means that non-Web-based resources such as people, books, museums, TV programs and films can be considered in the same fashion. CoFIND is thus a repository for references to various forms of help. Once again, the rating mechanism makes it easy to identify reliable and well-thought-of resources.

A guided path through new materials and resources

Requirement: to find a path through a subject or topic that matches our needs as a learner, not just the level but also the kind of resources that will help us.

Through the process of rating and the provision of qualities learners are able to identify relevant resources that fit their particular needs at each stage of the educational process. We know that they are relevant because they have been considered so by other learners with similar needs and wishes.

Guidance is thus born collaboratively through interaction with the system. Different learners will discover different aspects of a subject and their combined efforts will structure that information into a whole which others with similar needs can follow. Currently this provides less of a path than a model of the current context, but future versions of CoFIND will allow collaborative linking of resources, creating a structured path to follow.  

Inspiration and motivation

Requirement: to provide motivation and find links that match understanding and perceptions at a given time.

The capacity to inspire was not one of CoFIND’s design goals. However, it can help and motivate users in at least three related ways:

1. It can provide a range of resources that are more relevant and appropriate to learn from than would be provided by conventional search systems and directories, avoiding less useful resources and thus making learning a more pleasurable experience. As with all pleasurable experiences, there is an incentive to get more of the same. It is thus helping to increase the learners’ intrinsic motivation to learn.
2. It provides a means to communicate with other users and to benefit from their experiences. Inspiration and motivation need not come from the professionals. The act of participating in a learning community provides greater motivation than working alone. This mechanism provides some extrinsic motivation to learn. It also helps to reduce one of the demotivational pressures identified by Herzberg, that of isolation. (Herzberg 1966)
3. As noted below, feedback and assessment can provide motivation through positive reinforcement. The ability to discuss and publish combined with a feedback mechanism that gives a fairly precise and anonymous evaluation of what is discussed and published may provide senses of what Maslow describes as belonginess and esteem (Maslow 1954). Of course there is always the danger that the opposite may occur. CoFIND does not embody tact.

Feedback and assessment

Requirement: to provide a mechanism for providing effective feedback that does not rely on tutor assessment and yet which is not undermined by the uncertain understanding of individuals in self-help groups or the limitations of machines .

By allowing users to add free text resources as well as providing a means of making Web-based resources available, work can be shared. Qualities provide the criteria for a collaborative assessment system, especially powerful as those criteria are generated by the combined intellects of the learners themselves. Along with comments and discussions, anonymous and collaborative quality ratings provide a powerful evaluation mechanism, where the combined critical understanding of the users of the system can be brought to bear on any resource. Individual variations in understanding are evened out as they combine with others. If required, this could be used to provide summative assessments, although this is not the intention nor the wish of the authors of this system.

The ability to comment or discuss issues related to a resource allows more subtle personal evaluations and critiques if required, more in keeping with traditional assessment methods. This mechanism is most suited to small groups where a personal seal of approval from a known fellow-user will count for more than that of a stranger.

Establishing the boundaries

Requirement: To provide help with finding an appropriate path through a topic, ideally by combining the best of what is already provided, or at least by identifying a consensus of what is valuable and what is not.

We are of the slightly contentious opinion that no one is better qualified than an individual learner to recognise what is and is not useful at a given stage of that learner's development. CoFIND assumes that there are commonalities between such learners and that these commonalities are captured by qualities and ratings applied to them. The fact that the available qualities have evolved to be the most commonly used ensures that the qualities are more likely to be relevant. As an increasing number of resources become available and the selection of those resources is honed by the appropriate use of qualities and classification into topics, so the limits of a given subject become more firmly established. The boundaries of the topic are established by an evolving consensus rather than an individual's expert opinion.

An interesting outcome of early experiments has been to discover at least some of what is sought in an educational resource for our test groups. A group of Masters students using CoFIND to help with a case study in network management came up with the following list: "informative", "useful", "free", "interesting", "of broad coverage", "accessible", "reliable", "a good gateway to further resources", "good for beginners", "searchable", "about firewalls", "a good starting place for designing networks", "about cold fusion", and "about FV" (the company used in the case study). Although some of these qualities seem to be of poor predictive or descriptive power (for instance "useful" and "interesting") qualities such as "reliable" and "a good starting place for designing networks" have clearly identifiable ecological niches. The iteration of CoFind that the students were using did not include topics, so it is interesting to note that some of the students were adding topic-like phrases as qualities. Towards the end of the assignment students were required to post their work to the CoFIND system, whereupon there was an  explosion of qualities along the lines of " Spot On" , "A good read", "brilliant!", "assignment" and "Top!". Although added in the last two days of the assignment, "brilliant!" went on to be used to rate other resources, implying broader scope than its original use (Dron et al, 1999). We hope that more widespread use and adoption of CoFIND will lead to more useful and generally applicable qualities.

There are several assumptions that have been made in the construction of this tool. Currently the best proof we have that they are valid is mainly anecdotal.

Five wild assumptions in search of a proof:

1: That groups and subgroups of learners share similar conceptual views of the world, at least when averaged out over time.

This is a manifestly dubious assumption, but it provides us with a number of interesting potential experiments as CoFIND is implemented across different cultures and subject areas. As Lakoff has demonstrated, even those categories that we take to be basic and self-evident are not necessarily shared with other societies or even with other individuals (Lakoff, 1987). It is assumed that one of two things will take place:

1. that speciation occurs and that different qualities will be used by different subgroups of users in an unequal fashion. This would be a desirable result, creating extra competition and a richer feedback system
2. that the differences average out. This would suggest homogeneity, providing less of the variation hoped for and fewer ecological niches. Should this occur it suggests that our algorithms for positioning qualities and resources may need tuning.

Should averaging occur, a potential solution might be to use a conventional Automated Collaborative Filter (ACF) to identify commonalities between users. Our experiments so far are inconclusive and this remains the subject of further research.

2: That groups remain homogenous over time.

CoFIND is targeted at groups of learners with shared learning objectives and assumes that a particular cohort will be using it. What happens if a CoFIND system develops with a single group of users, then another individual joins late? As with traditional forms of teaching the teaching material may be difficult to follow and irrelevant, the range of available qualities adapted to a particular group, not the latecomer. The problem is similar to that of shared categorisation and the solution is similar. What we would expect to see is some qualities being popular with some groups, some with others. Adapting the list of qualities to be more relevant might require a traditional ACF, or an algorithm that includes variables relating the age of qualities to the length of time a user has been subscribed to the system. Alternatively,  an archived history of system states could be kept, allowing new learners to get up to speed by following the paths trodden by their forbears. Once again, this remains an agenda item for further research.

3: That resources remain unaltered over time

A problem that faces any cataloguer of Web-based resources lies in the dynamic nature of the Web. Not only do resources go away, they also change. A reference that was useful yesterday may be useless or positively harmful today. For example, there is nothing to stop learners posting work on the Web then changing it completely in response to quality ratings. Indeed, this is probably to be encouraged. However, it does mean that there is a perceptible time-lag between a resource changing and the accompanying ratings changing too. We are watching out for this closely and may need to provide an ageing mechanism for resources and quality ratings to ensure that they remain constantly fresh.

4: That learners know best what they need

There is a case to be made that the teacher knows best. Indeed, this is the assumption which keeps those of us employed as teachers and may even sometimes be true. A teacher is aware of the big picture and may be able to help the learner to reach places they might not go to unguided. It is important to note however that CoFIND does not seek to replace the teacher in this role. Someone has to produce whatever resources are used, to provide the content with which the learner interacts. CoFIND is merely there to help the user decide which of many resources are best, to identify the best teacher.

5: That evolution results in survival of the fittest, and that the fittest in these circumstances are the resources that help us learn best.

If "good" use is made of qualities then the desired results may be achieved. However, this relies upon a significant degree of self-knowledge on the part of the learner, particularly about their own learning methods and styles. Without the differentiation provided through the multiple dimensions of qualities the fitter resources might be those that are not necessarily better learning resources but that are more colourful, better presented and so on. Given the choice between chocolate and cabbage, many people would choose the former. How can we evaluate what is fittest for learning? One of CoFIND's raisons d'être is that it is not only unethical but also impractical to perform the kinds of tests that would be needed to establish empirical evidence that one way of learning is better than another. We are all familiar with comparative tests that draw on interviews and test scores to provide some justification for assuming that one learning method is better than another, but the predictive power of such methods is limited, inasmuch as they cannot provide adequate controls. Like the weather, teaching is a dynamic system and is susceptible to small changes in original conditions. Thus although we can observe broad patterns, we cannot make predictions based on individual instances and experiments any more than we can say that because we went out without our umbrellas that it rained. Combine this with an extremely hazy idea of what constitutes success (grade-based arguments are circular and interview-or questionnaire-based methods bear only a passing resemblance to a study of learning) and we are left with a slightly circular argument if we are hoping to demonstrate the effectiveness of a methodology. If a resource has a large number of high ratings across a range of qualities we hope that this means that the resource is useful, but there is no guarantee that this is for pedagogically sound reasons. Although it would be interesting to investigate this issue further, it is currently not on our research agenda.

HCI issues

One of the hardest things we have found so far is to encourage learners to use the system. A common problem to be overcome by collaborative filtering systems is the cold-start phenomenon, whereby a collaborative system requires a certain level of input before its outputs are useful to its users. Once the system is rolling, users begin to see its advantages. The experiment cited earlier to support a case study with a group of forty-two Masters students over two weeks provided a list of over eighty relevant and useful resources, but the list of qualities grew slowly until the last few days, when the students added their own pages to the system for appraisal by the group. With the incentive of enticing other students to view their resources, the list of qualities grew rapidly. Without such an incentive, use of the system is not great. Although the interface has been refined since then and the process of adding qualities has been made easier, a similar group without a specific case-study-based goal has only added four qualities over a six week period, despite adding some forty-odd resources and a wide range of comments. More depressingly, a CoFIND system set up to support the needs of teachers using learning technologies has hardly been touched, whilst a Microsoft Exchange public group supporting the same group of users has been used to recommend sites, suggesting that users will use a familiar (if less functional) technology if it is available. Even within the cohort of students who have been using CoFIND, one or two have posted messages to the course newsgroup providing recommendations of sites.

A solution that we have attempted to implement is to position CoFIND as a personal bookmark repository, available across the Internet, with the added benefits gained from collaboration with others. To this end it is possible to select a personal view, where only the resources which the user has entered or rated are displayed. A similar philosophy has caused us to provide options for the user to customise the stylesheet and number of qualities, resources and topics displayed.  However, CoFIND still suffers from usability problems.

Providing the incentive to participate and making that participation easy, transparent or desirable has to be a key goal for any future developments. One way of improving things would be to simplify the process of adding resources, which currently requires the user to open a form and manually enter the resource's details.  If a user could browse freely through Web pages and press a single button to add those of interest to CoFIND then we could achieve a much more rapid acceleration from the cold-start problem than is currently possible. It is easy to understand why the security model for browser scripting languages prevents one page in a frameset from discovering details of the URL and title of another page from a different site within the same frameset, but it prevents us from easily implementing a single-click solution. We are therefore planning to implement a proxy server to redirect all page requests reached through CoFIND so that any page's title and URL can automatically be captured and added to the database.

Conclusions

Embodying evolutionary principles, CoFIND is itself an evolving system, a snapshot of which is captured in this paper. Based on feedback and experience with users we are adapting it constantly to allow it to better suit the needs of learners.

Over the course of the coming year, students from a variety of cohorts in a number of circumstances, ranging from those studying network management at masters level to under-graduates assessing each others’ work to a group of HCI students rating various screens according to their own generated criteria will test the system and provide us with feedback.

One of the most interesting outcomes so far has been a clue as to what learners seek in learning resources, although the slow growth in qualities prevents any notable conclusions being drawn. Collecting such metadata may yet prove to be the most useful function of this tool, particularly if it can be used by a wider audience than the small experimental groups it has so far been tested on.

We are still some way off providing a perfect environment that develops ecological niches to suit all of its users. As we incorporate subtler relationships between qualities, resources and topics so we expect to see the unexpected emergent behaviour, for patterns and ‘courses’ to develop that could not have been foreseen. From these vague beginnings we can perhaps see a glimmer of how an organisational intelligence, distinct from yet composed of the minds that form it will provide a new way of amplifying our understanding of the complex world in which we live.

References

Ackerman, Mark S. & McDonald, David W., 1996, Answer Garden 2: Merging Organizational Memory with Collaborative Help, Proceedings of the ACM Conference on Computer-Supported Cooperative Work (CSCW'96), November 1996, pp. 97-105, also online at http://www.ics.uci.edu/~ackerman/docs/cscw96.ag2/cscw96.ag2.html, (1998 April 5)

Brusilowsky, P(1996) A Tool for Developing Adaptive Electronic Textbooks on WWW , Proceedings of WebNet 96, AACE also online at http://www.contrib.andrew.cmu.edu/~plb/WebNet96.html   (1999 November 19)

Bush, Vannevar (1945), As we may think, Atlantic Monthly, republished as hypertext at http://www.isg.sfu.ca/~duchier/misc/vbush/vbush.shtml   (1999 November 19)

Calvin, W.H. (1997). "The Six Essentials? Minimal Requirements for the Darwinian Bootstrapping of Quality," Journal of Memetics 1

The CASTLE Toolkit Home Page,http://www.le.ac.uk/cc/ltg/castle (1999 November 19)

Hot Potatoes Home Page, http://web.uvic.ca/hrd/halfbaked (1999 November 19)

Zellouf, Yamina, Jaillon P, Girardot J J (1999) Providing Rated Documents on the Net, Proceedings of WebNet 99, AACE

Vitae

Jon Dron is a Senior Lecturer in Networking and Network Management for the School of Information Management at the University of Brighton. With a first degree in Philosophy and a Masters degree in Information Systems followed by many years in network and systems management and support and an active involvement with the Web since 1993 (including from 1995-97 with Phil Siviter as a team member of the W3Lessonware project developing Web courseware development tools for UKERNA), he is now a PhD candidate researching self-organising network-based learning systems.

The remaining authors are Jon's  PhD supervisors and advisor:

Dr Richard Mitchell (supervisor) is Professor of Computing for the School of Computing and Mathematical Sciences at the University of Brighton. His wide ranging research interests include design by contract and object oriented design, in which areas he chairs/is on various programme committees throughout the world.

Dr Chris Boyne (supervisor) is a Senior Lecturer for the School of Information Management at the University of Brighton. His broad research interests range from philosophy to interface design.

Phil Siviter (advisor) is a Senior Lecturer for the School of Computing and Mathematical Sciences at the University of Brighton. He is a  researcher in the field of educational technologies, with an involvement stretching continuously back to the early 1980s. His main current research interests lie in the area of standards and methodologies for open courseware development and he is currently a PhD candidate, which involves him in developing a framework for interoperable courseware objects.

.