Skip to main content

Workshop: Multiplier Event 2

Schedule​

  • Date: Monday, February 27, 2023
  • Place: University of Iceland

Meeting Notes​

Background​

The existing SmileyTutor (formerly tutor-web) has been in development and use at the University of Iceland (UI) for two decades and used for thousands by students in Kenya as well. In M2 and in the Icelandic intellectual output in general it is therefore of primary interest to check out synergies between SmileyTutor and OpenEdu as well as how the projects complement each other. This includes feedback on how much of the SmileyTutor functionality can and should be implemented in OpenEdu as well as how the experience in either system can be used to affect the development of the other.

Another technology, EDBOOK, has been developed at UI as a content storage system for books and handouts related to courses in mathematics and statistics.

The multiplier event was used to extensively demonstrate the second content repository of OpenEdu, namely the primary Icelandic intellectual output which is the course Computer and Calculus for Applied Statistics (CCAS).

Material in the course originally includes course notes in SmileyTutor, student submissions of answers to multiple-choice drills, student-submissions of examples and drills as well as peer-review. It was decided to put this course at the centre of discussions since it contains all issues relevant to comparing, linking or migrating the SmileyTutor functionality with or into OpenEdu.

For M2, a general discussion was organised to go through the OpenEdu principles in the light of all the requirements implied by CCAS as this is the one course with the most extensive use of all SmileyTutor functionality.

On Formats and Content Types​

A part of the M2 discussion was on data structures. It was noted that it is generically important to try to enforce a split between the presentation mechanism (e.g. PDF, HTML) and internal data (text format content, images, etc). The "data" should be according to a prespecified format, aka a database of content, and the user should be able to choose an appropriate "view" from the database.

This approach was taken when initially designing SmileyTutor but was always problematic and has not been followed up on in recent generations of the system. Specifically, in the earliest implementations of SmileyTutor, the system included a mechanism for storing general educational content. This content included presentation slides as well as course notes in a unified form and was subsequently presented as web pages or PDF versions of slides and course notes (an example of which can be found here). The approach was eventually abandoned and the current aim is for OpenEdu to serve this role instead.

CCAS is an example of a course which extensively uses this now-obsolete feature of SmileyTutor. The process of migrating the CCAS course notes from SmileyTutor into OpenEdu is therefore a particularly important part of the intellectual output.

This is the first important link between the two systems: migrating course notes from SmileyTutor into OpenEdu.

Choice of Presentation Framework and Hurdles to Adoption​

A discussion was had as to what a sensible presentation framework would be for the CCAS course notes. The OpenEdu framework is currently using a custom presentation framework, revealMD, that allows for significantly more functionality than a typical Markdown file. However, some participants had previous experience with trying to get others to adopt various digital solutions and each additional system that a creator needed to understand decreased the likelihood of adoption.

However, even "out of the box" viewers do not always give the same results for plain" Markdown. In particular, GitHub's viewer had significant trouble with even simple LaTeX code. While workarounds were found, when GitHub fixed their rendering errors, they broke some of the workarounds, demonstrating the difficulties of working with views that might change with time.

Purpose of Student Submissions​

An extensive discussion was undertaken on the nature of student submissions in SmileyTutor and in OpenEdu.

Options in SmileyTutor have included student submission of material for a very long time. The submitted material is typically in the form of examples or drill items. The material is typically peer-reviewed by other students and may also be reviewed by a vetted reviewer (usually the instructor).

All of the above is used extensively in the CCAS course.

As assignments, authoring and submitting new material such as drills and examples is fundamentally different from solving existing drills. The most common use of material submission is solely to establish a grade in a course. However, alternate uses exist and have been implemented in conjunction with SmileyTutor. One approach is the use of peer-review as a teaching tool, as the act of grading can give a student important insight not covered by traditional problem-solving.

A considerable and very useful discussion ensued on whether students are good "test designers". It should be noted that the SmileyTutor drills are for practice and not intended for exams or tests, but participants' experience of grade inflation in Moodle peer reviews is also applicable in the OpenEdu project as well as in the SmileyTutor.

This general discussion on the nature or quality of student submissions was quite useful. Examples and research results on the lack of quality and general usefulness of student submissions abound. A typical example of a student-authored drill in an introductory calculus course is What is 1+1?. Another example is a common "Adequate" or even "Good" grade for a "Quality of descriptive graph" when no graph was in fact submitted.

Methods in SmileyTutor have been implemented to force students to use the full grading scale for peer-review, to fight grade inflation and to avoid some of these issues.

In spite of all of this, a large number of submissions which receive high peer-grades are not of the quality to be acceptable as a part of course material in any sense.

To counter the last of these issues, submissions which have received a good peer-review (possibly after resubmission) may be subjected to a review by a vetted reviewer (the instructor) in SmileyTutor. An exercise or drill which has been deemed of high quality following these reviews can be considered for inclusion in current or future course material.

These multiple uses of authoring, peer-review and vetted review are not seen as fundamental or obvious parts of OpenEdu.

However, OpenEdu has at its core a storage mechanism which can be used to store vetted student-submissions into SmileyTutor. This leads to the second important link between the two systems: storing SmileyTutor student submissions in OpenEdu. By this we are referring to submissions to be peer-reviewed and optionally reviewed by a vetted reviewer.

This second link has not been implemented yet and was therefore not discussed further at M2.

Migration Issues: The CCAS Course Notes​

During the meeting, a large number of issues involved in migrating the CCAS course notes into the current OpenEdu system were presented. Chief among them being the interplay between Markdown and LaTeX within GitHub's Markdown viewer.

A short demonstration was given on the conversion workflow, which utilised a reasonable automatic LaTeX-to-Markdown converter, but had to be both pre- and post-processed by some fairly sophisticated Python scripts.

In addition, almost all of the images in the notes were being generated from R scripts. As such functionality is obviously not reproducible with base Markdown, the scripts for the images were run locally and the resulting images could be referenced by Markdown.

Some comments followed on the difficulties and nature of repurposing content within education.

Rewarding Students​

Students are commonly rewarded using grades but can also be rewarded for work using other currencies such as smiley faces or points.

The SmileyCoin cryptocurrency is well established as a reward mechanism in SmileyTutor and will be included in OpenEdu. A real-world course can use either system and students will be able to redeem SmileyCoin for appropriate answers or submissions as the case may be. The SmileyCoin is an example of transferring technology from SmileyTutor to OpenEdu.

Within SmileyTutor an elaborate item allocation algorithm has been established for assigning the next drill to a student and a correspondingly elaborate weighting mechanism for grading has been established to emphasise learning-by-doing rather than examining students.

The approaches taken by SmileyTutor and OpenEdu are fundamentally different.

Using GitHub as Content Storage: Course Notes​

Once course notes have been set up in OpenEdu, it may be beneficial to be able to produce course notes / handout slides from the content. They may be provided to students before lectures so they may follow along and for self-study.

Having OpenEdu as a content storage for this type of material will be very useful and will save on SmileyTutor development. This is the third link between the systems: Students will be able to retrieve course handouts from OpenEdu using a single click from SmileyTutor.

Using GitHub as Content Storage: Drill Collections​

SmileyTutor has no mechanism for migrating vetted student-submitted items into the database of drill items, nor a facility for storing it for easy access. SmileyTutor does, however, have the mechanism for reviewing said material. It is therefore important to set up the fourth link: A flow mechanism of vetted student submissions into the OpenEdu git repository.

Courses at a post-graduate level such as the CCAS course are particularly likely candidates for the generation of high-quality drill items by students. Rather than duplicate OpenEdu development, it is natural to provide a pipeline between the system for vetted items, using OpenEdu for storage and SmileyTutor for drill delivery.