We know what you (think you) don't know

This project aims to use Sintella/ASP to track the resources that COMP115 students are using for completing their workshop exercises.  This will allow us to collect live data telling us what topics students are looking up, i.e. what things they are not confident of (yet).

Sintella/ASP

Sintella/ASP is a research tool developed at Macquarie University for investigating information assimilation.  It has been used in many experiments in the past but not in learning analytics.

  • Wastell, C. A.,Weeks, N., Wearing, A., Duncan, P., & Ebrahimi, W. (2013). The impact of closed mindedness on the assessment of threat: An empirical study. Open Psychology Journal6(1), 10-19.
  • Wastell, C. A.,Weeks, N., Wearing, A., & Duncan, P. (2012).  Identifying hypothesis confirmation behaviors in a simulated murder investigation: Implications for practice. Journal of Investigative Psychology and Offender Profiling. 9, 184-198
  • Weeks, N., Wastell, C. A., Taylor, A., Wearing, A & Duncan, P. (2012). Tracing decision processes in complex, ambiguous, information-rich environments. International Journal of Psychological Studies 4(1), 158-173.
  • Wastell, C. A.,Weeks, N., & Duncan, P. (2009). SINTELLA: Simulation of INTELLigence Analysis. Journal of Policing, Intelligence and Counter Terrorism. 4, 71-82.

In this project we will use Sintella to present a matrix of "tips" related to that week's material.  Each has a short heading to prompt the student's interest and then the long text within each cell is a tip or trick or reminder of something from the readings that should help the student complete their work.  We are aiming for each workshop to be accompanied by a 5 by 5 or larger matrix.

Experiment Design

For each workshop in COMP115, a Sintella/ASP matrix will be created containing one cell for each little piece of information the students might need to know for that workshop.  We will have to make an educated guess about what these would be and while we would like to be comprehensive, it is not vital to the experiment design.  We don't attempt to stop students from looking up information in other sources, so we are not trying to capture everything they are using to do their work.  So what are we collecting?  We are collecting a comparison between all the topics we put in the matrix to see which one students most often looked up, i.e. an estimate of what topic the students don't yet understand (more accurately feel like they don't know).

In this project, which is really a pilot of the idea, we will test some simple hypothesis:  For each workshop we will identify the cells we most expect students will look at and which we least expect they will look at.  If our expectations are not met, we have learned something we didn't already guess about what students were not picking up.

There are many other hypothesis that could be run, but won't be in this initial project.  Compiling a list for future experiments is a part of this project.  Here are some for starters:

  • If we mix student (last year's students) generated cells with instructor generated cells, which ones are used more?
  • If a topic comes up again later in the course where it is assumed to have been learned earlier, does that cell get only a few visits as we expect?
  • Are there "information assimilation personalities" in the IT student cohort that are separate from those we have found in the other cohorts that have participated in Sintella/ASP experiments?  You can expect a higher level of completionists I would imagine.

Personnel

  • Matt Roberts, Department of Computing
  • <accomplished COMP229 student working in COMP350 S3>
  • <need someone from learning and teaching>
  • <anyone else interested?>

Dates

We will run the experiment in S1 2017 so we need the matrices all built and tested by the start of S1 2017.  We also need to connect each of the COMP115 workshops to Sintella/ASP in the same time-frame.

Dissemination Venues

Icon

Who can help me out? No-one in computer science will be interested but I imagine the learning analytics people should be interested - is there something in psychology?

 

 

  • No labels

1 Comment

  1. attaching  comment from Matt Bower "Does the system allow students to ask questions and other students to answer them? And most viewed questions bubble to the top? If so, most tutors would want it! Could be some interesting questions around learning efficiency for such a system? Just pipe dreaming."