Category Archives: projects

Assessment in a digital age: Rethinking multimodal artefacts in higher education

I presented a very short ‘pecha kucha’ style talk yesterday at the inaugural Learning and Teaching Conference at the University of Edinburgh. I was speaking on behalf of two colleagues, Dr Amani Bell and Dr Jen Scott Curwood, who are members of the Centre for Research on Learning and Innovation at the University of Sydney. This super quick talk was a summary of research we’ve been doing into how teachers and students understand the assessment of students’ digital multimodal work – work that incorporates multiple modes such as images, text, sound, video and hyperlinks.

(Jen is presenting a poster of this work at the Festival of Learning next week in London: (poster number 1900) – if you are there, do stop by to talk to her!)

Willi Heidelbach, Metal movable type

This research is supported by an Edinburgh-Sydney Partnership Collaboration Award, which funded travel between Sydney and Edinburgh and some of our research activity. The overall project is about methodological innovations for assessing learning in digital spaces, and it was designed to bring our two centres together to develop partnerships and research collaborations.

Our strand of the project took a closer look at how digital assignments are being assessed and how this is working for students and teachers. These kinds of assignments are pretty common in a lot of disciplines now, and they can range from making diagrams to making films, web essays, infographics and portfolios. In some disciplines this is well established, but in others it’s emerging as institutions look for ways to translate students’ engagement with visual, interactive media spaces outside formal education, to critical capacities within it.

However, when it comes to assessing these capacities, teachers might find their practices are still rooted in what Jen calls ‘a paradigm of assessment rooted in print-based theoretic culture’ (Curwood, 2012, p. 232). This can happen because of the constraints of assessment systems, or because of assumptions that teachers and students have about what constitutes legitimate knowledge production in universities, or a combination of things. So, we wanted our project to look specifically at how assessment – and particularly rubrics – takes account of multimodality.

Our research questions were about how students use assessment criteria, how teachers design and assess these kinds of assignments, and how theories of mobilities and place-based learning could inform our thinking about these issues:

  • How do university students use assessment criteria for self and peer assessment of multimodal work?
  • How do teachers in higher education effectively design and assess students’ multimodal work?
  • How can theories of mobilities and place-based learning inform research into and assessment of multimodal student work?

Exploring the literature around grade descriptors and rubrics, there is a wide range of student responses:

  • some students are able to use them to accurately assess their peers’ work, to guide and structure their own work, and as a checklist (Bloxham & West, 2004; O’Donovan, Price & Rust, 2001; Bell, Mladenovic & Price, 2013).
  • many students find the language used in rubrics and grade descriptors to be subjective and vague (Price & Rust 1999).
  • providing more detailed criteria can paradoxically increase students’ anxieties and “lead them to focus on sometimes quite trivial issues” (Norton, 2004, p. 693), with some students leaning heavily on rubrics and exemplars as ‘recipes’ (Bell et. al., 2013). As we’ll see, this was an issue in the course we examined this year.

The answer to the last research question is still very much in development. However, thinking about multimodality, it’s really useful to also consider how students are producing materials that reflect changing dynamics of space and time, and the significance of how these digital assignments circulate:

The multimodal production of culture [is] characterised by changing dynamics of space and time, dynamics that are changing the meanings and effects of cultural production and distribution (Leander and Vasudevan 2009, p. 130).

Production of assignments can feel even higher stakes when they are public in some form – for example, the course we looked at this year asked all students to upload their final videos to youtube. These kinds of mobilities intersect with technical skills, composition elements, modes and meaning to determine what ends up in these assignments, and a need for a nuanced understanding of the “complex ways in which technical skills, composition elements, modes, and meaning interact” in student work (Curwood 2012, p. 242). Greater attention to materiality, including artefacts (Pahl & Rowsell, 2011), movement (Leander & Vasudevan, 2009), and place (Ruitenberg, 2005) enriches this understanding.


Over two semesters we analysed the creation and assessment of a single assignment on an undergraduate course about film and theatre at the University of Sydney. About 130 students take this course each year, mostly study abroad or international students. The assignment we looked at was the final assignment – a three minute film made in pairs about some aspect of students’ ‘Australian cultural experience’. There were specific technical requirements for the film, and an emphasis on ‘narrative’.

  • Stage 1: analysing existing processes and assignments, conducting interview and focus groups with students, interviews with tutors, and developing an assessment framework.
  • Stage 2: redesign of the assessment task, building on the new framework.
  • Stage 3: comparative analysis of the old and new assignments, further interviews.

The original assessment rubric for the course was divided into three sections – cultural narrative experience, cinematic elements, and collaboration. The second section – on cinematic elements – had the most detail and specificity. The ‘narrative’ section referred to a ‘sophisticated’, ‘adequate’ or ‘not adequate’ narrative. The collaboration section talked about ‘high order’ personal statements. Unsurprisingly, students tended to focus much more on meeting the criteria in the second section than the first – this meant that while often very technically proficient, the narratives varied considerably in the extent to which they connected with the critical themes of the course, for example.


Here are some of the things that people said in our initial interviews. This first quote comes from a tutor, who talked about how he used the criteria, but also how he used his own judgement when something might not have exactly followed the technical criteria but was ‘absolutely brilliant’:

[one group] used one interview but used it extremely well. I’m quite flexible and adaptable when it comes within the criteria. So if something is absolutely brilliant, of which this one was overall, then I wouldn’t penalise them. They really still came up here in the ‘exceeds criteria’ which is why they ended up getting a high distinction. (Tutor, Interview 1)

The question of what can be contained within the rubric and what, by necessity, goes beyond it in these types of assessments, is a central one for this project.

One of the students on the course talked about struggling to understand what was meant by ‘narrative’ and what was expected in this respect. They were clear about the technical expectations, but unclear about what to do about the narrative dimension.

We knew we needed a lot of cool angles, and different shots, so we started thinking ‘What would be really neat and catching to eye?’ The thing we struggled with looking at the rubric was the narrative, having a narrative, but everything else we were able to look at and make sure was in the project. (Carla, Focus Group 2)

The rubric guided students in the use of discipline-specific vocabulary and highlighted the importance of collaboration in reflecting on the meaning of Australian culture and representing it within a multimodal composition. Students felt, though, that the rubric ‘left a lot of room for interpretation’. As Carla added, “The Australian cultural experience from the videos [viewed as a class after submission] meant so many different things. I liked that it was open…but then again that’s also the challenge…” This highlights the importance of agency and creativity, but a tension exists with the tutor’s responsibility to communicate expectations and fairly assess student learning. One student noted, after the marks were returned and she knew she had done very well on the assignment:

when it says ‘the video demonstrates a sophisticated Australian cultural experience narrative, I don’t really know what [the tutor] means by sophisticated. Personally our project was more humorous, I don’t think you’d look at our video and say ‘That’s a sophisticated piece of art’. …But I still got really high marks on my assignment, and so really vague words like ‘sophisticated’, I think really limits people’s creativity. …[the students] don’t exactly know what [the tutors] want.(Sarah, focus group 1)

She was clear that their assignment wasn’t ‘sophisticated art’, but it still did well – so what was going on here?

It was clear from the reflective interviews students included at the end of their assignments that they were trying to grapple with both form and content, and how these intersect. In looking at these artefacts and talking to students and teachers, we think there are five main things that teachers need to take account of in developing multimodal assessments.

  1. Students need support to develop multimodal assessment literacy. This means being able to parse rubrics and criteria to understand what is being looked for, and how to identify it, and also to understand this process as a dialogue rather than a fixed and objective measurement. So for example where teachers say ‘sophisticated’, they need to be clear what they mean by this, and how they judge it when they see it.
Screenshot from the video of the Manifesto for Teaching Online, James Lamb.
  1. Some students talked about feeling constrained by the language of the rubrics, or sometimes by the fairly rigid instructions about what kind of technology to use. Where constraints can be removed that might be a good thing, sometimes, but sometimes those constraints can interact with creativity in quite generative ways. Teachers might think about and keep an eye on how these creative constraints are operating in the assignments students produce.
  2. The intra-action of form and content – and I mean this in Karen Barad’s sense, in terms of how they create and shape one another rather than existing separately and interacting. This is a really important site of criticality in multimodal work – as my colleague James Lamb says, there is a need to consider the coherence between modes (and, where there is dissonance, to be deliberate about this). We should consider if our students know how to create a ‘multimodal argument’.
  3. Trying to look holistically at multimodal assignments is challenging if teachers are also working with rubrics. Those rubrics – especially where they specify technical elements – can easily tend towards what Bateman calls ‘multimodal decomposition’, and that can leave students trying to follow recipes, as Amani noted earlier.
  4. Last but not least, teachers have to consider what they are asking students to do, and how to value it appropriately. A digital assignment isn’t a throwaway task – it often involves substantial learning, work and creativity, and its weighting within the course – in terms of time and assessment – needs to be carefully considered.

We’re working now on our framework, which has four dimensions – criticality, cultivating creativity, taking a holistic approach, and valuing multimodality. More to come on this! Our next steps are to finalise this framework, write and publish from this initial stage of the research, and develop some new ideas and use cases. We’re having conversations with our colleagues who led the other strand of the project, about automating feedback. We think it would be really useful to explore the potential for supporting these kinds of complex assignments through automated processes – lots to discuss!


Bateman, J. (2012). The decomposability of semiotic modes. In K. O’Halloran and B. Smith (Eds.), Multimodal studies: Exploring issues and domains. New York: Routledge.

Bell A, Mladenovic R and Price M (2013). Students’ perceptions of the usefulness of marking guides, grade descriptors and annotated exemplars. Assessment & Evaluation in Higher Education. 38(7), 769-788.

Bloxham, S., and A. West. 2004. Understanding the rules of the game: Marking peer assessment as a medium for developing students’ conceptions of assessment. Assessment & Evaluation in Higher Education29(6): 721–733

Curwood, J.S. (2012). Cultural shifts, multimodal representations, and assessment practices: A case study.  E-Learning and Digital Media, 9(2), 232-244.

Leander, K. M., & Vasudevan, L. (2009). Multimodality and mobile culture. In C. Jewitt (Ed.), Handbook of multimodal analysis(pp. 127-139). New York, NY: Routledge.

Miles, M.B., Huberman, A., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook. Thousand Oaks, CA: Sage.

Norton, L. (2004). Using assessment criteria as learning criteria: A case study in psychology. Assessment & Evaluation in Higher Education,29(6), 687–702.

O’Donovan, B., Price, M. & Rust, C. (2001). The student experience of criterion-referenced assessment. Innovations in Education and Teaching International,38(1), 74–85.

Pahl, K.H. & Rowsell, J. (2011). Artifactual critical literacy: A new perspective for literacy education. Berkeley Review of Education, 2(2), 129-151.

Price, M., & Rust, C. (1999). The experience of introducing a common criteria assessment grid across an academic department. Quality in Higher Education, 5(2), 133–144.

Ruitenberg, C. (2005). Deconstructing the experience of the local: Toward a radical pedagogy of place. Philosophy of Education Archive, 2005, 212-220.


surveillance, trust & technology: higher education futures

Several of the research and writing projects and discussions I’ve been involved with this year have directly or indirectly addressed issues of trust and surveillance, and this is an area of work I’m planning to develop further, along with a number of great colleagues.

In several talks this year, at Strathclyde in Glasgow; the University of Sydney; and the Networked Learning Conference in Zagreb, I’ve been drawing on our Manifesto for Teaching Online to develop my thinking about plagiarism detection and attendance monitoring – two routine processes which are becoming increasingly intensive and often invasive in most universities. For example, Hamish Macleod and I recently argued that

Logics of surveillance are strongly at work in practices which attempt to regulate student behaviour through the exposure of their writing to algorithmic scanning and monitoring. These logics frame students as in need of careful monitoring to ensure learning and teaching runs smoothly, and framing academic writing as a space of dishonesty which is both rampant and solvable through technology. Routines of plagiarism detection intervene negatively in one of the central facets of student-teacher relationships: the production and assessment of student work. Where these relationships become risk-averse and mutually suspicious, trust is blocked or lost and not easily regained. (Ross and Macleod 2018, 235)

We end that paper by arguing that we need to find “ways to re-sensitise ourselves and our students to the values we want to prioritise in our classrooms, and offering means by which students can voice their responses to surveillance cultures in higher education; and [address] issues at strategic levels within our institutions and the sector more widely by developing robust mechanisms for engaging in critical debate, discussion about and review of technology platforms and practices”. This second point echoes one of the key observations made by Laura Czerniewicz at her keynote address in Zagreb last month:

My talk in Zagreb (Hamish couldn’t make it, being retired and all!) generated quite a lot of interest, and it’s clear this is resonating with people from across a number of higher education contexts:

I’ve been involved in a number of intense discussions on these issues, during my seminars in Glasgow and Sydney, and with colleagues in the UK and North America (including Amy Collier and George Veletsianos). A highlight was a discussion with David Lyon and Sava Saheli Singh at Queen’s University in Canada last month. David’s latest work on surveillance culture is greatly informing my thinking. Colleagues closer to home are also doing fantastic work in related areas – Sian Bayne on anonymity; Jeremy Knox on learning analytics, Phil Sheail on data bodies in the library, Ben Williamson on monitoring in the school classroom.

The relationship of trust and technology in higher education is one that, in my view, requires a lot more attention.

It’s emerging as a key issue not only in the context of technologies of monitoring and surveillance, but in other more surprising places – like the current blockchain craze, which is generating new discussions about all sort of things…

…but perhaps less than it should about what kinds of models of trust (or trustlessness) we are inviting into the academy with these new technologies. Helen Murphy, one of the participants on my Digital Futures for Learning course this year, developed an excellent Open Educational Resource on this topic:

She says:

the blockchain is often described as ‘trustless’. We might think of what it means to be ‘trustless’ in two ways:
First, data.
The data in the blockchain is immutable and transparent, permanent and unalterable. Data in the blockchain is added by consensus, and because it is distributed there is no single copy of the data. In theory, it can be accessed by anyone. So it is trustless in the sense that no trust is required: it can be taken for granted that the data is accurate and permanent.
​Second, decentralisation.
With the blockchain, there is no need for a centralised authority (such as a bank or university) to verify transactions. Instead all of this verification can be done independently, by the technology and the mechanisms by which it works. So it is trustless because it does not require any trust in these third-party institutions.

(Murphy 2018)

I’m doing some work with colleagues in Digital Education and Information Services on exploring the potential and issues around blockcert technology, and these questions of trust and trustlessness are figuring strongly.

So, look out for more writing on this from me and others in the near future!

One of the most important things, I think, is beginning to speculate and design futures for higher education that are ‘beyond surveillance’. Importantly (and thanks to Peter Goodyear in Sydney for highlighting how important this point is), this doesn’t mean alternatives that require us to return to imagined better analog times  – instead, we need to develop trusting digital futures and approach our technologies critically and creatively to help us do so.  I, and others, will be working on this over the next year – get in touch if you want to discuss!


Lyon, D. 2018. The Culture of Surveillance: Watching as a Way of Life. Polity Press.

Murphy, H. 2018. Education and the Blockchain.

Ross, J. and Macleod, H. 2018. Surveillance, (dis)trust and teaching with plagiarism detection technology. Proceedsing of Networked Learning 2018, Zagreb.

innovating with assessment methods – a project with the University of Sydney

Along with Professor Dragan Gasevic here at Edinburgh, and  Dr Jen Scott Curwood, Associate Professor Abelardo Pardo, and Dr Amani Bell from the Centre for Research in Learning and Innovation at the University of Sydney, this year I’ve been awarded a Partnership Collaboration Award on the topic of Methodological Innovations for Assessing Learning in Digital Spaces. The project is building on research connections between the two Centres, weaving together two complementary strands: an approach to multimodal assessment and a framework to analyse learning strategies in digital spaces. Jen, Amani and I are working on the first strand, developing new insights into the nature of digital assignments and methodologies for their design and assessment, drawing on theories of place-based learning, mobilities and multimodality.

Along with getting to work closely with Jen and Amani (and returning to the lovely Sydney in March), and to think very broadly about assessment methodologies with all the partners, this project is also a brilliant chance to bring together expertise from a number of colleagues here in the Centre: James Lamb, Sharon Boyd, Yi-Shan Tsai and Sian Bayne are all contributing their insights to the multimodal assessment strand.

There’s a bit more information about this collaboration on the Digital Education site, and I’ll share findings and materials here as they develop.



digital cultural heritage research update

One of the four main strands of research I pursue  is about digital cultural heritage engagement and learning, and lots has been happening!

In London for the Critical Heritage Futures conference.

The Artcasting project continues to generate great speaking, writing and conversation opportunities – most recently at the AHRC and Association of Critical Heritage Studies’ Critical Approaches and New Directions conference in London in early October. My colleague Michael Gallagher and I gave a talk about mobilities, mobile technologies and heritage futures. Particular highlights for me from the event were Chris Whitehead‘s keynote, and Hayden Lorimer’s fascinating overview of work at St Peter’s Seminary at Kilmahew. It was also fantastic to see so many of the people I first met last year at the ACHS conference in Montreal, and to hear that Liz Stainforth is going to be spending a few months in Edinburgh as an IASH fellow at the start of next year.

Also re Artcasting, I’ve just finished and submitted a paper about hospitality and digital co-production; and the team recently reconvened with a number of others who have helped us a lot in thinking about Artcasting futures, to talk about a whole bunch of fascinating issues still to be explored. There is a summary of these on the Artcasting site blog.

Beyond Artcasting and its ripples, London also allowed a bit of time for Koula Charitonos and I to talk through some ideas – including to propose a symposium for next year’s ICLS conference on museum visitor experiences in the digital age. We hope this will be able to come together!

Here in Edinburgh, the new academic year has brought a number of new members to the Digital Cultural Heritage Research Network – not least our new chair of digital cultural heritage, Melissa Terras, who joined the University of Edinburgh this week. We’ve already tempted her to get involved with a couple of projects in development – it’s going to be great to have her here. Sian, Chris, James, Kirsty, Melissa and I hope to organise a few DCHRN events in the new year.

Serendipitously, I’ve got the opportunity to supervise not one but two masters students this year who are exploring aspects of 3d printing, scanning and visualisation in cultural heritage contexts – I look forward to learning lots from both of them about this topic.

Last but not least, I’ll be in Manchester for the Researching Digital Cultural Heritage conference on 30 November-1 December (sadly only for the second day) – the programme looks amazing.

Introducing the artcasting project

The Artcasting projectartcastinglogo is a collaboration between Digital Education and Design Informatics at the University of Edinburgh, Scotland. Working together with the National Galleries of ScotlandTate, and the ARTIST ROOMS Research Partnership, the project team is developing a new digital and mobile form of evaluation of arts-based engagement, in the context of ARTIST ROOMS On Tour.

Along with developing and testing artcasting, the project will engage with the cultural heritage education community to share and generate ideas about evaluation, digital engagement and learning.

Funding for artcasting comes from the AHRC, and the project runs from May 2015-April 2016.