I presented a very short ‘pecha kucha’ style talk yesterday at the inaugural Learning and Teaching Conference at the University of Edinburgh. I was speaking on behalf of two colleagues, Dr Amani Bell and Dr Jen Scott Curwood, who are members of the Centre for Research on Learning and Innovation at the University of Sydney. This super quick talk was a summary of research we’ve been doing into how teachers and students understand the assessment of students’ digital multimodal work – work that incorporates multiple modes such as images, text, sound, video and hyperlinks.

(Jen is presenting a poster of this work at the Festival of Learning next week in London: https://www.londonfestivallearning.com/agenda/session/6811 (poster number 1900) – if you are there, do stop by to talk to her!)

This research is supported by an Edinburgh-Sydney Partnership Collaboration Award, which funded travel between Sydney and Edinburgh and some of our research activity. The overall project is about methodological innovations for assessing learning in digital spaces, and it was designed to bring our two centres together to develop partnerships and research collaborations.

Our strand of the project took a closer look at how digital assignments are being assessed and how this is working for students and teachers. These kinds of assignments are pretty common in a lot of disciplines now, and they can range from making diagrams to making films, web essays, infographics and portfolios. In some disciplines this is well established, but in others it’s emerging as institutions look for ways to translate students’ engagement with visual, interactive media spaces outside formal education, to critical capacities within it.

However, when it comes to assessing these capacities, teachers might find their practices are still rooted in what Jen calls ‘a paradigm of assessment rooted in print-based theoretic culture’ (Curwood, 2012, p. 232). This can happen because of the constraints of assessment systems, or because of assumptions that teachers and students have about what constitutes legitimate knowledge production in universities, or a combination of things. So, we wanted our project to look specifically at how assessment – and particularly rubrics – takes account of multimodality.

Our research questions were about how students use assessment criteria, how teachers design and assess these kinds of assignments, and how theories of mobilities and place-based learning could inform our thinking about these issues:

• How do university students use assessment criteria for self and peer assessment of multimodal work?
• How do teachers in higher education effectively design and assess students’ multimodal work?
• How can theories of mobilities and place-based learning inform research into and assessment of multimodal student work?

Exploring the literature around grade descriptors and rubrics, there is a wide range of student responses:

• some students are able to use them to accurately assess their peers’ work, to guide and structure their own work, and as a checklist (Bloxham & West, 2004; O’Donovan, Price & Rust, 2001; Bell, Mladenovic & Price, 2013).
• many students find the language used in rubrics and grade descriptors to be subjective and vague (Price & Rust 1999).
• providing more detailed criteria can paradoxically increase students’ anxieties and “lead them to focus on sometimes quite trivial issues” (Norton, 2004, p. 693), with some students leaning heavily on rubrics and exemplars as ‘recipes’ (Bell et. al., 2013). As we’ll see, this was an issue in the course we examined this year.

The answer to the last research question is still very much in development. However, thinking about multimodality, it’s really useful to also consider how students are producing materials that reflect changing dynamics of space and time, and the significance of how these digital assignments circulate:

The multimodal production of culture [is] characterised by changing dynamics of space and time, dynamics that are changing the meanings and effects of cultural production and distribution (Leander and Vasudevan 2009, p. 130).

Production of assignments can feel even higher stakes when they are public in some form – for example, the course we looked at this year asked all students to upload their final videos to youtube. These kinds of mobilities intersect with technical skills, composition elements, modes and meaning to determine what ends up in these assignments, and a need for a nuanced understanding of the “complex ways in which technical skills, composition elements, modes, and meaning interact” in student work (Curwood 2012, p. 242). Greater attention to materiality, including artefacts (Pahl & Rowsell, 2011), movement (Leander & Vasudevan, 2009), and place (Ruitenberg, 2005) enriches this understanding.

Methodology:

Over two semesters we analysed the creation and assessment of a single assignment on an undergraduate course about film and theatre at the University of Sydney. About 130 students take this course each year, mostly study abroad or international students. The assignment we looked at was the final assignment – a three minute film made in pairs about some aspect of students’ ‘Australian cultural experience’. There were specific technical requirements for the film, and an emphasis on ‘narrative’.

• Stage 1: analysing existing processes and assignments, conducting interview and focus groups with students, interviews with tutors, and developing an assessment framework.
• Stage 2: redesign of the assessment task, building on the new framework.
• Stage 3: comparative analysis of the old and new assignments, further interviews.

The original assessment rubric for the course was divided into three sections – cultural narrative experience, cinematic elements, and collaboration. The second section – on cinematic elements – had the most detail and specificity. The ‘narrative’ section referred to a ‘sophisticated’, ‘adequate’ or ‘not adequate’ narrative. The collaboration section talked about ‘high order’ personal statements. Unsurprisingly, students tended to focus much more on meeting the criteria in the second section than the first – this meant that while often very technically proficient, the narratives varied considerably in the extent to which they connected with the critical themes of the course, for example.

Findings:

Here are some of the things that people said in our initial interviews. This first quote comes from a tutor, who talked about how he used the criteria, but also how he used his own judgement when something might not have exactly followed the technical criteria but was ‘absolutely brilliant’:

[one group] used one interview but used it extremely well. I’m quite flexible and adaptable when it comes within the criteria. So if something is absolutely brilliant, of which this one was overall, then I wouldn’t penalise them. They really still came up here in the ‘exceeds criteria’ which is why they ended up getting a high distinction. (Tutor, Interview 1)

The question of what can be contained within the rubric and what, by necessity, goes beyond it in these types of assessments, is a central one for this project.

One of the students on the course talked about struggling to understand what was meant by ‘narrative’ and what was expected in this respect. They were clear about the technical expectations, but unclear about what to do about the narrative dimension.

We knew we needed a lot of cool angles, and different shots, so we started thinking ‘What would be really neat and catching to eye?’ The thing we struggled with looking at the rubric was the narrative, having a narrative, but everything else we were able to look at and make sure was in the project. (Carla, Focus Group 2)

The rubric guided students in the use of discipline-specific vocabulary and highlighted the importance of collaboration in reflecting on the meaning of Australian culture and representing it within a multimodal composition. Students felt, though, that the rubric ‘left a lot of room for interpretation’. As Carla added, “The Australian cultural experience from the videos [viewed as a class after submission] meant so many different things. I liked that it was open…but then again that’s also the challenge…” This highlights the importance of agency and creativity, but a tension exists with the tutor’s responsibility to communicate expectations and fairly assess student learning. One student noted, after the marks were returned and she knew she had done very well on the assignment:

when it says ‘the video demonstrates a sophisticated Australian cultural experience narrative, I don’t really know what [the tutor] means by sophisticated. Personally our project was more humorous, I don’t think you’d look at our video and say ‘That’s a sophisticated piece of art’. …But I still got really high marks on my assignment, and so really vague words like ‘sophisticated’, I think really limits people’s creativity. …[the students] don’t exactly know what [the tutors] want.(Sarah, focus group 1)

She was clear that their assignment wasn’t ‘sophisticated art’, but it still did well – so what was going on here?

It was clear from the reflective interviews students included at the end of their assignments that they were trying to grapple with both form and content, and how these intersect. In looking at these artefacts and talking to students and teachers, we think there are five main things that teachers need to take account of in developing multimodal assessments.

1. Students need support to develop multimodal assessment literacy. This means being able to parse rubrics and criteria to understand what is being looked for, and how to identify it, and also to understand this process as a dialogue rather than a fixed and objective measurement. So for example where teachers say ‘sophisticated’, they need to be clear what they mean by this, and how they judge it when they see it.

2. Some students talked about feeling constrained by the language of the rubrics, or sometimes by the fairly rigid instructions about what kind of technology to use. Where constraints can be removed that might be a good thing, sometimes, but sometimes those constraints can interact with creativity in quite generative ways. Teachers might think about and keep an eye on how these creative constraints are operating in the assignments students produce.
3. The intra-action of form and content – and I mean this in Karen Barad’s sense, in terms of how they create and shape one another rather than existing separately and interacting. This is a really important site of criticality in multimodal work – as my colleague James Lamb says, there is a need to consider the coherence between modes (and, where there is dissonance, to be deliberate about this). We should consider if our students know how to create a ‘multimodal argument’.
4. Trying to look holistically at multimodal assignments is challenging if teachers are also working with rubrics. Those rubrics – especially where they specify technical elements – can easily tend towards what Bateman calls ‘multimodal decomposition’, and that can leave students trying to follow recipes, as Amani noted earlier.
5. Last but not least, teachers have to consider what they are asking students to do, and how to value it appropriately. A digital assignment isn’t a throwaway task – it often involves substantial learning, work and creativity, and its weighting within the course – in terms of time and assessment – needs to be carefully considered.

We’re working now on our framework, which has four dimensions – criticality, cultivating creativity, taking a holistic approach, and valuing multimodality. More to come on this! Our next steps are to finalise this framework, write and publish from this initial stage of the research, and develop some new ideas and use cases. We’re having conversations with our colleagues who led the other strand of the project, about automating feedback. We think it would be really useful to explore the potential for supporting these kinds of complex assignments through automated processes – lots to discuss!

References

Bateman, J. (2012). The decomposability of semiotic modes. In K. O’Halloran and B. Smith (Eds.), Multimodal studies: Exploring issues and domains. New York: Routledge.

Bell A, Mladenovic R and Price M (2013). Students’ perceptions of the usefulness of marking guides, grade descriptors and annotated exemplars. Assessment & Evaluation in Higher Education. 38(7), 769-788.

Bloxham, S., and A. West. 2004. Understanding the rules of the game: Marking peer assessment as a medium for developing students’ conceptions of assessment. Assessment & Evaluation in Higher Education29(6): 721–733

Curwood, J.S. (2012). Cultural shifts, multimodal representations, and assessment practices: A case study.  E-Learning and Digital Media, 9(2), 232-244.

Leander, K. M., & Vasudevan, L. (2009). Multimodality and mobile culture. In C. Jewitt (Ed.), Handbook of multimodal analysis(pp. 127-139). New York, NY: Routledge.

Miles, M.B., Huberman, A., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook. Thousand Oaks, CA: Sage.

Norton, L. (2004). Using assessment criteria as learning criteria: A case study in psychology. Assessment & Evaluation in Higher Education,29(6), 687–702.

O’Donovan, B., Price, M. & Rust, C. (2001). The student experience of criterion-referenced assessment. Innovations in Education and Teaching International,38(1), 74–85.

Pahl, K.H. & Rowsell, J. (2011). Artifactual critical literacy: A new perspective for literacy education. Berkeley Review of Education, 2(2), 129-151.

Price, M., & Rust, C. (1999). The experience of introducing a common criteria assessment grid across an academic department. Quality in Higher Education, 5(2), 133–144.

Ruitenberg, C. (2005). Deconstructing the experience of the local: Toward a radical pedagogy of place. Philosophy of Education Archive, 2005, 212-220.

Along with Professor Dragan Gasevic here at Edinburgh, and  Dr Jen Scott Curwood, Associate Professor Abelardo Pardo, and Dr Amani Bell from the Centre for Research in Learning and Innovation at the University of Sydney, this year I’ve been awarded a Partnership Collaboration Award on the topic of Methodological Innovations for Assessing Learning in Digital Spaces. The project is building on research connections between the two Centres, weaving together two complementary strands: an approach to multimodal assessment and a framework to analyse learning strategies in digital spaces. Jen, Amani and I are working on the first strand, developing new insights into the nature of digital assignments and methodologies for their design and assessment, drawing on theories of place-based learning, mobilities and multimodality.

Along with getting to work closely with Jen and Amani (and returning to the lovely Sydney in March), and to think very broadly about assessment methodologies with all the partners, this project is also a brilliant chance to bring together expertise from a number of colleagues here in the Centre: James Lamb, Sharon Boyd, Yi-Shan Tsai and Sian Bayne are all contributing their insights to the multimodal assessment strand.

There’s a bit more information about this collaboration on the Digital Education site, and I’ll share findings and materials here as they develop.