Along with some fellow members of the Higher Education After Surveillance Network, Anna Wilson, Amy Collier and Martin Hawksey, as well as Jane McKie, I’ve just finished work on a small research project which aimed to facilitate the creation of short pieces of speculative fiction by people with an interest in the growing use of surveillance technologies in Higher Education.  

The Speculative Data Storytelling project‘s purpose was to facilitate stories that explored possible futures, in order to give expression to perhaps previously un-recognised hopes, concerns and fears.  

Initial work focused on the development of face-to-face co-design activities, but we shifted approach as a result of the Covid-19 pandemic, revising our plans to create a remote method of generating data, enabling participants to engage in brief, asynchronous ways.

 

Over the project period, we explored how speculative data stories can be scaffolded and created. Anna led on designing and testing a methodology to help participants create data stories. Working with Pat Lockley, we mapped this methodology onto a web based interface (in the form of a WordPress plugin, built by Pat). An iterative process of building, testing and refining led to a three-part data storytelling tool: prompts, mapping and writing. Prompts and mapping help users identify actors and explore possible interactions between them, while the writing section gives a space to write an anonymous multimedia story (text, images, video, tweets and GIFs are all possible elements of the story). The finished story can be saved, and also (optionally) submitted to be shared publicly on the data stories site.

Like a lot of things this year, this project did not go as planned, but I am really grateful to the team, the network, and all the people who participated in the testing phase, for being involved in creating something that I think is really worthwhile, and I hope will be of interest and use to others. Thanks, too, to the Edinburgh Futures Institute Research Awards for the funding that supported this project.

I thought it was time to summarise some of the work I’ve been doing with colleagues in and beyond Edinburgh on the topic of higher education and surveillance. I’ve touched on these issues in the context of my doctoral research (trying to understand how students viewed the ‘audience’ for their high-stakes reflection) and in relation to work around plagiarism detection, surveillance and trust (see the conference paper Hamish Macleod and I wrote, and the forthcoming book on the Manifesto for Teaching Online). In early 2019, Amy Collier and I began to put together an international network of people – so far mostly in the UK, US and Canada) to explore possible futures for surveillance in universities – we’ve called this network Higher Education After Surveillance, and it is full of truly brilliant people doing urgently needed work in this area.

Amy and I wrote a commentary for Postdigital Science and Education, explaining some of our thinking behind the idea of ‘after surveillance;’. We explain that when we say ‘after surveillance’ we are not looking back, but instead:

gesturing toward a future that involves a deeper understanding of the role surveillance has played and continues to play in universities and tactics and strategies for interrupting and perhaps reducing or reconfiguring its impacts. This requires a willingness to speculate that some of the surveillance roles we have come to accept could be otherwise, along with an acknowledgment that we are implicated in what Lyon terms ‘surveillance culture’ in education. What can we do with that knowledge, and what culture shifts can we collectively provoke?

Two new things have so far emerged from the network. One is a research project called Co-designing with Speculative Data Stories. This was funded by the Edinburgh Futures Institute Research Awards scheme, and the research team (me, Amy, Anna Wilson, Jane McKie and Martin Hawksey) proposed to run ‘speculative data stories’ workshops with groups of colleagues in UK universities whose roles involve supporting, promoting and working with learning technologies. With the current closure of campuses and intense pressures on those very colleagues to support their institutions to move considerable amounts of university work online, we have had to put this project on hold – but we are hoping to be able to reimagine it in some form, soon.

The second new thing – and the prompt to write this blog post today – is that Karen Gregory and I have been successful in securing one of five new funded PhD studentships on the theme of data ethics (Edinburgh Futures Institute/Baillie Gifford). These projects will be advertised in the next few weeks – ours is called The University of Data: Ethical and Social Futures of Data-Driven Education. I am very happy, as is Karen, to hear from anyone who might be interested in applying for this! More info to follow soon, in a separate blog post.

References

Collier, A. and Ross, J. (2020). Higher education after surveillance? Commentary, Postdigital Science and Education. Online First. https://link.springer.com/article/10.1007/s42438-019-00098-z

Ross, J. and Macleod, H. (2018). Surveillance, (dis)trust and teaching with plagiarism detection technology. Proceedings of the 11th International Conference on Networked Learning 2018, Edited by: Bajić, M, Dohn, NB, de Laat, M, Jandrić, P & Ryberg, T. ISBN 978-1-86220-337-2.

image source: Unsplash. Photographer: Franki Chamaki.

I’ve meant to post about this for a while, but things have been hectic! Higher Education After Surveillance is a new project that my colleague Amy Collier (Middlebury College) and I dreamed up last year, in light of work we are each doing around issues of surveillance, trust, visibility, digital sanctuary and more, and as a way of trying to think big about some of the challenges we are currently facing. We enlisted the help of a small but mighty group of colleagues around the UK, US and Canada  to get involved, and we hosted a virtual roundtable in March as a way of beginning to scope what such a group might do.

We’re aiming to develop this project in a sustainable way – it isn’t funded (or not yet, anyway), and everyone involved is already incredibly busy – so that it can become something genuinely meaningful, critical and impactful. We also need to think carefully about the scope (geographical and otherwise) of this work. It’s all very exciting, and timely, if daunting – it seems there are new stories, questions and areas for attention emerging most days (this week in the UK it was the announcement of a collaboration involving JISC and the Office for Students, led by Northumbria University, to ‘lead transformation in how the Higher Education sector identifies mental health issues in students‘. It… has not gone over well).

If you are or know someone who needs to be involved in what we’re doing, please do get in touch.

I presented a very short ‘pecha kucha’ style talk yesterday at the inaugural Learning and Teaching Conference at the University of Edinburgh. I was speaking on behalf of two colleagues, Dr Amani Bell and Dr Jen Scott Curwood, who are members of the Centre for Research on Learning and Innovation at the University of Sydney. This super quick talk was a summary of research we’ve been doing into how teachers and students understand the assessment of students’ digital multimodal work – work that incorporates multiple modes such as images, text, sound, video and hyperlinks.

(Jen is presenting a poster of this work at the Festival of Learning next week in London: https://www.londonfestivallearning.com/agenda/session/6811 (poster number 1900) – if you are there, do stop by to talk to her!)

This research is supported by an Edinburgh-Sydney Partnership Collaboration Award, which funded travel between Sydney and Edinburgh and some of our research activity. The overall project is about methodological innovations for assessing learning in digital spaces, and it was designed to bring our two centres together to develop partnerships and research collaborations.

Our strand of the project took a closer look at how digital assignments are being assessed and how this is working for students and teachers. These kinds of assignments are pretty common in a lot of disciplines now, and they can range from making diagrams to making films, web essays, infographics and portfolios. In some disciplines this is well established, but in others it’s emerging as institutions look for ways to translate students’ engagement with visual, interactive media spaces outside formal education, to critical capacities within it.

However, when it comes to assessing these capacities, teachers might find their practices are still rooted in what Jen calls ‘a paradigm of assessment rooted in print-based theoretic culture’ (Curwood, 2012, p. 232). This can happen because of the constraints of assessment systems, or because of assumptions that teachers and students have about what constitutes legitimate knowledge production in universities, or a combination of things. So, we wanted our project to look specifically at how assessment – and particularly rubrics – takes account of multimodality.

Our research questions were about how students use assessment criteria, how teachers design and assess these kinds of assignments, and how theories of mobilities and place-based learning could inform our thinking about these issues:

• How do university students use assessment criteria for self and peer assessment of multimodal work?
• How do teachers in higher education effectively design and assess students’ multimodal work?
• How can theories of mobilities and place-based learning inform research into and assessment of multimodal student work?

Exploring the literature around grade descriptors and rubrics, there is a wide range of student responses:

• some students are able to use them to accurately assess their peers’ work, to guide and structure their own work, and as a checklist (Bloxham & West, 2004; O’Donovan, Price & Rust, 2001; Bell, Mladenovic & Price, 2013).
• many students find the language used in rubrics and grade descriptors to be subjective and vague (Price & Rust 1999).
• providing more detailed criteria can paradoxically increase students’ anxieties and “lead them to focus on sometimes quite trivial issues” (Norton, 2004, p. 693), with some students leaning heavily on rubrics and exemplars as ‘recipes’ (Bell et. al., 2013). As we’ll see, this was an issue in the course we examined this year.

The answer to the last research question is still very much in development. However, thinking about multimodality, it’s really useful to also consider how students are producing materials that reflect changing dynamics of space and time, and the significance of how these digital assignments circulate:

The multimodal production of culture [is] characterised by changing dynamics of space and time, dynamics that are changing the meanings and effects of cultural production and distribution (Leander and Vasudevan 2009, p. 130).

Production of assignments can feel even higher stakes when they are public in some form – for example, the course we looked at this year asked all students to upload their final videos to youtube. These kinds of mobilities intersect with technical skills, composition elements, modes and meaning to determine what ends up in these assignments, and a need for a nuanced understanding of the “complex ways in which technical skills, composition elements, modes, and meaning interact” in student work (Curwood 2012, p. 242). Greater attention to materiality, including artefacts (Pahl & Rowsell, 2011), movement (Leander & Vasudevan, 2009), and place (Ruitenberg, 2005) enriches this understanding.

Methodology:

Over two semesters we analysed the creation and assessment of a single assignment on an undergraduate course about film and theatre at the University of Sydney. About 130 students take this course each year, mostly study abroad or international students. The assignment we looked at was the final assignment – a three minute film made in pairs about some aspect of students’ ‘Australian cultural experience’. There were specific technical requirements for the film, and an emphasis on ‘narrative’.

• Stage 1: analysing existing processes and assignments, conducting interview and focus groups with students, interviews with tutors, and developing an assessment framework.
• Stage 2: redesign of the assessment task, building on the new framework.
• Stage 3: comparative analysis of the old and new assignments, further interviews.

The original assessment rubric for the course was divided into three sections – cultural narrative experience, cinematic elements, and collaboration. The second section – on cinematic elements – had the most detail and specificity. The ‘narrative’ section referred to a ‘sophisticated’, ‘adequate’ or ‘not adequate’ narrative. The collaboration section talked about ‘high order’ personal statements. Unsurprisingly, students tended to focus much more on meeting the criteria in the second section than the first – this meant that while often very technically proficient, the narratives varied considerably in the extent to which they connected with the critical themes of the course, for example.

Findings:

Here are some of the things that people said in our initial interviews. This first quote comes from a tutor, who talked about how he used the criteria, but also how he used his own judgement when something might not have exactly followed the technical criteria but was ‘absolutely brilliant’:

[one group] used one interview but used it extremely well. I’m quite flexible and adaptable when it comes within the criteria. So if something is absolutely brilliant, of which this one was overall, then I wouldn’t penalise them. They really still came up here in the ‘exceeds criteria’ which is why they ended up getting a high distinction. (Tutor, Interview 1)

The question of what can be contained within the rubric and what, by necessity, goes beyond it in these types of assessments, is a central one for this project.

One of the students on the course talked about struggling to understand what was meant by ‘narrative’ and what was expected in this respect. They were clear about the technical expectations, but unclear about what to do about the narrative dimension.

We knew we needed a lot of cool angles, and different shots, so we started thinking ‘What would be really neat and catching to eye?’ The thing we struggled with looking at the rubric was the narrative, having a narrative, but everything else we were able to look at and make sure was in the project. (Carla, Focus Group 2)

The rubric guided students in the use of discipline-specific vocabulary and highlighted the importance of collaboration in reflecting on the meaning of Australian culture and representing it within a multimodal composition. Students felt, though, that the rubric ‘left a lot of room for interpretation’. As Carla added, “The Australian cultural experience from the videos [viewed as a class after submission] meant so many different things. I liked that it was open…but then again that’s also the challenge…” This highlights the importance of agency and creativity, but a tension exists with the tutor’s responsibility to communicate expectations and fairly assess student learning. One student noted, after the marks were returned and she knew she had done very well on the assignment:

when it says ‘the video demonstrates a sophisticated Australian cultural experience narrative, I don’t really know what [the tutor] means by sophisticated. Personally our project was more humorous, I don’t think you’d look at our video and say ‘That’s a sophisticated piece of art’. …But I still got really high marks on my assignment, and so really vague words like ‘sophisticated’, I think really limits people’s creativity. …[the students] don’t exactly know what [the tutors] want.(Sarah, focus group 1)

She was clear that their assignment wasn’t ‘sophisticated art’, but it still did well – so what was going on here?

It was clear from the reflective interviews students included at the end of their assignments that they were trying to grapple with both form and content, and how these intersect. In looking at these artefacts and talking to students and teachers, we think there are five main things that teachers need to take account of in developing multimodal assessments.

1. Students need support to develop multimodal assessment literacy. This means being able to parse rubrics and criteria to understand what is being looked for, and how to identify it, and also to understand this process as a dialogue rather than a fixed and objective measurement. So for example where teachers say ‘sophisticated’, they need to be clear what they mean by this, and how they judge it when they see it.

2. Some students talked about feeling constrained by the language of the rubrics, or sometimes by the fairly rigid instructions about what kind of technology to use. Where constraints can be removed that might be a good thing, sometimes, but sometimes those constraints can interact with creativity in quite generative ways. Teachers might think about and keep an eye on how these creative constraints are operating in the assignments students produce.
3. The intra-action of form and content – and I mean this in Karen Barad’s sense, in terms of how they create and shape one another rather than existing separately and interacting. This is a really important site of criticality in multimodal work – as my colleague James Lamb says, there is a need to consider the coherence between modes (and, where there is dissonance, to be deliberate about this). We should consider if our students know how to create a ‘multimodal argument’.
4. Trying to look holistically at multimodal assignments is challenging if teachers are also working with rubrics. Those rubrics – especially where they specify technical elements – can easily tend towards what Bateman calls ‘multimodal decomposition’, and that can leave students trying to follow recipes, as Amani noted earlier.
5. Last but not least, teachers have to consider what they are asking students to do, and how to value it appropriately. A digital assignment isn’t a throwaway task – it often involves substantial learning, work and creativity, and its weighting within the course – in terms of time and assessment – needs to be carefully considered.

We’re working now on our framework, which has four dimensions – criticality, cultivating creativity, taking a holistic approach, and valuing multimodality. More to come on this! Our next steps are to finalise this framework, write and publish from this initial stage of the research, and develop some new ideas and use cases. We’re having conversations with our colleagues who led the other strand of the project, about automating feedback. We think it would be really useful to explore the potential for supporting these kinds of complex assignments through automated processes – lots to discuss!

References

Bateman, J. (2012). The decomposability of semiotic modes. In K. O’Halloran and B. Smith (Eds.), Multimodal studies: Exploring issues and domains. New York: Routledge.

Bell A, Mladenovic R and Price M (2013). Students’ perceptions of the usefulness of marking guides, grade descriptors and annotated exemplars. Assessment & Evaluation in Higher Education. 38(7), 769-788.

Bloxham, S., and A. West. 2004. Understanding the rules of the game: Marking peer assessment as a medium for developing students’ conceptions of assessment. Assessment & Evaluation in Higher Education29(6): 721–733

Curwood, J.S. (2012). Cultural shifts, multimodal representations, and assessment practices: A case study.  E-Learning and Digital Media, 9(2), 232-244.

Leander, K. M., & Vasudevan, L. (2009). Multimodality and mobile culture. In C. Jewitt (Ed.), Handbook of multimodal analysis(pp. 127-139). New York, NY: Routledge.

Miles, M.B., Huberman, A., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook. Thousand Oaks, CA: Sage.

Norton, L. (2004). Using assessment criteria as learning criteria: A case study in psychology. Assessment & Evaluation in Higher Education,29(6), 687–702.

O’Donovan, B., Price, M. & Rust, C. (2001). The student experience of criterion-referenced assessment. Innovations in Education and Teaching International,38(1), 74–85.

Pahl, K.H. & Rowsell, J. (2011). Artifactual critical literacy: A new perspective for literacy education. Berkeley Review of Education, 2(2), 129-151.

Price, M., & Rust, C. (1999). The experience of introducing a common criteria assessment grid across an academic department. Quality in Higher Education, 5(2), 133–144.

Ruitenberg, C. (2005). Deconstructing the experience of the local: Toward a radical pedagogy of place. Philosophy of Education Archive, 2005, 212-220.