Assessment in a digital age: Rethinking multimodal artefacts in higher education

I presented a very short ‘pecha kucha’ style talk yesterday at the inaugural Learning and Teaching Conference at the University of Edinburgh. I was speaking on behalf of two colleagues, Dr Amani Bell and Dr Jen Scott Curwood, who are members of the Centre for Research on Learning and Innovation at the University of Sydney. This super quick talk was a summary of research we’ve been doing into how teachers and students understand the assessment of students’ digital multimodal work – work that incorporates multiple modes such as images, text, sound, video and hyperlinks.

(Jen is presenting a poster of this work at the Festival of Learning next week in London: https://www.londonfestivallearning.com/agenda/session/6811 (poster number 1900) – if you are there, do stop by to talk to her!)

Willi Heidelbach, Metal movable type https://en.wikipedia.org/wiki/Typesetting#/media/File:Metal_movable_type.jpg

This research is supported by an Edinburgh-Sydney Partnership Collaboration Award, which funded travel between Sydney and Edinburgh and some of our research activity. The overall project is about methodological innovations for assessing learning in digital spaces, and it was designed to bring our two centres together to develop partnerships and research collaborations.

Our strand of the project took a closer look at how digital assignments are being assessed and how this is working for students and teachers. These kinds of assignments are pretty common in a lot of disciplines now, and they can range from making diagrams to making films, web essays, infographics and portfolios. In some disciplines this is well established, but in others it’s emerging as institutions look for ways to translate students’ engagement with visual, interactive media spaces outside formal education, to critical capacities within it.

However, when it comes to assessing these capacities, teachers might find their practices are still rooted in what Jen calls ‘a paradigm of assessment rooted in print-based theoretic culture’ (Curwood, 2012, p. 232). This can happen because of the constraints of assessment systems, or because of assumptions that teachers and students have about what constitutes legitimate knowledge production in universities, or a combination of things. So, we wanted our project to look specifically at how assessment – and particularly rubrics – takes account of multimodality.

Our research questions were about how students use assessment criteria, how teachers design and assess these kinds of assignments, and how theories of mobilities and place-based learning could inform our thinking about these issues:

  • How do university students use assessment criteria for self and peer assessment of multimodal work?
  • How do teachers in higher education effectively design and assess students’ multimodal work?
  • How can theories of mobilities and place-based learning inform research into and assessment of multimodal student work?

Exploring the literature around grade descriptors and rubrics, there is a wide range of student responses:

  • some students are able to use them to accurately assess their peers’ work, to guide and structure their own work, and as a checklist (Bloxham & West, 2004; O’Donovan, Price & Rust, 2001; Bell, Mladenovic & Price, 2013).
  • many students find the language used in rubrics and grade descriptors to be subjective and vague (Price & Rust 1999).
  • providing more detailed criteria can paradoxically increase students’ anxieties and “lead them to focus on sometimes quite trivial issues” (Norton, 2004, p. 693), with some students leaning heavily on rubrics and exemplars as ‘recipes’ (Bell et. al., 2013). As we’ll see, this was an issue in the course we examined this year.

The answer to the last research question is still very much in development. However, thinking about multimodality, it’s really useful to also consider how students are producing materials that reflect changing dynamics of space and time, and the significance of how these digital assignments circulate:

The multimodal production of culture [is] characterised by changing dynamics of space and time, dynamics that are changing the meanings and effects of cultural production and distribution (Leander and Vasudevan 2009, p. 130).

Production of assignments can feel even higher stakes when they are public in some form – for example, the course we looked at this year asked all students to upload their final videos to youtube. These kinds of mobilities intersect with technical skills, composition elements, modes and meaning to determine what ends up in these assignments, and a need for a nuanced understanding of the “complex ways in which technical skills, composition elements, modes, and meaning interact” in student work (Curwood 2012, p. 242). Greater attention to materiality, including artefacts (Pahl & Rowsell, 2011), movement (Leander & Vasudevan, 2009), and place (Ruitenberg, 2005) enriches this understanding.

Methodology:

Over two semesters we analysed the creation and assessment of a single assignment on an undergraduate course about film and theatre at the University of Sydney. About 130 students take this course each year, mostly study abroad or international students. The assignment we looked at was the final assignment – a three minute film made in pairs about some aspect of students’ ‘Australian cultural experience’. There were specific technical requirements for the film, and an emphasis on ‘narrative’.

  • Stage 1: analysing existing processes and assignments, conducting interview and focus groups with students, interviews with tutors, and developing an assessment framework.
  • Stage 2: redesign of the assessment task, building on the new framework.
  • Stage 3: comparative analysis of the old and new assignments, further interviews.

The original assessment rubric for the course was divided into three sections – cultural narrative experience, cinematic elements, and collaboration. The second section – on cinematic elements – had the most detail and specificity. The ‘narrative’ section referred to a ‘sophisticated’, ‘adequate’ or ‘not adequate’ narrative. The collaboration section talked about ‘high order’ personal statements. Unsurprisingly, students tended to focus much more on meeting the criteria in the second section than the first – this meant that while often very technically proficient, the narratives varied considerably in the extent to which they connected with the critical themes of the course, for example.

Findings:

Here are some of the things that people said in our initial interviews. This first quote comes from a tutor, who talked about how he used the criteria, but also how he used his own judgement when something might not have exactly followed the technical criteria but was ‘absolutely brilliant’:

[one group] used one interview but used it extremely well. I’m quite flexible and adaptable when it comes within the criteria. So if something is absolutely brilliant, of which this one was overall, then I wouldn’t penalise them. They really still came up here in the ‘exceeds criteria’ which is why they ended up getting a high distinction. (Tutor, Interview 1)

The question of what can be contained within the rubric and what, by necessity, goes beyond it in these types of assessments, is a central one for this project.

One of the students on the course talked about struggling to understand what was meant by ‘narrative’ and what was expected in this respect. They were clear about the technical expectations, but unclear about what to do about the narrative dimension.

We knew we needed a lot of cool angles, and different shots, so we started thinking ‘What would be really neat and catching to eye?’ The thing we struggled with looking at the rubric was the narrative, having a narrative, but everything else we were able to look at and make sure was in the project. (Carla, Focus Group 2)

The rubric guided students in the use of discipline-specific vocabulary and highlighted the importance of collaboration in reflecting on the meaning of Australian culture and representing it within a multimodal composition. Students felt, though, that the rubric ‘left a lot of room for interpretation’. As Carla added, “The Australian cultural experience from the videos [viewed as a class after submission] meant so many different things. I liked that it was open…but then again that’s also the challenge…” This highlights the importance of agency and creativity, but a tension exists with the tutor’s responsibility to communicate expectations and fairly assess student learning. One student noted, after the marks were returned and she knew she had done very well on the assignment:

when it says ‘the video demonstrates a sophisticated Australian cultural experience narrative, I don’t really know what [the tutor] means by sophisticated. Personally our project was more humorous, I don’t think you’d look at our video and say ‘That’s a sophisticated piece of art’. …But I still got really high marks on my assignment, and so really vague words like ‘sophisticated’, I think really limits people’s creativity. …[the students] don’t exactly know what [the tutors] want.(Sarah, focus group 1)

She was clear that their assignment wasn’t ‘sophisticated art’, but it still did well – so what was going on here?

It was clear from the reflective interviews students included at the end of their assignments that they were trying to grapple with both form and content, and how these intersect. In looking at these artefacts and talking to students and teachers, we think there are five main things that teachers need to take account of in developing multimodal assessments.

  1. Students need support to develop multimodal assessment literacy. This means being able to parse rubrics and criteria to understand what is being looked for, and how to identify it, and also to understand this process as a dialogue rather than a fixed and objective measurement. So for example where teachers say ‘sophisticated’, they need to be clear what they mean by this, and how they judge it when they see it.
Screenshot from the video of the Manifesto for Teaching Online, James Lamb. https://vimeo.com/222406740
  1. Some students talked about feeling constrained by the language of the rubrics, or sometimes by the fairly rigid instructions about what kind of technology to use. Where constraints can be removed that might be a good thing, sometimes, but sometimes those constraints can interact with creativity in quite generative ways. Teachers might think about and keep an eye on how these creative constraints are operating in the assignments students produce.
  2. The intra-action of form and content – and I mean this in Karen Barad’s sense, in terms of how they create and shape one another rather than existing separately and interacting. This is a really important site of criticality in multimodal work – as my colleague James Lamb says, there is a need to consider the coherence between modes (and, where there is dissonance, to be deliberate about this). We should consider if our students know how to create a ‘multimodal argument’.
  3. Trying to look holistically at multimodal assignments is challenging if teachers are also working with rubrics. Those rubrics – especially where they specify technical elements – can easily tend towards what Bateman calls ‘multimodal decomposition’, and that can leave students trying to follow recipes, as Amani noted earlier.
  4. Last but not least, teachers have to consider what they are asking students to do, and how to value it appropriately. A digital assignment isn’t a throwaway task – it often involves substantial learning, work and creativity, and its weighting within the course – in terms of time and assessment – needs to be carefully considered.

We’re working now on our framework, which has four dimensions – criticality, cultivating creativity, taking a holistic approach, and valuing multimodality. More to come on this! Our next steps are to finalise this framework, write and publish from this initial stage of the research, and develop some new ideas and use cases. We’re having conversations with our colleagues who led the other strand of the project, about automating feedback. We think it would be really useful to explore the potential for supporting these kinds of complex assignments through automated processes – lots to discuss!

References

Bateman, J. (2012). The decomposability of semiotic modes. In K. O’Halloran and B. Smith (Eds.), Multimodal studies: Exploring issues and domains. New York: Routledge.

Bell A, Mladenovic R and Price M (2013). Students’ perceptions of the usefulness of marking guides, grade descriptors and annotated exemplars. Assessment & Evaluation in Higher Education. 38(7), 769-788.

Bloxham, S., and A. West. 2004. Understanding the rules of the game: Marking peer assessment as a medium for developing students’ conceptions of assessment. Assessment & Evaluation in Higher Education29(6): 721–733

Curwood, J.S. (2012). Cultural shifts, multimodal representations, and assessment practices: A case study.  E-Learning and Digital Media, 9(2), 232-244.

Leander, K. M., & Vasudevan, L. (2009). Multimodality and mobile culture. In C. Jewitt (Ed.), Handbook of multimodal analysis(pp. 127-139). New York, NY: Routledge.

Miles, M.B., Huberman, A., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook. Thousand Oaks, CA: Sage.

Norton, L. (2004). Using assessment criteria as learning criteria: A case study in psychology. Assessment & Evaluation in Higher Education,29(6), 687–702.

O’Donovan, B., Price, M. & Rust, C. (2001). The student experience of criterion-referenced assessment. Innovations in Education and Teaching International,38(1), 74–85.

Pahl, K.H. & Rowsell, J. (2011). Artifactual critical literacy: A new perspective for literacy education. Berkeley Review of Education, 2(2), 129-151.

Price, M., & Rust, C. (1999). The experience of introducing a common criteria assessment grid across an academic department. Quality in Higher Education, 5(2), 133–144.

Ruitenberg, C. (2005). Deconstructing the experience of the local: Toward a radical pedagogy of place. Philosophy of Education Archive, 2005, 212-220.

 

surveillance, trust & technology: higher education futures

Several of the research and writing projects and discussions I’ve been involved with this year have directly or indirectly addressed issues of trust and surveillance, and this is an area of work I’m planning to develop further, along with a number of great colleagues.

In several talks this year, at Strathclyde in Glasgow; the University of Sydney; and the Networked Learning Conference in Zagreb, I’ve been drawing on our Manifesto for Teaching Online to develop my thinking about plagiarism detection and attendance monitoring – two routine processes which are becoming increasingly intensive and often invasive in most universities. For example, Hamish Macleod and I recently argued that

Logics of surveillance are strongly at work in practices which attempt to regulate student behaviour through the exposure of their writing to algorithmic scanning and monitoring. These logics frame students as in need of careful monitoring to ensure learning and teaching runs smoothly, and framing academic writing as a space of dishonesty which is both rampant and solvable through technology. Routines of plagiarism detection intervene negatively in one of the central facets of student-teacher relationships: the production and assessment of student work. Where these relationships become risk-averse and mutually suspicious, trust is blocked or lost and not easily regained. (Ross and Macleod 2018, 235)

We end that paper by arguing that we need to find “ways to re-sensitise ourselves and our students to the values we want to prioritise in our classrooms, and offering means by which students can voice their responses to surveillance cultures in higher education; and [address] issues at strategic levels within our institutions and the sector more widely by developing robust mechanisms for engaging in critical debate, discussion about and review of technology platforms and practices”. This second point echoes one of the key observations made by Laura Czerniewicz at her keynote address in Zagreb last month:

My talk in Zagreb (Hamish couldn’t make it, being retired and all!) generated quite a lot of interest, and it’s clear this is resonating with people from across a number of higher education contexts:

I’ve been involved in a number of intense discussions on these issues, during my seminars in Glasgow and Sydney, and with colleagues in the UK and North America (including Amy Collier and George Veletsianos). A highlight was a discussion with David Lyon and Sava Saheli Singh at Queen’s University in Canada last month. David’s latest work on surveillance culture is greatly informing my thinking. Colleagues closer to home are also doing fantastic work in related areas – Sian Bayne on anonymity; Jeremy Knox on learning analytics, Phil Sheail on data bodies in the library, Ben Williamson on monitoring in the school classroom.

The relationship of trust and technology in higher education is one that, in my view, requires a lot more attention.

It’s emerging as a key issue not only in the context of technologies of monitoring and surveillance, but in other more surprising places – like the current blockchain craze, which is generating new discussions about all sort of things…

…but perhaps less than it should about what kinds of models of trust (or trustlessness) we are inviting into the academy with these new technologies. Helen Murphy, one of the participants on my Digital Futures for Learning course this year, developed an excellent Open Educational Resource on this topic: https://educationandtheblockchain.weebly.com

She says:

the blockchain is often described as ‘trustless’. We might think of what it means to be ‘trustless’ in two ways:
First, data.
The data in the blockchain is immutable and transparent, permanent and unalterable. Data in the blockchain is added by consensus, and because it is distributed there is no single copy of the data. In theory, it can be accessed by anyone. So it is trustless in the sense that no trust is required: it can be taken for granted that the data is accurate and permanent.
​Second, decentralisation.
With the blockchain, there is no need for a centralised authority (such as a bank or university) to verify transactions. Instead all of this verification can be done independently, by the technology and the mechanisms by which it works. So it is trustless because it does not require any trust in these third-party institutions.

(Murphy 2018)

I’m doing some work with colleagues in Digital Education and Information Services on exploring the potential and issues around blockcert technology, and these questions of trust and trustlessness are figuring strongly.

So, look out for more writing on this from me and others in the near future!

One of the most important things, I think, is beginning to speculate and design futures for higher education that are ‘beyond surveillance’. Importantly (and thanks to Peter Goodyear in Sydney for highlighting how important this point is), this doesn’t mean alternatives that require us to return to imagined better analog times  – instead, we need to develop trusting digital futures and approach our technologies critically and creatively to help us do so.  I, and others, will be working on this over the next year – get in touch if you want to discuss!

References:

Lyon, D. 2018. The Culture of Surveillance: Watching as a Way of Life. Polity Press.

Murphy, H. 2018. Education and the Blockchain. https://educationandtheblockchain.weebly.com

Ross, J. and Macleod, H. 2018. Surveillance, (dis)trust and teaching with plagiarism detection technology. Proceedsing of Networked Learning 2018, Zagreb.

innovating with assessment methods – a project with the University of Sydney

Along with Professor Dragan Gasevic here at Edinburgh, and  Dr Jen Scott Curwood, Associate Professor Abelardo Pardo, and Dr Amani Bell from the Centre for Research in Learning and Innovation at the University of Sydney, this year I’ve been awarded a Partnership Collaboration Award on the topic of Methodological Innovations for Assessing Learning in Digital Spaces. The project is building on research connections between the two Centres, weaving together two complementary strands: an approach to multimodal assessment and a framework to analyse learning strategies in digital spaces. Jen, Amani and I are working on the first strand, developing new insights into the nature of digital assignments and methodologies for their design and assessment, drawing on theories of place-based learning, mobilities and multimodality.

Along with getting to work closely with Jen and Amani (and returning to the lovely Sydney in March), and to think very broadly about assessment methodologies with all the partners, this project is also a brilliant chance to bring together expertise from a number of colleagues here in the Centre: James Lamb, Sharon Boyd, Yi-Shan Tsai and Sian Bayne are all contributing their insights to the multimodal assessment strand.

There’s a bit more information about this collaboration on the Digital Education site, and I’ll share findings and materials here as they develop.

 

 

digital cultural heritage research update

One of the four main strands of research I pursue  is about digital cultural heritage engagement and learning, and lots has been happening!

In London for the Critical Heritage Futures conference.

The Artcasting project continues to generate great speaking, writing and conversation opportunities – most recently at the AHRC and Association of Critical Heritage Studies’ Critical Approaches and New Directions conference in London in early October. My colleague Michael Gallagher and I gave a talk about mobilities, mobile technologies and heritage futures. Particular highlights for me from the event were Chris Whitehead‘s keynote, and Hayden Lorimer’s fascinating overview of work at St Peter’s Seminary at Kilmahew. It was also fantastic to see so many of the people I first met last year at the ACHS conference in Montreal, and to hear that Liz Stainforth is going to be spending a few months in Edinburgh as an IASH fellow at the start of next year.

Also re Artcasting, I’ve just finished and submitted a paper about hospitality and digital co-production; and the team recently reconvened with a number of others who have helped us a lot in thinking about Artcasting futures, to talk about a whole bunch of fascinating issues still to be explored. There is a summary of these on the Artcasting site blog.

Beyond Artcasting and its ripples, London also allowed a bit of time for Koula Charitonos and I to talk through some ideas – including to propose a symposium for next year’s ICLS conference on museum visitor experiences in the digital age. We hope this will be able to come together!

Here in Edinburgh, the new academic year has brought a number of new members to the Digital Cultural Heritage Research Network – not least our new chair of digital cultural heritage, Melissa Terras, who joined the University of Edinburgh this week. We’ve already tempted her to get involved with a couple of projects in development – it’s going to be great to have her here. Sian, Chris, James, Kirsty, Melissa and I hope to organise a few DCHRN events in the new year.

Serendipitously, I’ve got the opportunity to supervise not one but two masters students this year who are exploring aspects of 3d printing, scanning and visualisation in cultural heritage contexts – I look forward to learning lots from both of them about this topic.

Last but not least, I’ll be in Manchester for the Researching Digital Cultural Heritage conference on 30 November-1 December (sadly only for the second day) – the programme looks amazing.

Activist arts and youth inequalities workshop, 7 June 2017

I’m spending the day with a fantastic group of artists, youth organisation representatives, researchers and arts experts for a workshop at the Scottish National Portrait Gallery, aiming to collaboratively explore and develop ideas for a project to explore the potential of contemporary art to raise awareness, provoke thought and motivate change in relation to youth inequalities. I’ll post some notes and other materials later, but we’ll be using the hashtag #artineq.

The other organisers of the event are Marlies Kustatscher and Alan Brown.

Notes from keynote lecture, “Learning with Digital Provocations”

I was delighted to give a keynote talk at the Digital Day of Ideas here at the University of Edinburgh on 17 May 2017. Here are the slides, notes and references from my talk.

“Learning with Digital Provocations”

Abstract: One of the most significant tensions in the convergence of technology and education is how the promise/threat of ‘disruption’ comes up against theories, practices and structures of formal and informal education. Disruption in educational technology contexts has come to be aligned with neo-liberal discourses of efficiency, enhancement, personalisation, scale and automation; and we can be forgiven for cynicism about its critical and creative potential in education. This talk aims to reanimate the debate by reframing disruption in terms of inventiveness, provocation, uncertainty and the concept of ‘not-yetness’. Focusing on the recent AHRC-funded Artcasting project, and with other examples drawn from the work of the Centre for Research in Digital Education at the University of Edinburgh, it argues that inventive digital approaches can help us develop critical responses to assumptions about the role of the digital in contexts including higher education, museums and galleries.

Slides:

 

Notes:

This talk brings together several ideas I’ve been working on over the past few years, alongside the best colleagues anyone could hope for, some of whom are in the room today. The message I’ll leave you with is ultimately a hopeful one, so you can look forward to that. But before we get there, I need to talk about the challenges of teaching and researching in the context of other things that are going on in the sphere of educational technology. I’m going to start by telling you about the concept of ‘disruption’ and why, for a lot of us teaching and researching in digital education, it’s become a dirty word. You can play along if you know what to look out for.

So: next time you read a news article about educational technology, see how well you do in disruption bingo:

  • Universities and schools are broken, failing, out of date
  • Digital natives/millenials/?? demand, expect, deserve
  • Teachers resist
  • Efficiency, speed, simplicity through better technology!
  • Personalisation/individualisation is key
  • Satisfaction guaranteed

Harvard Business School professor Clay Christensen has been an important promoter of ideas about how education is fundamentally broken and in need of disruption through technology. We’re not going to dwell on him for long, but his framing of disruption has been very influential. Here’s Professor Christensen writing in 2014 about massive open online courses – the huge, free courses offered by universities through platforms like Futurelearn, Edx and Coursera.

“In 2013, we witnessed aggressive discounting strategies as well as schools experimenting with lowering net — not sticker — prices in an effort to recruit students.”

“Free access to content from prestigious institutions revealed that content didn’t need to be proprietary.”

“Faculty have been forced to reassess how and why they teach the way they do.”

…“Many colleges and universities resist the idea of training students for jobs. Yet it is employers who are truly the ultimate consumers of degree-holders.”

The first three points might or might not seem objectionable to you, but I do want to draw attention to the fourth, which was buried rather far down in the article, right near the end. The idea that students are the product and employers are the consumers is a rather striking point from which to disrupt/fix/reimagine education, don’t you think?

When you see headlines like this – “many universities are resisting online learning”  – I would encourage you to consider that resisting educational technology and online learning, at least in the form it’s being offered by thinkers like Christensen and the companies, institutional actors and others that embrace these types of philosophies, might be a highly principled position. We’ll come back to this.

Another common line of reasoning in the disruptive technology for education genre is the analogy to failed business models in other sectors. A well-known one was Clay Shirky’s analogy of how napster destroyed the music industry, and why universities need to take note of this. Here’s another one – Blockbuster. Where is blockbuster today? And what does this have to do with education? Well, according to Sheninger, incremental change is not enough to save the education system. Only disruptive strategies will save education from being the next Blockbuster.

There is literally no area of education that is immune from the idea that technology will disruptively improve it. This is from this year’s Future EdTech conference web site:

Digital Disruption in Education: Institutions have far understood that traditional means of communications are not efficient anymore when engaging with generation Y. Reaching out to recruits, current students and alumni, positioning your brand and communication can be massively assisted with the use of new tech that offers unprecedented insights and opportunities to personalise communication.

Note the corporate language and the appearance of generation y and personalisation to talk about communicating with students.

Pervasive rhetoric of disruption makes those in the education technology sector especially open to ‘the next big thing’, whatever it might be. As an example that emerged on the scene last year, I want to talk about the emergence of the concept of ‘blockchain’ for education.

Bitcoin is a digital currency in which transactions can be performed without the need for a credit card or central bank… The blockchain is a public ledger of all transactions in the Bitcoin network. –  https://blockchain.info/wallet/bitcoin-faq

Blockchain is really fascinating, and if you want to learn more about it, you might like to check out Professor Chris Speed’s ESRC project called ‘after money’ (http://aftermoney.design ).

There are some big promises being made for what blockchain could do for society:

let’s put health records, voting, ownership documents, marriage licenses and lawsuits in the blockchain. Eventually, every dataset and every digital transaction could leave a “fingerprint” there, creating an audit trail for any digital event throughout history, without compromising anyone’s personal privacy. [blockchain] could introduce a level of democracy and objective “truth” to the digital world that even the physical world can’t match. Its promise involves a future in which no one has absolute power online, and no one can lie about past or current events. (http://recode.net/2015/07/05/forget-bitcoin-what-is-the-blockchain-and-why-should-you-care/ )

Education is no exception. This is a design fiction, based on what the creators called ‘edublocks’. This video was made by the Institute for the Future and the Act Foundation to generate discussion and debate, so we can assume it is aiming to be at least a bit controversial. Bear that in mind as we enter a world where ‘learning is earning’.

Controversial or not, the concept of blockchain for education has caught on, and Tapscott & Tapscott assure us, disruption is sure to follow.

“the blockchain represents nothing less than the second generation of the Internet, and it holds the potential to disrupt money, business, government, and yes, higher education.”

Big players are getting into this. Last year Sony moved into ‘blockchain for education’, and two weeks ago Google announced that its Classroom platform will be opened up so that ‘anyone’ can become a teacher. The infrastructure, the funding and the desire is lining up behind making edublocks, or something like them, a reality. What does it mean to work critically in such a space?

I think that pervasiveness of the discourse of disruption means that it is something educators and educational researchers have to grapple with. I’ve hinted at some of the difficulties I have with the underpinning values being expressed through the concept of disruptiveness. Now I want to make explicit what we need, as digital educators, scholars and researchers, to take into account if we want to reconfigure this debate.

Neil Selwyn offers a useful set of critical questions we can ask when faced with calls for disruptive educational technology:

What is actually new here?

What are the unintended consequences or second-order effects?

Who is pushing these ideas in education? What are their reasons for doing so? What wider agendas are attached to these conversations?

What is being said about education that might be useful? What is being said about education that teachers might wish to challenge and talk back to?

(Selwyn 2015, p183)

We shouldn’t necessarily take claims about disruption at face value – I would argue, with Sian Bayne, that a lot of the current ideas about disruptive technology in education are popular not because they are so radical, but precisely because they leave what she calls ‘deeply conservative assumptions’ about education in place:

[Technology Enhanced Learning] carries with it a set of discursive limitations and deeply conservative assumptions which actively limit our capacity to be critical about education and its relation to technology. At the same time, it fails to do justice equally to the disruptive, disturbing and generative dimensions of the academy’s enmeshment with the digital. (Bayne 2015, p.7)

Rhetoric of disruption, like that of enhancement, doesn’t often signal a willingness to grapple with what is genuinely ‘disruptive, disturbing and generative’. Lesley Gourlay suggests that one of the big ‘disruptive’ ideas in education technology, openness, often carries with it some really troubling fantasies, including one of moving away from engagement with contestable knowledge, towards access to content:

“The fantasy [of openness] appears to be one of total liberation from the perceived constraints of formal study, the rigours of assessment and engagement with expertise and established bodies of (contestable) knowledge, all of which are activities deemed hierarchical and repressive of creativity. The emphasis is instead reduced to access and the online generation of content– which carries with it a further powerful fantasy of unfettered human potential which can be unlocked unproblematically in informal lay interaction. (Gourlay 2015, p.8)

(Jeremy Knox has also done excellent work around the problems of framing openness as ‘access alone’ – see references for more on this.)

Audrey Watters – talking about Sebastian Thrun (Udacity for-profit educational platform founder), and the context of market demands, precarious labour and automation – has a lot to say about the ‘uberification’ of education. As she points out, Universities are holding on to accreditation and other protections… for now (but see blockchain above).

Too often, what we see when we go digging into disruption is an impulse for change that isn’t honest about its assumptions, and that can be destructive. That’s why, for me, the most important thing about working in scholarly areas that focus on the digital is how we bring together sharp and critical analysis with creativity. That is: how do we work in a way that shows how things are, but also makes space to actively explore how they might be otherwise? What kinds of provocations can help us do this? I think what we need are approaches that are more surprising, and subversive, and imaginative.

I first started thinking seriously about the role of provocation in 2011, when my colleagues and I were working on a project about online writing, assessment and feedback. We thought about writing some principles, but we decided that what was needed in our field at that time was not ‘principles’ or yet another version of ‘best practice’, but something altogether more provocative – a manifesto. The 2012 and 2016 versions of the manifesto for teaching online have been really important sources of discussion, debate and inspiration for us and our students, colleagues and others in the field.  The series of short statements are grounded in our own and others’ research, and they open up a range of critical questions and ideas about digital education, framing these in active ways (we tried not to simply repudiate everything). This video interpretation of the manifesto, made by our excellent colleague and PhD student, James Lamb, gives a good way in.

The Manifesto for Teaching Online (2016) from james858499 on Vimeo.

The manifesto and an invitation it generated in 2012 led directly to meeting Dr Amy Collier, now at Middlebury College in Vermont. Amy and I have been working for the past few years on what we’ve found to be a generative way of thinking about the future of educational technology. As Neil Selwyn points out:

‘technologies are subjected continually to complex interactions and negotiations with the social, economic, political and cultural contexts into which they are situated.’(Selwyn 2012, 214–15)

The category of ‘emerging technologies’ is especially complex, since it brings the future into the equation. Our colleague George Veletsianos talks about these as “not yet fully understood” and “not yet fully researched, or researched in a mature way” (Veletsianos 2010, 15). Amy and I have coined the term ‘notyetness’ to capture something of this complexity and uncertainty:

Practices, identities, pedagogies and technologies can be marked by this ‘not-yetness’ (Ross & Collier 2016)

Not-yetness works in the service of a messier understanding of what constitutes higher education, and how technologies act in this space; and it engages with complexity, uncertainty and risk, not as factors to be minimised or resolved, but as necessary dimensions of technologies and practices which are unknown and in flux. We want to continually centre this complexity, mess and uncertainty when we think about the future of education and its technologies.

For me, that applies equally to research, and I’ve been developing my own understanding of what that means over the past few years. It’s helping me understand what ‘creatively critical’, provocative methods can do when applied to digital education. The best way I’ve found so far to think about this is through the lens of speculative method. These methods are increasingly used in design disciplines, and they include approaches like design fictions. They’ve also been taken up in the social sciences in a range of conceptual and empirical ways. Speculative (or inventive method)

  • is ‘explicitly oriented towards an investigation of the open-endedness of the social world. … the happening of the social world – its ongoingness, relationality, contingency and sensuousness’ (Lury and Wakeford 2012, 2).
  • is aimed at envisioning or crafting futures or conditions which may not yet currently exist.
  • provokes new ways of thinking and brings particular ideas or issues into focus.
  • may blur boundaries between research, design and teaching.
  • involves considerations around epistemology, temporality and performativity.

For me, three key elements to speculative method are epistemology, temporality and performativity. I’ve written about this in a recent journal article in Learning Media and Technology which you might like to check out if you want to read more.

These methods challenge linearity and replicability – and they are explicitly about the kinds of questions being asked. Questions about the future, provocative questions, questions that create their own conditions of answerability – conditions that didn’t exist before:

the ‘answerability’ of a problem is introduced by crafting a method specifically to address that problem. (Lury & Wakeford 2012)

Wilkie, Michael, and Plummer-Fernandez (2015) argue that methodology itself is ‘a process of asking inventive, that is, more provocative questions’ (p.4)

Speculative methods are also, in their focus on the future, very enmeshed in ideas about time. Visions of the future generate effects in the present, and our fictions and inventions are shaped by issues we inherit, and closed off from futures we can’t yet imagine. Furthermore, the effectiveness of inventive methods ‘cannot be secured in advance’ (Lury & Wakeford 2012).

Importantly, speculative method relies on engaging with and provoking various kinds of publics – and how those publics, or audiences, or participants respond to ‘objects to think with’ determines the nature of the problem and its answerability.

I’m going to spend the last part of my talk sharing some examples of projects in the Centre for Research in Digital Education  that have used speculative method to introduce creative criticality into areas that really needed it! I’m going to mention two briefly, then talk in a little more detail about the third, the Artcasting project.

The first of these projects tackled automation and the role of the teacher. This was the ‘teacherbot’ project, led by Sian Bayne, with colleagues from Digital Education, Informatics and Design Informatics. In this project we worked together to create an automated twitter bot that would respond to participants on the E-learning and Digital Cultures MOOC. Its responses were written by the course team and triggered by particular keywords that appeared in the #edcmooc hashtag. Participants engaged with it in all kinds of ways, both playful and serious. Teacherbot helped us explore, alongside MOOC participants, responses to automation and different ways of thinking about how humans and machines might teach together.

The second project I want to mention is the “learning analytics report card”, or LARC, project, led by Jeremy Knox: https://larcproject.wordpress.com . LARC was designed to explore how Universities can develop critical and participatory approaches to educational data analysis. It did this by creating an interface for data from the Moodle virtual learning environment, where students could chose the types of reports they wanted to have generated about them, and would get ‘plain english’ results that we’ve seen from engaging in them with students can be really generative and thought-provoking. By revealing data that is usually hidden, and presenting it in playful and provocative ways, the project is getting people more critically engaged with ideas around algorithmic culture and educational policy.

Finally, I want to tell you about the Artcasting project. This was an AHRC funded project that set out to find more inventive ways of evaluating how visitors engage with artworks. We did this by designing and trialling a mobile method for inviting, capturing and analysing people’s connections between art and place through an app we called artcasting. The academic team was Chris Speed, Claire Sowton, Jeremy Knox and me, with Chris Barker providing the software engineering, and we worked with National Galleries of Scotland, Tate and the ARTIST ROOMS programme over the course of a year.

ARTIST ROOMS:

  • is jointly owned and managed by Tate & National Galleries of Scotland
  • a collection of more than 1,600 works of international contemporary art was acquired in 2008 by National Galleries of Scotland and Tate.
  • is shared throughout the UK in a programme of exhibitions organised in collaboration with local associate galleries.
  • aims to ensure the collection engages new, young audiences.

Evaluation is a really thorny issue in the cultural sector at the moment, and there has been a lot of critique of instrumental approaches to evaluation and to measuring engagement, learning and impact. Belfiore and Bennett are two of the key people working on this, and they are pretty stark about the issues at stake.

with the present levels of knowledge around aesthetic reception, it is not possible to make any meaningful broad generalization about how people respond to the arts, and if or how they might be affected by the experience. Even less plausible is the possibility of actually “measuring” any of these aspects. (Belfiore & Bennett 2010, p.126)

Claire Sowton brought these and a lot of other critical perspectives on evaluation to the project from her doctoral work.

Galleries and museums are generally good at thinking about how people get to the building, but other kinds of movement are less visible. We were hugely interested in understanding museum and gallery learning from a theoretical perspective that took into account social, spatial and technological mobilities. We saw the movement of artworks, people, ideas, inspiration and technologies as a really central part of what it means to understand the impact of art on visitors.

So we built this app as a way of testing out whether more inventive ways of approaching evaluation could be productive for visitors and for gallery professionals. Artcasting was developed as a methodology that could capture articulations of engagement with artworks. We tested it out in two exhibitions – Robert Mapplethorpe at the Bowes Museum in Co Durham, and Roy Lichtenstein at the National Galleries of Scotland. Read more about how Artcasting worked: https://www.artcastingproject.net/what-is-artcasting/

In total we ended up with about 170 casts that we could work with to try to understand what this approach could do. Here are a few examples of the kinds of artcasts people sent.

The Artcasting data could be seen collectively as more or less stable ‘traces’ of memory, insight and message. But we also explored how those traces can be a volatile assemblage of engagement that can move in multiple directions away from a single point (for example a single artwork cast to many places and times). Interpretation, like the objects that spark it, is ambiguous and shifting – and the impacts that artworks and galleries have are never going to be easy to pin down.

“it is always possible to take an individual object and place it in a new framework or see it in a new way. The lack of definitive and final articulation of significance keeps objects endlessly mysterious – the next person to attach meaning to it may see something unseen by anyone else before.” (Hooper-Greenhill 2000, 115)

This is really consequential for how institutions talk about and engage with evaluation, and this project has helped us and our partners and others in the sector to think more imaginatively about these issues.

I’ve also been working on how to conceptualise ‘digital co-production’, using ideas about artcasting as a stimulus. Artcasting content is requested and is able to be interpreted by gallery professionals for accountability, audience development, and other purposes. But artcasting is also a form of public interpretation of the artwork, and visitors are creating new encounters with art in new places and times. The guest becomes the host of a new exhibition. This has implications for what I’m calling ‘digital co-production’, which

  • unfolds across multiple times and spaces
  • involves the ‘unknowable other’
  • challenges the stability of relationships
  • invites a rethinking of hospitality

There is a lot we can say in 2017 at a digital day of ideas about how some of the digital ideas that we’ve been engaging with over the years look pretty scary when socio-political landscapes shift. (digital labour is another context where the rhetoric and practices of digital disruption are wreaking havoc – I’d refer you to the excellent work of our colleague Karen Gregory for more on this)

For me, this makes it even more important that we use technology provocatively rather than instrumentally, to explore big questions and possible futures, and to challenge assumptions. Some of those assumptions seem to be baked right into the DNA of devices, services, and organisational policies, but I believe there is (in the immortal words of Leonard Cohen) a crack in everything, and that’s how the light gets in. I’d like to encourage us all to keep working on how to keep being creative, to keep doing things, making things and trying things – as a way of bringing criticality to the urgent and interesting questions we are facing.

References:

Bayne, S. (2015). What’s the matter with ‘technology-enhanced learning’? Learning, Media and Technology, 40(1), 5–20.

Bayne, S. (2015). Teacherbot: interventions in automated teaching. Teaching in Higher Education, 20(4), 455–467.

Belfiore, E., & Bennett, O. (2010). Beyond the ‘Toolkit Approach’: arts impact evaluation research and the realities of cultural policy-making. Journal for Cultural Research, 14(2), 121–142.

Collier, A., & Ross, J. (2017). For whom, and for what? Not-yetness and thinking beyond open content. Open Praxis, 9(1), 7–16.

Gourlay, L. (2015). Open education as a ‘heterotopia of desire’. Learning, Media and Technology, 40(3), 310–327.

Hooper-Greenhill, E. (2000). Museums and the Interpretation of Visual Culture. Abingdon: Routledge.

Isard, A., & Knox, J. (2016). Automatic Generation of Student Report Cards. In The 9th International Natural Language Generation conference (p. 207).

Knox, J. (2016). Posthumanism and the Massive Open Online Course: Contaminating the Subject of Global Education. Routledge.

Knox, J., & Ross, J. (2016). ‘Where does this work belong?’ New digital approaches to evaluating engagement with art. Presented at the MW2016: Museums and the Web 2016, Los Angeles. Retrieved from http://mw2016.museumsandtheweb.com/paper/where-does-this-work-belong-new-digital-approaches-to-evaluating-engagement-with-art/

Lury, C., & Wakeford, N. (2012). Inventive Methods: The Happening of the Social. London: Routledge.

Ross, J. (2016). Speculative method in digital education research. Learning, Media and Technology, 0(0), 1–16.

Ross, J., & Collier, A. (2016). Complexity, mess and not-yetness: teaching online with emerging technologies. In G. Veletsianos (Ed.), Emergence and Innovation in Digital Learning: Foundations and Applications. Athabasca University Press.

Ross, J., Sowton, C., Knox, J., & Speed, C. (in press). Artcasting, mobilities, and inventiveness: engaging with new approaches to arts evaluation. In L. Ciolfi, A. Damala, E. Hornecker, M. Lechner, & L. Maye (Eds.), Cultural Heritage Communities: Technologies and Challenges. London: Routledge.

Selwyn, N. (2015). Never believe the hype: questioning digital ‘disruption’and other big ideas. Teaching and Digital Technologies: Big Issues and Critical Questions, 182.

Selwyn, N. (2012). Ten suggestions for improving academic research in education and technology. Learning, Media and Technology, 37(3), 213–219.

Veletsianos, G. (2010). Emerging Technologies in Distance Education. Athabasca University Press. Retrieved from http://www.aupress.ca/index.php/books/120177

Watters, A. (2017). Driverless Ed-Tech: The History of the Future of Automation in Education. http://hackeducation.com/2017/03/30/driverless

Wilkie, A., Michael, M., & Plummer-Fernandez, M. (2015). Speculative method and Twitter: Bots, energy and three conceptual characters. The Sociological Review, 63(1), 79–101.