{"id":13227,"date":"2018-06-21T14:28:49","date_gmt":"2018-06-21T14:28:49","guid":{"rendered":"http:\/\/jenrossity.net\/blog\/?p=13227"},"modified":"2018-06-22T08:16:19","modified_gmt":"2018-06-22T08:16:19","slug":"assessment-in-a-digital-age-rethinking-multimodal-artefacts-in-higher-education","status":"publish","type":"post","link":"http:\/\/jenrossity.net\/blog\/?p=13227","title":{"rendered":"Assessment in a digital age: Rethinking multimodal artefacts in higher education"},"content":{"rendered":"<p>I presented a very short \u2018pecha kucha\u2019 style talk yesterday at the inaugural <a href=\"http:\/\/www.docs.hss.ed.ac.uk\/iad\/Learning_teaching\/Academic_teaching\/Events\/LT_Conference\/Conference_Programme_FINAL_web_copy.pdf\">Learning and Teaching Conference at the University of Edinburgh<\/a>. I was speaking on behalf of two colleagues, Dr Amani Bell and Dr Jen Scott Curwood, who are members of the <a href=\"http:\/\/crlionline.net\/\">Centre for Research on Learning and Innovation at the University of Sydney<\/a>. This super quick talk was a summary of research we\u2019ve been doing into how teachers and students understand the assessment of students\u2019 digital multimodal work \u2013 work that incorporates multiple modes such as images, text, sound, video and hyperlinks.<\/p>\n<p>(Jen is presenting a poster of this work at the Festival of Learning next week in London: <a href=\"https:\/\/www.londonfestivallearning.com\/agenda\/session\/6811\">https:\/\/www.londonfestivallearning.com\/agenda\/session\/6811<\/a> (poster number 1900) &#8211; if you are there, do stop by to talk to her!)<\/p>\n<figure id=\"attachment_13230\" aria-describedby=\"caption-attachment-13230\" style=\"width: 474px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-13230 size-large\" src=\"http:\/\/jenrossity.net\/blog\/wp-content\/uploads\/Picture2-1024x680.png\" alt=\"\" width=\"474\" height=\"315\" srcset=\"http:\/\/jenrossity.net\/blog\/wp-content\/uploads\/Picture2-1024x680.png 1024w, http:\/\/jenrossity.net\/blog\/wp-content\/uploads\/Picture2-300x199.png 300w, http:\/\/jenrossity.net\/blog\/wp-content\/uploads\/Picture2-768x510.png 768w, http:\/\/jenrossity.net\/blog\/wp-content\/uploads\/Picture2.png 1622w\" sizes=\"auto, (max-width: 474px) 100vw, 474px\" \/><figcaption id=\"caption-attachment-13230\" class=\"wp-caption-text\">Willi Heidelbach, Metal movable type https:\/\/en.wikipedia.org\/wiki\/Typesetting#\/media\/File:Metal_movable_type.jpg<\/figcaption><\/figure>\n<p>This research is supported by an <a href=\"https:\/\/global.ed.ac.uk\/edinburgh-sydney-funding\">Edinburgh-Sydney Partnership Collaboration Award<\/a>, which funded travel between Sydney and Edinburgh and some of our research activity. The overall project is about methodological innovations for assessing learning in digital spaces, and it was designed to bring our two centres together to develop partnerships and research collaborations.<\/p>\n<p>Our strand of the project took a closer look at how digital assignments are being assessed and how this is working for students and teachers. These kinds of assignments are pretty common in a lot of disciplines now, and they can range from making diagrams to making films, web essays, infographics and portfolios. In some disciplines this is well established, but in others it\u2019s emerging as institutions look for ways to translate students\u2019 engagement with visual, interactive media spaces outside formal education, to critical capacities within it.<\/p>\n<p>However, when it comes to assessing these capacities, teachers might find their practices are still rooted in what Jen calls \u2018a paradigm of assessment rooted in print-based theoretic culture\u2019 (Curwood, 2012, p. 232). This can happen because of the constraints of assessment systems, or because of assumptions that teachers and students have about what constitutes legitimate knowledge production in universities, or a combination of things. So, we wanted our project to look specifically at how assessment \u2013 and particularly rubrics \u2013 takes account of multimodality.<\/p>\n<p>Our research questions were about how students use assessment criteria, how teachers design and assess these kinds of assignments, and how theories of mobilities and place-based learning could inform our thinking about these issues:<\/p>\n<ul>\n<li>How do university students use assessment criteria for self and peer assessment of multimodal work?<\/li>\n<li>How do teachers in higher education effectively design and assess students\u2019 multimodal work?<\/li>\n<li>How can theories of mobilities and place-based learning inform research into and assessment of multimodal student work?<\/li>\n<\/ul>\n<p>Exploring the literature around grade descriptors and rubrics, there is a wide range of student responses:<\/p>\n<ul>\n<li>some students are able to use them to accurately assess their peers\u2019 work, to guide and structure their own work, and as a checklist (Bloxham &amp; West, 2004; O\u2019Donovan, Price &amp; Rust, 2001; Bell, Mladenovic &amp; Price, 2013).<\/li>\n<li>many students find the language used in rubrics and grade descriptors to be subjective and vague (Price &amp; Rust 1999).<\/li>\n<li>providing more detailed criteria can paradoxically increase students\u2019 anxieties and \u201clead them to focus on sometimes quite trivial issues\u201d (Norton, 2004, p. 693), with some students leaning heavily on rubrics and exemplars as \u2018recipes\u2019 (Bell et. al., 2013). As we\u2019ll see, this was an issue in the course we examined this year.<\/li>\n<\/ul>\n<p>The answer to the last research question is still very much in development. However, thinking about multimodality, it\u2019s really useful to also consider how students are producing materials that reflect changing dynamics of space and time, and the significance of how these digital assignments circulate:<\/p>\n<blockquote>\n<p style=\"padding-left: 30px;\">The multimodal production of culture [is] characterised by changing dynamics of space and time, dynamics that are changing the meanings and effects of cultural production and distribution (Leander and Vasudevan 2009, p. 130).<\/p>\n<\/blockquote>\n<p>Production of assignments can feel even higher stakes when they are public in some form \u2013 for example, the course we looked at this year asked all students to upload their final videos to youtube. These kinds of mobilities intersect with technical skills, composition elements, modes and meaning to determine what ends up in these assignments, and a need for a nuanced understanding of the \u201ccomplex ways in which technical skills, composition elements, modes, and meaning interact\u201d in student work (Curwood 2012, p. 242). Greater attention to materiality, including artefacts (Pahl &amp; Rowsell, 2011), movement (Leander &amp; Vasudevan, 2009), and place (Ruitenberg, 2005) enriches this understanding.<\/p>\n<p><strong>Methodology:<\/strong><\/p>\n<p>Over two semesters we analysed the creation and assessment of a single assignment on an undergraduate course about film and theatre at the University of Sydney. About 130 students take this course each year, mostly study abroad or international students. The assignment we looked at was the final assignment \u2013 a three minute film made in pairs about some aspect of students\u2019 \u2018Australian cultural experience\u2019. There were specific technical requirements for the film, and an emphasis on \u2018narrative\u2019.<\/p>\n<ul>\n<li>Stage 1: analysing existing processes and assignments, conducting interview and focus groups with students, interviews with tutors, and developing an assessment framework.<\/li>\n<li>Stage 2: redesign of the assessment task, building on the new framework.<\/li>\n<li>Stage 3: comparative analysis of the old and new assignments, further interviews.<\/li>\n<\/ul>\n<p>The original assessment rubric for the course was divided into three sections \u2013 cultural narrative experience, cinematic elements, and collaboration. The second section \u2013 on cinematic elements \u2013 had the most detail and specificity. The \u2018narrative\u2019 section referred to a \u2018sophisticated\u2019, \u2018adequate\u2019 or \u2018not adequate\u2019 narrative. The collaboration section talked about \u2018high order\u2019 personal statements. Unsurprisingly, students tended to focus much more on meeting the criteria in the second section than the first \u2013 this meant that while often very technically proficient, the narratives varied considerably in the extent to which they connected with the critical themes of the course, for example.<\/p>\n<p><strong>Findings:<\/strong><\/p>\n<p>Here are some of the things that people said in our initial interviews. This first quote comes from a tutor, who talked about how he used the criteria, but also how he used his own judgement when something might not have exactly followed the technical criteria but was \u2018absolutely brilliant\u2019:<\/p>\n<blockquote>\n<p style=\"padding-left: 30px;\">[one group] used one interview but used it extremely well. I&#8217;m quite flexible and adaptable when it comes within the criteria. So if something is absolutely brilliant, of which this one was overall, then I wouldn&#8217;t penalise them. They really still came up here in the \u2018exceeds criteria\u2019 which is why they ended up getting a high distinction. (Tutor, Interview 1)<\/p>\n<\/blockquote>\n<p>The question of what can be contained within the rubric and what, by necessity, goes beyond it in these types of assessments, is a central one for this project.<\/p>\n<p>One of the students on the course talked about struggling to understand what was meant by \u2018narrative\u2019 and what was expected in this respect. They were clear about the technical expectations, but unclear about what to do about the narrative dimension.<\/p>\n<blockquote>\n<p style=\"padding-left: 30px;\">We knew we needed a lot of cool angles, and different shots, so we started thinking \u2018What would be really neat and catching to eye?\u2019 The thing we struggled with looking at the rubric was the narrative, having a narrative, but everything else we were able to look at and make sure was in the project. (Carla, Focus Group 2)<\/p>\n<\/blockquote>\n<p>The rubric guided students in the use of discipline-specific vocabulary and highlighted the importance of collaboration in reflecting on the meaning of Australian culture and representing it within a multimodal composition. Students felt, though, that the rubric \u2018left a lot of room for interpretation\u2019. As Carla added, \u201cThe Australian cultural experience from the videos [viewed as a class after submission] meant so many different things. I liked that it was open\u2026but then again that\u2019s also the challenge\u2026\u201d This highlights the importance of agency and creativity, but a tension exists with the tutor\u2019s responsibility to communicate expectations and fairly assess student learning. One student noted, after the marks were returned and she knew she had done very well on the assignment:<\/p>\n<blockquote>\n<p style=\"padding-left: 30px;\">when it says \u2018the video demonstrates a sophisticated Australian cultural experience narrative, I don\u2019t really know what [the tutor] means by sophisticated. Personally our project was more humorous, I don\u2019t think you\u2019d look at our video and say \u2018That\u2019s a sophisticated piece of art\u2019. \u2026But I still got really high marks on my assignment, and so really vague words like \u2018sophisticated\u2019, I think really limits people\u2019s creativity. \u2026[the students] don\u2019t exactly know what [the tutors] want.(Sarah, focus group 1)<\/p>\n<\/blockquote>\n<p>She was clear that their assignment wasn\u2019t \u2018sophisticated art\u2019, but it still did well \u2013 so what was going on here?<\/p>\n<p>It was clear from the reflective interviews students included at the end of their assignments that they were trying to grapple with both form and content, and how these intersect. In looking at these artefacts and talking to students and teachers, we think there are five main things that teachers need to take account of in developing multimodal assessments.<\/p>\n<ol>\n<li>Students need support to develop multimodal assessment literacy. This means being able to parse rubrics and criteria to understand what is being looked for, and how to identify it, and also to understand this process as a dialogue rather than a fixed and objective measurement. So for example where teachers say \u2018sophisticated\u2019, they need to be clear what they mean by this, and how they judge it when they see it.<\/li>\n<\/ol>\n<figure id=\"attachment_13231\" aria-describedby=\"caption-attachment-13231\" style=\"width: 474px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-13231 size-large\" src=\"http:\/\/jenrossity.net\/blog\/wp-content\/uploads\/Picture1-1024x546.png\" alt=\"\" width=\"474\" height=\"253\" srcset=\"http:\/\/jenrossity.net\/blog\/wp-content\/uploads\/Picture1-1024x546.png 1024w, http:\/\/jenrossity.net\/blog\/wp-content\/uploads\/Picture1-300x160.png 300w, http:\/\/jenrossity.net\/blog\/wp-content\/uploads\/Picture1-768x410.png 768w, http:\/\/jenrossity.net\/blog\/wp-content\/uploads\/Picture1.png 1995w\" sizes=\"auto, (max-width: 474px) 100vw, 474px\" \/><figcaption id=\"caption-attachment-13231\" class=\"wp-caption-text\">Screenshot from the video of the Manifesto for Teaching Online, James Lamb. https:\/\/vimeo.com\/222406740<\/figcaption><\/figure>\n<ol start=\"2\">\n<li>Some students talked about feeling constrained by the language of the rubrics, or sometimes by the fairly rigid instructions about what kind of technology to use. Where constraints can be removed that might be a good thing, sometimes, but sometimes those constraints can interact with creativity in quite generative ways. Teachers might think about and keep an eye on how these creative constraints are operating in the assignments students produce.<\/li>\n<li>The intra-action of form and content \u2013 and I mean this in Karen Barad\u2019s sense, in terms of how they create and shape one another rather than existing separately and interacting. This is a really important site of criticality in multimodal work \u2013 as my colleague <a href=\"https:\/\/www.james858499.net\">James Lamb<\/a> says, there is a need to consider the coherence between modes (and, where there is dissonance, to be deliberate about this). We should consider if our students know how to create a \u2018multimodal argument\u2019.<\/li>\n<li>Trying to look holistically at multimodal assignments is challenging if teachers are also working with rubrics. Those rubrics \u2013 especially where they specify technical elements \u2013 can easily tend towards what Bateman calls \u2018multimodal decomposition\u2019, and that can leave students trying to follow recipes, as Amani noted earlier.<\/li>\n<li>Last but not least, teachers have to consider what they are asking students to do, and how to value it appropriately. A digital assignment isn\u2019t a throwaway task \u2013 it often involves substantial learning, work and creativity, and its weighting within the course \u2013 in terms of time and assessment \u2013 needs to be carefully considered.<\/li>\n<\/ol>\n<p>We\u2019re working now on our framework, which has four dimensions \u2013 criticality, cultivating creativity, taking a holistic approach, and valuing multimodality. More to come on this! Our next steps are to finalise this framework, write and publish from this initial stage of the research, and develop some new ideas and use cases. We\u2019re having conversations with our colleagues who led the other strand of the project, about automating feedback. We think it would be really useful to explore the potential for supporting these kinds of complex assignments through automated processes \u2013 lots to discuss!<\/p>\n<p><strong>References<\/strong><\/p>\n<p>Bateman, J. (2012). The decomposability of semiotic modes. In K. O\u2019Halloran and B. Smith (Eds.), <em>Multimodal studies: Exploring issues and domains<\/em>. New York: Routledge.<\/p>\n<p>Bell A, Mladenovic R and Price M (2013). Students&#8217; perceptions of the usefulness of marking guides, grade descriptors and annotated exemplars. <em>Assessment &amp; Evaluation in Higher Education<\/em>. 38(7), 769-788.<\/p>\n<p>Bloxham, S., and A. West. 2004. Understanding the rules of the game: Marking peer assessment as a medium for developing students\u2019 conceptions of assessment. <em>Assessment &amp; Evaluation in Higher Education<\/em>29(6): 721\u2013733<\/p>\n<p>Curwood, J.S. (2012). Cultural shifts, multimodal representations, and assessment practices: A case study.\u00a0 <em>E-Learning and Digital Media, 9<\/em>(2), 232-244.<\/p>\n<p>Leander,\u00a0K. M., &amp; Vasudevan,\u00a0L. (2009).\u00a0Multimodality and mobile culture. In C. Jewitt (Ed.), <em>Handbook of multimodal analysis<\/em>(pp. 127-139). New York, NY: Routledge.<\/p>\n<p>Miles, M.B., Huberman, A., &amp; Salda\u00f1a, J. (2014). <em>Qualitative data analysis: A methods sourcebook<\/em>. Thousand Oaks, CA: Sage.<\/p>\n<p>Norton, L. (2004). Using assessment criteria as learning criteria: A case study in psychology. <em>Assessment &amp; Evaluation in Higher Education,<\/em>29(6), 687\u2013702.<\/p>\n<p>O\u2019Donovan, B., Price, M. &amp; Rust, C. (2001). The student experience of criterion-referenced assessment. <em>Innovations in Education and Teaching International,<\/em>38(1), 74\u201385.<\/p>\n<p>Pahl, K.H. &amp; Rowsell, J. (2011). Artifactual critical literacy: A new perspective for literacy education. <em>Berkeley Review of Education, 2<\/em>(2), 129-151.<\/p>\n<p>Price, M., &amp; Rust, C. (1999). The experience of introducing a common criteria assessment grid across an academic department. <em>Quality in Higher Education, <\/em>5(2), 133\u2013144.<\/p>\n<p>Ruitenberg, C. (2005). Deconstructing the experience of the local: Toward a radical pedagogy of place. <em>Philosophy of Education Archive<\/em>, 2005, 212-220.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I presented a very short \u2018pecha kucha\u2019 style talk yesterday at the inaugural Learning and Teaching Conference at the University of Edinburgh. I was speaking on behalf of two colleagues, Dr Amani Bell and Dr Jen Scott Curwood, who are members of the Centre for Research on Learning and Innovation at the University of Sydney. &hellip; <a href=\"http:\/\/jenrossity.net\/blog\/?p=13227\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Assessment in a digital age: Rethinking multimodal artefacts in higher education<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[87],"tags":[],"class_list":["post-13227","post","type-post","status-publish","format-standard","hentry","category-projects"],"_links":{"self":[{"href":"http:\/\/jenrossity.net\/blog\/index.php?rest_route=\/wp\/v2\/posts\/13227","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/jenrossity.net\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/jenrossity.net\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/jenrossity.net\/blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/jenrossity.net\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=13227"}],"version-history":[{"count":6,"href":"http:\/\/jenrossity.net\/blog\/index.php?rest_route=\/wp\/v2\/posts\/13227\/revisions"}],"predecessor-version":[{"id":13237,"href":"http:\/\/jenrossity.net\/blog\/index.php?rest_route=\/wp\/v2\/posts\/13227\/revisions\/13237"}],"wp:attachment":[{"href":"http:\/\/jenrossity.net\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=13227"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/jenrossity.net\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=13227"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/jenrossity.net\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=13227"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}