Blog
Rethinking Learning through Annotation Studio
By Rachel Schnepper on July 3, 2014
In recent years, academic professional organizations have adopted guidelines for evaluating scholars’ work in the digital humanities. The MLA, for example, after exploring the challenge in a series of articles in a 2011 issue of its journal Profession, adopted a series of guidelines in 2012 that encourages digital humanists to be “prepared to make explicit the results, theoretical underpinnings, and intellectual rigor of their work.” The same year, the AHA acknowledged that with the continued growth of digital humanities, history departments needed to “establish rigorous peer-review procedures to evaluate new forms of scholarship.” How to evaluate and assess the work of digital humanists continues to be an thriving discussion on an international, multi-disciplinary scale, with special issues of the Journal of Digital Humanities devoted to it and countless columns in The Chronicle.
As digital humanities scholarship grows, so too does teaching with digital humanities tools and methods. Just as the monograph as the preferred presentation of research is being challenged with digital humanities projects, so too is the traditional term paper with multimodal work. Increasingly, new technologies, media, and tools are being integrated into classrooms, producing a lively and expanding discourse on digital pedagogy in forums such as HASTAC and the Journal of Interactive Technology and Pedagogy.
Just as new forms of scholarship require new approaches to assessment, so too do innovative digital pedagogies. Unfortunately, even as the discourse on evaluating digital humanities projects as scholarship for tenure continues to grow, the same cannot be said of evaluating the effects of using digital tools as pedagogical resources. In the lush landscape of literature on digital humanities, this is a glaringly depressing bald patch.
This is precisely the challenge we at HyperStudio face as we scale-up the use of our digital anntotation tool, Annotation Studio, in classrooms from local high schools to national and international universities. When Annotation Studio was in use exclusively in classes at MIT in 2012, we conducted a preliminary assessment investigation using surveys, focus groups, and interviews. The results of this research revealed that most students, despite a lack of experience with both analog annotation and online annotation, were extremely interested in using Annotation Studio. By the end of the term, students readily acknowledged the value of annotating, crediting Annotation Studio with helping them collect evidence from the texts to then construct better arguments in their papers.
The results of our initial assessment were encouraging.
But we want to know more.
One of HyperStudio’s goals in all our projects is to enable users to have a more meaningful experience with, to dig deeper into, whatever form of media they are engaging with. This is true whether it is a document from Mohammed Khatemi’s presidency in Iran from our US-Iran Relations Project or a piece of art. With Annotation Studio, however, in order to develop a tool that provides this deeper, more meaningful experience, we need to fully understand how instructors and students are using it in the classroom.
Consequently, HyperStudio is conducting an ongoing and intensive inquiry in several classes at MIT. We are beginning with extended discussions with faculty members who are using Annotation Studio in their classes, going over reading and writing assignments that use Annotation Studio. Instructors have identified the promotion of collaborative learning and evidence based classroom debate, careful examination and incorporation of textual evidence in writing assignments, and foreign language reading comprehension as some of the specific performance tasks they have used Annotation Studio for. By pinpointing the exact pedagogical purposes the instructor had in mind when assigning Annotation Studio, we can better evaluate the tool’s effectiveness through classroom observation, surveys, focus groups, and interviews with students.
While this past semester we conducted assessment with writing and foreign language classes here at MIT, we also recognize that Annotation Studio is increasingly being used in classes outside of MIT. Accordingly, we were excited to talk with professors from Yale University and the University of Washington about their experiences with Annotation Studio as well.
Our goal in conducting these assessment exercises is twofold. First, feedback from instructors determines the directions and priorities of our development of Annotation Studio. At HyperStudio, we believe that the functionality of the application should not determine its pedagogical uses. Rather, the pedagogical uses of Annotation Studio inform its development.
Second, we are committed to uncovering and enhancing student learning. If we want students to become more sophisticated and critical readers, learn how to work with textual evidence when writing essays, and develop enduring skills to increase understanding, then it is vital that we appreciate how these processes are taking place when students use Annotation Studio.
We are eager to unpack and analyze the results of our assessment this spring, to learn how Annotation Studio is transforming and enhancing student learning. As we move through the summer, we are developing with instructors an even more extensive assessment strategy to begin at the start of the fall.
Beyond evaluating the students’ transferability of the skills cultivated through Annotation Studio and their endurability, we are also examining the analytics data the tool generates. We are capturing the stats and metrics of student use of Annotation Studio, including timestamps on annotations, the use of tags, private versus public annotations. This data provides users of Annotation Studio with unqiue insights into how students are reading texts and interacting with one another in the social environment of Annotation Studio. With this data, we can produce visualizations of the reading process, which reveal trends in the reading process, the evolution of interactions with passages in the text, and provide practical feedback for instructors.
As we do learn more about Annotation Studio, we look forward to sharing our research. We intend to share the results of our research with instructors, so they can refine their assignments and improve their pedagogy. We intend to share the results of our research with students, so they can understand better their own learning processes. And, finally, we intend to share the results of our research with the growing Annotation Studio community, so they can be inspired.
