Module 9. Incorporating interactivity
Module 9. Incorporating interactivity
Embedded Files
By the end of this module you should be able to do the following:
Define hypermedia and hypertext
Describe the World Wide Web as a hypermedia system
List some examples of different interaction patterns that can be incorporated into multimedia products or software
Discuss how the arts contributed to the development of interactivity in contemporary multimedia
Contents
Contents
These days, there is a wide range of ways to interact with digital text, audio, graphics, animation, and video. Many of these interaction techniques were not popularly available years ago, although many of them were first conceived in research laboratories decades before they became mainstream. For instance, touchscreens allow users to directly manipulate elements on a screen, and while touchscreens were first invented in mid-1960s, it was only in the late 1990s when the first commercially-available products incorporated them ().
Recall from Module 1 that interactivity is a key feature of multimedia. While Unit 2 looked at the technical foundations of the media modalities of image, sound, and the moving image, we did not cover the technical foundations of interactivity. We will, however, look at some basic ways in which interactivity can be incorporated in a multimedia experience.
One fundamental property of many interactive features in a multimedia product is that they afford the user a choice in how they can experience that multimedia product. The simplest example of this is how webpages have embedded links that allow users to visit the linked page--this is called hypertext.
Hypertext and hypermedia
Hypertext and hypermedia
The World Wide Web Consortium defines both hypertext and the more general term hypermedia:
Hypertext is text which is not constrained to be linear. Hypertext is text which contains links to other texts. The term was coined by Ted Nelson around 1965 (see History). HyperMedia is a term used for hypertext which is not constrained to be text: it can include graphics, video and sound, for example. Apparently Ted Nelson was the first to use this term too. Hypertext and HyperMedia are concepts, not products. (links provided in the original definition)
The World Wide Web (or Web for short) is a very special case of hypermedia. It is a system of interlinked hypertext documents that are located on various computers that are connected to each other. The network that is formed by the computers is called the Internet. So you see, technically, the Web and the Internet are two different things. However, in common usage, most people use the two terms interchangeably. Learn more about the Web by doing the following activity.
Activity 9.1. Learn about the World Wide Web
Activity 9.1. Learn about the World Wide Web
Allot 10 minutes
Read the BBC Bitesize introduction to the Web.
The World Wide Web (or the Web, for short) is the largest hypermedia system, but you can make your own hypermedia system that is not part of the Web. For example, you can host the hypertext documents and hypermedia files in a local network that is not connected to the Internet.
Typically, you use a Web browser to browse a hypermedia system, although you can build hypermedia systems that don't run on a Web browser. However, because tools, products, and standards for hypertext and hypermedia support in Web browsers are so advanced, it often makes sense to develop material with the Web and typical Web browsers in mind.
Activity 9.2. The contribution of the arts and other disciplines to the development of interactivity in multimedia
Activity 9.2. The contribution of the arts and other disciplines to the development of interactivity in multimedia
Allot 1 hour
One of the areas of practice that has been most influential to the development of interactivity and hypermedia is that of experimental and conceptual art. Browse through From Wagner to Virtual Reality, a website by Randall Packer and Ken Jordan (2000). This website highlights artists that the authors consider important to the history of multimedia. Don't feel obliged to read through the entire site (although you are very welcome to do so). A more useful approach would be to only read articles that you find interesting, and to follow the path that appeals to you most.
What can you say about the structure of the website? Does it follow a linear structure, or would you say it was more non-linear? What do you think the structure of the website implies about the development of contemporary multimedia?
Choose a concept, person, or historical development discussed in the resource that you interesting or surprising, and share with the class how it influenced the development of contemporary multimedia.
Designing hypermedia material
Designing hypermedia material
There are a few issues that you should keep in mind when designing hypermedia material. Two of these are directionality and structure.
Directionality
Directionality
In general, hypermedia tends to be unidirectional. Imagine if you had a link to a page that no longer existed. Unfortunately, there is no way to alert all the Web pages that reference that missing page that the link is no longer valid. This points to a significant defect in the way that links are implemented on the Web: they are unidirectional. Webpages don't usually know which other webpages are pointing to them. This irked information technology pioneer Ted Nelson so much that he published a paper in 2007 describing a system called Xanadu which illustrates his original vision for hypermedia, where all content is not only linkable but "transclusible", meaning that the content knows where it comes from (Nelson, Smith, & Mallicoat, 2007). If the original content changes, then any other documents that link to that content automatically get updated.
Figure 9.1. A visual representation of Ted Nelson's Project Xanadu
Structure
Structure
You have to be careful with how you link together documents and media to create a hypermedia structure. This involves carefully planning the structure of your hypermedia system is and matching it to your intended experience for the user. For example, think about the difference between the structure you would build in Activity 8.2 (a tutorial) versus the one you might build for Activity 8.3 (an interactive story). The simplest hypermedia system would have a linear structure, where you can visit every node in the system in a straightforward, step-wise fashion. Some hypermedia systems are structured like a tree (hierarchical); others are like a starfish (radial); still others are structured in a way that shows little hierarchy and that is very highly connected (Figure 9.2). Barfield (2004) includes a useful chapter that deals with planning and implementing structure in hypermedia systems.
While it seems like a good idea to connect all relevant information to each other, some critics of technology argue that the highly-connected and information-rich structure of the Web has had some unexpected (and unwanted) effects in the cognitive abilities of people. We look at this argument in Unit 4.
Figure 9.2. From upper left, clockwise: a) A hierarchical structure; b) A radial structure; c) A highly-connected structure; d) A loosely-connected structure. Not shown: a simple linear structure.
Activity 9.3. Different structures for different purposes
Activity 9.3. Different structures for different purposes
Allot 10 minutes
Consider Figure 9.2.
Using the same system of nodes and connecting links (arrows), draw what a simple linear structure would look like.
No one structure is better than the other. Under what circumstances do you think it might be useful to design a hypermedia system with ...
... a simple linear structure?
... a hierarchical, tree-like structure (Figure 9.2a)?
... a radial structure (Figure 9.2b)?
Activity 9.4: Mapping the structure of an interactive film (optional)
Activity 9.4: Mapping the structure of an interactive film (optional)
Allot 4+ hours (i.e., more hours than you care to imagine!)
In December 2018, the movie streaming service Netflix released Bandersnatch, an interactive movie. An interactive movie (or interactive cinema) is a hybrid between digital games and traditional cinema, allowing the viewer to have control over the narrative of the movie (Vosmeer and Schouten, 2014; MediaLAB Amsterdam, 2015). While interactive cinema is not new, Bandersnatch was the first interactive movie to produced and released by a major mainstream Hollywood media company.
Within hours of its release, some viewers had claimed to have completely mapped the structure of Bandersnatch. The most definitive map to date was generated by a contributor to the site Reddit.com, shown in Figure 9.1.
Working in groups of no more than five, can you confirm that this map is accurate by finding any gaps or errors in this map? That is to say, the goal of this activity is to find at least one way in which this map is inaccurate or incomplete. Working in small groups helps distribute the work because you will be spending a lot of time watching Netflix to work on this activity!
Video 9.1. The official trailer of Bandersnatch (Netflix, 2018)
Figure 9.3. The most definitive known map of the structure of Bandersnatch (alpine-, 2018)
Interaction patterns
Interaction patterns
In the early days of the computer and the Internet, interactivity was mostly associated with navigation. That is, because of the concept of hypermedia, people were allowed to traverse through documents and media in a nonlinear fashion, which was remarkable at the time. Today, however, a much wider array of interactions are possible with digital objects, which includes not only text, audio, 2D graphics, 2D animations, but also 3D graphics and 3D animations. Interaction patterns that pertain to the manipulation information include undo, redo, and 'drilling down'. Interaction patterns for spatial software (e.g., Google Earth, Microsoft Photosynth, VRML) include panning, zooming, rotating. Interactions pertain to playing temporal media (media that unfolds through time, like videos or audio) include stopping, playing, slowing down, speeding up, and pausing. Interaction patterns for virtual physical environments (e.g., digital games, 3d modeling tools like Google Sketchup, as shown in Video 9.2) including hitting, pulling, carving, and nudging.
Video 9.2. Some interaction possibilities in Google Sketchup
New interaction types are constantly evolving, and some of these interaction types can be both surprisingly simple but also game-changing. For instance, consider the interaction pattern of swiping right or left in dating apps, to indicate whether you like someone or not; this was first popularized by the dating app Tinder, and it has revolutionized online dating apps since then (Pierce, 2012).
Activity 9.5. Gestural interactions
Activity 9.5. Gestural interactions
Allot 10 minutes
The advent of touchscreen devices has generated a whole range of gestural interactions. Figure 9.4 lists some common touchscreen gestures.
Gestures and interaction patterns have a close relationship. Gestures are mapped to interaction patterns, and this mapping is often popularized in apps and products. We already mentioned Tinder's popularizing of swiping left or right to indicate rejection or acceptance. As another example, you can see in Video 9.3 some interesting interaction patterns in a list app called Clear, such as spreading to insert an item between two other items in a list, and pinching close to collapse a list and return to a broader overview of the program.
Identify and discuss with the class an interesting gestural interaction that you have come across.
Video 9.3. The Clear app (Realmac Software, 2012)
Activity 9.6. Other interactive media forms
Activity 9.6. Other interactive media forms
Allot 60 minutes
Other genres of interactive media exist, such as visual novels (Cavallaro, 2010) or hypertext fiction (Malloy, 1998). Report on one of these genres to the class.
Conclusion
Conclusion
Interactivity in multimedia was influenced by concepts and theories advanced by a variety of artists, scientists, philosophers, and innovators. Indeed, the arts played a major role in the development of interactivity in contemporary multimedia, and will continue to influence how media technologies will evolve in the future. We will revisit the role of the arts in the development of multimedia in the last module of this Course Package.
Hypermedia and hypertext provided some of the earliest interaction possibilities in multimedia systems, especially in webpages that were part of the World Wide Web. As genres of digital tools and products evolve and cross-pollinate, you should expect to see new types of interaction possibilities in multimedia. We will revisit the future of interactivity in Unit 5.
References
References
alpine-. (2018, December 28). r/blackmirror - Almost 4 hours after release and I think I have mapped Bandersnatch. Throw tea over computer. Retrieved January 17, 2019, from https://www.reddit.com/r/blackmirror/comments/aa9oym/almost_4_hours_after_release_and_i_think_i_have/
BBC. (2018, October 23). What is the world wide web? Retrieved February 1, 2019, from https://www.bbc.com/bitesize/articles/z2nbgk7
Black Mirror: Bandersnatch. (n.d.). Retrieved from http://www.imdb.com/title/tt9495224/
Cavallaro, D. (2010). Anime and the visual novel: Narrative structure, design and play at the crossroads of animation and computer games. Jefferson, N.C: McFarland & Co.
Graham-Rowe, D. (2007, June 19). Mapping the Internet. Retrieved January 17, 2019, from https://www.technologyreview.com/s/408104/mapping-the-internet/
Ion, F. (2013, April 4). From touch displays to the Surface: A brief history of touchscreen technology. Retrieved January 17, 2019, from https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/
Kleinman, J. (n.d.). Reddit Already Knows How to Get Every Ending in “Black Mirror Bandersnatch.” Retrieved January 17, 2019, from https://www.inverse.com/article/52130-black-mirror-bandersnatch-every-ending-reddit-how-to-get-the-best-ending
Malloy, J. (1998). Hypernarrative in the Age of the Web. Retrieved November 13, 2019, from https://people.well.com/user/jmalloy/neapaper.html
MediaLAB Amsterdam. (2015). Interactive Cinema: A mini-guide to understanding and producing a 360° interactive movie. Retrieved January 17, 2019, from https://issuu.com/medialab_amsterdam/docs/interactivecinema_guideline_v3
Nelson, T. H., Smith, R. A., & Mallicoat, M. (2007). Back to the future: hypertext the way it used to be. In Proceedings of the eighteenth conference on Hypertext and hypermedia (p. 228). Retrieved from http://xanadu.com/XanaduSpace/btf.htm
Netflix. (2018). Black Mirror: Bandersnatch | Official Trailer [HD] | Netflix. Retrieved from https://www.youtube.com/watch?v=XM0xWpBYlNM
Packer, R., & Jordan, K. (2000). Multimedia - From Wagner to Virtual Reality. Retrieved April 21, 2008, from http://www.artmuseum.net/w2vr/contents.html and archived on https://web.archive.org/web/20090330143805/http://www.artmuseum.net/w2vr/contents.html
Pierce, D. (2016, September 28). The Oral History of Tinder’s Alluring Right Swipe. WIRED. Retrieved from https://www.wired.com/2016/09/history-of-tinder-right-swipe/
r/blackmirror - Almost 4 hours after release and I think I have mapped Bandersnatch. Throw tea over computer. (n.d.). Retrieved January 17, 2019, from https://www.reddit.com/r/blackmirror/comments/aa9oym/almost_4_hours_after_release_and_i_think_i_have/
Vosmeer, M., & Schouten, B. (2014). Interactive Cinema: Engagement and Interaction. In A. Mitchell, C. Fernández-Vara, & D. Thue (Eds.), Interactive Storytelling (Vol. 8832, pp. 140–147). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-12337-0_14
Report abuse