What is the meaning of it all? The simplest answer to this question is that multimedia is the bringing together of pictures, video, images, sound effects (such as a bang or an excerpt of Bach), animations and text within the confines of a computer screen. And perhaps more importantly, multimedia is fun!
Now add to these images and sound a method of navigating your way electronically through a myriad of information and multimedia really starts to take shape.
Storytelling – An Overview
The opening credit sequence of Steven Spielberg’s short-lived television series Amazing Stories (1985-1987) begins with a shamanistic figure telling a story to a group of people around a campfire. The sequence moves through a variety of images out of stories, ending with a family watching a television on which is an image of…the shamanistic figure telling a story to a group of people around a campfire. In this way, Spielberg encapsulated thousands of years of storytelling tradition. There was no little hubris in this: Spielberg, a film director who had been criticized for relying on technical wizardry rather than compelling storytelling for his popularity, was placing himself at the end of a line of historically impeccable storytellers as if he unquestionably belonged there. Nonetheless, the credit sequence is a good reminder that telling stories is something that people from all human cultures have done.
The first stories we have records of human beings telling are cave paintings of buffalo whose hides have been pierced by spears. These stories have a definite protagonist/hero in the human being who threw the spears (whether they are actually pictured on the wall or we have to use our imaginations to see them) and antagonist (the buffalo). The conflict is primal: two beings fight each other in order to see which will live and which die. Moreover, the conflict is portrayed at its climactic moment. We can imagine the long hours of the hunter stalking his prey, honing his weapons and otherwise preparing for this moment, but the artist has not chosen to show us any of this; instead, he or she has painted the moment of greatest tension, the moment at which the two forces finally come into contact, the moment which will determine which will survive and which will not. As Alexander Marshak has written: “The art is often so strong, and the examples so numerous, one practically hears the ancient clashing of horns and tusks and the belling and bellowing of the males, year after year.” [1]
Human beings probably told stories long before they started putting them down on the walls of caves; such stories were transmitted orally, passed on from one generation to the next in the telling. Storytelling is a particularly human activity, part of what makes us unique as a species: “Within this limited but complex world – from our point of view – skills and knowledge were taught by word of mouth, by use of story and symbol, by ceremony and rite, and by showing and doing,” Marshak wrote of early human societies. [2] Everybody tells stories: when your parents told you about how they met, it was likely in the form of a story; when you trade gossip with a friend, you are telling a story; when your child tells you about his or her day at school, you are probably hearing a story. In fact, our memories are frequently structured like narratives, with heroes (usually us), villains (usually people who annoy us), introductory events, escalating conflicts and a climax. If the results of our remembered narratives tend to favor us more than the actual experiences, well, we all seek comfort where we can.
Most stories (in the way of most human experience) are ephemeral; they are told and are then lost to time. However, some stories are so important that they must be repeated often enough that they survive over time. For instance, from what we know of oral stories (from those which survived in cultures before they had been contacted by “print” societies and from those whose oral stories were eventually written down), they contained answers to the existential questions of the purpose of human existence, demonstrated norms of acceptable behavior and gave their listeners other information which was necessary for group and individual survival.
Each medium of storytelling develops its own forms. Oral storytelling, for the most part, involved short stories with little detail beyond what the teller, with his or her individual ability, was able to improvise. This was necessary in order to accommodate the memories of most storytellers, which focused on the broadest details so that they could memorize the large number of stories which made up their society’s culture. Longer stories (the Epic Poems of the ancient Greeks, for instance) required a different form: the rhyming scheme and meters of varying lengths of poetry are believed to have been introduced as aids to the memory of the epic storytellers.
Our own experience of modern media supports this view. Books contain a wealth of detailed information which would be impossible in oral stories, both long descriptions of physical things and internal monologues of characters. Film and television, being primarily visual media, tell their stories primarily through images. And so on.
Spielberg was quite mistaken: listening to an oral story is not the same as watching one being told on television, even if the outward manifestations of the experience (the storyteller, the story being told, etc.) are the same. McLuhan encapsulated
this view with his famous aphorism that “the medium is the message.” [3] A corollary of this is that the medium affects the form of the stories which are told in it.
This is important because we are beginning to see the emergence of a new medium of communication: the computer. The computer has been traditionally seen as a simple storage and retrieval device for information; the (rather limited) vision of the computer and traditional arts is that you will be able to use it to access a wide range of books and films stored on computers at distant locations. What this doesn’t take into account is the fact that the computer is controlled, not by the person who wrote the book or directed the film, but by the person sitting at his or her terminal. Thus, the potential for a whole new kind of book or film is brought into being by the computer, an interactive kind where the participant has choices over the direction the experience will take.
Interactive art, like any other new medium of expression, will have new properties and offer artists new opportunities for creative storytelling. As Wimberley and Samsel put it: “the integration of choice with dramatic presentation will make new demands and will create new paradigms for interactive storytellers.” [4] Exploring such potential new paradigms is the subject of this thesis.
What is Interactivity?
A punch is pretty interactive. To start with, you are immersed in a particular environment (say, a bar). There is a stimulus to which you must respond (say, the insult of another patron). You have a choice of possible responses (to hit the person with your fist or to slap the person with your open hand…well, okay, to fight or turn away and ignore the insult). Once you have acted, you can be sure that the environment will provide you with immediate feedback.
Interactivity is a fact of life which we take for granted. To reproduce it in our media of communications, however, requires the most advanced technology at our disposal, and still we are able to create only the most rudimentary experiences. Generally,
we can say that interactivity requires an environment in which the participant is free to explore and make decisions which will direct the course of the experience. As Pimentel and Teixeira write: “There are two unique aspects of interactivity…navigation within the world and the dynamics of the environment. Navigation is simply a user’s ability to move around independently… The dynamics of an environment are the rules for how its contents (people, rocks, doors, everything) interact in order to exchange energy or information.” [5]
Advances in computer technology have made increasingly more sophisticated forms of interactivity possible. Researchers at Apple Computer describe interactivity as a continuum on which every form of communications can be found. They describe three levels of interactivity on this continuum: passive, interactive and adaptive. Passive media “are presented in a linear manner… Users do not interact except to perform basic actions of control such as starting or stopping.” [6] This describes most traditional media, from books to television.
The next level of interactivity describes media which “allow users to control the action and chart a personal course through the content.” [7] As the Apple Researchers point out, this general definition of interactive media takes in everything from an electronic book, where the reader can jump between linked pages, to video games, where participants are immersed in an environment where their every decision affects the outcome.
Finally, adaptive media, the highest level of interactivity, “allow users to enter their own content and control how it is used. This content can be original text, illustrations, sound, video, commentary on content already in the project, or a new sequence or arrangement of the existing content. The user may be expected to add voice, provide text input, create graphics, or appear in a video. All of this input then becomes part of the product.” [8] We have yet to reach the stage where films are adaptive; for most of us, the addition of sound and video to existing works still lies in the future. At the moment, the best known example of an adaptive medium is the World Wide Web, where anybody can develop information which can, theoretically, be connected to any other bit of information which has been previously placed on the system.
Underlying the WWW is a straightforward concept: hypertext. According to Stuart Moulthrop, hypertext is “a collection of writings through which the reader is free to move in almost any sequence. But unlike a printed encyclopedia, the hypertext does not come to the reader with a predefined structure. The ‘articles’ in a hypertext are not arranged by title or subject; instead, each passage contains links or reference markers that point towards other passages. These markers may be words in the text, keywords implied by the text, or special symbols. Invoking the link, by typing a phrase on a keyboard or sending some indication through a pointing device (or ‘mouse’), brings the indicated passage to the screen.” [9] A footnote in a traditional book is a form of hypertext, giving the reader the opportunity to move to information in a different part of the book (whether it is at the bottom of the page, the end of the chapter or the end of the work). A footnote is usually a reference to another text, which the reader can look up if it is close to hand. Computer-based hypertext allows readers to move more easily between texts than traditional print media.
According to Stuart Moulthrop, hypertext pioneer Michael Joyce believes the medium can take two distinct forms: “Generally speaking, exploratory texts allow readers to navigate through fixed bodies of material, while constructive texts represent ‘structures for what does not yet exist,’ open-ended, contingent forms (‘Siren Shapes’ 10-12). In exploratory hypertext, the distinction between primary author and subsequent reader-explorer remains clear. In constructive hypertext, anyone is free to change the nature of the text. There can be many authors, or perhaps it is more accurate to say that no author retains that status absolutely. This account distorts Joyce’s actual argument somewhat. In fact his terms are more continuous than exclusive – even most commercial hypertexts retain some traces of constructive form. On the other hand, most ventures in open, collaborative electronic writing betray some lingering elements of authorial control…” [10] This is another way of stating the Apple Computer Research Group’s three levels of interactivity, and, in fact, this same idea appears in a number of guises throughout the literature on interactivity. My own preference is to refer to interactive works to which the participant cannot add further information “closed,” while works to which the participant can directly contribute are “open.” Whatever terms are used, this distinction has important implications for creators of interactive works, as Moulthrop’s quote indicates.
Information in hypertext is not gathered in a page or a chapter, but a “node,” which can be defined as “a collection of data organized around a specific topic.” [11] The terms page and screen are often used interchangeably to describe how a node is displayed on a computer screen, although they have distinct meanings. A screen is everything which can be contained in the space on the user’s computer screen; since different programs may display information differently, offering more or less of it on a single screen, and, in any case, computer users can customize how they see information on their screen, there is no way to determine how much information is on a screen. The term page, by way of contrast, refers to all the information contained at a single location in a computer memory; a page may be contained entirely in a single screen, but it may also extend for many screens, forcing the user to scroll or otherwise move to a different part of the document to see it in its entirety. Following common usage, I will also use the two terms more or less interchangeably, noting the distinction only when it is important.
It follows, then, that: “A link is information embedded in a node that connects it in some way to another node.” [12] The term is confusingly used to refer to both the device on the page which must be activated to move from one node to another, and the actual process by which the reader moves from one node to another. In addition, the term is also used as a verb, as in “I want to link my footnote on Brecht to the closing number in The Threepenny Opera.”
Links are often activated by placing a mouse or cursor on a word which stands out from surrounding text because it is underlined, in a different color or both and clicking or pressing the return key. As computer interfaces become increasingly visual, links are becoming more commonly activated by buttons. A button is “a visual cue in a node that alerts a user that a link exists. In other words, a button is a visual representation of a link in a node.” [13]
The term hypertext refers to the entire amalgam of textual nodes and links between them. A path through a hypertext document refers to the specific sequence of nodes through which a reader travels; an interactive story may contain a large number of potential paths, to be determined by the choices made by the reader. Paths can also be referred to as a “threads” – I will be using the two terms interchangeably throughout this thesis.
Adding graphic elements, as well as sound and, some day, film and video, to hypertext gives us a whole new medium of communication which many refer to as ‘hypermedia.’ “‘[H]ypertext’ is used to refer to hyper-structures consisting exclusively of written texts, while ‘hypermedia’ denotes similar structures built around multiple media,” writes Kathleen Burnett. [14] A hypermedia node, therefore, may contain data in any combination of media.
In postmodern literary theory, the term text is often used to encompass a work of art in any medium; thus, while a book is traditionally considered a text, a film, a television show, a t-shirt or a billboard could also be a text for the purpose of analysis. For this reason, some people argue that the term hypertext should be used to describe any interactive work regardless of the media which are contained within it. However, I find the distinction between text-based media and media which combine different art forms (including text) to be a useful one, so I will be using the terms hypertext and hypermedia as I have described them.
As Theodore Nelson, who coined the term hypertext, points out,
the word ‘hypertext’ has become generally accepted for branching and responding text; but the corresponding word ‘hypermedia,’ meaning complexes of branching and responding graphics, movies and sound – as well as text – is much less used. Instead they use the strange phrase ‘interactive multimedia’ – four syllables longer, and not expressing the idea it extends hypertext.
Why? “Multimedia” is a resurrected term from the sixties, when it meant slideshows with sound. Why bring back such an irrelevant term? [15]
I agree that interactive multimedia has unfortunate connotations, and, therefore, will not be using it to describe hypermedia.
On the other hand, I find the term hyperfilm an inelegant way to describe film and video projects which allow the participant to choose the course of the narrative. For this reason, I will use the term “interactive film” to describe such works. Foolish consistency, you understand. Interactive film can be applied to movies seen in theatres where the audience gets to choose how the story will development, but I use the term more generally to cover any film or video based work regardless of the means by which it is delivered. Thus, I consider a CD-ROM where a participant has to navigate between digitized film or video segments to be a form of interactive film.
In describing the different kinds of interactive media, Philip Barker makes an important distinction: “Non-linear arrangements of textual material are called ‘hypertext.’ Similarly, non-linear arrangements of multimedia information (combinations of text, pictures and sound) are known as ‘hypermedia.'”16 We are used to linear media, books which begin at the first page and are meant to be read in order until the final page, or films which start with the opening credits and are meant to be watched to the end of the closing credits. Barker’s point, that the hypertext and hypermedia creations made possible by the computer do not progress in a linear fashion, has many vital consequences for artists who want to create works in emerging, interactive media. (In fact, the difference between linear narrative, as we have traditionally experienced it, and non-linear narrative, as we will experience it through emerging technologies, is the primary focus of this thesis.)
Non-linear narratives can be described as “interactive fiction,” which is, according to the rec.arts.int-fiction page on Frequently Asked Questions, “Narrative based experiences that tend to be either puzzle solving or plot branching. This term is usually applied to the ‘first generation’ Adventure games – those developed in the last decade. The second generation of IF is moving towards the development of Interactive Fantasy as a new genre.” [17] Interactive fantasy is subsequently described as a ” first person dramatic experience. Achieved through a combination of physical modeling, character modeling, and dramatic modeling.” [18] There is some truth to the writer’s claim that interactive stories are becoming more sophisticated, but I find the distinction arbitrary and unhelpful. I will be using the term interactive fiction to denote any fictional work which allows the participant some control over the direction of the narrative.
A Final Note About Terms
At Interactive 96, a conference on interactivity held at York University in 1996, I mentioned to one of the delegates that I was working on an interactive screenplay. He asked me what I meant by the term. To me, it seemed obvious: I was writing a script for an interactive work of fiction whose primary medium would be film. I use this term to differentiate it from interactive prose (whose primary medium is text) and interactive multimedia (which combines various artistic media). However, it was a reminder that, because interactive media are so new, there is no agreed upon language to talk about them.
I came upon this problem when I was trying to determine what to call a person who experiences an interactive work of art. In computer circles, the common term for such a person is a “user.” For me, this conjures up images of people in rags on street corners whining, “Hey, man! I need a fix, man! I need 80 Megs of RAM real bad!” Even without such an unfortunate connotation, the word would still be inappropriate, since it reflects the idea that computers are tools to be used and, in fact, interactive art offers a much different experience. (This also assumes that computers will be the only delivery system for interactive works, a highly debatable point; trials with interactive movies in theatres are already taking place, while the future of interactive media is currently being fought over by those in the computer and television industries. Depending upon who wins this battle, the term could prove to be highly inappropriate.)
Another common term, taken from the gaming industry, is “player.” There is some justification for using this word, since interactive fiction certainly offers a potentially playful experience. However, games, although currently the leading edge of the interactive industry, are only one type of interactive experience, and not the kind which I will be focusing on here. Inasmuch as I want to expand the idea of what interactivity can be, especially as it moves into realms of fictional storytelling, this term strikes me as too restrictive. (For similar reasons, I will only refer to interactive works as “games” when their creator or other authority identifies them as such.)
Although the computer industry doesn’t have any terms which I find suitable, looking at traditional media for an acceptable word is equally fruitless. For instance, “viewer,” which is a common way of referring to somebody watching film or television, denotes passive looking at an object of art; it is an inappropriate term for somebody experiencing an interactive art form because the whole point is to be actively involved in the work.
I have chosen to use the word “participant” in most cases where I want to describe somebody who is experiencing an interactive work of art. I believe the word comes closer to what the person is actually doing than any of the other words currently in use. (In sections dealing specifically with video and computer games, however, I may use the term player, and in sections dealing specifically with computers, I may use the term user.)
This may strike some as academic. I don’t think so. The words we use color the way we perceive the world. If we think of ourselves as creating works for “players,” we may tend to think more in terms of puzzles or game play than plot and character development or underlying themes and motifs.