Taking and Making Meaning:

Semiotics and New Media

Andy Clarke and Grethe Mitchell

This paper is part of an ongoing project, researching the language of videogames, virtual environments and interactive narrative in relation to film theory and practice. Because of its basis in film theory, our project draws upon a number of theories including structuralism and post-structuralism, semiotics, spectator theory and psychoanalysis.

We believe that the formal similarities between film and computer-based media mean that film theory is a particularly useful approach in looking at digital work: they are both screen-based media; they are both time-based media; they both convey meaning predominantly though image and montage. This makes the comparison with film more appropriate than, say Laurel’s use of theatre theory.

Structuralism and semiotics have proven to be particularly useful in this investigation, providing a theoretical framework in which to deconstruct and analyse the techniques used in these works; structuralism and semiotic theory are not specific to any one medium, and so can be applied as easily to computer-based media - such as virtual environments, videogames, interactive narratives, web pages and other applications - as they have been to novels, films, photography etc.

One area of our research has been the use of viewpoint in videogames and virtual environments. A game such as Quake uses a first person viewpoint, seeing the action through the eyes of the player’s character; Tomb Raider, on the other hand, has a third person perspective, following the character. Each of these shots has a different effect on the player - the former gives a greater sense of "being there" in the world of the game, while the latter gives a greater sense of following a narrative.

It is important to note that the way in which videogames use viewpoint is different to the way in which films use viewpoint, and this is what makes the semiotics of videogames and virtual environments a particularly rich and interesting area of study. There is a "tension" within these works - they use the established conventions of a variety of different media including films, comic strips, etc., but because of the added elements of interactivity and non-linearity, need to modify these conventions.

One particularly interesting "tension" within these games is between realism and non-realism. A game such as Quake provides a realistic, three-dimensional, world to wander around in and yet the objects within this world - such as keys, guns, bullets - are not realistic, nor are they presented realistically. In stead, we have a simplified gun floating in mid-air so that you don’t miss it. Essentially, what these games show is the icon of a gun, rather that a real gun.

We have speculated that this deliberate non-realism is a result of the difficulty in presenting a narrative within a continuous, three-dimensional, space with a constant first or third-person viewpoint (as is the case with most videogames and virtual environments). This is a point that we will expand upon in our presentation to the workshop.

As we are practitioners as well as theoreticians, we know that an understanding of structuralism and semiotics can help, as part of a conscious self-reflexive creative process, in the production of work. It is tempting, therefore, to speculate as to whether they could form the basis of an automated computer-based creative system. Could one, for example, create a videogame (or virtual world) in which the user could go anywhere and do anything, and the system would create a sufficiently rich and engaging set of new characters, environments, scenarios and dialogue on the fly (and decide how to show it)?

We believe that there are problems with this. Semiotics has been referred to as a "science of signs", but it is an imprecise science. Signs do not have a single, stable, archetypal meaning that can be programmed into this hypothetical system - they have meanings layered one on top of another, and these meaning change according to the context. It is best to think not to think of signs individually, in isolation, but rather as part of a "web" or a "matrix" of signifiers and signifieds - each sign can only have meaning through its relation to other signs (similarity to, opposition to, use with, etc.) and carries both desired and undesired meanings because of these relationships.

The difficulties in programming thus become more apparent. It is difficult to get back to meanings that are not dependent on other meanings, so any attempt to encode what a sign means tends to leads to circular definitions. Likewise, because signs refer to - and are modified by - one another, one would rapidly end up with unmanageable tables trying to cross-reference the effect of one sign on another.

This is not to say that the task is impossible for every scenario. Genre has an important role in limiting and fixing meaning, and this makes the problem much simpler. Similarly, what we know about character and narrative in film can possibly suggest shortcuts in this process as well.

The classic western, for example, works with a very restricted number of characters (sheriff, outlaw, indian, etc.), objects (horses, guns, etc.), plots (cleaning up town, crossing hostile territory, etc.), situations (bar-room brawl, indian raid, etc.) and locations (bars, jails, brothels, desert, etc.). We know, therefore, that certain modern objects - such as mobile phones - will not appear in the western and that objects such as stars have a clear meaning (the sheriff) and that all other meanings can be ignored.

We can also look at the classic western as an archetypal narrative and see that certain characters have certain roles: the sheriff is the hero, a hot-headed young gunslinger may be a "false villain" (initially appears bad, but then helps the hero). This likewise restricts how characters interact with one another, the objects that they use (and how they use them), and how they react in certain situations.

All of these techniques can simplify the task of mapping meaning. It is worth bearing in mind, however, that when a work obeys the conventions of genre too closely, or when its characters (and their interactions) become too formulaic, it becomes boring, thereby losing narrative "charge".

To conclude, we consider that structuralism and semiotics have an crucial role to play in the study of digital media. We would go on to re-iterate our belief that because of their formal similarities - the fact that they are both screen-based and time-based, for example - film theory in general has particular relevance to the study of digital media.

We feel, however, that structuralism and semiotics are of limited use on their own in automating the creative process on computer, possibly being restricted to certain areas - fairy tales being an obvious example - where there are strong, clearly-defined, genre conventions or simple, strict, narrative structures that simplify the task and limit possible misreadings.

We also believe that while there has been some success with the automated generation of text-based stories, the increased complexity, sophistication and subtlety of the visual language of cinematic (2D) or immersive (3D) stories may present intractable problems.

It is possible, however, that other theories of film - such as those of character and narrative - may provide alternative or complementary techniques to those already apparent from structuralism and semiotics. These investigations also form part of our continuing research.