Particip@tions Volume 1, Issue 1 (November 2003)

Media Effects: Redux or Reductive?

Against a backdrop of fifty years of public controversy concerning children’s use of violent entertainment, a group of 33 cultural studies scholars have become supporters of the video game industry in its legal battle against media censorship in St. Louis. As friends of the court they have backed the industry’s de-regulatory legal doctrine by dismissing the concerns of some psychological researchers about media violence saying there is no proof of the ‘media effects’ hypothesis. This paper analyzes the Amici Curiae’s interpretation of  effects research by putting it within the context of a long-standing political struggle over the ‘rights’ in children’s cultural products markets in order to highlight audience researchers' contentious place within the modern matrix of socialization.

The Janus Face of Communications Research

During the second half of the twentieth century,  as our interest in understanding the role of media grew, the traditions of studying communication diverged. Worried by an ever-widening epistemological divide in contemporary communication studies Raymond Williams (1974) expressed his concern about the tensions mounting between social science and humanistic perspectives in media studies: on one side, stood the hermeneutic traditions of arts and humanities who critically interpreted media texts and what people did with them in context. On the other, stood the social sciences (especially in America), which emphasized the generalizations about the social effects of  the mass media.   The  thinking about audiences, he argued, seemed especially bifurcated by epistemological  rifts.  Each methodological community tended to police its sub-disciplinary boundaries by refusing to dialogue about the points of convergence of their perspectives.

Williams knew there were fundamental differences underlying the theories, interests, research methods and philosophies in the diverging streams of communication studies. The humanities had evolved its critical ‘interpretive’ approach from methods of the exegesis of texts which emphasized the insightful interpretive analysis of specific cultural artifacts both among analysts and audiences, thus situating the meaning-making experience in specific socio-historical contexts. These scholars contributed "sustained and detailed analysis of actual cultural works" he argued.  Williams was also critical: "what was much more open to question was the extension of this kind of analysis and insight to matters of cultural and social generalization."  On their part, the American social sciences  were uncritically steeped in narrow quantitative operationalism which seemed to Williams reductionist and anti-historical. Their obsession with studying general laws, structures and  impacts  constrained  their empirical inquiries  to questions which were easily "observable" often resulting in facile conclusions.

Although identified with the humanities, Williams worried that culturalist scholars were becoming overtly hostile to the social science method. He personally refused however to write-off  the American ‘effects  research tradition’, much of which he found  “useful” because it at least kept alive the complex critical question of  determinacy relations.  Williams proposed instead, a hybrid discipline called "cultural science" emerging from an ongoing dialogue between humanistic and scientific researchers about media effects. The rethinking of audiences, he hoped, would begin healing the epistemic fissures in media studies by focusing researchers on questions of social structure and  ‘determinacy’ relationships while acknowledging the diversity and  'agency' of audiences who consume media in diverse circumstances.   

Effects Theory Goes to Washington

Michele Foucault notes that, however nobly stated, our scientific quests for truth remain social discourses deeply embedded in the broader struggles over social power in a politicized world.  Foucault suggests that however tempting it is to take sides in the contested discourses of a science "drawing the line between that in a discourse which falls under the category of scientificity or truth, and that which comes under some other category” can be pointless.  Instead he suggests that we  examine “historically how effects of truth are produced within discourses which in themselves are neither true nor false." (Foucault 1980). Those interested in social science must recognize that  "in societies like ours, the 'political economy' of truth … is subject  to constant  economic and political incitement”, he claimed. This statement is especially true of media effects science.

For media effects research, the incitement began in earnest shortly after TV’s post-war diffusion. As a commercial interest operating within the cultural domain of socialization, children’s television became the focal point of a protracted struggle over post-war family values and lifestyles. Given the crucial symbolic space that childhood occupies in western cultures, it is hardly surprising that children’s fascination with TV was viewed with both optimism and horror, not only by social theorists, but by the public at large (Spigel 1998). 

The public debates about children’s TV became a highly contested zone of “social regulation”  that grew ever more controversial as children’s enthusiasm for it grew. Although the concerns were at first wide-ranging, the press repeatedly associated  media violence with the specter of mounting juvenile delinquency and youth aggression. Sociologists and psychologists were gradually drawn into the controversy to add the scientific perspective on whether media prompted anti-social behavior or act as a cathartic release?  Giving testimony at the Kefauver inquiry in 1954, Paul Lazarsfeld claimed his agnosticism: it was too early to determine the impact of TV violence on children because  there simply wasn’t sufficient evidence. There were no firm grounds therefore for regulation in the public interest chimed in the increasingly profitable entertainment industry in defense of their rights of free expression ( Murray 1995, Minow 1995).

Despite the industries protestations, concerns about TV’s effects on the young  wouldn’t go away. Prompted by repeated instances of spectacular youth violence (the Charles Manson killings for example) the effects of  children’s exposure to violence has become one of  the most researched issues  in media studies.  Anyone interested in this field must now confront shelves of books, studies and numerically sophisticated research reports which provide evidence about the impact of violent entertainment on children’s learning of aggressive and anti-social behavior from film and TV. 

In spite of numerous scientific panels  and hundred’s of scientific papers, there is no definitive account about how viewing violence impacts on children. This substantial literature remains as conflicted as it is tentative, concluding that heavy consumption of media violence can contribute to aggression in some circumstances (Comstock et. al 1972, NIMH 1982).  Although the effects are generally considered to be  small and  difficult to specify, psychologists and health professionals suggest the evidence shows that in some circumstances heavy viewers can become more aggressive because of their consumption of violent entertainment (Huston et. al. 1992; American Academy of Pediatrics  2001, American Psychological Association 2001).  But these conclusions have been critiqued by other social scientists who point out that the size of the effects found were often too small and the designs too flimsy to be scientifically convincing. (McGuire 1986, Freedman 1984, Wober 1988, Durkin 1995, Fowles 1999, Goldstein 1998). As Tony Reichardt (2003) writing in Nature recently asked: “So who is right? The answer is far from clear, partly because the two sides are engaged in a war of words that can be as combative as some of the games being studied”.

War and Peace in the Hallowed Halls

Indeed, Raymond Williams’ call for a unified cultural science of audiences seems long forgotten now, as the rift between humanist and social science approaches gradually escalated into a fire-fight about violent entertainment’s  contribution  to youth aggression and crime. Particularly after the brutal Jamie Bulger killing, British media studies scholars voiced ever stronger opposition to what seemed to be the psychologists’ overly hasty blaming of the media for child slayings and murders.  A group of media scholars published a collection of essays which challenged the validity of the scientific evidence which ‘proves’ the effects of media violence on children, calling the public demands for  regulation of media violence unnecessary when there are no ‘ill effects’ of media violence. Popular entertainment has never been shown to  be harmful to children they argued.   Noting a number of critiques of media effects research they argued that the psychologists calls for regulation are  not only  ‘false and misleading’ but also “daft” and “mischievous”.  Their claims are false because there is “no such thing as violence in the media’ which can have either harmful or beneficial effects” in the first place. Mischievous because the ‘alarmism’ precipitated by “effects science” contributes to public censorship of children’s pleasures. (Barker and Petley: Ill Effects 1997/ 2002).  

Increasingly British media scholars dismissed the whole body of American media effects research for its quantitative methodology which seemed to ignore the fact that children are active and savvy audiences who can tell the difference between fictional violence and  reality. As David Buckingham claimed: “Within Cultural Studies, there is a long tradition of damning this work, not just as positivist and empiricist, but also for conceiving of children (and audiences generally) as merely passive victims of the media.” Buckingham (2001) feels that the “denial of children’s agency at the heart of this approach” explains why psychologists attempts to substantiate powerful media effects have largely been unsubstantiated and why their claims about media effects must be criticized.

Effects Research Goes to Washington

During the 1980’s and 1990’s  the popularity of  realistic fighting, combat and shooting video games added fuel to the public  concerns with  violent entertainment  (Murray 1995).  Coming off a period of Reaganite deregulation, the media industries mobilized against the mounting pressure from legislators. Rallying behind the twin flags of freedom of expression and corporate responsibility the fast growing gaming industry launched a self-regulatory code in 1994 launching the ESRB under the auspices of the ISDA to classify each game according to their content. The industry agreed to display the codes and descriptors as product warnings and parental advisories in accordance with the public interest in helping parents make informed decisions about what games were appropriate for various age groups. (www.ESRB.com )

Yet after a particularly nasty school massacre at Jonesboro the debate about media violence and its impact on young people spilled into the media again.  On the news children’s advocates were  blaming drugs, parents, families teachers and of course video games for the rise of school shootings. Perhaps not coincidentally Jonesboro was also the place where Dr. Dave Grossman, author of On Killing (1995) and a leading critic of the media industries, had retired. Grossman had been a lieutenant colonel who had built a career figuring out how to train soldiers to kill.  As a retired US army officer, Grossman seems well positioned to comment on the similarity between the tactics used in the army to train soldiers and the use of violent video games among children today. The US military has long used simulation training for its soldiers because the “repetition and desensitization” of simulated killing influences the kill rates (the actual percentage of soldiers that will pull the trigger in real life combat). Recently he has become a leading US advocate of media regulation  arguing that “the main concern is that these violent video games are providing military quality training to children”. Like the training of these soldiers, Grossman believes that violent video games may have a similar effect on young people who play them a lot, not because they create models or templates for children’s  behaviour, but because they help break down the psychological barriers that prevent killing: “children don’t naturally kill; they learn it from violence in the home and…from violence as entertainment in television, movies and interactive video games”.  The self-regulatory code didn’t seem to be very effective claimed Grossman who set out to persuade Americans that it was time to do something about the ‘virus of violence’ infecting America (Grossman  and DeGaetano1999).     

The following years slayings at Columbine high school where Harris and Klebold’s fascination with playing Doom were well publicized, provoked renewed calls from politicians  to regulate video games in a manner similar to film and TV. The newer video games like Street Fighter, Mortal Kombat and Doom involve realistic and graphic representations of killing, while positioning players as active rather than voyeuristic perpetrators of virtual violence, claimed the psychological critics (Funk 1996, Dill and Dill 1998, Anderson and Bushman 2001).  Congress (1999) was prodded by this growing public anxiety about playground shootings to once again hold hearings on the gaming industry. At these proceedings, David Grossman expressed his strong views to the committee based on his simulation training (he is in fact not a researcher). Psychologists Anderson and Funk also summarized their views of the scientific evidence supporting the concern that violent entertainment contributed to aggressive and anti-social behavior. On the other side, ISDA president Doug Lowenstein and psychologist Jeffrey Goldstein both explained  that there was in fact very little empirical evidence that indicated that video games harmed children. The battle lines within the effects research camp were drawn around what the empirical evidence implied.

Also headed to Washington to testify, cultural studies scholar Henry Jenkins feared that the scientific debate had turned into a witch hunt  which served only to intensify the surveillance and control of children. Jenkins articulated for the committee the cultural studies scholars view that the social scientific  evidence was not consistent enough to warrant regulation. Effects researchers who study of media violence in laboratories by counting how many times a child hits a Bobo doll, were overstating their case he claimed. Moreover children’s play, and game play particularly, is complex: children don’t make literal sense of the violence in their video games. Children know that video games are simply environments for playful exploration of a sometimes difficult ‘adult’ world. They choose to watch  horror films or play violent video games for many reasons  including the potential to fantasize  ‘empowerment’ and transgression’ and to experience ‘intensified emotions’ which help them cope with the confusions of grown up world. Unfortunately, explained Jenkins, to a reporter afterward: "In the case of Harris and Klebold they drew into their world the darkest, most alienated, most brutal images available to them and they turned those images into the vehicle of their personal demons."  Still, is it not far better  to recognize that the roots of their pathology lies with dysfunctional families, drugs and impoverished communities, and not with the media content.  Eliminating violence from the screens will therefore have absolutely no impact on aggressive and antisocial behavior, he argued. And for all the public hand wringing about the industries marketing of violent video games to children (FTC 2000), there is still no federal legislation (Kline 2001).

Effects Theory Goes to Court

The diverging visions of humanist and social scientific media scholars were again mustered  when a legal battle broke out over a St. Louis ordinance which would restrict the sale of violent video games to children based on the age related criteria developed by the ESRB.  A similar ordinance had recently been defeated by the ISDA (Independent Software Developers Association) in Minneapolis on censorship grounds so alarm bells rang through the industry when Judge Lindbaugh ruled that even if local ordinance infringe on the free speech rights,  these rights must be weighed against the interests of parents who wished to control their children’s media use. St. Louis County  counselor Michael Shuman told a newspaper reporter ''It drives these people crazy that Judge Limbaugh thought these games are not protected by the First Amendment,'' in isolation of the “right of the parent to make decisions for their kids?'' (Jurowitz 2002). Needless to say, the ISDA launched an appeal of this decision and the case became a minor ‘cause celeb’.

The industry’s appeal was supported by Marjorie Heins, the director of the Free Expression Policy Project representing the anti-censorship lobby. But perhaps more at issue from an effects researchers point of view is that this court case was also given the willingly assistance of  33 academic Amici Curiae – including prominent media studies scholars Henry Jenkins, Jib Fowles, Todd Gitlin, Martin Barker and David Buckingham – who have taken up the cudgel, not only against the effects scientists, but also local elected officials who decided to regulate the sale of violent games to children according to the ESRB age ratings developed by the industry. As ‘friends of the court’ the Amici Curiae’s brief claims that the St. Louis regulations arise from “commonly held but mistaken beliefs about a proven causative link between violent  entertainment and violent behavior to uphold a censorship law.”

In their brief they advise the court not to accept that the ‘effects hypothesis is proven’ on the basis of psychologist Dr. Craig Anderson’s (2000, 2001, 2002) expert testimony  that "there is a causal connection between viewing  violent movies and TV programs and violent acts" and that video games "provide  a complete learning environment for aggression."  They condemn the body of evidence gathered by effects research over the years citing  reviews of this literature by Jeffrey Goldstein, Kevin Durkin, Mark Griffiths, Mallory Wober and Jonathon Freedman which they believe will  “assist the court in understanding the media effects debate”. The Amici’s contention is that  psychological researchers like Anderson have no proof that violent entertainment “causes – or is even a risk factor for actual violent behavior”.  All this research actually indicates, they claim is that  “researchers who attempt to reduce the myriad effects of art and entertainment to numerical measurements and artificial laboratory experiments are not likely to yield useful insights about the way that viewers actually use popular culture".  

The Question of Proof

For those who have not, or will not wade through the literature  this brief builds on, the following discussion adds  a crypto-cultural sciences perspective to the scientific objections raised by the Amici’s intervention in this longstanding legal battle over what cultural products children should have access to in the media marketplace. Given their opposition to positivism and quantitative methods generally, what first seems curious is that  the Amici’s  brief relies so heavily on the arguments of  ‘behavioralist’  commentaries on the empirical evidence. 

Much of what these critics have to say about the inconsistent frameworks, dubious findings and design short-comings of the experimental  literature on video gaming, is cogent. The early experimental studies conducted in labs and lecture halls with psychology students are particularly problematic: In these studies the operationalization of violent content is crude and of aggressive behavior often confusing. Moreover, at least half of  the  25 + studies, are out of date:   experimental comparisons of playing Space Invaders for 10 minutes can provide no insight into the consequences of playing  Quake for  25 hours a week.  The indicators of aggressive consequences are inconsistent,  including hitting and fighting, verbal taunts, feelings of hostility, moral judgments of others behaviour, as well as playful enactments of conflict?  Moreover, a 10 minute exposure to a space invaders game or a questionnaire conducted in a lecture halls provide limited understanding of the complex processes of internalizing representations of imaginary conflict in video games. Even supportive readers of this literature would probably agree that the evidence has not provided convincing proof of the ‘causal hypothesis’. 

So far Griffiths sums up the empirical record best stating flatly that “all the published studies on video game violence have methodological problems and only include possible short-term measures of aggressive consequences” (Griffiths 1997). Other reviewers are equally scathing about the design and measurement issues plaguing experimental video game research (Durkin 1995, Goldstein 1999, 2002, Griffiths 1997).  Since male aggression is so deeply embedded in our contemporary culture, Goldstein (1998, 1999) suggests there is no reliable finding beyond the male fascination with human conflict and combat: what these positive results actually show is that boys enjoy playing out their action adventure  fantasies. And Goldstein is right that it would take evidence gathered not in the lab, but in longitudinal studies to sort out the causal links between aggressive predisposition, interest in violent games, or gamers experience of playing them over time. The limitations of these study designs means that our understanding of causal links between game play and aggression is debatable: but one wonders why they are being tallied as failed tests of the causal hypothesis.

To agree that there are severe limitations of research design in the experimental literature is not tantamount to confirming that psychological research reveals “absolutely nothing” about children’s use of violent video games.  Goldstein (2002) criticizes the video game research literature for at best demonstrating that video games only influence the way children talk and play together aggressively. Griffiths similarly finds that “one consistent finding is that the majority of the studies on very young children – as opposed to those in their teens upwards-tend to show that children do become more aggressive after either playing or watching a violent video game” when the research observes children’s ‘free play’.  In short, the Amici’s behavioral critics imply that there is evidence of influence of violent entertainment on children’s play, attitudes and skills, but this evidence of ‘effects’ does not constitute in their mind solid evidence of aggression.  

Beyond Generalization

Because the study of video games is relatively new, the Amici’s brief does not confine its criticisms to the video game literature claiming that there is insufficient scientific proof that violent entertainment has any harmful effects either. The brief cites Jonathon Freedman’s (2002) recent substantial and complex book length assessment of the  television literature to dispute Anderson’s claim that there is  proven “causal link” between violent entertainment and aggression. Although Freedman agrees that there is some evidence suggesting a relationship exists between preferences for and exposure to violent entertainment and aggressive and anti-social behavior, he argues that on close examination the evidence is exaggerated and inconsistent.  He notes that in many cases the statistical significances are “vanishingly” small and that the measures of aggression so badly defined that they include play, feelings of hostility, or willingness to hurt, all which he considers “dubious” indicators of a direct causal hypothesis. Although he acknowledges there is some evidence of effects, Freedman points out at best surveys can account for only 10% of the variance of  aggressive behavior, which he considers rather unimportant, compared with other determinants. 

Since the Amici exhibit a badly disguised antipathy to the use of probability statistics in the empirical sciences stating that “significant” does not mean ‘important’. It means simply ‘not likely to happen by chance.’ Since the meaning of statistical inference is raised by the Amici it may be useful to put Freedman’s debate about ‘variance explained’  in a population statistics perspective.  For example the Youth Risk Behavior Surveillance data for 2001 of over 13000 teenagers finds that 33 % report getting in a fight during the last year. Since 16% of the US population of 276 million  is between 12-20 we can estimate that .16 x .33 x 276 = 14, 572, 800 fights take place each year. Using Freedman’s estimate that 10% of those fights can be accounted for by the statistical relationship between violent media consumption and aggression, we can estimate that about 1.45 million more fights take place every year than would happen by chance, or for other reasons.

Freedman’s review is too complex to contest in more detail here.  But I find the applicability of his work to the video games debate marginal for three reasons: First he simply does not address differences between playing games and watching narratives which means that experiential differences in modes of engagement with meaning making (implied in an ‘active audience’ model) need to be taken into account (Dill and Dill 1998, Kline 2001). Secondly, by applying  behaviorist rules of evidence to the media effects literature, Freedman excludes all evidence which does not prove a significant direct effect on real aggression.  Applying this behavioral yardstick to experiments, surveys and longitudinal research, he blithely dismisses all kinds of evidence of media influences on intervening variables such as perceptions, learning, attitudes, peer relations, feelings or play, because they  fail to confirm the ‘causal hypothesis’. But most importantly, his study is well past its best-by date:  there have been a growing number of well designed longitudinal studies (by everyone’s admission the most suited for establishing how significant the contribution of media violence to aggressiveness) which have provided compelling findings.   (Johnson et al. 2002, Anderson. D et al, 2001, Huesmann et al. 2003, Endstad and Torgersen 2003). Johnson et al. (2002) for example, report that even after controlling for other factors known to contribute to aggressiveness in young people “like childhood neglect, growing up in an unsafe neighborhood, low family income, low parental education and psychiatric disorders” there remain “significant associations between television viewing during early adolescence and subsequent aggressive acts against other persons” later in life. Their data show for example that young boys who watch more television are particularly at risk for aggressive behavior media: whereas 45% of the boys who watched television more than 3 hours per day at age 14, subsequently committed aggressive acts involving others, only 8.9%, who watched television less than an hour a day were aggressive later in life.

Whose causal hypothesis anyway?

One of the favorite tropes in the attack on effects research is that an impact of  media violence on aggressive play, attitudes, bullying, or hostile thoughts and feelings are not considered valid indicators of  real aggression.  This argument manifests itself in a persistent maligning of the 1963 Bobo doll experiments conducted by Albert Bandura.  Freedman and Goldstein both discount this piece of research, claiming that in spite of impressively clear findings, the study provides no ‘proof’ that media violence causes aggression, because ‘hitting’ a Bobo doll cannot be considered an aggressive act.  This insistence on a behavioralist test of the causal hypothesis was invoked recently when in response to data from a recent survey of 9000 Norwegian high school students, ISDA president Doug Lowenstein  protested that evidence of a relationship between video game play and self reports of bullying and threatening behaviors at school could not be considered as supporting the causal hypothesis because bullying is not aggression (Endstad and Torgerson 2003).

A less antagonistic reading of his 1963 study however reveals that Bandura never set out to prove the behavioralist version of the causal hypothesis.  Rather, he set out to examine whether observational learning processes extended to modeling behavior portrayed on television. Hitting of the Bobo doll is never interpreted by Bandura as a measure of aggression per se, but rather, as evidence  that a child, having observed the particular pattern of modeled behavior, incorporates that behavioral  construct into their play routines.  His study indicates that they are more likely to do so, when the behavior is modeled on television than even in the classroom -- both by cartoon characters as well as real ones.  His conclusion states that it is hard to believe that watching television doesn’t provide some children with an opportunity to learn behavioral scripts from media representations of conflict.

In his review Griffiths (1997) is reluctant to dismiss evidence of learning: he concludes that  “the question of whether video games promote aggressiveness cannot be answered at the present because the available literature is relatively sparse and conflicting, and there are many different types of video games which probably have different effects”. It is not that there is no proof of effects, but rather that the evidence is not yet consistent enough to know what those effects are. Without that evidence, its better to say the case is not closed. Yet  Griffiths maintains  that the research on video games suggests that they can have both positive and negative consequences for children’s learning: “If care is taken in the design, and if games are put in the right context, they have the potential to be used as training aids in classrooms and therapeutic settings, and to provide skills in psychomotor coordination in simulations of real life events, for example, training recruits for the armed forces”. Pace David Grossman. Isn’t that exactly what is meant by video games being “environments in which children can learn about aggression”. 

But Griffiths is here making a good point. Should we believe that playing video games in school can have positive learning outcomes, but violent gaming at home has no negative ones?  Anyone who has talked to young gamers (as I have) will know that they do learn about military tactics from playing games, as well as becoming computer literate.  This doesn’t mean that playing them will ultimately make them better able to kill someone, but it does imply there are plausible psychological explanations (identification, aggressive cognitions) suggesting why conflict gaming influences some children. At the moment we know very little about media’s role in the socialization of aggression.  Even the American military, who now use combat games as part of the military recruitment, strategy and training programs, have not conducted scientific evaluations of their role in training soldiers (Mark Prensky 2003). But that does not mean that they are wasting this investment. We simply don’t know. Unfortunately, there are simply no longitudinal field studies to reveal the part played by video gaming in the socialization of  aggressive and anti-social attitudes. And this absence of scientific knowledge is what is being contested. 

Criticizing the critics critiques

I must point out that my methodological criticisms of the Amici brief should not be taken as a wholesale endorsement of Anderson’s testimony. His claim that video games provide a ‘complete’ training environment clearly overstates what we know so far. The OED defines a science as "a branch of study which is concerned either with a connected body of demonstrated truths or with observed facts systematically classified and more or less colligated by being brought under general laws and which includes trust worthy methods for the discovery of new truths within its own domain". The first rule underlying all scientific discourse, like that of the courts, is empiricism: the conclusions reached must be based on evidence established through a rigorous process of gathering and evaluating facts systematically. On these criteria, both lawyers and scientists would agree. But according to this definition, scientific discourses should be disciplined by "trustworthy methods" that help adjudicate the best explanation of those empirical observations.  In the twentieth century, those who practiced quantitative social science established inferential rules of evidence based on probabilistic tests of a null hypotheses formulated by the proposition that no relationship exists between cause and effect. These inferential procedures used to assess a result maintain that research can never prove a hypothesis true, but rather only convince us to reject the ‘null hypothesis’.

Judged by Karl Popper’s epistemological doctrine of falsifiability,  Anderson’s claim that there is already sufficient evidence to claim a ‘proof of harm’ also does not rest, to my mind, on  credible scientific grounds. Unfortunately the Amici’s insistence that unless experiments show  that after playing a murderous video game a significant number of children jump up and kick or hit another child, there is no proof of a harmful effect on children’s behavior is equally incompatible with the ‘falsifiability’ principle subsumed in scientific rules of evidence. No experiment can ever prove media violence effects behavior, but rather only weaken our belief that there are no consequences from persistent exposure to media violence. That is generally the conclusion reached by the American Psychological Association in their public review of the issue (2001).

Much of the research reviewed by the critics never set out to test the causal hypothesis in the first place. Rather, their purpose was to investigate to what degree and how children learn about their conflicted social world from media. In such research, the observation of play, peer teasing, moral judgments, desensitization, attitudes to crime etc. can provide useful insights into psycho-social processes. This is because, unlike the behaviorists, most social psychologists are interested in the learning of aggressive behavior as meaningful pattern of social action that is performed in different situations which have implicit rules and sanctions. However allegorical, media narratives remain one place where children can learn about the conflict world they live in – its values and ideologies – even while playing games.  What is learned from using media  will depend  on  children’s disposition, patterns of use, identification with and interpretation of the violent scenarios they encounter. Moreover, there are many factors besides media, which contribute to children’s learning about and performance of anti-social behavior:  Personal experience, peer relations, identification with role models, intelligence, sex roles, and  parenting styles are all documented factors in the development of social skills and aggressive  dispositions. Given the diversity in children’s circumstances,  there is little reason to expect uniform behavioral responses to violent entertainment among children whose circumstances and experiences are diverse.

This is also why most contemporary effects researchers do not predict that  a majority of children will be negatively influenced by media violence.  It is only by factoring in environmental factors numerically that social psychological researchers have gradually will be able to explain why not all heavy consumers of violent entertainment grow up in some situations to be aggressive and anti-social while non-gamers become serial killers. There are of course many complex factors to consider not all of them predicting the same outcome  (Ehron 1997).   Rather than the causal hypothesis, the driving force behind the risk factors approach is the quest to understand what it all depends on. Garbarino (2001), for example has shown that once the researcher accounts for other social context variables which he calls  “developmental assets”  the media’s influence becomes clearer.  “Assets are found throughout the social ecology of the child—family, school, neighborhood, and community. Among asset-rich children the impact of violence is low while among asset-poor children the rate is high: the rates are 6% for kids with 31 to 40 assets, 16% for those with 21 to 30, 35% for those with 11 to 20, and 61% for those with 0 to 10. Risk and opportunity interact in the field of social action which is why, Garbarino says “an accumulation-of-risk model is essential if we are to understand where televised violence fits into the learning and demonstration of aggressive behavior.”   

The Amici’s blurring of the arguments about the causal hypothesis and risk factors proves extremely reductionist in my view. In a world where multiple social determinants are mutually interacting in highly varied circumstances we obviously want more sophisticated diagnostic tools which can map systematically and tease out the various factors. The risk factors model, based on epidemiology, although not perfect, is a step in the direction of complexity.  In the history of science this distinction is akin to those between biology and ecology in the natural sciences, or between a mechanical Newtonian physics and systems quantum science. The tools for assessing risks developed by epidemiology are still under development, but it does no service to science to dismiss the insights these scientists offer.

Clearly, the advantage of this model is that it is less concerned with generalizing, than with understanding whether, how and why the patterned use of media contributes to young peoples attitudes, interactions and social behavior. It acknowledges many environmental factors besides media that contribute to the socialization of aggression ranging from family dysfunction to anti-bullying programs in schools. And it is not unidirectional: It is equally interested in both the circumstances in which an aggressive disposition can lead some children to prefer  violent entertainment and identify with aggressive characters more as well as why fascination with violent entertainment can teach or consolidate anti-social attitudes and aggressive peer interactions in some children. And it acknowledges and accounts for social diversity recognizing the circumstances such as family mediation, peer groups and neighborhoods can be both a precipitating or moderating influence over the way children use and interpret media. Learning all depends on circumstance and context, so the consequences of heavy media use cannot be understood independently of these contextual factors.

Conclusion

I began this argument by claiming that scientific discourses on media and aggression have been increasingly articulated within a context which is highly politicized because the question lay at the centre of the children’s cultural marketplace where state control of the potential risks of media conflicts with the industry’s prerogatives of commercial speech.  Worried about matters of legal liability and refusing all constraints on their rights of ‘free artistic expression’, the gaming industry maintains that without ‘proof of harm’ the state has no grounds for regulation of this medium of communication. This legal doctrine, favored by private interests defending themselves in risk controversies (i.e. tobacco, GM foods, irradiated foods), extends what Tindale (1998:59) calls the Panglossian principle meaning "the burden of proof is with those trying to prove that there is a risk”.

The Amici weighed in to the legal battle claiming that empirical researchers have not been able to sustain broad generalizations about the causal processes involved, so the St. Louis ordinance is tantamount to censorship. And herein we see the hidden agenda behind this scientific intervention: although, on the surface the Amici’s brief challenges the evidence of a causal link, its intent is to short-circuit precedent setting regulation of the gaming industry in America by throwing their academic weight behind the industries’ claim there is no proof of harm.  In the process, the more complex scientific discourses on media effects were subverted by this battle about legal doctrines adjudicating liabilities and responsibilities in the production and selling of cultural products to children.

What has been lost in this battle over the limits of scientific certainty is the Amici’s assumption that all regulation in the children’s cultural marketplace -- even that directed at protecting children -- is tantamount to censorship.  In this sense, what purports to be a scientific aide memoir for a judge turns out to be a  political tract defending the industry’s de-regulationist ideology. Although I am not a constitutional lawyer, I think the assumptions implied in this assertion bear on some very important general questions, concerning the rights of free speech enjoyed by cultural industries and the ambiguous status of entertainment as a cultural product. Far from Draconian, the purpose of this legislation seems to be to insure that parents have sufficient information about morally controversial and potentially risky cultural products being sold to children.  The industry itself, espousing its desire to ensure parental informed consent and responsibility, claim this is why they created the ESRB ratings in the first place. So why are they opposed to having these ratings mandated in a manner similar to cinema and television? 

The reason is, the industry maintains that video games are an art form.  The anti-censorship lobby represented by Marjorie Heins of the Free Expression Policy Project supports this claim telling reporters that the St. Louis “legislators would like to censor violent media content based on a notion of moral harm or moralism''.  Others, among the Amici have claimed that a ‘media panic’ about violence has been primed by moral entrepreneurs who are opposed to violence and sexuality in the media on taste grounds (Buckingham 2000, Davies et al. 2001). But this obscures the status of entertainment as commercial popular arts – that is as cultural artefact and commodity circulated in the market to children. Even assuming the law was intended to maintain moral and artistic taste, is there to be no state interest in maintaining community standards in cultural markets? Such provisions are thought to be especially important for children 12 years and younger who for developmental reasons are not considered adult subjects in most other nations. Difficult as they are to administer,  in Canada, as well as many European countries, make provisions for sanctioning offensive (sexist, racist) material in children’s cultural markets (Broadcast Standards Council) – which could be applied to the marketing of violent games to children for example.

The Amici also assert that the St. Louis legislation is a censorship law.  Censorship implies a prohibition. But what St. Louis proposed was not a ban, but a commercial ordinance which enforces the ESRB ratings system (constructed by the gaming industry) to advise parents about the potentially offensive content of these games.[i]  Of course, the games industry’s legal position is that their rights to free speech are paramount and over-riding the interests of parents and children. Their lobby group Media Coalition Inc. has stated in opposition to the Minnesota Senate Bill: “While voluntary ratings exist to help parents determine what is appropriate for their children, government enforcement of existing rating system is constitutionally impermissible”. Perhaps this is true in America, which has not signed the UN convention on the rights of the child, and where freedom of the press amounts to constitutional dogma demanding absolute proof of harm under the Ginsberg test. But elsewhere mandated warning labels for risky consumer goods (cigarettes, peanuts, GM foods) have been based on the precautionary principle.  The principle establishes that even without proof of harm, warning labels are justified as a right of consumers to have a reasonable knowledge about risks associated with the products they use. In short if the intent of the law, is not to regulate morality or even control risks, but  to help parents make appropriate choices about the games they or their children purchase, how is it censorship? 

So was the actual intent of the legislation to censor taste and values, or to regulate a medium because of health and safety risks?  If the former, then why is there a need to prove harm? And if the later, then why is a precautionary principle (as suggested by the Surgeon General) not the better policy framework to apply? The dual status of cultural products – as speech acts and as commodities exchanged in the market is the key to the underlying legal ambiguity. In many countries, the precautionary principle has been extended to cultural goods, such as film and TV, because these states feel that mandated product advisories, as applied to the movie, video and television industries, help parents take responsibility for their children’s media use. In the USA, similar product warning systems have been recently implemented technologically on television by the V-chip. Perhaps that is why Judge Lindbaugh found the limit on commercial speech acceptable. So why should cultural products like video games (and the internet) be exempted from this established principle of mandated product advisories?

While preparing this response, I have been corresponding with the Amici through Marjorie Heins. Her response to my explication of the risk factors model indicates that convergence is possible on many of the methodological issues dividing audience research. Acknowledging that “media messages have a powerful impact”  lawyer Heins maintains that it is still too “difficult to generalize about - no less quantify - these effects; that is the essential point of the brief - particularly when the subject under examination is something as broad and vague and varying in its context as "media violence". But she also concedes that there are other ways of evaluating the body of evidence:  “As you say, the more sophisticated contemporary debates about media effects don't talk in terms of "direct causes of aggressive behaviour. But this brief wasn't responding to that more nuanced debate; it was responding to claims made by scholars such as Craig Anderson and relied upon by political bodies such as the St. Louis County Council, that speak very much in terms of direct causes.” 

I point this out here to help put  the arguments about the scientific evidence of a causal link  in the context of the legal doctrines impinging on state regulation of cultural markets in a society where values and community standards are diverse and constantly changing. In the heat of a legal battle, both sides seem to have stretched their scientific credibility by making sweeping declarations about this limited body of scientific research.  In the absence of sound empirical research, the industry’s demand for proof of a ‘direct causal relationship’ sets an impossibly high bar for demonstrating there are risks associated with long term media consumption while discarding all evidence that children can learn about aggression from media. In response the researchers overstate their case about learning and exaggerate what we actually understand of this complex phenomenon. The net effect is a lot of rhetoric obfuscating the policy debates about the risks encountered in cultural markets behind a smokescreen of scientific uncertainty.

In my critique I have been careful not to side with the claims of Dr. Anderson. I personally find the scientific interpretation offered by the U.S. Surgeon General (2001) in a recent report on youth aggression the most thoughtful account to date on the issues of media and aggression.   This report suggests that video games  can be a risk factor, if a not a cause, which in addition  to and  interacting with many other social factors contributes to socialization of aggression:

“Unlike earlier Federal research reports on media violence and youth (National Institute of Mental Health, 1982; U.S. Surgeon General's Scientific Advisory Committee on Television and Social Behavior, 1972), this discussion takes place within a broader examination of the causes and prevention of youth violence. This context is vital. It permits media violence to be regarded as one of many complex influences on the behavior of America's children and young people. It also suggests that multilayered solutions are needed to address aggressive and violent behavior."

The Surgeon General’s  media risk model does not predict that  young  people  will uniformly commit  aggressive acts immediately after watching because media effects interact with other risk factors experienced within peer groups, schools, families, communities. Weighing up the available evidence according to well established epidemiological criteria for studying causality in multiple and interacting determinacy relations (Bradford-Hill 1965), he recommends a precautionary rather than panglossian principle  stating that  "Research to date justifies sustained efforts to curb the adverse effects of media violence on youths. Although our knowledge is incomplete, it is sufficient to develop a coherent public health approach to violence prevention that builds upon what is known, even as more research is under way.” 

It is interesting to note, than in a recent review of a similarly controversial literature – the effects of food marketing on children’s diet and obesity, that a very similar conclusion was reached. Applying the risk model to a equally complex influence process the review concludes that “promotion is just one part of the complex process of marketing and that measuring its effects on consumer behaviour (and disentangling these from other influences) is notoriously difficult.” Yet reviewing the field of alcohol and tobacco promotion they maintain that although “hard and fast proof about promotional effects will never emerge” that in the case of tobacco promotion “judgements have to be made on the balance of probabilities” . Noting a “clear link between television viewing and diet, obesity and cholesterol levels” the conclude that the literature “does suggest food promotion is influencing children’s diet in a number of ways”. The researchers are careful note that the research is complex and inconsistent stating that “incontrovertible proof simply isn’t attainable”. Yet they report concludes that “Nonetheless, many studies have found clear effects and they have used sophisticated methodologies that make it possible to determine that 1) these effects are not just due to chance; ii) they are independent of other factors that may influence diet, such as parents’ eating habits or attitudes; and iii) they occur at a brand and category level.”

This is not to say there is no room for healthy scientific debate about risks associated with media whether it be violence or obesity: but perhaps once we acknowledge that there is a complex relationship between media violence and aggression, researchers get on with diagnosing the reasons for this relationship, its magnitude, its implications and what we can do about it. It is true we don’t know very much about the role that violent video games play in the socialization of aggression. We know that young males prefer violent games and play them more. We also know that some gain a sense of power and standing in the peer groups from their skill in gaming. We know that games can teach, and that they seem to be an important part of some youthful male peer groups. But we don’t know what those drawn to these agonic experiences actually learn in their play. The Amici  speculate that gaming’s  transgressive experiences can have a cathartic effect by providing a harmless outlet for fantasy offering children a coping mechanism for dealing with youthful frustration and anomie. Grossman speculates alternatively that heavy players will become desensitized by the repeated experiences of killing, and more skilled in the strategies of hunt and shoot. Others believe that games can in some instances habituate players to the affect of killing or reinforce pathological fascinations with conflict and death.   We simply don’t know very much about the part played by violent and transgressive gaming and gamer cultures in the socialization of aggression.  

Being a researcher rather than a lawyer, I can only point out that the Amici’s  arguments about this policy issue are framed by distinctly American legal doctrine. In Canada for example, mandating a ratings system is a legal constitutional restraint on commercial speech under certain circumstances. The Supreme Court has already ruled that freedom of commercial speech is a right which must be weighed against the state’s interest in the socialization of the children under thirteen years of age. But when the British Columbia government passed a law extending the film classification act to video games, the ISDA came to Canada protesting they were being censored. The export of American legal doctrine in global cultural markets is a form of cultural imperialism, that I for one, find as offensive as the rifle associations lobby against Canadian gun control laws. But our constitution embodies different legal doctrine.

One of the most intellectually paralyzing assumptions of the Amici’s brief is that violence has always been with us throughout history and is so pervasive in our culture that there is nothing we can do about it. A recent natural experiment conducted by Tom Robinson in San Jose suggests otherwise. Robinson (2001, 2000)  reasoned that if the amount of media use really is a factor in the violence effect (because of increased exposure) then reducing that media consumption should reduce the risk. He tested this causal hypothesis finding that schools that participated in the media education program not only reduced their media consumption by 25% but also enjoyed in a significant reduction in playground aggression and had more children with a lower rate of increase of body fat.  I have replicated Robinson’s finding (Kline 2003). This is the contribution that cultural science of audiences can make to the debate. But without a more complex and robust study of processes involved in learned aggressiveness (i.e. identification with aggressors; peer relations and lack of social skills) we will never be able to understand the long term contribution of these meaningful agonic experiences to social cognitions, peer relations, moral thinking and masculine identities.   Isn’t it time the industry put some of the money it devotes to court battles, into finding out?

References

American Academy of Pediatrics. (2001). "Media violence. Committee on Public Education." Pediatrics 108(5): 1222-6.

Amici Curiae, Interactive Digital Software Assn., et al., v. St. Louis County, et al., Anderson, C. A. and B. J. Bushman (2001). "Effects of violent video games on aggressive behavior, aggressive cognition, aggressive affect, physiological arousal, and prosocial behavior: a meta-analytic review of the scientific literature." Psychol Sci 12(5): 353-9.

Anderson, C. A. and B. J. Bushman (2002). "Psychology. The effects of media violence on society." Science 295(5564): 2377-9.

Anderson, C. A. and K. E. Dill (2000). "Video games and aggressive thoughts, feelings, and behavior in the laboratory and in life." J Pers Soc Psychol 78(4): 772-90.

Anderson, D. R., A. C. Huston, et al. (2001). "Early childhood television viewing and adolescent behavior: the recontact study." Monogr Soc Res Child Dev 66(1): I-VIII, 1-147.

APA American Psychological Association (2003). Violence on television - what do children learn? What can parents do?

Bandura, A., D. Ross, et al. (1963). "Imitation of film-mediated aggressive models." Journal of Abnormal & Social Psychology 66(1): 3-11.

Barker, M. and J. Petley, Eds. (1997/ 2001). Ill effects : the media/violence debate. London, Routledge.

Barkin, S., S. Kreiter, et al. (2001). "Exposure to violence and intentions to engage in moralistic violence during early adolescence." J Adolesc 24(6): 777-89.

Bradford-Hill, A. (1965). "The environment and disease: association or causation?" President's Address. Proc Royal Soc Med. 9: 295-300.

Buckingham, D. (2000). After the death of childhood : growing up in the age of electronic media. London, Polity Press.

Buckingham, D. (2000). Studying children's media cultures: A new agenda for cultural studies. Children and media : multidisciplinary approaches. B. v. d. Bergh and J. v. d. Bulck. Leuven, Garant: 49-66.

Bushman, B. J. and C. A. Anderson (2001). "Media violence and the American public. Scientific facts versus media misinformation." Am Psychol 56(6-7): 477-89.

Comstock, G. A., E. A. Rubinstein, et al. (1972). Television and social behavior : reports and papers ; a technical report to the Surgeon General's Scientific Advisory Committee on Television and Social Behavior, National Institute of Mental Health.

Davies, H., David Buckingham and Peter Kelly. (2000). In the Worst Possible Taste: Children, Television and Cultural Value. European Journal of Cultural Studies, 3(1), 5-26.

Dill, K. E. and J. C. Dill (1998). "Video game violence: a review of the empirical literature." Aggression and Violent Behavior 3(4): 407-428.

Durkin, K. (1995). Computer games: their effects on young people. Sydney, NSW, Office of Film and Literature Classification.

Endstad and Torgersen 2003, Violent behaviour and violent video games: is there really a connection. Paper given at Level Up Conference, University of Utrecht Nov 4. 2003

Eron, L. (1997). The Development of antisocial Behavior From a Learning Perspective. In D. Stoff, J. Breiling, & J. Maser (Eds.), Handbook of Antisocial Behavior (pp. 140-148). New York: Wiley.

ESRB (1998). ESRB Parent's Guide, ESRB. 2003.

FTC. (2000). Marketing Violent Entertainment to Children:  A review of self regulation and industry practices in the motion picture, music recording and electronic game industries. Washington, DC: Federal Trade Commission.

Foucault, M. and C. Gordon (1980). Power/knowledge : selected interviews and other writings, 1972-1977. New York, Pantheon Books.

Fowles, J. (1999). The case for television violence, Sage Publications.

Freedman, J. L. (1984). "Effect of television violence on aggressiveness." Psychol Bull 96(2): 227-46.

Freedman, J. L. (2002). Media violence and its effect on aggression : assessing the scientific evidence. Toronto, University of Toronto Press.

Funk, J.-B., G. Flores, et al. (1999). "Rating electronic games: Violence is in the eye of the beholder." Youth-and-Society. 1999 Mar; Vol 30(3): 283-312.

Funk, Jeanne et. al. (1999). Playing Violent Electronic Games and Indices of
Psychopathology in Adolescent. Paper accepted for presentation at the Annual
Meeting of the American Psychological Association, august 20, 1999.

Funk, J. B. (2000). Testimony regarding The Impact of Interactive Violence on Children before the United States Senate Commerce Committee.

Garbarino, J. (2001). "Violent children: where do we point the finger of blame?" Arch Pediatr Adolesc Med 155(1): 13-4.

Goldstein, J. H. (1998). Why we watch : the attractions of violent entertainment, Oxford University Press.

Goldstein, J. H. (2000). Effects of Electronic Games on Children. 2003.

Goldstein, J. H. (2001). Does playing violent video games cause aggressive behavior? Cultural Policy Center, University of Chicago. 2003.

Grossman, D. and G. DeGaetano (1999). Stop teaching our kids to kill: a call to action against TV, movie & video game violence. New York, Crown Publishers.

Griffiths, M. (1997). "Video games and aggression." Psychologist. 1997 Sept. Vol 10(9): 397-401.

Gunter, B. and J. Harrison (1998). Violence on television : an analysis of amount, nature, location, and origin of violence in British programmes, Routledge.

Hastings,Gerard, Martine Stead, Laura McDermott, Alasdair Forsyth, Anne Marie MacKintosh, Mike Rayner, Christine Godfrey, Martin Caraher and Kathryn Angus. Review of Research on the Effects of Food Promotion to Children Final Report, Prepared for the Food Standards Agency, accessed http://www.foodstandards.gov.uk/healthiereating/promotion/readreview/

Huesmann, L. R., J.-F. Moise, et al. (1997). The effects of media violence on the development of antisocial behavior. Stoff, David M. (Ed); Breiling, James (Ed); et-al. (1997). Handbook of antisocial behavior. (pp. 181-193). New York, NY, USA: John Wiley & Sons, Inc. xxii, 600 pp.

Huesmann, L. R., J. Moise-Titus, et al. (2003). "Longitundinal  Relations Between Children's Exposure to TV Violence and Their Aggresive and Violent Behavior in Young Adulthood: 1977- 1992." Developmental Psychology 39(2): 201-221.

Huston, A. C. (1992). Big world, small screen: the role of television in American society. Lincoln, University of Nebraska Press.

Jenkins, H. (1999). Professor Jenkins Goes to Washington.

Johnson, Jeffrey G. Patricia Cohen, Elizabeth M. Smailes, Stephanie Kasen, and Judith S. Brook Television Viewing and Aggressive Behavior During Adolescence and Adulthood Science 295: 2468-2471 2002

Jurkowitz, Mark  Appeals court holds key in battle over regulation of violent video games, Boston Globe 10/2/2002

Kline, S. (2000). Moral Panic and Video Games Research in Childhood, A Collection of Papers from the Sociology, Culture and History Conference. Odense University, Denmark, Department of Child and Youth Culture.

Kline, S. (2000)    "Killing Time? A Canadian Meditation on Video Game Culture"  in Children and Media Violence. UNESCO Clearinghouse Annual.

Kline, S. and K. Stewart (2000). Family Life and Media Violence: A Qualitative Study of Canadian Mothers of Boys. Children and Media: Multidisciplinary Perspectives. B. v. d. Bergh and J. v. d. Bulck. Garant, Lueven: 89-110.

Kline, S. (2003): Learners, Spectators or Gamers in the Media Saturated Household?   An Investigation of the Impact of the Internet on Children’s Audiences in Goldstein, J. Buckingham, D. Brougere, G. (Eds.) Toys, Games and Media. Lawrence Ehrlbaum.

Lowenstein, D. (2000). Testimony of Douglas Lowenstein, President International Digital Software Association before the Senate Committee on Commerce on the Effects of Interactive Violence on Children. 2003.

Prensky, Mark, Defense combat Sim Olympics-methodologies incorporating the cybergaming culture, Tues Nov 4 Level Up Conference, Utrecht University. 2003

McGuire, W. J. (1986). The myth of massive media impact: savagings and salvagings. Public communication and behavior. G. Comstock. New York, Academic Press. 1: 173-257.

Minow, N. N. and C. L. LaMay (1995). Abandoned in the wasteland: children, television, and the First Amendment. New York, Hill and Wang.

Murray, J. P. (1995). "Children and Television Violence." Kansas Journal of Law & Public Policy 4(3): 7-14.

Murray, J. P. (2001). "TV violence and children’s brains: More reasons for advocacy and policy reform." The Child, Youth, and Family Services Advocate 24(2): 1-4.

NIMH 1982 Television and growing up: the impact of televised violence; report to the Surgeon General, United States Public Health Service. Rockville, Md.,, National Institute of Mental Health.

Reichardt Tony 2003  Playing with Fire, Nature, Vol 424:367-368

Robinson, T. N., M. L. Wilde, et al. (2001). "Effects of reducing children's television and video game use on aggressive behavior: a randomized controlled trial." Archives of Pediatrics & Adolescent Medicine 155(1): 17-23.2000

Sherry, J. L. (2001). "The effects of violent video games on aggression: A meta-analysis." Human Communication Research 27(3): 409-431.

Spigel, L. 1998). Seducing the Innocent: Childhood and Television in Postwar America. The Children's Culture Reader. H. Jenkins. New York, New York University Press: 110-135.

Surgeon General of the USA (2001). Youth Violence: A Report of the Surgeon General.

Tindale, S. (1998). Procrastination and the Global Gamble. The Politics of Risk. J. e. Franklin. Cambridge, Polity: 54-69.

United States Court of Appeals for the Eighth Circuit. United States Department of Justice (2002).

United States Court of Appeals for the Eighth Circuit (2002).  Interactive Digital Software Assn., et al., v. St. Louis County, et al.,

Williams Raymond (1974) Television: Technology and Cultural Form. London: Fontana

Williams, R. (1974). "Communications as cultural science." Journal of
Communication 24(3): 17-25.

Wober, J. M. (1988). The use and abuse of television: a social psychological analysis of the changing screen. Hillsdale, N.J., L. Erlbaum Associates.

▲ ◄