How to measure media bias?

Introduction

Media bias is under heavy discussion at the moment, especially relating to the on-going presidential election in the US.

However, the quality of discussion is not the way it should be; I mean, there should be objective analysis on the role of the media. Instead, most comments are politically motivated accusations or denials.

This article aims to be objective, discussing the measurement of media bias; that is, how could we identify whether a particular media outlet is biased or not?

The author feels there are not generally acknowledged measures for this, so it is easy to claim or deny bias without factual validation. Essentially, this erodes the quality of the discussion, leading only into a war of opinions.

Second, without the existence of such measures, both the media and the general public are unable to monitor the fairness of coverage.

Why is media fairness important?

Fairness of the media is important for one main reason: the media have a strong influence on the public opinion. In other words, journalists have great power, and with great power comes great responsibility. The existence of bias leads to different standards of coverage depending on the topic being reported. In other words, the information is being used to portray a selective view of the world. This is analogous to confirmation bias; a person wants to prove a certain point, so he or she only acknowledges evidence supporting that point.

Such behavior is very easy for human beings, for which reason journalists should be extra cautious in letting their own opinions influence the content of their reportage.

In addition to being a private problem, the media bias can also be understood as a systemic problem. This arises through 1) official guidelines and 2) informal group think. First, the official guidelines means that the opinions, beliefs or worldviews of the particular media outlet are diffused down the organization. Meaning that the editorial board communicates its official stance (“we, as a media outlet, support a political candidate X”) which is then taken by the individual reporters as their ethos.

When the media outlet itself, or the surrounding “media industry” as a whole, absorbs a view, there is a tendency to silence the dissidents. This, again, can be reduced to elementary human psychology, known as the conformity bias or group think.

Because others in your reference group accept a certain viewpoint, you are more likely to accept it as well due to social pressure. The informal dynamics are even more dangerous to objective reporting than the official guidelines because they are subtle and implicit by nature. In other words, journalist may not be aware of bias and just consider their worldview “normal” while arguments opposing it are classified as wrong and harmful.

Finally, media fairness is important due to its larger implications on information sources and the actions taken by citizens based on the information they are exposed to. It is in the society’s best interest that people resort to legitimate and trustworthy sources of information, as opposed to unofficial, rogue sources that can spread misinformation or disinformation.

However, when the media becomes biased, it loses its legitimacy and becomes discredited; as a form of reactance to the biased stories, citizens turn to alternative sources of information. The problem is that these sources may not be trustworthy at all.

Therefore, by waving their journalistic ethics, the mass media become at par with all other information sources; in a word, lose their credibility. The lack of credible sources of information leads into a myriad of problems for the society, such as distrust in the government, civil unrest or other forms of action people take based on the information they receive. Under such circumstances, the problem of “echo chamber” is fortified — individuals feel free to select their sources according to their own beliefs instead of facts.

After all, if all information is biased, what does it matter which one you choose to believe in?

How to measure media bias?

Overview

While it may not be difficult to define media bias at a general level, it may be difficult to observe an instance of bias in an unanimously acceptable way. That is where commonly accepted measures could be of some help. To come up with such measures, we can start by defining the information elements that can be retrieved for objectivity analysis.

Then, we should consider how they can best be analyzed to determine whether a particular media outlet is biased.

In other words, what information do we have? Well, we can observe two sources: 1) the media itself, and 2) all other empirical observations (e.g., events taking place). Notice that observing the world only through media would be inaccurate testimony of human behavior; we draw a lot from our own experiences and from around us. By observing the stories created by the media we know what is being reported and what is not being reported.

By observing things around us (apart from the media), we know what is happening and what is not happening. By combining these dimensions, we can derive

1. what is being reported (and happens)
2. what is being reported (but does not happen)
3. what is not being reported (but happens), and
4. what is not being reported (but does not happen).

Numbers 2 and 4 are not deemed relevant for this inquiry, but 1 and 3 are. Namely, the choice of information, i.e. what is being reported and what is being left out of reporting.

Hence, this is the first dimension of our measurement framework.

1. Choice of information

• topic inclusion — what topics are reported (themes –> identify, classify, count)
• topic exclusion — what topics are not reported (reference –> define, classify, count)
• story inclusion — what is included in the reportage (themes –> identify, classify, count)
• story exclusion — what is left out of the reportage (reference –> define, classify, count)
• story frequency — how many times a story is repeated (count)

This dimension measures what is being talked about in the media. It measures inclusion, exclusion and frequency to determine what information the media disseminates. The two levels are topics and stories — both have themes that can be identified, then material classified into them, and counted to get an understanding of the coverage.

Measuring exclusion works in the same way, except the analyst needs to have a frame of reference he or she can compare the found themes with. For example, if the frame of reference contains “Education” and the topics found from the material do not include education, then it can be concluded that the media at the period of sampling did not cover education. Besides themes, reference can include polarity, and thus one can examine if opposing views are given equal coverage.  Finally, the frequency of stories measures media’s emphasis; reflecting the choice of information.

Because all information is selected from a close-to-infinite pool of potential stories, one could argue that all reportage is inherently biased. Indeed, there may not be universal criteria that would justify reporting Topic A over Topic B. However, measurement helps form a clearer picture of a) what the media as a whole is reporting, and b) what does each individual media outlet report in comparison to others.

A member of the audience is then better informed on what themes the media has chosen to report. This type of helicopter view can enhance the ability to detect a biased information choice, either by a particular media outlet or the media as a whole.

The question of information choice is pertinent to media bias, especially relating to exclusion of information. A biased reporter can defend himself by arguing “If I’m biased, show me where!”. But bias is not the same as inaccuracy. A biased story can still be accurate, for example, it may only leave some critical information out. The emphasis of a certain piece of information at the expense of other is a clear form of bias.

Because not every piece of information can be included in a story, something is forcefully let out. Therefore, there is a temptation to favor a certain story-line. However, this concern can be neutralized by introducing balance; for a given topic, let there be an equal effort for exhibiting positive and negative evidence.

And in terms of exclusion, discarding an equal amount of information from both extremes, if need be.

In addition to measuring what is being reported, we also need to consider how it is being reported. This is the second dimension of the measurement framework, dealing with the formulation of information.

2. Formulation of information

• IN INTERVIEWS: question formulation — are the questions reporters are asking neutral or biased in terms of substance (identify, classify, count)
• IN REPORTS: message formulation — are the paragraphs/sentences in reportage neutral or biased in terms of substance (classify, count)
• IN INTERVIEWS: tone — is the tone reporters are asking the questions neutral or biased (classify count)
• IN REPORTS: tone — are the paragraphs/sentences in reportage neutral or biased in terms of tone (classify, count)
• loaded headlines (identify, count)
• loaded vocabulary (identify, count)
• general sentiment towards key objects (identify, classify: pos/neg/neutral)

This dimension measures how the media reports on the topics it has chosen. It is a form of content analysis, involving qualitative and quantitative features. Measures cover interview type of settings, as well as various reportages such as newspaper articles and television coverage. The content can be broken down into pieces (questions, paragraphs, sentences) and their objectivity evaluated based on both substance and tone.

An example of bias in substance would be presenting an opinion as a fact, or taking a piece of information out of context. An example of biased tone would be using negative or positive adjectives in relation to select objects (e.g., presidential candidates).

Presenting loaded headlines and text as percentage of total observations gives an indication of how biased the content is. In addition, the analyst can evaluate the general sentiment the reportage portrays of key objects — this includes first identifying the key objects of the story, and then classifying their treatment on a three-fold scale (positive, negative, neutral).

I mentioned earlier that agreeing on the observation of bias is an issue. This is due to the interpretative nature of these measures; i.e., they involve a degree of subjectivity which is generally not considered as a good characteristic for a measure. Counting frequencies (e.g., how often a word was mentioned) is not susceptible to interpretation but judging the tone of the reporter is. Yet, those are the kind of cues that reveal a bias, so they should be incorporated in the measurement framework.

Perhaps we can draw an analogy to any form of research here; it is always up to the integrity of the analyst to draw conclusions. Even studies that are said to include high reliability by design can be reported in a biased way, e.g. by re-framing the original hypotheses. Ultimately, application of measurement in social sciences remains at the shoulder of the researcher. Any well-trained, committed researcher is more likely to follow the guideline of objectivity than not; but of course this cannot be guaranteed.

The explication of method application should reveal to an outsider the degree of trustworthiness of the study, although the evaluation requires a degree of sophistication.

Finally, using several analysts reduces an individual bias in interpreting content; inter-rater agreement can then be calculated with Cohen’s kappa or similar metrics.

After assessing the objectivity of the content, we turn to the source. Measurement of source credibility is important in both validating prior findings as well as understanding why the (potential) bias takes place.

3. Source credibility

• individual political views (identify)
• organizational political affiliation (identify)
• reputation (sample)

This dimensions measures why the media outlet reports the way it does. If individual and organizational affiliations are not made clear in the reportage, the analyst needs to do work to discover them.

In addition, the audience has shaped a perception of bias based on historical exposure to the media outlet — running a properly sampled survey can provide support information for conclusions of the objectivity study.

How to prevent media bias?

The work of journalists is sometimes compared to that of a scientist: in both professions, one needs curiosity, criticality, ability to observe, and objectivity. However, whereas scientists mostly report dull findings, reporters are much more pressured to write sexy, entertaining stories. This leads into the the problem of sense-making, i.e. reporters create a coherent story with a clear message, instead showing the messy reality.

The sense-making bias in itself favors media bias, because creating a narrative forces one to be selective of what to include and what to exclude. As long as there is this desire for simple narratives, coverage of complex topics cannot be entirely objective.

We may, however, mitigate this effect by upholding certain principles. I suggest four principles for the media to uphold in their coverage of topics.

• criticality
• balance
• objectivity
• independence

First, the media should have a critical stance to its object of reportage. Instead of accepting the piece of information they receive as truth, they should push to ask hard questions. But that should be done in a balanced way – for example, in a presidential race, both candidates should get an equal amount of “tough” questions.

Furthermore, journalists should not absorb any “truths”, beliefs or presumptions that affect in their treatment of a topic. Since every journalist is a human being, this requirement is quite an idealistic one; but the effect of personal preferences or those imposed by the social environment should in any case be mitigated.

The goal of objectivity should be cherished, even if the outcome is in conflict with one’s personal beliefs.

Finally, the media should be independent. Both in that it is not being dictated by any interest group, public or private, on what to report, but also in that it is not expressing or committing into a political affiliation. Much like church and state are kept separate according to Locke’s social contract as well as Jefferson’s constitutional ideas, the press and the state should be separated. This rule should apply to both publicly and privately funded media outlets.

Conclusion

The status of the media is precious. They have an enormous power over the opinions of the citizens. However, this is conditional power; should they lose objectivity, they’d also lose the influence, as people turn to alternative sources of information. I have presented that a major root cause of the problem is the media’s inability to detect its own bias. Through better detection and measurement of bias, corrective action can be taken.

But since those corrective actions are conditioned to willingness to be objective, a willingness many media outlets are not signalling, the measurement in itself is not adequate in solving the larger problem. At a larger scale, I have proposed there be a separation of media and politics, which prevents by law any media outlet to take a political side.

Such legislation is likely to increase objectivity and decrease the harmful polarization that the current partisan-based media environment constantly feeds into.

Overall, there should be some serious discussion on what the role of media in the society should be. In addition, attention to journalistic education and upholding of journalistic ethics should be paid. If the industry is not able to monitor itself, it is upon the society to introduce such regulation that the media will not abuse its power but remains objective. I have suggested the media and related stakeholders provide information on potential bias. I have also suggested new measures for bias that consider both the inclusion and exclusion of information.

The measurement of inclusion can be done by analyzing news stories for common keywords and themes. If the analyst has an a prior framework of topics/themes/stories he or she considers as reference, it can be then concluded how well the media covers those themes by classifying the material accordingly.

Such analysis would also reveal what is not being reported, an important distinction that is often not taken into account.