Lecture 12: Qualitative Data Analysis

Qualitative Data Analysis

This lecture for social science research methods at the University of Maine at Augusta presents resources on qualtitative data analysis to augment the Chapter 13 reading in your Dixon et al. textbook. Please be sure you’ve read Chapter 13 before you begin this lecture. The qualitative approach is quite different from the quantitative approach, and not just in the fact that quantitative approaches to research can involve numbers.  Qualitative research involves collecting and exploring data before any variables and perhaps even before any theories are identified, which means that the examination of data to identify important variables and generate theories from observed patterns is a much more intuitive and creative process.  Like the creative process of art, the creative process of qualitative research can often seem to escape any effort to be restricted to a single, predictable, step-by-step algorithm for research. Where an individual researcher’s interpretive leaps are an essential part of the research project, the assumptions and biases of the researcher become more important but more difficult to identify and address than the assumptions and biases of a quantitative computational procedure.

Although this is the last full lecture of the semester, it isn’t quite the end of the semester itself.  This week, it’ll be your turn via a DIY Activity to engage in work with some qualitative data considering the website of one of the world’s largest retailers as a source of text. Next week, we’ll complete on our group research project having to do with campaign signs. Now that the class has identified its most popular hypotheses, we’ll run regressions to assess those hypotheses. Can you interpret results of those quantitative regressions correctly?  Finally, you should be working on your research proposal, and sending any questions you may have to me (e-mail james.m.cook@maine.edu) during the process.

Checking Conclusions: Evaluating the Validity of Qualitative Findings

In your textbook, Dixon et al. describe many technical means by which notes can be transcribed into longer forms and themes can be described with organizational forms such as flow charts, taxonomies and data matrices.  A question apart from techniques of organization, however, is the evaluation of findings.  One of the strengths of quantitative research is the long project of developing robust indications of the validity, reliability, and generalizability of quantitative data itself and the patterns found in quantitative data.  So long as assumptions regarding measurements are valid, quantitative researchers are able to describe in detail the extent to which data is biased in one manner or another.  In contrast, it is fair to say that qualitative research is still in the early days of developing such indicators.

Part of the difficulty is philosophical: some qualitative researchers, members of the “antirealist” camp, resist any effort to develop measures of the soundness of conclusions because they do not agree that there is an external social reality that exists outside of the social perceptions of individuals (Mays and Pope 2000).  Rather, anti-realists argue, multiple perspectives regarding the nature of social reality may co-exist with an equal claim to respect, even if the tenets of those perspectives are mutually inconsistent.  In its strongest form, philosophical anti-realism rejects outright the basic idea of quantitative research that there exist any claims at all that can be definitively shown to be true or not: all that exists, according to this position, are a set of different stories.  Taken to such absolutist extremes, qualitative work regarding society ceases to become social science research, since the essence of the idea of “social science” is that knowledge is possible.  Instead, such inquiry becomes part of the humanities, in which concern for truth is secondary and concern for the building of a coherent, convincing narrative story becomes paramount.

On the other hand, a more moderate form of this argument, referred to as “subtle realism” by Hammersley (1992), notes that (as quantitative researchers would agree) any truth claim about the nature of a possibly objective world involves a number of assumptions about the nature of the social world and about how one would go about measuring the social world.  It is impossible, according to this position, for the researcher to completely remove herself from the process and therefore the product of research.  If the researcher is an important subject in the sense of a being who does and observes, research is always subjective, and so subjectivity cannot be avoided.  The important thing about making truth claims, according to this position, is to take subjectivity into account as fully as possible.  This means, on the one hand, that the researcher should be as transparent as possible about her prior stances, assumptions, and biases.  On the other hand, it means that other subjects in the world should be consulted to check any truth assertions about a situation.  If a number of different observers from a number of different perspectives note the emergence of the same themes, it does not necessarily mean that these themes must reflect objective reality.  However, at the very least they indicate the emergence of an intersubjectivity — a consensus about accounts regarding reality — that is worthy of note (Ritchie et al. 2003).  The remainder of this lecture will elaborate steps that a qualitative researcher can take in pursuit of higher-quality work.

 

Reflexivity: Acknowledging Paradigms, History and Assumptions

One of the core observations social scientists make about human beings is that they suffer from confirmation bias — the tendency to see the world in ways that are compatible with their prior worldview.  We have a tendency to defend ideas that we cherish and dismiss observations that threaten them; we also tend to see what we expect to see. Because social scientists are human beings, we suffer from conformation bias too, no matter how much we might protest that we are objective, highly-trained and therefore beyond such errors.  An honest social scientist should acknowledge the prior assumptions that she or he brings to the research enterprise rather than dismissing them as irrelevant.

One variety of prior assumptions involves personal history and identity.  Regional, religious, political, and other sources of distinct culture in our upbringing may affect the way we see and judge qualitative research subjects who are like ourselves or who are from another culture.  Even if we attempt to remain objective, our ability to understand individuals who are different from us may be more difficult because we do not possess cultural fluency, and our ability to understand individuals who share our culture may suffer from an inability to see the latent assumptions that make our culture possible.  These challenges should be noted.

Another variety of prior assumptions involves the very essence of the research process: the subscription to paradigms and the development of theories.  No matter how much a qualitative researcher may intent to enter a situation without any pre-existing theories or paradigmatic points of view in mind, there really is no purely inductive research because prior ideas always slip in.  At a minimum, the very decision to study a particular subject is the result of a decision that this subject is “important.”  Beyond that, our educational history leads us to be exposed to a series of paradigms and theories (yes, I’ve infected you) that reflect a particular worldview and lead us to make assumptions when generating research questions.  This influence is inescapable; it should be acknowledged rather than ignored, and the literature review of a research paper is a great place for such an acknowledgement.

 

Striving for Intersubjectivity

Although qualitative research is not objective, it is possible for the intuitions, inspirations and organization of raw observation into patterns to be checked for credibility.  Here are a few ways that qualitative researchers bolster the credibility of their work, as mentioned in qualitative research guides by Creswell and Miller (2000) and Mays and Pope (2000).

Triangulate.  Triangulation is the process by which multiple methods are used to try to observe the same phenomenon in different ways.  Your text acknowledges the triangulation work in Snow and Anderson’s study of the homeless in Austin, Texas, when Leon Anderson engaged in direct observation of homeless individuals while David Snow searched through records and interviewed local authorities (Snow and Anderson 1993).

Search for the Negative Case. Once qualitative researchers have developed an account or story to explain an observed social phenomenon, it is useful for those researchers to pivot and look for the “negative case” — that is, evidence in their own observations that would violate the trend.  This is difficult for researchers to accomplish because of confirmation bias — we tend to be attached emotionally, even subconsciously, to the stories we have developed. For that reason, it is often a good idea to bring an independent researcher into the team to look over notes and find patterns that contradict the burgeoning story of the research.

Member Checking.  Does the story written by a qualitative researcher accurately reflect life as lived by the people she or he has studied?  Some say there’s no better way to find out than to share results with the very same people who have been studied.  There’s an open, democratic nature to research in which those being studied have a light “veto” power to tell researchers they’ve gotten it wrong.  A weakness of this approach, however, is that it assumes that people fully understand not only the nature of their own lives but the nature of the social circumstance in which they’ve been living. It may be the case that researchers coming in from the outside may be able to see patterns that those being studied just don’t see for themselves. Nevertheless, it is useful to get a “gut check” from those being studied in order to provoke reflection.

Try it Yourself! Put a Target on Target

For this DIY Activity, I would like you to engage in qualitative analysis (using the two weeks’ worth of focus we’ve placed on qualitative research and qualitative data analysis) to address the question, “how are toys marketed on Target’s website?”  As reported last this year, the department store Target has decided to stop formally labeling its toys as “for boys” or “for girls” in both its brick-and-mortar stores and its web-based store, but what are the remaining patterns in the way that Target markets toys on its webpage?  To answer this question, first develop a method for observing toys at Target.com and gathering qualitative data related to the marketing of toys in Target’s toy aisles.  Then actually go to Target.com — the web address for the toys section is http://www.target.com/c/toys/-/N-5xtb0 — and make qualitative observations of marketing for a number of Target web pages for toys, using a method described in your course textbook.  Finally, organize your observations into a set of findings using either the taxonomy or typology format. Report your findings and your method for obtaining those findings in a 3-4 page report that you upload to using the Blackboard DIY Activities section labeled “DIY Activity #10: A Target on Target”.  This DIY Activity is due Saturday, December 10.

References

Creswell, John W. and Dana L. Miller. 2000. “Determining Validity in Qualitative Inquiry.” Theory into Practice 39(3): 124-130.

Hammersley, Martyn. 1992. What’s Wrong with Ethnography? London: Routledge.

Mays, Nicholas, and Catherine Pope. 2000. “Assessing Quality in Qualitative Research.” British Medical Hournal 320(7226): 50-52.

Nickerson, Raymond. 1998. “Confirmation Bias: A Ubiquitous Phenomenon in Many Guises.” Review of General Psychology 2(2): 175-220.

Ritchie, Jane, Jane Lewis, Carol McNaughton Nicholls, and Rachel Ormston. 2003. Qualitative Research Practice: A Guide for Social Science Students and Researchers. Sage.

Snow, David and Leon Anderson. 1993.  Down on their Luck: A Study of Homeless Street People. Berkeley: University of California Press.