Lecture 5: Measurement

Welcome — Observe the World!

Welcome to the fifth online lecture for SSC 320, the Research Methods in Social Science course at the University of Maine at Augusta. This week, we’re going to move beyond abstract ideas of research and start to observe the world around us.  To do that, we’ll need to think about how observations can help build theory through induction or be used to test hypotheses through deduction.  Those may be dry words, but the striving toward understanding that lies behind the research process is anything but dry.  It is a process of eager anticipation, willing frustration, sacrifice of the obvious.  It is a commitment to let go of what we think we know to know better.  Although T.S. Eliot is a literary rather than an empirical talent, his poem East Coker contains a passage that I think captures the giddy, disorienting and sometimes disturbing turbulence of the journey that is research:

In order to arrive there,
To arrive where you are, to get from where you are not,
You must go by a way wherein there is no ecstasy.
In order to arrive at what you do not know
You must go by a way which is the way of ignorance.
In order to possess what you do not possess
You must go by the way of dispossession.
In order to arrive at what you are not
You must go through the way in which you are not.
And what you do not know is the only thing you know
And what you own is what you do not own
And where you are is where you are not.

Are you ready to take this path?

Please be sure to participate fully, reading and thinking about each element of the lecture you see below.  At the end of the lecture is a comments section; I encourage you to post any questions you might have there.  Keep in mind that to protect your privacy, I don’t require you to sign your name to any comment or lecture response on this public page — use a pseudonym instead.

How’s the Course Going? Let Me Know

We’re not quite at mid-semester, but we’re getting there.  As we settle in to the course, I’d like to know how it’s going for you.  The following is a survey with anonymous responses — I won’t know who’s said what, so feel free to express yourself with honesty! I look forward to reading your thoughts on the research methods course.

Nominal, Ordinal, Interval, Ratio: Untangling the Four Levels of Measurement

In previous semesters, one of the subject areas in which students seemed to be most likely to get stuck was the four levels of measurement: nominal, ordinal, interval and ratio.  In order to help clarify the difference between these four levels, I’ve taken to the woods in order to find real, physical examples of the differences in measurement.  Watch this video if you feel the need for clarity:

Precision, Accuracy, Reliability and Validity

In their textbook, Jeffrey Dixon and his co-authors spend a good amount of time discussing reliability and validity — and rightfully so, since they are such important qualities to have when developing variables’ operational definitions.  I’d like to add two other indicators of high-quality measurement: precision and accuracy.  The precision of a measurement is the level of detail gathered by that measurement.  The accuracy of a measurement, on the other hand, is the extent the measurement obtained is true.  Consider the variable of age.  We could measure age by saying Edna Crabapple is “old.”  That may accurate, but it isn’t very precise.  To become more precise, we could say she is 74 years old, but we could measure age even more precisely than that.  How about 892 weeks?  How about 892 weeks, 3 days and 1 hour?  That’s a very precise measurement, and in general the more precise a measurement of a variable we can make, the better.  There are times, however, when high precision leads to lower accuracy.  If we say Edna Crabapple is “old,” that will continue to accurately describe her until the day she dies.  If we say she is 74 years old, that measurement will be accurate for a year.  If we very precisely declare Edna Crabapple to be 892 weeks, 3 days and 1 hour old, on the other hand, that measurement won’t remain accurate for long!

Precision is not the same as accuracy.  Accuracy is not the same as reliability.  And neither precision nor reliability matters if what we’re measuring is not a valid indicator of the concept we’re trying to explain.  In the following video I use the example of geographic location to make the different considerations of precision, accuracy, reliability and validity a bit more clear:

Overgeneralization and Selective Observation

Two problems in research related to reliability and validity are overgeneralization and selective observation.

Overgeneralization is the problem of making a conclusion based on only a small number of observations.  For an example of overgeneralization, consider the job website Glassdoor.  Glassdoor shares reports about the “average” pay earned in an occupation, but I really have to put the word “average” in quotes in this sentence, because the “average” values are often based on the reports of very few people.  Take, for instance, Glassdoor’s reported “average” compensation for catastrophe adjusters, specialists who visit scenes of devastation and help people make insurance claims:

Catastrophe Adjusters' Salaries, averaging from very low numbers

Or how about the “average” wage of an embalmer?  Glassdoor reports an “average” of $16.74 for embalmers who work at SCI Funeral and Cemetary, based on just two workers there:

Embalmers' average salary? Don't rely on Glassdoor's overgeneralized average based on just two observations

These overgeneralizing reports are not reliable because there tends to be a lot of variation in how people in any one job are paid, based on length of experience in the current job, previous experience, education, training, workplace effectiveness and so on.  The chances that just two, or three, or even eight people will offer an accurate glimpse of the typical experience of a job are not very good.  We’ll cover the problem of overgeneralization in greater detail in our course module on sampling.

Selective observation, on the other hand, is the problem of making a conclusion based on only paying attention to certain observations and ignoring other observations. For an example of selective observation, read Iggyman, who in 2004 posted the following message to an online bulletin board system:

“i never fail to be amazed at the stupidity of americans….any of you watch street smarts or jaywalking on jay leno? AHHHH!”

Comedian and Tonight Show host Jay Leno got lots of laughs with his regular “Jaywalking” feature in which he could be seen on the streets asking people simple questions and getting hilariousy wrong answers:

But on another bulletin board, Brighteyes asked the right question:

“But did you ever stop to think that the Jay Leno skits may be orchestrated?”

The answer to that question comes from a third bulletin board discussion of the Jaywalking segments, where Cervaise points out:

“Yeah, some people really are that stupid. And yeah, the editing of that segment is self-selecting; if he asks an obvious question to fifty people, and forty-eight get it right, the two dumb guys with funny responses are picked for the show.”

Jay Leno’s Tonight Show was an entertainment show, not a documentary.  What Leno showed in his Jaywalking segments were the most entertaining answers to his questions, not the most representative answers.  Because of Leno’s approach, his results are highly reliable (always laughably ignorant) but probably not very high in validity (since Leno rejected reasonable people from being shot for his video).

If you’re looking for a more valid and reliable indication of the civic awareness of Americans, try the Pew News IQ Quiz.  The Pew News IQ Quiz is an online news quiz that you can take to see how knowledgeable you are about current events.  When you’re done taking the quiz you can see the pattern of results from a random sample of thousands of U.S. adults and compare yourself to that random sample.  The Pew Research Center follows a rigorous random sampling method to get around the problem of selectivity in perception.

 

Get Ready for Main Street Sociology (DIY Activity #3)…

It’s time to apply these ideas in practice! For your DIY Activity #3, I would like you to select a town with a “Main Street” (the main commercial street in a town) that has at least twenty visible storefronts.  You may either visit this town personally or use Google Street View to observe the storefronts virtually (see this web page for information on how to use Google Street View with Google Maps).  In a word processing document of at least 3 pages, complete the following:

1.  Identify at least three variables that can be used to describe storefronts.

2.  Fully describe and justify a conceptual definition and an operational definition of each of these three variables.

3.  Indicate the level of measurement, precision, accuracy, reliability and validity of the operational definition of each of these three variables.

4.  For no more than forty storefronts but no fewer than twenty storefronts, make an observation of each of the three variables.  Write down your observations in a data set.  Your dataset should have the following tabular format in which variables are in columns and individually observed cases are in rows:

Example dataset observing kittens:

Kitten ID Tail Length Fur Color Number of Toys in Mouth
Kitten 1 10 cm gray 0
Kitten 2 8 cm tabby 1
Kitten 3 9 cm orange 1
Kitten 4 8 cm white 0

5.  Include a few paragraphs in which you name and describe the town you’ve observed and reflect on the experience of crafting a plan, defining variables, and gathering a dataset.  Did all go as planned?  If not, how not?  Would you conduct the research in the same way again?  Tell me a good story of the research process to help me understand how the experience went and what you learned.

6.  Upload your work as a word-processing document by following the “DIY Activity #3: Main Street Sociology” link via the “DIY Activities” area on our course Blackboard page.

This work is due by October 1.

 

… And Get Ready for Our Group Research Project Next Week

As the air gets crisp, as the airwaves get hot, something’s appearing by the side of the roads.  As the syllabus says, we’re going to be engaged in a group research project in this class (one for which your participation will be graded). I can promise you it’ll be timely!  We’re finishing up the skills needed to get started this week; we’ll talk in greateer detail about our group research project it next week.

Remember, if you have any questions, please send them my way, by leaving a comment below (feel free to use a pseudonym for your anonymity), by phone (207-621-3190), by e-mail (james.m.cook@maine.edu), or in office hours (see the top of the syllabus).

Assistant Professor James CookBest,

James Cook

Assistant Professor of Social Science

University of Maine at Augusta

5 thoughts to “Lecture 5: Measurement”

  1. Jay Leno’s did an awesome but I hope if someone came up to me and asked the same questions I would be able to at least get milky way correct. lol

  2. Wow… this week’s problem set was PAINFUL! I’m embarrassed by the number of attempts I made. I feel like I’ve tried all of the answers for question 10 multiple times without successfully getting it correct. Lesson learned for next time… write down what I submitted for answers originally when they feel like a guess. Can you elaborate on questions 6, 8 and 10? Even though I stumbled across some answers by MANY multiple attempts, I still don’t understand why. This week’s reading was tough for me, too! Feels like a lot of terminology that I’d have to memorize before I could understand.

    Feeling very nervous about the upcoming test and other assignments.

    1. Hi, V. Sometimes there is a lot of terminology to get to know to really understand a subject well. Let’s look at the meat of questions 6, 8 and 10.

      6. Number of hours spent in an activity during a day is a ratio measure because it has a meaningful zero point, because the distances between the numbered responses are equivalent in size, and because a ratio of two measures of hours spent in an activity are meaningful. If Jack spends 2 hours petting a dog and Jane spends 4 hours petting a dog, 2/4 = 1/2. We can say with meaning that Jack spent half as much time petting a dog as Jane did.

      8. Asking the same question of the same research subject at two points during a research project is called test & retest. If subsequent questioning changes the answer, then the question is NOT test-retest reliable.

      10. If two observers record the same response, then we have an indication of reliability. They could both be very, very wrong, so we don’t know that the responses recorded are valid just because they’re the same.

      Best,
      Prof. Cook

Leave a Reply

Your email address will not be published. Required fields are marked *