Are You Capturing What You Are Really Trying to Measure?

You started with a model of what you wanted to measure. Then you planned to collect data based on the logistics of practice and game schedules throughout a season, or long training days and crucible events in a military course designed to assess and select candidates. With conflicting schedules and time constraints, this was no small feat. But have you considered if your plan will allow you to get at what you are really trying to measure?

Think about a situation where you are wanting to gather data from the peers of the person you are assessing. Peer data will be an added benefit by providing a different perspective on that candidate than coaches or instructors might offer. There may be certain events in the data collection plan where the peers will interact with each other much more so than other times. If there is a particularly difficult event, collecting peer data immediately after that event will elicit raw feelings about how that individual performed. However, if you wait, allowing those feelings to become less intense, you will likely get peer data which may be less emotion-based and more reflective. Deciding when to collect information from peers could yield different data. The choice is yours — which kind of peer data are you looking for?

Now imagine a scenario where you have administered a long day of physical events and students are separated into two groups. Because there is a staggered schedule after the physical events, one of the groups must attend class after and take a knowledge test, whereas the other group is done for the day and will be administered a test the following day. Do you think test scores are likely to be the same across groups? Or do you think one group may perform better?

The schedule of events can have a larger effect on the data than you might anticipate. It is questions such as these that must be answered when you are planning your data collection process and procedures. The missing piece here is strategy to your data collection plan. Once you have your plan in place, pause to think if you are lacking the strategy that will enhance what you are really trying to measure.