“Evaluators” have the REAL Power
(but who evaluates them?)
Who evaluates the evaluator?
As a homeland security professional, and efficiency expert, I am frequently called on to evaluate test results, programs, methods, or processes in order to determine their validity or value. The skill of “evaluation” is often misused as, or confused with “analysis”, when in reality “evaluation” by definition requires a much more focused mission than just breaking down a problem or process down into constituent parts. And though they are actually very different, it is fair to say that proper evaluation requires a good bit of analysis. The difficulty of understanding “evaluation” is partly due to the infinite number of definitions, but even more so to its infinite types of application. There are many commonalities among the different definitions that speak to the “focused” nature of evaluation, such as: informed, systematic, quantitative, objective, meticulous, and rigorous. These common elements indicate that an uninformed evaluator cannot conduct effective evaluations. As I come across more and more biased reports, contextually inaccurate “evaluations”, and expert-less opinion, it occurs to me that the position of “evaluator” (while not a high ranking position) is a position of great power.
This got me thinking….how do watchdog agencies such as the Government Accountability Office (GAO) evaluate all of these dissimilar programs without informed subject matter expertise, and who ensures that these are indeed unbiased, non-skewed evaluations? The evaluator must have intimate knowledge (or access to that knowledge) of the subject (process, idea, method, solution) being evaluated, and must have a clear understanding of the goals of the subject in order to determine subject effectiveness. So how valuable is an audit done by a human being? I guess it depends on the combination of capability and impartiality of the auditor.
Additionally, the perspective (or tasking) of the evaluator may also skew the evaluation, as different stakeholders producing evaluators may have different ideas of what is of value. For example, a company accountant may evaluate a process and determine it is very effective, but their evaluation is purely from a financial perspective; that is what has merit in their opinion. On the other hand, the quality control manager may evaluate the same process and determine that it is not effective, as it produces a low quality product, this opinion is based what he deems to have merit, the quality of the end product. Thus, it is of utmost importance that the evaluator defines and states what is of value prior to starting the evaluation.
This also got me thinking, how does the GAO conduct “independent evaluations” when they are issued a heavily biased request from say…a member of congress? In some cases it seems like the wording of the request, or the source of the request can predict the outcome of the audit. Is this conspiracy theory? GAO would not be so bold as to manipulate the system, would they? Well, it certainly seemed that way when a known TSA opponent in congress requested an audit of one of TSAs programs. Predictably this report was unfavorable to TSA, and used cherry picked research to imply the program was not valuable (but again, the value was never defined). Anyone heard of the IRS? They were recently accused of having a culture of corruption, targeting conservative groups for audits, and seeing themselves as above the law. What do you do when you are asked to evaluate a subject, but also given the outcome before you start? If you deny, you may lose your job, if you comply, you are part of the problem. So it begs the questions – who decides what the IRS audits, who audits them, the GAO, or (insert other government agency with arbitrary and easily abused power)??????
My conclusion….the checks and balances system is susceptible to the same corruption as the subjects they are investigating. Seems like the “system” is a house of cards, and we are all at the mercy of our “evaluator”. Is it just me or is corruption at an all-time high these days…
Luckily your Supreme Court justices are not normal human beings, and are not subject to the standard human biases, selfish urges, and temptations that force us to set up checks and balances. They are, after all, supreme, but that’s another post….
