Great questions Jesse. While I don’t presume to have the complete answers, here are my initial thoughts:
I think a good learning measurement, like any good measurement, reliably and validly measures whatever learning construct you want to measure. In most educational settings I would suggest this involves, at a minimum, measuring the long-term, post-instruction, retention and transfer of knowledge and skills. The actual measurement instrument, evaluative context, and timing will necessarily vary depending on the construct chosen. I think carefully defining and validating one’s dependent learning measures is one of the most challenging and, as I mention in my piece, neglected tasks in ed research.
You make a great point about balancing evidence-based practices and student motivation. It seems the goal should be to try to maximize student motivation to engage in the desired cognitive processing that engenders learning while avoiding adding extraneous seductive elements that can distract and interfere with student efforts to construct knowledge. How to do this successfully is an empirical question that I believe requires a lot more research to answer, along with better (i.e., non-self report) methods for measuring student engagement/cognitive load in a learning experience.
I’d love to hear your thoughts on these issues — especially if you have a different perspective!