Lecture Capture: dubious scholarship and market forces
Number 62: #USSbriefs62
Listen to this brief on SoundCloud (in production, available soon)
In the past few years, the UK Higher Education (UKHE) sector has implemented Lecture Capture (henceforth LC) as a standard e-technology across academic disciplines. Targeted rollouts or full-scale adoptions of LC have happened at Oxford, Cambridge, and most Russell Group institutions. One recent review claims that nearly three-quarters of UK higher education institutions reported using the technology as of 2016, and some universities have proposed default use of LC for all teaching formats, including — astonishingly — discussion seminars.
LC’s critics warn of its damaging pedagogical side-effects — notably, the encouragement of rote learning and lecture absenteeism — and its facilitation of institutional skulduggery during industrial disputes. The USS strike of February-March 2018 exposed such underhanded behaviour on several campuses (for example, Edinburgh) where university administrators encouraged both department heads and students to use captured lectures from previous years to replace struck teaching.
The technology’s advocates, meanwhile, point to the way that LC ostensibly expands classroom accessibility while providing an additional study aid for students. They dismiss worries about absenteeism — to which reality almost any lecturer who has used LC can attest — by claiming that the research literature on LC shows those worries to be baseless.
Resistance to LC on pedagogical or professional grounds often meets with insistence that LC is urgently necessary to provide equal access. That urgency, however, is in part the consequence of recent austerity policy’s damage to alternative, less invasive equal-access solutions. The UK government — when Jo Johnson was the Minister for Universities and Science — slashed the Disabled Students Allowance (DSA), which disabled students had previously used to employ note-takers and study assistants. In this context, LC has become a ‘quick-fix’ for many university administrators.
It often goes unacknowledged amid this debate that introducing LC requires a significant devotion of human and financial resources. At a minimum, it involves evaluating, purchasing, and installing LC hardware and software; integrating new tasks within teaching staff’s established pedagogical practice; and charging IT staff with oversight. Across UKHE, this adds up to thousands of hours and millions of pounds in start-up costs alone.
One would hope and expect that such resource-intensive initiatives — controversial as they are in pedagogical and institutional terms — would have a firm basis in robust research demonstrating the e-technology’s unequivocal benefits to student learning.
This, however, is not the case.
The most influential educational research literature on LC does not demonstrate any real enhancements to learning. In fact, it generally fails to pose questions regarding the technology’s potential relationship to common institutional-level learning outcomes such as critical thinking, independent research skills, or effective collaborative work. Neither does it allay the common concern over LC’s adverse effects on attendance, despite the fact that one of its most emphatically repeated and popularised ‘findings’ is that faculty concerns over such effects are without foundation.
Instead, the dominant literature bases its positive recommendations about the value of LC’s adoption on pervasive misinterpretation and misrepresentation or misuse of previous studies (or, in at least one case examined below, of the authors’ own data). In many cases, those underlying studies make claims that their method and evidence do not support.
A preponderance of the research encouraging LC’s adoption also seems to have been published in journals devoted specifically to the promotion of e-technology tools. This raises serious questions regarding possible conflicts of interest in the editorial or peer-review processes that have produced the works underlying the technology’s rise.
Two recent reviews of the literature on LC, examined below, expose and reflect a variety of deep methodological and analytical problems in the scholarship more broadly. These problems — alongside the dubious conditions of the scholarship’s production — lead to a basic conclusion: LC has been sold on false premises by interested parties despite evidence, on balance, that indicates damage to educational outcomes.
The enthusiasm for LC’s adoption reflects a research literature that skews superficially positive when evaluating the technology’s benefits. A 2015 study by Kinash et al. claiming to offer the most comprehensive review of that literature to that date issues a clear recommendation: ‘the benefits of online lectures outweigh the disadvantages’.
Another frequently cited study, a 2013 review of the research literature by Arun Karnad at the London School of Economics, offers positive bullet-points regarding LC’s effects on student achievement in its Executive Summary, while downplaying the credibility of evidence that LC negatively affects student attendance.
Above all, what nearly every study shows — and emphasises — is how much students like LC. ‘There is strong evidence’, as one study puts it in its introduction, ‘that students place significant value on lecture recordings deployed via the Internet’. Such surveys of the existing research literature give the impression that LC can only help, not hurt student learning.
In fact, neither existing surveys of the literature nor the literature itself substantiates claims that adopting LC or webcasting improves overall student performance. At best, those that attempt to claim a positive effect on student achievement do so in very narrow terms. At worst, they display methodological flaws or analytical contradiction so apparent as to render their conclusions useless, if not actively deceptive.
Arun Karnad’s review article, which includes a bibliography of thirty-three studies relating to the topic, is typical of the literature it surveys: its own analysis contradicts or undermines its more prominent ‘Executive Summary’ and conclusions. Addressing the question of achievement in the conclusion, Karnad writes, ‘The perception that access to recorded lectures lowered student attainment has…been disputed by most of the papers considered in this review, with lecture recordings having a slightly positive, or negligible effect on student attainment, and even a rise in student grades and lecture attendance in some cases’. (It is worth noting that despite his reference to ‘most of the papers’, Karnad only cites one paper, David S. Franklin et al.’s ‘Use of lecture recordings in medical education’, which claims in its abstract that ‘lecture recordings did not have an impact, either in a positive or negative direction on exam performance across seven first- and second-year basic science courses’.) This leaves readers with the impression that the bulk of the literature he has reviewed claims net positive effects of LC on student achievement.
Yet, of the six studies Karnad considers in the section of his review devoted to evaluating LC’s effects on achievement, he only claims that one shows a positive correlation. Another returns ambiguous results; a second shows no statistically significant difference; two show correlation between more LC viewing and lower achievement; and the last he dismisses as methodologically irrelevant to his question.
And what of the one study that Karnad cites to claim a positive relationship between webcasted lecture recordings and student achievement? Karnad cites this study — by Tomoko Traphagan, John V. Kucsera and Kyoko Kishi (henceforth, Traphagan), writing in Education Technology Research and Development — five times in his ‘Executive Summary’ to buttress his position extolling the benefits and minimising the dangers of LC. Indeed, the Traphagan paper is one of the most frequently cited and influential in the field.
The paper itself, however, will not bear the weight of Karnad’s — or anyone else’s — claims about the value of LC. Addressing the question, ‘How does webcasting affect students’ performance?’ Traphagan begins by acknowledging, ‘Mean scores for tests and quizzes were consistently higher for students in the no-webcast section than for those in the webcast section’. In other words, it states from the outset that students in the study who lacked LC achieved greater success with the same curriculum than those who had it.
So how did Karnad get the impression that Traphagan supports the opposite conclusion?
The answer is that Traphagan obscures its observed evidence by filtering it through a mathematical model of its own design, allegedly to account in part for differences in student quality and motivation, or ‘covariants’. Taking GPA and student absence into consideration in its ‘model’, Traphagan claims, indicates that ‘students in the webcast and no-webcast sections performed similarly’.
What does this mean in plain English? It means that according to Traphagan, students in the no-webcast section began the term with higher GPAs and greater levels of motivation than those in the webcast section. Therefore, their higher achievement in the course probably has nothing to do with the webcasted lectures.
This would seem to raise basic questions about the study’s ability to trace cause and effect, indicating crippling flaws in the relationship between its research inquiry and its methods. Nevertheless, the authors forge ahead, writing, ‘we conducted path analyses with the webcast section data using our predicted path model. The results suggest that webcast viewing has positive and significant effects’.
Again, what does this mean in layman’s terms? It appears to mean that although students in the webcast section may have underperformed students in the no-webcast section in real life, they did better than the authors’ model had predicted they would perform on the basis of their assumptions. Therefore, according to the authors, viewing of lectures on the web provides ‘positive and significant effects’.
To the authors’ dubious credit, long after making such bold assertions on such thin evidence, Traphagan includes a section entitled, ‘Limitations and future research’, in which they write:
First, the generalizability of this study is limited because it was conducted at a single institution and with only two course sections…Second, our research was a quasi-experimental design rather than a true experiment with randomization…Third, some of the data collected were self-report [sic], such as absence behavior. In addition, the webcasting access log could not determine the duration of students’ webcast views, only access frequency.
In other words, the authors themselves — after deploying a superficially impressive array of mathematical symbols, figures, and regression analyses to claim ‘positive and significant effects’ of webcasting lectures — acknowledge that their study has effectively zero representational value, does not qualify as properly empirical, and has gaping methodological holes.
According to Google Scholar, the Traphagan article has been cited nearly three hundred times. As noted above, Karnad of the London School of Economics cites it five times in the ‘Executive Summary’ of his literature review. A basic initial sampling of other citations suggests that, as Karnad does, other authors consistently use Traphagan’s ‘study’ to emphasise or substantiate the alleged positive effects of LC on student achievement, and its lack of effect on attendance. Not one of these citations mentions Traphagan’s disavowal of the study’s generalisability!
The Karnad survey bears the imprimatur of the London School of Economics and has been correspondingly influential (cited by staff and management at UK universities such as Sheffield, Oxford, Sussex, Edinburgh, Loughborough, Hertfordshire). It was not, however, actually peer-reviewed (though most of its sources, including the Traphagan paper, allegedly were). Indeed, Karnad himself is not a professional educator or researcher of education, but a Clinical Trial Administrator at the University of Oxford who has an MSc in Biomedicine, Bioscience and Society from the LSE, and was a Research and Evaluation Assistant in the LSE’s Centre for Learning Technology at the time he wrote his literature review.
What of the other major survey of LC literature mentioned above, from 2015, which claims so definitively that ‘the benefits of online lectures outweigh the disadvantages’?
This study, by Shelley Kinash, Diana Knight, and Matthew McLean (henceforth, Kinash), did appear in a peer-reviewed journal, and purports to conduct a synthetic analysis of nineteen studies published between 2006 and 2013 that address their article’s title question, ‘Does scholarship through online lectures affect student learning?’ The authors contextualise those studies with over forty further texts in an extensive bibliography. The centrepiece of the article is an impressive-looking table, ‘Table 2’, which categorises the results of the nineteen studies under closest scrutiny. The authors present the table, explain its contents, and conclude, ‘The weight of the evidence is that providing lectures online does not decrease student on-campus attendance and that it increases achievement’.
These two conclusions seem as definitive as can be. Yet even a cursory glance at the table — much less a perusal of the authors’ attempts to explain it, or a review of the underlying literature — renders them immediately problematic.
The authors acknowledge that nearly half of the studies or papers on which they base these conclusions use methods that would disqualify them as ‘empirical’ or ‘experimental’. Putting aside the question of why the authors made the choice to include studies they themselves deem non-empirical as equally weighted evidence for their analyses, the attentive reader will notice that nine of nineteen studies in their table do not even address LC or its effects!
In other words, Kinash offers answers to a question about LC’s effects on student learning (as part of a slightly broader question about ‘online lectures’, which might mean the provision of lecture texts, for example) based on a synthesis of research, half of which does not explicitly relate to LC.
Of the six studies in ‘Table 2’ that the authors rate as ‘empirical’ or ‘experimental’ and that do specifically address the effects of LC, the authors claim that two of five find LC reduces on-campus attendance (the sixth didn’t measure it), and half find that it has no effect on ‘achievement’. These results appear inconclusive, not positive. There is a prima facie contradiction between the authors’ evidence and their conclusion.
When one begins to read Kinash’s exposition of the table’s data against the articles that this data ostensibly represents, however, their conclusive interpretations and recommendations begin to appear not just unwarranted, but bizarre.
Reviewing a paper by Grabe and Christopherson (which Kinash adjudges empirical but which does not address LC) from the Journal of Computer Assisted Learning, for instance, Kinash asserts, ‘Notably, there was a positive relationship between student attendance and use of digital resources. Further, this positive relationship extended to achievement as demonstrated through exam performance’. Again, the authors are stating unambiguously that scholarship they rate as empirical establishes the benefits of digital resources.
But this characterisation of Grabe and Christopherson is doubly false. First, Grabe and Christopherson offer no empirical basis for any general conclusion. Their study was limited to a single Introductory Psychology module of 329 students over five weeks. The authors assert in the article’s abstract, ‘Because students in this study were not asked to explain their use of these [digital] resources, the present findings are regarded as speculative’. Such an admission renders any assertions that their study is empirical, or that it documents a ‘positive relationship…to achievement’ in the use of ‘digital resources’, untenable.
Second, the module under study by Grabe and Christopherson had an in-built incentive for student attendance that was unrelated to the availability of digital resources: unannounced, in-class writing assignments whose completion and marks constituted 5% of the students’ final mark for the module. ‘Admittedly’, Grabe and Christopherson concede, ‘awareness of the writing assignment may have influenced lecture attendance’. This concession to the absence of empirical control vacates any conclusion — either in the original article, or its representation in Kinash — of a ‘positive relationship’ between the use of digital resources and student attendance.
In at least one case, one need not return to the original study to find serious misunderstanding in Kinash’s characterisation of prior research. In this same expository passage, the authors assert, ‘Williams, Birch and Hancock (2012) provided empirical evidence that some students use captured lectures as a replacement for on-campus attendance and others as a supplement or revision tool, thereby attendance is indicated in Table 2 as neutral’.
Readers must not be blamed if they feel the urge to read that sentence again. If Williams et al., writing in the Australasian Journal of Education Technology, show that ‘some students used captured lectures as a replacement for on-campus attendance’, how can Kinash’s registering the effect on attendance as ‘neutral’ make sense?
The more reasonable conclusion would be that attendance diminished from the levels it would have reached without LC. This means that 60% — not 40% as represented in Table 2 — of the studies Kinash considers as both ‘empirical’ and addressing LC record a net negative effect of the technology on in-person lecture attendance.
Returning to the original article only confirms this suspicion: Williams et al. assert explicitly, ‘[M]any students appear to be using the online recordings often as a substitute for lectures’. What part of this appears ‘neutral’ to Kinash et al.?
And what of Kinash’s claim that Williams et al. offer evidence of LC’s improvement to student achievement? One need read no further than Williams et al.’s abstract to find this: ‘The main finding is that students using the online lectures as a substitute for attending lectures are ultimately at a fairly severe disadvantage in terms of their final marks’.
If the article shows that ‘many students’ in its study were using captured lectures as a replacement for in-person lectures, and the ‘main finding’ is that such students are ‘at a fairly severe disadvantage in terms of their final marks’, how can Kinash justify categorising it as an empirical study demonstrating the opposite?
Kinash appears to ignore what Williams et al. describes as its ‘main finding’. It selectively focuses instead on Williams et al.’s subordinate assertion, that ‘students who attend the majority of lectures in person do receive a benefit from additional use of the lecture recordings’. This point — which Williams et al. clearly views as less significant than its ‘main finding’ — does not demonstrate, or even claim, that LC has positive overall effects on student achievement. It claims that for students who would likely have been high-achieving anyway based on attendance patterns, providing captured lectures can offer ‘a benefit’ — but only while simultaneously encouraging behaviour by ‘many students’ that puts them at ‘a fairly severe disadvantage’.
What serious educator would make this trade-off? What serious scholar of pedagogical practice would fail to recognise it? What fair-minded interpreter of the English language would suggest that the combination of these findings could be understood as an endorsement of LC’s overall positive effects on student achievement?
One might begin to wonder whether Kinash has correctly categorised any of the studies in ‘Table 2’ by empiricism or experimentalism. Investigation of the articles it cites relating directly to LC offers no reassurance. One of the studies marked as ‘empirical’ — and one of only three claiming LC’s positive effects on student achievement — is Traphagan! As noted above, Traphagan derives its claims of LC’s positive effects on achievement from a speculative methodology, and the authors themselves disavow the empiricism of their approach.
Another of the three studies that Kinash evaluates as empirically demonstrating LC’s positive effects on student achievement is entirely based on student self-reporting. That article, by Hasan al Nashash and Cindy Gunn in the Journal of Education Technology & Society, also cites Traphagan in its introduction, ostensibly to buttress claims that LC has no effect on student attendance.
But Traphagan, whatever its other sins, asserts the opposite. Among its bullet-pointed conclusions, Traphagan states, ‘students with webcast access attended class less frequently than students without webcast access’, and ‘students who viewed webcasts more frequently had more absences’. How can Al Nashash and Gunn cite this as evidence for LC’s lack of effect on attendance? Forget classifying their study as ‘empirical’; they appear not to have read the literature they cite to buttress one of their central conclusions!
The only remaining ‘empirical’ study in Kinash’s table supporting the conclusion that LC has a positive effect on student achievement is Williams et al.’s 2012 study from the Australasian Journal of Education Technology. As the discussion of Williams et al. above makes clear, that study shows nothing of the kind.
In other words, not a single study that Kinash rates as both empirical and addressing LC actually demonstrates a positive effect on student achievement. Astonishingly, Kinash’s authors remain comfortable referring to ‘an overall positive relationship between digital scholarship and achievement’, and frame their recommendations regarding LC accordingly.
This must be the sort of thing that philosophers and Hindu mystics have in mind when they invoke the phrase, ‘turtles all the way down’.
Market influence over the literature?
How can we explain this phenomenon of misrepresentation and positive spin for LC, pervading both the literature and previous critical literature reviews, without making accusations of incompetence or fraud?
One way might be by pointing out the editorial incentives for positive spin when writing for journals or employers who have a vested interest in promoting the supposed benefits of e-technology. One article cited on Kinash’s first page (Yuan & Powell 2013) as evidence of the ‘emerging innovations in educational technology… fostering efficient, effective and often interactive online learning environments’ was published by Cetis LLP, ‘an independent strategic IT consultancy’. Moreover, Cetis claims:
[We] have been instrumental in developing and promoting the adoption of technology and standards for course advertising, open education resources, assessment, and student data management, opening new markets and creating opportunities for innovation.
This is hardly an impartial, scholarly source when it comes to evaluating the possible adverse effects of e-technology in the classroom.
A similar case is Educause Quarterly, a frequently cited journal in the research literature on LC. EQ is published by Educause, ‘a nonprofit association and the foremost community of IT leaders and professionals committed to advancing higher education’, according to its own website. Is it possible that ‘IT leaders and professionals’ might be committed to advancing something other than higher education? Such as, for instance, the adoption of e-technology systems that require purchase from IT leaders and servicing by IT professionals?
The Journal of Education Technology & Society, a formerly prolific publisher of peer-reviewed and oft-cited studies in the field, announced in December 2016 that it was no longer accepting submissions because it ‘accomplished what it aimed to do’. Over its twenty-year run, the journal was published by the International Forum of Educational Technology & Society (IFETS). IFETS was a consortium of academics encouraging discussions of educational technology, but its website (now defunct) was hosted and owned by the Institute of Electrical and Electronics Engineers (IEEE):
The IEEE Technical Committee on Learning Technology’s website states that it ‘has been founded on the premise that emerging technology has the potential to dramatically improve learning’. This would appear to be a transparent avowal of partiality for e-technology’s adoption in the classroom, a predictable position for a group of electrical and electronics engineers to hold.
But intellectual or professional partiality is not the only possible influence here. The International Journal of Information and Learning Technology, for instance, is published by Emerald Publishing, which ‘was founded in 1967 to champion new ideas that would advance the research and practice of business and management’, according to its website. LC is among the most promising products in a business — education technology — whose market the Financial Times has estimated will grow to £129bn in the UK by 2020. It is in the financial interests of those who would support that market’s growth that the prevailing research reflects positively on the pedagogical effects of LC. It seems a safe assumption that Emerald Publishing is in that camp.
They are not, however, without even more powerful company. The potential economic growth of the education technology sector has long since drawn favourable attention in Westminster. According to the Financial Times’ Jonathan Moules in 2015, the central government has played an active role in encouraging the expansion of this sector in the UK. ‘The government is backing Edtech UK’, writes Moules, ‘a lobbying body, to help the country gain an even greater share of [the] market’.
The central government, of course, remains among the primary funders of UKHE, so the fact that it is ‘backing’ a lobbying body seeking to cultivate UK universities as customers for e-technology companies sends a fairly powerful message to university administrators making budgeting decisions.
Business observers, however, have identified yet another major driver of LC’s adoption in UKHE, an ace up the sleeve that might well render the mass of flawed research literature and the cosy relationship between government and the e-technology lobby superfluous: student demand in the wake of the central government’s introduction of student fees.
‘Back in the day’, writes Karen Mitchell of AVNetwork.com, ‘institutions provided all academic funding. Now, with student fees, [student] input is needed, and they like lecture capture’. Mitchell goes on to quote an education-technology industry source: ‘Campus-wide lecture capture is mainstream and gaining momentum, fuelled by student demand and decreasing tech costs’.
The sole claim of the research literature on LC that survives close scrutiny remains: ‘There is strong evidence that students place significant value on lecture recordings deployed via the Internet’. Obviously, enthusiasm for LC is overdetermined among those (such as students and technology professionals) who lack the teaching expertise necessary to anticipate its pitfalls.
But if the preferences of teenagers, early-twenty-somethings, and professional boosters can dictate the universal adoption — or imposition — of a consequential pedagogical tool in UKHE, then why even bother with all this research?
The fact that so much of the research literature on LC appears to be so deeply flawed — and that its prevailing conclusions ignore many of its own dubious findings — has not undermined its influence. Nor has the plain fact that much of the literature has been written, published, or edited by parties eager to encourage educational technology’s widespread adoption for reasons entirely unrelated to the supposed pedagogical benefits.
At the risk of seeming cynical, however, it probably would not matter if the research literature on LC’s effects were a model of intellectual rigour offering incontrovertible evidence that the technology damages both student attendance and learning. With business interests, government, university administrators, and students lined up behind LC for non-pedagogical reasons (i.e., profit, economic growth, industrial-relations control, and preference, respectively), the technology’s adoption is well beyond questions of intellectual justification. It is now a question of power.
In historian David Montgomery’s classic study of the turn-of-the-century U.S. labour movement, The Fall of the House of Labor (1988), he writes that mid-nineteenth century iron-rollers in Columbus, Ohio ‘exercised an impressive degree of collective control over the specific productive tasks in which they were engaged and the human relations involved in the performance of those tasks’. This control was no accident, Montgomery explains. It was a function of the workers’ willingness to demand it. ‘They exercised this control because they fought for it’, he writes, ‘and their position in that struggle drew strength from the workers’ functional autonomy on the job, from the group ethical code that they developed around their work relations, and from organisations they created for themselves in order to protect their interests and values’.
Ultimately, as Montgomery chronicles in that book, business leaders joined with powerful state agents to impose schemes of scientific management and automation on these workers. This alliance of business and government undid the workers’ ‘functional autonomy’, eroded their ‘ethical code’, and sapped the strength of their organisations to the breaking point. That business-government alliance of yesteryear appealed both to a voluminous academic literature extolling the benefits of scientific management’s ‘efficiency’, and to the consuming public’s seemingly insatiable appetite for the appearance of greater value at a lower price.
The twenty-first century university is not a nineteenth-century iron-rolling mill, but it bears unsettling comparisons. Academic teachers and researchers in the early twenty-first century UK still enjoy more ‘functional autonomy’ on the job, however attenuated, than practically any other category of wage or salaried worker. Yet they are now facing a similar conundrum to that of Ohio iron-rollers in the 1870s and 1880s. The imposition of LC is merely one skirmish of a larger campaign in which an alliance of business and government, empowered by a reigning ideology of global consumerism, aims to exert ever more methodical control over academic labour — ostensibly to improve ‘efficiency’ and student satisfaction, but more clearly to lower costs while increasing opportunities to extract private profit from public education.
The question is, will academic workers sit back and allow their functional autonomy (control over their own classrooms and pedagogical practice), ethical code (commitment to education as its own value), and organisations (intellectual disciplines with pedagogical integrity or collective organisations defending academic freedom) be eroded? And will they, to add insult to injury, allow this erosion to be underwritten by a body of scholarship undeserving of the name?
We shall see. In the meantime, immediate responsibility for the implementation of LC still rests in the hands of university administrators who, by the most generous interpretation, have been systematically misled by that very same ‘body of scholarship’. Even if they are sceptically inclined, they may simply not be up to the task of resisting the pressure to pay millions of pounds for fractionally higher institutional scores on the National Student Survey (NSS) and Teaching Excellence Framework (TEF), even when the non-monetary cost is the actual pedagogical ‘excellence’ they so frequently invoke as their highest priority.
The author wishes to thank Leon Rocha for his acute editing and extensive improvements to this USSbrief; colleagues at the University of York for indulging frequent discussions over the past three years about the dangers of LC; and the Postgraduate Certificate in Academic Practice (PGCAP) program at York, which offered an opportunity to research and write an earlier, unpublished version of this essay.
This paper represents the views of the author only. The author believes all information to be reliable and accurate; if any errors are found please contact us so that we can correct them. We welcome discussion of the points raised and suggest that the discussants use Twitter with the hashtags #USSbriefs62; the author will try to respond as appropriate. This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.