Open Recognition and its Enemies (4) — Quality Assurance

“There being no recognition that each individual constitutes his own class, there could be no recognition of the infinite diversity of active tendencies and combinations of tendencies of which an individual is capable. There were only three types of faculties or powers in the individual’s constitution [reason, passion and appetite]. Hence education would soon reach a static limit in each class, for only diversity makes change and progress.”

John Dewey, Democracy and Education

After a quick pause with authentic friends of Open Recognition, we are now back on the tracks of its enemies — we will come back to more friends in the conclusion of this series of posts. This time we will focus on quality, or more precisely, how certain views on quality and Open Badges might have a damaging impact on the idea of Open Recognition.

This post will refer to the following definitions:

• Quality: “degree to which a set of inherent characteristics fulfils requirement.” (source ISO 9000)
• Quality assurance (QA) “part of quality management focused on providing confidence that quality requirements will be fulfilled.” (source ISO 9000)
• Quality management (QM) includes all the activities related to quality planning, quality control, quality assurance and quality improvement.

In search of Quality

Quality is a subject of reflection, when not of concern, in the Open Badge community. A search on Google returns the following results: ”open badges” “Quality assurance” 17,500 entries, “open badges” “Quality” 630,000.

The first item returned by the query is a 2016 paper: Quality considerations in Open Badge initiatives, an introduction to a discussion paper “present[ing] data gathered from a “Quality Survey” and provid[ing] recommendations for quality assurance of Open Badge initiatives.” Although the low number of respondents (39 with 25 complete responses, mainly from the formal education sector) would not be sufficient to generate any significant statistical data or meaningful conclusions, the format and content of the survey, the responses collected and their analysis provide a useful insight on how the relationship between quality and Open Badges is perceived by segments of the Open Badge community and analyse the consequences of those views on the possibility to make Open Recognition a reality.

The first striking element in the Open Badge Network “Quality Survey” is its understanding of Open Badges: each of the respondents had to define themselves as either badge issuer, earner or viewer. The problem with this division into three classes, without reflecting on what led to it, is that it takes for granted the structural asymmetry of the 1.x badge infrastructure. Although it is something I have discussed in a number of posts and was the theme of An API of One’s Own, a presentation I gave with Nate Otto at the 2014 OpenED conference in Washington DC, I need to repeat one more time that before the innovation brought in by the 2.0 version of the Open Badge specification, the Open Badge Infrastructure, nolens volens, did not empower badge earners, but badge issuers — the power was fully in the hands of the issuers.

To paraphrase John Dewey’s quote (written in his critic of Plato’s vision of education), one could write:

“There being no recognition that each individual constitutes his own class, there could be no recognition of the infinite diversity of active tendencies and combinations of tendencies of which an individual is capable. The [1.0] Open Badge Infrastructure only knew three types of faculties or powers in the individual’s recognition [issue, earn and view]. Hence recognition would soon reach a static limit in each class, for only combining powers empowers individuals to be the active contributors to change and progress.”

Discussing the “Quality” of Open Badges without having a critical view of a one sided infrastructure is a bit like discussing the quality assurance of carpet bombing and asking the pilots, civilians and journalists what is the quality of a carpet bombing you can trust? For example, the “Quality Survey” indicates that one of the “Key desired strategic outcomes for badges” is “Badges to empower people to forge their own skill paths” but that is a hollow statement if the only tool they have at their disposal is the Mozilla Backpack that can only be used to collect badges designed by others. With the 1.0 specification, you have institutions establishing their own “skill paths” and individuals invited to walk the skill paths defined for them. It is about conformance, not empowerment, unless sharing the authoritarian view that the power to conform is what we should be aiming at.

Going through the list of statements/questions and the author’s comments, it is quite clear that the “Quality Survey” had no interest whatsoever in informal recognition. This could make sense if the idea was that quality only applies to formal recognition, something I might partly subscribe to, but the problem is that 1) it is not explained, 2) it does not do justice to the explicit aims of the Open Badge Network, the organiser and publisher of the “Quality Survey”:

“The specific aims of OBN are to […] place formal and informal recognition of learning on a par” (source: http://www.openbadgenetwork.com/about-us/).

If the aim of the OBN was genuinely to “place formal and informal recognition of learning on a par”, how could it not address the very concept of informal recognition: the number of occurrences of the word ‘informal’ in the “Quality Survey” is… 1 (one!)… And it is to refer to the introductory question “please choose the sector descriptor that is most relevant.” In the analysis of the responses to the survey there are 4 (four) more occurrences of the word, all in reference to a document produced by the Centre for the Development of Vocational Training (Cedefop): “European Guidelines for Validating Non-Formal and Informal Learning” — there are also 14 occurrences of “non-formal,” all in relation to [formal] recognition of non-formal learning.

No, Quality assurance is not “vital to create badges that are considered of quality”!

To the question “Do you think having quality assurance processes in place for your badges are vital to create badges that are considered of quality?” the responses were: Yes 60.0%, No 20.0%, Don’t know 20.0%. This question was followed by another: “If you answered yes, what would the vital components of a quality assurance process be?” There was no follow-up question for those responding “No.”

When reading the sequence of questions I wondered: why are the “Quality Survey” authors solely interested in knowing more from those responding “Yes” and not from those who responded “No”? Why not ask those who responded “No” why they think that it is not vital to have a quality assurance process?

From a methodological point of view, the fact that one of the responses, not the others, led to additional questions creates a bias. The implicit message to the respondents is: if you do not think that having a quality process is important, just move to the next question, we just don’t care for your reasons. On the other hand, if you think that having a quality process is important, stay with us a bit longer because we think that you probably have something important to share with us!

May be, some (like me) might have responded that it is vital not to have a quality assurance process, that it depends on the context whether what you are looking for is formal or informal recognition, that while quality assurance makes sense in the context of formal recognition it could have many adversarial effects on informal recognition. This might explain why to the question “What makes a good quality badge?” the responses were:

• “The badge issuer must be a well-established organisation” 87.5% agree.
• “The badge issuer must be an organisation or someone I’ve heard of “ 50.0% agree

To the instruction “Please rank in order how much you would trust a badge was of quality”, the top three criteria selected were “being aware of the quality assurance procedures that had been implemented by the issuer, if the badge earner had received endorsements for their badge evidence from appropriately qualified people and if the badge indicated a level of competency [my highlights]. From an informal recognition perspective the statement could have been put more simply: I trust a badge if the badge has been endorsed by appropriately qualified people. I understand that the nuance might sound subtle (endorsing evidence vs. endorsing a badge or a person) but it is exactly where one can capture the superiority of informal recognition over formal recognition over time.

Without informal recognition, formal recognition is worthless

The subtext of most, not to say nearly all, discourses on (formal) recognition is the superiority “Quality assured recognition” has over any other form of recognition. My claim is that informal recognition is a far superior form of recognition to formal recognition — in fact, it underpins, precedes and follows it!

Formal recognition is what will lead to a diploma, a certificate or what is too often conflated with Open Badges: micro-credentials. What is the value of a formal recognition five years (even 6 months!) after having been issued? Probably nil, if what led to its delivery has not been practiced. What keeps a formal recognition current is the informal recognition of its bearer by a community of peers, clients and contractors. Without informal recognition, formal recognition becomes simply worthless.

One of the conclusions of the “Quality Survey” is:

“We believe the overall outputs of the survey show that most agree that some level of process is important to ensure validity, consistency and for the reputation of a badge scheme to be upheld, so perhaps quality assurance processes should be designed on a sliding scale, ensuring that the process is appropriate for the particular badge initiative’s setting. Some may need to be in-depth while others could be light touch, depending on considerations such as: if they include high stakes assessment; the value proposition to the earner (which may be to use badges for developing skills awareness and understanding of intrinsic motivation, rather than to necessarily share with prospective employers).”

What is stated implicitly is “in-depth” quality assurance might be more useful when badges are shared with prospective employers, which is equivalent to saying that employers are more interested in formal qualifications than informal recommendations (by previous employers, colleagues or clients). This might apply to people going for their first job (a proposition that could also be challenged!) but certainly not to most of us. Recommendations letters, endorsements, job certificates are informal documents produced with no “in depth” quality assurance. The only things that really matter are: is the content authentic? Are the claimant and issuer real? It the issuer trustworthy? all questions Open Badges are particularly apt at addressing, when used properly.

In the context of recognition, whether formal or informal, the quality people are looking for is first and foremost trustworthiness: can I trust the information I’m provided with? Are those who have produced that information trustworthy? In the context of formal recognition, quality assurance is a means to an end, not a goal, it is ancillary to trust. In the context of informal recognition, trustworthiness does not have to be connected to a formal process of quality assurance. The quality assurance infrastructure resides within the people, within their trust networks. It is an organic and emerging property.

What does quality mean in the context of informal recognition? Should quality assurance be a requirement? Quality Assured Informal Recognition? This sounds very much like an oxymoron or the marriage of a carp and a rabbit… Or that of a scorpion and a frog…

“Trust is good but control is better” (a sentence attributed to Vladimir Ilitch Oulianov, better known as Lenin) would be a good moto for any of the tenants of quality assurance as sole extrinsic mechanism for establishing trust. One key characteristic of trust, in relation to security is that security is intrinsic to trust (when trust is high, security is also high) and when (extrinsic) security measures are used as a means to heal declining trust, things generally get even worse (c.f. The Deleterious effects of Mistaking Security for Trust). And this is precisely the crux of the problem with the architecture resulting from the 1.x Open Badge specification: we did not entrust individuals with the ability to issue badges, and we did not ask institutions to prove their credentials with the badges they would have collected in their Mozilla Backpack. As I wrote before, we did not give people the power to trust, I would say today, we did not recognise in them power to recognise, we did not recognise them as recognisant agents — endorsement will bring us closer to a solution to that problem.

One major difference between formal and informal recognition is the power structure: formal recognition is by nature asymmetrical (hence the match with the 1.0 Open Badge specification), while informal recognition is more egalitarian, the power structure might not be homogeneous but, at least, it is distributed between all the agents (hence the match with the 2.0 specification). Moreover, in the informal world, every individual is actively involved in the recognition of others. I recognise myself through recognising others, not just by asking to be recognised. One could even state that recognition is THE key competency underpinning all other competencies. In the world of informal learning, the recognition of others occurs naturally and is generally tacit, embedded within social codes. In the world of formal recognition, recognition is artificial, explicit and often embedded within procedures regulated through quality management activities. The trustworthiness of formal recognition rests on extrinsic quality management mechanisms while that of informal recognition rests on informal trust networks. Here, quality is intrinsic, it is within the agents and their networks; there is no need for any form of extrinsic quality assurance.

As with learning, where extrinsic motivation has deleterious effects on intrinsic motivation, with informal recognition, (extrinsic) quality assurance mechanisms could also have deleterious effects on trust and learning altogether:

“The legendary statistical consultant W. Edwards Deming, with his characteristic gift for understatement, has called the system by which merit is appraised and rewarded “the most powerful inhibitor to quality and productivity in the Western world.” He adds that it “nourishes short-term performance, annihilates long-term planning, builds fear, demolishes teamwork, nourishes rivalry and … leaves people bitter.” To this we can add that it is simply unfair to the extent that employees are held responsible for what are, in reality, systemic factors that are beyond their control.” (Punished by Rewards, Alfie Kohn p. 129)

The systemic factors beyond the control of those using an Open Badge infrastructure based on the 1.0 specification, i.e. the participants in the “Quality Survey”, were precisely the fact that individuals, as badge earners, did not have their say in the system, had to carry a Mozilla Backpack to prove their credentials, while badge issuers had their say and did not have to carry a Mozilla Backpack to prove their own credentials. An outcome one might have expected from the implementation of quality management for Open Badges could have been the correction of that asymmetry. Unfortunately, the recommendations of the “Quality Survey” are to more or less replicate this faulty design into the model of quality assurance…

The missed opportunity: how to use Open Badges to improve quality management?

One point that should not escape the learning professional with a basic experience of quality management is the fact that the document entitled “O7A1 Open Badges and Quality Management” does not address… quality management.

• Quality management (QM) includes all the activities related to quality planning, quality control, quality assurance, and quality improvement.

The only activity overtly discussed is quality assurance, while quality control and quality improvement are ignored. Quality planning is implicitly discussed through the presentation of a “canvas” but not addressed as such. Addressing quality assurance without simultaneously addressing quality improvement (QI) means that there is no feedback loop that will preserve quality from being emptied from its substance. Understanding and properly implementing QI is essential to a well-functioning learning organisation, and is non negotiable for any practitioner interested in improving learning efficiency and impact.

While this point should be sufficient to invalidate the value of this document as a basis for thinking about the quality for Open Badges, there is a more patent issue, the missed opportunity of addressing the obvious question: how to use Open Badges to improve quality management? Had this angle been used, the issue of badges produced as the result of a formal or informal recognition processes would have had a chance to be properly addressed.

A genuine exploration of how Open Badges could contribute to the improvement of quality processes (and in doing so contribute to their own quality!) would require more than a single post so I will only initiate an unordered list of ideas to be reviewed and expanded:

• Open Badges as source of information for quality processes;
• Using Open Badges to deliver Quality Marks, so clients would be able to endorse (or not) those Quality Badges;
• Use badges and endorsements received to submit a portfolio of evidence to claim a quality mark;
• Analyse the endorsements received to the Open Badges delivered by an educational body to monitor the quality of its learning provision;
• Use Open Badges to measure the social capital of an organisation, its structure and growth to inform quality processes;
• Use Open Badges to recognise individual and collective contributions to quality improvement;
• Use Open Badges/endorsements to monitor the quality of customer service
• <your idea here!>

More ideas could be inferred from a paper I wrote a few years ago: From quality of eLearning to eQuality of learning where, under the influence of my work with ePortfolios, I explored how technologies could contribute to rethinking quality management. My guess is that we could do even more with Open Badges and Open Endorsements!

So, far from being an enemy of quality management (among other quality initiatives and projects I was at the origin of the creation of European Foundation of the Quality in E-Learning (EFQUEL), now merged with EDEN) I am also aware of what poor quality management practice can do (c.f. Deming’s quote above). My belief is that Open Recognition and its instruments are an opportunity, not just to equip existing processes, but to contribute to rethinking them altogether. This could be the starting point for a number of exciting action-research projects!

Leave informal recognition free from quality assurance!

In a project I designed 3 years ago (Badge Europe!) I had a similar discussion when one project partner decided to rename the deliverable entitled “Discussion Paper on Open Badges and Quality Management ” to “Quality Assurance of Open Badges” which totally changed the spirit of what I wanted to achieve when I designed the project. I am afraid that the Open Badge Network Erasmus project has fallen into the same trap and instead of exploring the tension between Open Badges and Quality Management and the opportunity Open Badges offer to rethink Quality Management, it has confined its work to defining what could be identified as good enough quality assurance for the delivery of Open Badges within a flawed infrastructure.

The criticisms made to the “Quality Survey” should be taken both seriously and lightly. Lightly first: this report should not be taken too seriously as it is the typical outcome of a European project where the covert objective is to be able tick the box and tell the project officer “job done.” As the project officer has probably no or little understanding of Open Badges, nor about the distinction between formal and informal recognition, it is very unlikely that anyone will find any discrepancy between what led the project to be accepted in the first place (and receive approximately € 450,000 / $ 528,000, for a 3 year project) and the actual results. After all, all the right words are there, sugar coated with figures and statistics, something generally sufficient to pass through the final review.

More seriously, the fact that the “Quality Survey” appears first when querying Google for “Open Badges” and “quality”, is fraught with the danger of disseminating harmful ideas on both quality (a term never defined in the survey nor its analysis) and Open Badges. Far from “plac[ing] formal and informal recognition of learning on a par”, the “Quality Survey” conveys the implicit message that formal recognition is the only meaningful form of recognition and that quality assurance is a must have, thus ignoring how informal recognition operates, what could make it more visible and valuable and what could result in adversarial effects.

My claim is that the models of quality assurance inherited from formal recognition are not just irrelevant but detrimental to informal recognition. The direction taken by the quality assurance thinking of the “Quality Survey” is one of the adverse consequences of confining the three roles (earner, issuer, viewer) into three independent and not overlapping spaces, thus creating the conditions for the emergence of a distrust infrastructure epitomised by the Mozilla backpack as a tool of institutional confinement, making learners the accomplices of their own disempowerment.

Does this mean that there should not be any form of quality management? Certainly not, but quality management processes should be confined to formal recognition — and to the design of ‘high quality’ open recognition enabling technologies. Just leave informal recognition free from (formal) quality management, please!

Next post: The Open Recognition and its Enemies (5) — Saved by Open Endorsement!

Originally published at www.learningfutures.eu on August 9, 2017.