Student Engagement and the New ASQA Auditing Model

January 2017 was a month of transition. While most eyes were trained on the USA inaugurating its 45th President, educators in Australia were focusing on the Australian Skills Quality Authority (ASQA) — and new changes to the auditing model for Registered Training Organisations (RTOs). The previous auditing approach focused on business systems, processes, compliance and methodologies. But a new breed of rogue, and in some cases, criminally negligent, RTOs — the so-called ‘cowboys’ of the new education landscape — have led the Commonwealth to impose new, tougher standards. The auditing process now focuses more on the student experience, questioning whether enrolled students are actively engaged and satisfied with their education provider. It’s a metric that will be harder to demonstrate, and could create potential headaches for RTOs who are not prepared.

But what do the regulatory changes actually mean for RTOs on the ground? That all depends on attitude. The new auditing era is either a source of administrative hassle, or an exciting opportunity to greatly enhance student engagement and their relationship with RTOs. RTOs may be required to pre-emptively and periodically assess their own training, and can pursue this through a number of methods. Unsurprisingly, technology will play a major role in boosting compliance.

Before we consider the new landscape, it’s worth asking: how did we get here?

I. What Went Wrong?I

In early 2016, a big education provider was placed under administration by the Federal Government, and became the subject of a major fraud investigation by the Australian Federal Police. In the lead-up to the investigation, ASQA had received numerous complaints from students of registered RTOs — ranging from students complaining they did not understand their course requirements, to more disturbingly, students reporting they had never knowingly signed up to a course in the first place. Subsequent investigations uncovered ‘significant non-compliance with the VET quality framework, which all registered training providers are required to satisfy to maintain registration as a training organisation.”[1] Something was seriously amiss.

As the true nature of these RTOs came to light, it became obvious that these weren’t examples of accidental non-compliance. Rather, it is alleged that brokers and agents were deliberately targeting students from low socioeconomic communities, focusing on obtaining VET FEE-HELP loans without a genuine belief the student could succeed in his or her program. Such conduct amounted to a serious defrauding of the government, and brought the rest of Australia’s bona fide and compliant RTOs under increased scrutiny. Part of the problem was that the old auditing system enabled RTOs to effectively model compliance on the outside without engaging with the student experience on the inside. ASQA, faced with the prospect of a repeat performance of this disreputable conduct, changed its focus. The watchdog, as it were, grew teeth.

II. The New Model

What does this mean for RTOs moving forward in 2017? The main change is that the focus of audits is now on behaviours and the student experience. The “student experience” has been usefully divided into five components, which are discussed below.

The marketing and recruitment component assesses whether an RTO’s marketing practices deliver factual information to potential students, allowing these students to make informed decisions about what training and benefits the course will deliver, and ultimately, whether the course is suitable for them. Enrolment will focus on whether the RTO is ensuring its’ students have the existing skills, knowledge, and experience required to successfully undertake the course. Support and progression assesses whether the RTO is providing appropriate support services to its students, and best enabling its students to undertake the course. Training and assessment is a two-pronged inquiry. Firstly, has the RTO appointed qualified trainers to deliver the training? Secondly, is the amount of training delivered to students, and the manner in which the training is delivered, consistent with requirements for the accreditation being offered? Finally, Completion assesses whether the RTO is awarding AQF certificates only to students who have met the course or Training Package requirements.

Each of these components will be measured via student surveys conducted by ASQA. For the full list of questions, see http://www.asqa.gov.au/verve/_resources/Fact_sheet_-_Student_Surveys.pdf#search=audit%20questions. Interestingly, ASQA has two modes of conducting its surveying; it may require RTOs to provide ASQA with a list of students who can then be surveyed, or ASQA may require the RTO to conduct the surveys on ASQA’s behalf.[2] The key message is that RTOs needs to be prepared for either scenario.

III. Engagement — And how to find it

From an academic perspective, ‘student engagement’ refers to the interaction between the time, effort and other relevant resources invested by both students and their institutions with their performance, and the reputation of their institution.”[3] This metric is in turn often divided into subcategories; intellectual engagement, emotional engagement, behavioural engagement, and so on. Engagement has been described as practically involving clear articulation of learning criteria. Do students get constructive feedback? Are learning outcomes clearly demonstrated? Are students developing inquisitive personalities as a product of their learning?[4] Engaged students are emotionally positive, ambitious to learn, and exert intense concentration in the implementation of learning tasks.[5] It is a key element of achieving equality and social justice, helping to level the ‘playing-field’ for students from low-income family backgrounds and others who have been historically undeserved.[6]

The underlying theme that connects these elements is student satisfaction, well-being and outcomes. What strategies and tools are actually available for RTOs to succeed with student engagement? Broadly, RTOs need to maximise reflexivity to student feedback (for example, on social media), as complaints are an early-warning sign to RTOs that there may be flaws in service delivery. If there’s anything to be learned from the recent spate of rogue RTOs, where there’s smoke there’s fire. Responsible RTOs need to address problems before they enter the public domain, by engaging with their students from the point of first interaction (the ‘marketing’ phase) to the point of exit (the ‘completion’ or ‘awarding’ phase’).

One traditional method of assessing student satisfaction is by interview-based surveys — either face-to-face or by telephone. This can provide insightful qualitative data, but is usually costly and time-consuming. Studies indicate that face-to-face interviews are the most expensive surveying option. Additionally, poor interviewing methods can introduce additional errors into the surveying process. Further, new research suggests that interviewees may be inclined to give overly polite, and in some cases, false appraisals of an RTO in a personal interview.[7]

A potentially more effective method — and one that is being increasingly deployed by RTOs, TAFEs, and tertiary institutions — is to measure student satisfaction by way of online surveying. Measuring and maximising student engagement through the use of surveys is an approach endorsed in the USA and Canada, and has found articulation in Australia through the Australasian Survey of Student Engagement (AUSSA). In AUSSA, six engagement scales are measured through surveying; academic challenge, active learning, student and staff interactions, enriching education experiences, supportive learning environment, and work-integrated learning. The AUSSA scales provide an illuminating academic reference point to the metric of engagement that ASQA will now be testing in its audits of RTOs under the new model.

Internal surveying’s primary advantage is that information provided by students can be directly used as evidence of student satisfaction in the event of an audit by ASQA. ‘Pre-course surveys’ provide a rich insight into the temperament of students and of trainers prior to commencement of a course, and initiate a channel of student engagement with the RTO that is invaluable for obtaining further data as the student progresses. Likewise, an ‘Early Stage Feedback Survey’ provides trainers with critical insights into student satisfaction during the first 2–4 weeks of a training program. ‘Course completion surveys’ and ‘Post-course evaluation surveys’ provide RTOs with sets of data that detail student satisfaction and any potential improvements that could be implemented.

Indeed, the effective use of surveys is precisely the mechanism that ASQA plans to utilise to conduct their audits. The ability for RTOs to quickly, cheaply and accurately administer surveys themselves will effectively put them one step ahead of any ASQA auditing process. At the same time, internal surveying creates and fosters an invaluable source of student data that can be used to continually improve services and boost compliance.

IV. New Opportunities

The new ASQA guidelines should be seen as a rare opportunity for RTOs to distinguish themselves from the pack, by maximising communication with students in order to measure, and increase, student engagement. This can involve implementing a range of pedagogical and technological innovations designed to strengthen training, assessment and evaluation metrics. The RTO landscape may be rapidly evolving, but with the right mixture of preparation and conviction, RTOs can take full advantage of the new opportunities.

[Editor’s note: The new audit model will be progressively rolled out across all VET and CRICOS audits in 2017. If your RTO is facing an audit, you will be notified about which audit approach is to be used and whether student data is required to conduct an online student survey.]

[1] Paul Karp. “Police raid vocational education provider after fraud allegations.” The Guardian. Accessed 12 December 2016 https://www.theguardian.com/australia-news/2016/apr/12/police-raid-vocational-education-provider-after-allegations

[2] Fact sheet — Student surveys. Date accessed: 12 December 2016. URL: http://www.asqa.gov.au/media-and-publications/student-surveys.html

[3] Trowler, Vicki. Student engagement literature review. Department of Education Research, Lancaster University, November 2010, p. 3

[4] Strong, R. Silver, H. and Robinson, A. (1995) “What do students want (and what really motivates them)?” Educational Leadership. September. p. 25.

[5] Skinner, E.A., & Belmont, M.J. (1993). “Motivation in the classroom: Reciprocal effects of teacher behavior and student engagement across the school year.” Journal of Educational Psychology, 85(4). p. 572

[6] Kuh, G .D . (2009) The National Survey of Student Engagement: Conceptual and Empirical Foundations. New Directions for Institutional Research. 141 (Spring 2009), pp . 5–20

[7] ‘Survey methods’. Queensland Government Statistician’s Office. Last reviewed 1 April 2015. << http://www.qgso.qld.gov.au/about-statistics/survey-methods/