Benchmarking of the e-Learning Quality Assurance in Vocational Education and Training: Project Results

Purpose: The paper aims to present project No. 2020-1-SK01-KA226-VET-094266 BEQUEL and partial results of the benchmarking questionnaire. Methodology/Approach: The starting point for the project solution was an analysis of the current state of ensuring the quality of e-learning in vocational training and education in the partner countries of the project (Slovakia, Greece, Spain, and Italy) and an overview of laws and regulations valid at the European level and in the world. A benchmarking survey was conducted to determine the level of e-learning quality assurance in these countries compared to good practices in the European Union. The Benchmarking Badge published monthly on the www.bequal.info portal tracks changes over time. Findings: The average standard for the four involved countries after the pandemic was the level for VET (Vocational Education and Training) providers: (1) strategy and policy for e-learning 71.6%, (2) support for trainers and trainees for e-learning 70.4, (3) infrastructure support for e-learning 74.9%, (4) program/course design and development and approval for e-learning provision 75.8%, and (5) e-learning training program evaluation procedures 67.9%. Research Limitation/Implication: On the one hand, the project was limited by the measures of the Covid-19 pandemic, during which face-to-face meetings and training were not allowed. Still, on the other hand, the VET providers recognised their weaknesses, strengths, and readiness for complete online education. Originality/Value of paper: Examples of good practice and video presentations inspire improving the quality of e-learning in VET.


INTRODUCTION
The  pandemic and the resulting social distancing and 'lockdown' measures have significantly affected the provision of vocational education and training (VET) internationally (Majumdar and Araiztegui, 2020).
In connection with this period, ensuring the quality of e-learning has become significantly relevant. Although several e-learning models have been created in the past, either by academic, interest or consulting organisations, no unified framework or model defines the requirements for ensuring the quality of elearning in VET, including education within the European Higher Education Area.
The crisis caused by the Covid-19 pandemic has brought a new reality to education. Many education providers, including VET organisations, have begun to make greater use of e-learning to ensure continuity of education for their students during the pandemic.
The goal of the project was to develop and implement an online benchmarking tool for comparison of the quality assurance in e-learning procedures of VET organisations. The project focuses primarily on VET providers of the participating European countries who have gone through the procedures for adopting e-learning programs and digitising education provision. The project also targets stakeholders, policymakers, researchers and practitioners.
The paper presents the results of project No. 2020-1-SK01-KA226-VET-094266 BEQUEL "Benchmarking for quality assurance in e-learning provision of vocational education and training" and the current results of a benchmarking survey carried out in five countries (Slovakia, Greece, Italy, Spain), which are independently supplemented by the results of other European countries.

STATE OF PLAY
The starting point for the e-learning quality assurance research in VET organisations was the analysis of the existing quality assurance frameworks for elearning practices in VET and other education levels developed by International, European, and National organisations.
Digital competencies have become vital for citizens' participation in today's social, economic, and civic life. Digitalisation is transforming the nature of work and poses new challenges (European Commission, 2018): 90% of future jobs will require digital skills; 44% of Europeans lack basic digital skills; more than 48,000 schools lack a broadband connection. Misinformation, cyberbullying, and data privacy issues threaten digital well-being.
We examined the available electronic resources to create an overview of the current state of knowledge and practice in the quality assurance of e-learning in schools and VET organisations. We grouped them according to ( Fig. 1) and (Tab. 1), and organised them into two closely related groups: VET and e-Learning. These resources are presented on the Global, European, and National levels.

E-LEARNING FRAMEWORK FOR VET ORGANISATIONS
Creating a unified framework for ensuring the quality of e-learning became the basis for the project's solution.
The Quality Assurance Framework (Fig. 2) is located in an external supportive environment that explicitly recognises quality as the value of work and enables the educational institution's objectives to be achieved. The external environment must provide support and advice at the national level for continuous improvement.

Figure 2 -Quality Assurance Framework for e-learning Provision in VET Organisations
Figure comments: • External environment: it is assumed that e-Learning for VET can only be implemented successfully if specific external requirements are met. These include legal frameworks being in place and the institution/organisation having access to the necessary hardware and software.
• Strategic management: means that the e-learning goal meets students' requirements from any location, uses and works with their knowledge and skills, and prepares them for the professional and academic arena to contribute to a democratic and sustainable global development within society in the 21st century. Further objectives are to encourage widening recruitment and lifelong learning, facilitate attractive study and learning environments both on Campus and virtually, and increase and encourage strong development of the leader, teacher, and employee excellence (Grifoll et al., 2010).
• Risk: means, according to (ISO 21001, 2018), "the effect of uncertainty where an effect is a deviation from the expected -positive or negative and uncertainty is the state, even partial, of deficiency of information related to, understanding or knowledge of, an event, its consequence, or likelihood".
• The curriculum design, course conception, and course delivery: mean continuous quality assurance through validation processes and planned according to student requirements, scientific development, and community needs (Grifoll et al., 2010).
• Staff support: means a greater focus on scholarships for teaching and learning, constructive alignment, and encouragement of pedagogical qualification and recognition (Grifoll et al., 2010).
• e-Learning inputs: o Experts for creating e-Learning courses, o Trained teachers in creating and providing e-Learning courses, o Students are involved and have access to electronic infrastructure (i.e. e-Resources and the e-Library independent of place, time, and languages).
• e-Learning outcomes: o Learning outcomes -a detailed description of what a learner knows, understands and can do at the end of the learning process to ensure that the individual learning objectives of the programme are fulfilled. It is stated in the content of knowledge, skills, and competencies, o Skills -the ability or art of easily and accurately applying knowledge and performing a specific cognitive, psychomotor or social activity, o Competency -the proven ability of a person to perform a specific professional activity, Skills, knowledge and competencies serve as the structural characteristics of learning outcomes.
• Monitoring and measurement: determining the status of a system, a process or an activity. Measurement process to determine a value.
• Limitations and risks: finance, knowledge, and technical support can limit the process of providing quality e-learning. The effect of uncertainty manifests itself as a risk.
The framework includes quality assurance criteria for the e-learning delivery of VET (both theoretical and practical). This framework became the starting point for the e-learning benchmarking survey.

E-LEARNING BENCHMARKING SURVEY
The purpose of the VET e-learning benchmarking survey was to get to know the current state of ensuring the quality of e-learning in vocational training and education in the partner countries of the project (Slovakia, Greece, Spain, Italy) and an overview of laws and regulations valid at the European level and in the world. To determine the level of e-learning quality assurance in these countries compared to good practice in the EU, a benchmarking survey was used, which was developed within the project and tested on a sample of five people from each country involved in the project. The survey has five modules and 46 statements: In each module, quality indicators are measured based on quality criteria. The quality criteria have been phrased as statements that the respondent has to review and assess on a four-level frequency Likert scale with the following score: "not at all true" (0-25), "somewhat true" (25-50), "mostly true" (50-75) and "completely true" (75-100). The system generates several reports after the usage of the benchmarking tool: • Benchmarking report: Provides the Statistics from all users.
• Individual report: Provides the overall score of the user resp. VET organisation) Furthermore, it gives the position of the user among overall users and the position among the country.
In order to carry out benchmarking and find a preliminary benchmark for the country, the minimum number of completed questionnaires was set at 20. The Benchmarking Badge published monthly on the www.bequal.info portal tracks changes over time. The result is calculated from the number of answers in the "completely true" category.

Table 2 -Module 1: VET Provider's Strategy and Policy for E-Learning
No Statement Result 1.1 The VET provider employs a distance education strategy, which is aligned with the institution's mission and strategic goals.

37%
1.2 The VET provider ensures that their e-learning strategy is public, transparent and available to any interested stakeholder.

42%
1.3 The VET provider periodically reviews and updates their distance education strategy. 20% 1.4 The e-learning quality policy is periodically reviewed to improve future e-learning programs.
29% 1.5 The VET provider has policies, procedures, and resources in place to support socially vulnerable groups (i.e. e-learners with diverse needs) among e-learning audiences.
27% 1.6 The VET provider has policies, procedures, and resources in place to support socially vulnerable groups (i.e. e-learners with diverse needs) among e-learning audiences.   3.4

25%
The VET provider has a documented plan for the support, maintenance, and upgrade of existing technologies.

29%
3.5 The VET provider has a documented plan for the creation of backup files at an LMS level (which allows the administrator to save anything related to the LMS).

3.6
The VET provider has a documented plan for the creation of backup files at a program level (which allows the administrator or the instructor to save anything related to a specific program).
32% 3.7 The VET provider has a documented plan for security issues (e.g., password protection, encryption, secure online exam participation, etc.).

40%
3.8 The LMS has the ability to directly connect and share data on the e-learning programs with the internal management system of the organisation.

30%
3.9 The LMS offers the ability to create, delete and customise the provision of programs (adding and deleting weeks/modules, hiding modules, etc.).

40%
3.10 The LMS offers the ability to schedule activities or divide work by weeks/units. 44% 3.11 The LMS offers the ability to embed and play multimedia material such as video and audio files.

49%
3.12 The LMS offers the ability to create tests and assessments. 57% 3.13 The LMS offers the ability to create work assignments and submit them. 58% 3.14 The LMS offers the ability to correct and grade. 57% 3.15 The LMS offers the ability to generate reports on the performance of trainees. 53% 3.16 The LMS offers the ability to generate a report on a program (including aggregate data on the access of trainees to the system at a program level).

5.1
The training program includes an evaluation process of all support services available to the teaching staff and the trainees in order to ensure the general satisfaction of all stakeholders with its implementation.

5.2
The training program includes a process for the institutional evaluation of the teaching staff with regard to their efficiency in distance learning, as well as the evaluation of the teaching material by the trainees.

5.3
The VET provider has an internal procedure for validating new training programs before they are officially launched. 27%

5.4
The VET provider has a policy for regularly improving the training program based on the evaluation and feedback collected.

5.5
The VET provider ensures that an external body officially accredits the training program before it is launched.

DISCUSSION AND CONCLUSION
The aim of the benchmarking was to identify internal opportunities to improve the provision of e-learning in the VET organisation after the Covid-19 pandemic.
By analysing the results from Slovakia, Greece, Italy and Spain, we found the best performance in the monitored category and found out what such performance enables (Tab. 7). Subsequently, the concrete processes can be compared with how a specific VET organisation works, and we have identified the possibilities of implementing changes that could significantly improve the quality assurance of e-learning. For a more detailed investigation, it is possible to select the data on the Bequal platform (2023) according to the type of VET organisation: • Educational Institution (secondary vocational education and training schools), • Educational Institution providing post-secondary vocational education and training, • Educational Institution providing continuing vocational education and training.
A limitation of the research was the restrictions of the Covid-19 pandemic, during which face-to-face meetings and training of the expert team were not allowed.