Share

Skills beyond school

AHELO Main Study

 

What is AHELO?

The principal objective of Assessment of Learning Outcomes in Higher Education (AHELO) is to provide data to governments, institutions and students themselves on what students at the end of their first (bachelor level) degrees know and are able to do. Such data can serve multiple purposes:

  • They allow governments to evaluate the quality of their tertiary educated human capital among the higher educated cohorts against the international standards.
  • They enable institutions to compare and benchmark the learning outcomes of their students against international standards in order to improve the quality of teaching and learning.
  • They empower students to weigh their learned skills against the distribution of learning outcomes in their own institution and country and against international standards.

AHELO Brochure

Why do we need AHELO?

The focus of higher education has changed significantly in the past thirty years, primarily in response to the changing nature of work. A rapid increase in jobs requiring higher order, cognitive skills has created a worldwide need for more graduate-level employees. As a result, the priority previously given by universities to inducting a small minority into research capabilities has given way in many countries to providing up to half the population with the skills and knowledge relevant to employability. This has been achieved through a rapid expansion of the higher education sector and the establishment of more diverse types of higher education institutions.

However, in a period when governance, accountability and transparency in higher education have been strengthened in many countries, and when data on the performance of institutions in research is acknowledged to be powerful and comprehensive, the loud demands for data on learning outcomes, expressed by students, institutional leaders, ministers and business, continue to go unmet. In this key area, there is a continuing and damaging absence of information, of a quality to ground credible benchmarking and comparison.   Without AHELO, judgements about the quality of teaching and learning at higher education institutions will continue to be made on the basis of flawed rankings, derived not from outcomes, nor even outputs – but from idiosyncratic inputs and reputation surveys.

In several countries initiatives have been taken to start developing the modelling of assessing learning outcomes of higher education programmes and institutions. These national initiatives are extremely valuable in further developing the value proposition of learning outcomes assessment, the conceptual frameworks and the methodologies associated. At the same time the variety in conceptual and methodological approaches adopted preclude the development of comparable metrics so desperately needed in an increasingly globalised environment. The AHELO seeks to develop a global platform of assessing higher education learning outcomes through which national systems, institutions and individual students can assess and benchmark the skills the learning outcomes of higher education.

What is the AHELO Feasibility Study?

Between January 2010 and December 2012, a major AHELO Feasibility Study was conducted, to test the scientific and practical feasibility of assessing what higher education students know and can do at graduation, across diverse national, cultural, linguistic and institutional contexts.

Feasibility Study Reports:

Volume 1  Design and Implementation (Full reportExecutive Summary)
Volume 2 Data Analysis and National Experiences (Full reportExecutive Summary
Volume 3 Value-added Measurement and the Conference proceeings (Full report).

Assessment frameworks were developed, assessment instruments developed or adapted and a proof of concept completed. Field implementation took place in 2012. Focusing on what students know and can do, the study demonstrated that a large scale comparative assessment of higher education learning outcomes is conceptually valid and for the most part technically feasible. However, a number of questions were left unanswered. These concerned the value proposition, the business case and a number of operational, technical and financial issues.  Find out more on the the assessment and the instruments here.

 

What is the AHELO Main Study?

 

The Main Study will build on the achievements of the Feasibility Study by anchoring the first cycle in the proven domains of civil engineering and economics.  And it should significantly improve the value it offers to participants and stakeholders, by learning lessons from the Feasibility Study in four areas: 

Benchmarking against agreed international standards

Firstly, the Feasibility Study compared the mean performance of students by institution against the mean performance of all students. The Main Study would introduce criterion referenced standards that help to explain the performance of students and institutions in substantive terms rather than just relative to other institutions, bringing AHELO into line with best international practice, including PISA. The effect would be to provide information not only about the performance of students in relation to each other but also about their performance in relation to agreed levels of competence.

Providing additional categories of comparative data

Secondly, the Main Study would provide a fuller and personalised break-down of comparative data. Students would be able to compare their performance with students at the same institution. Students, teaching staff, institutions and other participating entities would be able to see how the performance of their institution compared with that of other institutions, nationally, internationally and against institutions with similar characteristics.

Increasing the level of detail

Thirdly, the Main Study would provide reporting against performance bands, allowing students, faculty, institutions and other entities to see the distribution of student performance against, for example, bottom quartile and top deciles. It would also provide reporting against proficiency levels, showing what proportion of students sits at which level. Finally, it would provide a description of each test item or group of items and a detailed breakdown of student performance by rotation, enabling the identification of detailed patterns and the design of specific teaching strategies to address under-performance.

Strengthened reporting

Taken together, the above measures would allow major improvements in reporting. One immediate benefit would be to participating students, who could receive rapid online feedback following completion of the assessment. A second benefit would be a suite of new reports, available at the end of each assessment cycle, which can be tailored to serve the needs of different constituencies.

The proposal for an AHELO Main Study was discussed at a meeting of countries interested to pursue AHELO on 11 February 2015 hosted by the United Kingdom.

The AHELO Main Study would be designed in six stages:

Stage 0 (March-December 2015): Foundations.

Stage 1 (2016): Development.

Stage 2 (2017): Preparation.

Stage 3 (2018): Field Trial.

Stage 4 (2019): Main Test.

Stage 5 (2020): Analysis and Reporting.

The model below indicates the approach proposed for a main AHELO study.

 AHELO Main Study Model

  

How will it be governed?

 

The AHELO Main Study will be instituted by the participating national or subnational government authorities, who will form the AHELO Governing Board and who contribute to the development and financing of the Main Study.

Given the complexity of developing assessment frameworks and assessment instruments the AHELO Governing Board would benefit from technical assistance. A Technical Advisory Group would therefore be established as an early priority, with responsibility to consult widely with experts in countries and regions.

The AHELO Main Study will need to establish strong links with stakeholders in the higher education community. An Academic Advisory Group should be established representing the academic community (institutions, academic staff, students) with the mandate to advise the AHELO Governing Board. The Board should also reflect on the best ways to incorporate the voices of employers and other stakeholders into its work. 

 

 Some criticism in Media

 

The proposed work of the AHELO Main Study has been featured in the media recently and sometimes even in a negative light. 

Some issues were raised in a joint letter from the American Council on Education ACE and Universities Canada about the AHELO Main Study.  

The reply from the OECD's Secretary-General can be read here.

 

 Find Out More 

 

 Contact us

Contact Dirk.vandamme@oecd.org  if you have any questions.  

 

Last updated: July 2015