How important is the alignment of the learning outcome teaching learning activities and assessment tasks?
The Bologna process and the implementation of the European Higher Education Area (EHEA, 2014) brought a need for a substantive change in the pedagogical model in higher education, requiring improvement of the traditional ways of describing qualifications and qualification structures, modules and programmes. The focus moved from the mere accumulation of knowledge to skills acquisition by students. The result is a shift from a content-based approach to a learning-centred approach, a change from teaching objectives to students’ learning outcomes. Learning outcomes are becoming fundamental for structuring the standards and guidelines of quality assessment in higher education and continuing education institutions in Europe and worldwide. In this context, the assessment of learning outcomes becomes a crucial process in the education system. Show
With information and communication technologies increasingly used in education, new opportunities for improving teaching, learning and assessment are arising. Time to Assess Learning Outcomes in E-learning – TALOE (http://taloe.up.pt), is a project financed by the European Commission from January 2014 to December 2015 (Ref. 543097-LLP-1-2013-1-PT-KA3-KA3MP), which intends to promote the internal consistency of online courses. The main goal of the international project TALOE is to develop a web-based tool to help teachers and trainers decide on the e-assessment methods to use in their online courses. To be able to develop the practical tool, the TALOE consortiumFootnote 1 has set and worked on the following specific tasks:
This paper focuses on tasks 2 and 3, i.e. development and testing of the TALOE web-tool. The first task, selection and analysis of innovative e-assessment practices, was completed during 2014 and has been addressed in previous publications (Lõssenko, 2014; Gil-Jaurena et al. 2015). It consisted on the collection of 18 cases of assessment practice in online courses from different institutions around Europe, mostly from the project partners. They were selected according to a set of criteria, built upon the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ENQA, 2009) and some principles agreed within the project partnership that the selected e-assessment practices had to meet:
Those principles refer to “authentic” assessment with a learning-oriented approach (Rodríguez-Gómez & Ibarra-Saiz, 2015). The goal of this selection of case studies was to obtain examples that can be used as a showcase of current practice and also as testing material during the second year of the project (as will be explained in a later section). It as well enabled the insight how much teachers acquainted with learning outcomes and about the relation between learning outcomes and assessment methods. This selection also provided information about assessment methods that are usually used in online courses. The case studies are written on predefined templates and detail information about the learning outcomes, course content, teaching practice and assessment methods. In addition, a separate section was developed to linking the assessment to the selection criteria. Theoretical framework: the ALOA modelNot all assessment methods are valid for each type of learning outcomes. The TALOE project uses an existing framework called the ALOA model (Aligning Learning Outcomes and Assessment) (Falcão 2013). This model highlights the connection between the intended learning outcomes and the assessment strategy used during a course. So, it provides tools for linking learning outcomes and assessment tasks. The ALOA model was developed by Falcão (2013) from the concept of alignment (Anderson & Krathwohl, 2001; Biggs & Tang 2007; Bloom, 1956; Boud & Falchikov, 2006), which means that the learning outcomes of a course should be used to define the teaching and learning activities and the assessment tasks. The main principle that supports the ALOA model is that, in order to ensure the validity of assessment in relation to what is intended from a course, it is necessary that the outcomes measured by the assessment tasks are the same as the ones expressed in the learning outcomes. In this sense, the ALOA model matches the basis of learning-oriented assessment (Carless, 2007), “an attempt to emphasise the learning features of assessment and promote their development” (p. 58). The ALOA model uses the revised version of Bloom’s Taxonomy of the cognitive domain (Anderson & Krathwohl, 2001) to establish the link between the learning outcomes and general assessment methods. Thus, the model has three components:
Fig. 1 The cognitive process dimension, revised Bloom’s taxonomy. Adapted from Anderson and Krathwohl (2001), 67–68. Source: Iowa State University (2015). http://www.celt.iastate.edu/teaching-resources/effective-practice/revised-blooms-taxonomy/ Full size image The TALOE project intends to apply the ALOA model to the specific context of e-learning. Development and description of the TALOE web toolThe TALOE web tool (available at http://taloetool.up.pt) is being developed to help teachers and educators (users) decide on the assessment strategies that will be used in their online courses. It provides users with a possibility to analyse the provided courses’ or modules’ learning outcomes, and offer the most appropriate e-assessment methods consistent with the intended learning outcomes. The tool is envisioned to be used by teachers/faculty/trainers, either to check if the existing assessment methods of their course or module are consistent with the stated learning outcomes, or to be advised on the most appropriate assessment methods for the existing or new course or module. The TALOE project partnership produced a first version of the web-tool at the beginning of 2015. At that point it was decided by project partners to simplify the tool procedures during the first phase of testing due to the complexity of the ALOA model (Falcão 2013). The present version of the TALOE web tool does not discriminate between knowledge types, as suggested by the theoretical model. After initial testing was completed (phase 1, explained in the coming section), it has been confirmed that the setup matrix is working properly. The best (most appropriate) e-assessment methods are selected on the base of the absolute matches between input (learning outcome) and the e-assessment methods. The welcome page of the TALOE web tool (Fig. 2) explains how to use it. Fig. 2 First page of the TALOE web tool Full size image In the first step (Fig. 3), the user describes the learning outcome that students should achieve in the course or module. The learning outcome should be described in a clear way and kept simple. For example: Apply algorithms based on prime numbers on practical problems. Fig. 3 Describing the learning outcome in the TALOE web tool Full size image Basic information about writing learning outcomes and the ALOA model are available within the TALOE web tool. After describing a learning outcome, in the Step 2 the user chooses up to three verbs that best describe it (Fig. 3). This step also enables the user to check and review the stated learning outcome. After describing the learning outcome with verbs, the user sets the process in motion and receives an assessment advice for the defined learning outcome (Fig. 4). The received outcome provides a list of potential e-assessment methods to use, with the corresponding description. Fig. 4 The TALOE web tool page with results Full size image A specific section in the TALOE website provides information and examples about assessment methods. The decision engine consists of the estimation of the score that measures the best match between the cognitive processes submitted by the user and the specific ones of a given assessment method. The assessment methods are selected on the basis of score, calculated as the ratio between the number of matches between input, and the model assessment over the number of the selection of the assessment method. Based on the received inputs, the tool calculates the best assessment methods for the defined learning outcomes. For each suggested assessment method, a description is provided to give users a better idea on how to use and integrate it in the course. Therefore, the TALOE web tool can be used in two ways: to check if the existing assessment methods in a existing course are in line with the stated learning outcomes; and to help users decide on the most appropriate assessment method for a new course or module. The TALOE web tool can be used to learn more about learning outcomes, assessment and e-assessment, as it provides guidance to teachers to formulate the learning outcomes in accordance to Bloom’s revised taxonomy (Anderson & Krathwohl, 2001). Testing of the web toolOne of the commitments and tasks of the project was to test the implementation of the web tool. In this section, we present both the testing method and the results. MethodThe dimensions considered in the evaluation of the tool were:
These criteria were agreed among project partners after some working sessions (face-to-face in the project meeting that took place at the University of Padova (Italy) in February 2015, and virtually in the audio meetings we have held monthly). The instrument used was an evaluation form, which included nine close-ended questions (Likert scale) and three open-ended questions. It was first piloted within the project consortium (phase 1); annex 1 includes the evaluation form used in that first phase. Based on the results of phase 1, the web tool was improved and converted into a web form (phase 2). We used Google forms to collect the information. The structure of the evaluation form was the following:
The evaluation of the TALOE web tool was done in two phases.
The participants in the online testing could remain anonymous, as demographical questions were optional, except the question about the home institution. Nevertheless, the majority of participants left their contact and wanted to receive further information about the TALOE project and web tool. This was important since it reveals the relevance of the TALOE project results to the stakeholders and helps the Project Consortium build a database of stakeholders for dissemination and exploitation activities. Sample descriptionThe profile of the 42 respondents (i.e., stakeholders invited by the project partners) to the evaluation form in phase 2 is presented in Figs. 5, 6, 7, and 8. Fig. 5 TALOE web tool testers’ scientific field Full size image Fig. 6 TALOE web tool testers’ course level Full size image Fig. 7 TALOE web tool testers’ country Full size image Fig. 8 TALOE web tool testers’ academic profile Full size image Different scientific fields have been represented among the web tool testers: 38 % come from the field of Education, followed by 19 % from Computer Science (Fig. 5). Twelve percent come from Humanities and another 12 % from Social Sciences. Seven percent work in Health Studies and another 7 % in Mathematics. More than 92 % have tested the TALOE web tool in relation to a higher education level course (Fig. 6). Forty-five percent have tested it in a Bachelor or Undergraduate level course; 31 % in a Master or Postgraduate level course; 14 % in a continuing education level course. Testers of the web tool work in 11 different countries. Most respondents are from Hungary (24 %), Spain (21 %) and Portugal (14 %). All the project partners’ countries are represented in the sample, plus some stakeholders from other regions not directly involved in the project, such as Germany, Ireland, Finland and Serbia (Fig. 7). Almost 80 % are professors and/or researchers in a higher education institution (Fig. 8). Thus, they have direct and practical experience in defining learning outcomes and deciding about assessment methods in higher education. Besides this demographic information that was included in the feedback survey, given the home institution we can state that most respondents (specially those in higher education) have experience in e-learning: some of them work in distance education universities and others use a blended learning mode. In the next section we present the findings related to each dimension considered in the evaluation of the tool, as explained in “method” section. Research findingsThe testing done in the first phase (with the TALOE case studies) showed that the majority of the teachers have defined the appropriate e-assessment methods for the learning outcomes. In some cases, the TALOE web tool suggested additional possibilities in terms of e-assessment methods. For several cases the TALOE web tool indicated that the existing assessment methods should be revised. The analysis of received feedback in the second phase (through the web form) from 42 participants is presented in Figs. 9, 10, 11, 12, 13, 14, 15, 16, and 17 and explained in this section. The legend in those figures ranges from 1 (totally disagree) to 5 (totally agree). In the interpretation of results, for calculating percentages we have added replies 1 and 2 as disagreement and replies 4 and 5 as agreement. We have considered reply 3 as neutral. Fig. 9 Easiness of the TALOE web tool Full size image Fig. 10 Friendliness of the TALOE web tool Full size image Fig. 11 Usefulness of the TALOE web tool Full size image Fig. 12 Skilfulness of the TALOE web tool Full size image Fig. 13 Quality of the TALOE web tool results Full size image Fig. 14 Agreement of the TALOE web tool results with previous assessment plans Full size image Fig. 15 Validity of the TALOE web tool Full size image Fig. 16 Help provided by the TALOE web tool Full size image Fig. 17 Recommendation of the TALOE web tool Full size image The majority of participants found TALOE web tool easy to use (Fig. 9). This is an important feature as it influences teacher’s attitude towards the use of the web tool. Answers to the question about use of the web tool without written instructions are distributed along all ranges (Fig. 10). The majority of participants (64 %) did not need any written instruction to use the tool. This indicates that the tool is well designed, but also that participants may be experienced in defining learning outcomes and familiar with the revised Bloom taxonomy. Nineteen percent of participants needed additional instructions to be able to use the tool. Although some participants needed instructions to use the tool (Fig. 10), the majority of respondents found it is very useful (Fig. 11), which is very relevant for the project. Some participants (17 %) did not have an opinion about the usefulness of the tool. This perception can depend on their knowledge and experience in defining learning outcomes and the extent to which they have been achieved. Those more experienced may find it not as useful as they are quite experienced in writing learning outcomes and linking them with teaching and assessment. Eighty-six percent of respondents found the web tool easy to use or become quickly skilful with it after first use or by reading the posted instructions (Fig. 12). This result is relevant for the project, as it compensates to same extent the difficulties that first users may experience (Fig. 10). Results provided by the TALOE web tool, i.e, recommendations about assessments methods for each learning outcome, are of good quality for most testers (Fig. 13). Some participants (26 %) could not conclude about the quality of the received results. The majority of participants (81 %) found that received results are in agreement with their own assessment plans (Fig. 14). Only 5 % of respondents found that the assessment methods recommended by he TALOE web tool don’t match their methods. This is an interesting result that would require further research: at first sight it suggests that professors are correctly aligning their learning outcomes and assessment methods, and that the TALOE web tool identifies this alignment. In relation to the previous question (Fig. 14), most respondents state that the tool helps them to better align learning outcomes and assessment methods (Fig. 15). It could mean that the tool reinforces the assessment plan they were already using (given that there is a high match between the recommendation provided by the tool and the existing plans, Fig. 14). Almost 30 % of respondents don’t have a clear opinion about this question. The majority of participants (76 %) find the TALOE web tool can help them to understand how to better define the assessment methods in their courses (Fig. 16). Finally, the majority of participants would recommend the TALOE web tool to a friend; still, there are some of them (14 %) who are not sure about it (Fig. 17). The information collected from the stakeholders in the open-ended questions of the TALOE feedback form provides some clues about missing elements or interface recommendations that could help improve the tool. From the 42 respondents, 23 have submitted their comments about changes needed in the web tool, missing aspects and removable aspects. Regarding changes in the TALOE web tool procedure or interface, some relevant feedback is present in the following quotations:
About missing aspects in the web tool, we could highlight a demand for examples of use of the different assessment methods, expressed by the testers in the following terms:
There are not remarkable removable elements addressed by testers. In summary, the feedback shows a positive evaluation regarding the usability of the tool and the quality of the output/results, i.e. the e-assessment method suggestions. Invited users found this tool easy to use and useful, but would like to see it improved for more complex testing. These results also confirmed that the web tool provides support and guidance to teachers to formulate the learning outcomes in accordance to Bloom’s revised taxonomy, as intended and planned by the task. It also increases the accuracy of e-assessment methods received by the tool and the alignment between learning outcomes, assessment techniques and teaching methods. How important is to align the learning objectives teaching strategies and assessment tools?It is important to align learning objectives with instructional strategies and assessments to ensure that everyone involved is aware of the expectations. Both instructors and students should have a consistent understanding of what is going to be taught and how it will be evaluated.
Are the assessment tasks aligned with the learning outcomes?The teaching methods and the assessment are aligned to the learning activities designed to achieve the learning outcomes. Aligning the assessment with the learning outcomes means that students know how their achievements will be measured.
What is alignment of learning outcomes?Alignment refers to the way in which each of the elements of your course work together to support the intended learning outcomes.
Why is it important that learning targets and assessment task or activities are match?Without the alignment between learning objectives and learning activities/assessments, students will spend time on activities, assignments and assessments that do not lead to intended goals.
|