Читайте также:
|
Background reading:
1. Council of Europe 2009. Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR): A Manual. Language Policy Division, Strasbourg,2009 / www.coe.int/lang
L ecture STANDARD SETTING
LECTURE OUTLINE:
- Stages
- Methods
- concerns
‘ Standard’ means:
• a level of performance required or experienced
• a set of principles which can be used as a basis for evaluating what language testers do. Standards in this sense may lead to codification in an agreed set of guidelines or Code of Practice. Such codification indicates a concern to establish professional ethics. (Davis, 1999. Dictionary of language testing. Cambridge: Cambridge University Press, p. 185)
The issue of standard language is one of the most important in testing, because we should understand what language we are going to test. In other words, we should be aware of what should be considered the ‘standard English’ that is going to be tested. On the other hand, tests themselves impose ‘correct’ standards, i.e. the language that should be taught and learned. Standards are set both for test and testers and for test takers in terms of their performance. Performance standards are ‘ explicit definitions of what students must do to demonstrate proficiency at a specific level’ on a given aspect of a test’.
All of them were laid out in the Common European Framework of Reference for Languages has been set up with the aim to provide: “… a common basis for the elaboration of language syllabuses, curriculum guidelines, examinations, textbooks, etc. across Europe. It describes in a comprehensive way what language learners have to learn to do in order to use a language for communication and what knowledge and skills they have to develop so as to be able to act effectively. The description also covers the cultural context in which language is set. The Framework also defines levels of proficiency which allow learners’ progress to be measured at each stage of learning and on a life-long basis” (Council of Europe 2001a: 1).
But the CEFR is also specifically concerned with testing and examinations:
“One of the aims of the Framework is to help partners to describe the levels of proficiency required by existing standards, tests and examinations in order to facilitate comparisonsbetween different systems of qualifications. For this purpose the Descriptive Scheme and the Common Reference Levels have been developed. Between them they provide a conceptual grid which users can exploit to describe their system”
In the Common European Framework of Reference (CEFR) the levels of language proficiency are described with the help of “can-do” statements.
The “can do” statements refer to what learners are actually able to do. They are called “can do” statement because all of them start with the word “ can”.
The Common European Framework of Reference enables different examinations to be related to each other indirectly without any claim that two examinations are exactly equivalent. The Manual for Relating Language Examinations to the CEFR (see Background reading above, #2) specifies all the necessary steps to ensure validity of this linkage. The Manual identifies the stages in the process of relating language exams to the CEFR.
Relating an examination or test to the CEFR can best be seen as a process of “building an argument” (Slide 23) based on a theoretical rationale. The central concept within this process is “validity”. The Manual presents five inter-related sets of procedures that users are advised to follow in order to design a linking scheme in terms of self-contained, manageable activities:
· Familiarisation
· Specification
· Standardisation training/benchmarking
· Standard setting
· Validation.
Although they should not be seen as linear steps, the five sets of procedures follow a logical order. At all stages it is recommended that users start with the productive skills (speaking and writing) since these can be more directly related to the rich description in the CEFR, thus providing a clear basis for training, judgments and discussion.
Standard setting. ‘The crucial point in the process of linking examination to the CEFR is the establishment of the decision rule to allocate students to one of the CEFR levels on the basis of their performance in the examination. Usually this takes the form of deciding on cut-off scores, borderline performances’ (COE, 2009:11). The process of reaching a cut-off score is referred to as standard setting. A cut-off score is the border between the lowest acceptable score on a test to be assigned the relevant CEFR level and the highest score on that test to fail that level. For example, a cut-off score of 30 in indirect tests says that a score of 30 or more on the tests grants a level of a particular level (e.g. B1) or higher, while a lower score points to a level lower than the level of the cut-off score (here: B1).
Practically in all the methods the notion of a minimally accepted person is one of the basic notions. The participants are given the concept of the ‘minimally accepted person’ (MAP), also referred to sometimes as the ‘borderline person’ or a person ‘just barely passing’ or ‘minimally competent person’. Where a standard has to be set, for example, for CEFR level B1, a minimally acceptable person has the competences, skills and abilities to be labeled as ‘B1’, but only to such an extent that the slightest decrease in those competencies, skills and abilities would suffice in order not to grant this qualification
Standard-setting methods are divided into test-centered and examinee (learner)-centered methods.)
1. Test-based Methods
– Test-based methods -
- Angoff Method
- Yes/No Extension of Angoff
Considered from a purely formal viewpoint, one could say that to apply test-based methods, panel members need not to have any teaching or other experience with real students in the subject matter of the test, but in practice using such a panel might result in totally unacceptable standards. Even with experienced teachers, the task setting is quite abstract, and teachers usually find it quite difficult to give the required judgments.
Therefore, all variations on the method nowadays use several rounds and provide information about real students’
performances to moderate the standard setting. Yet, even with these provisions, the main focus of the method is on the
characteristics of the test, the qualification of the method as test centred remains justified.
Within the test-centred methods there is a group of tests, that is labelled IRT, meaning that panel members use a
summary of empirical data (usually provide via an IRT (item response analysis).. In these methods information derives
directly from the performance of a group of test takers
2. Test-based Methods (+IRT):
- Bookmark Method (IRT)
- Basket Method (IRT)
Learner-based (or examinee-based) methods differ from test-based methods in the sense that the panellists base their
judgements (almost exclusively) on the performances of real students on the test.
There are two basic requirements: (a) ‘test scores from a sample of students should be available and care must be taken that the sample of students should be representative o f the target population’, (b) ‘the students must be well- known by at least one of the panel members’. Thus, the panel consists of ‘the teachers of the sampled students and each student is well-known by … one of the panel members’ (COE, 2009:67).
3. Learner-based methods
- Borderline Group Method
- Contrasting Groups Method
· During and after the application of the procedures, one needs quality monitoring. The most important question is whether the results of the standard setting – allocating students to a CEFR level on the basis of their test score – is trustworthy, and the basic answer to this question comes from independent evidence which corroborates the results of a particular standard setting procedure. It is the task of everyone applying such a procedure to provide an answer to that question, and this is precisely what is meant by validation.
Дата добавления: 2015-09-11; просмотров: 422 | Поможем написать вашу работу | Нарушение авторских прав |
| <== предыдущая лекция | | | следующая лекция ==> |
| Junk Mail | | | Пробуждение. |