DonNTU Master’s Portal ACS Department

• Autobiography
• Abstract
• Individual task

Abstract

The general formulation of the problem. The knowlege control is an important part of the learning process, which gives a comprehensive assessment of the level of knowledge of trainees. The testing is one of the well-established forms of knowledge control. Testing has a lot of certain deficiencies, however, is the only truly technological means for measuring the level of knowledge and indispensable as a tool to properly organize the management of educational process and to ensure effective pedagogical control.

There is a certain variety of types of tests: closed (multi-alternative and one-alternative) open, for establishing a match between the elements, for establishing the correct sequence, situational tests. [2] The knowledge assessment is one of the difficult and controversial issues during testing. To effectively solve this problem, use polytomic rating system, which allows several categories of response to the task, each of which is estimated differently. [4]

Research in the field of building knowledge control systems have shown the need to separate tasks into complexity levels. Lack of segregation of the tasks on the levels of complexity leads to a lack of objectivity of knowledge evaluation and often do not correlate with the true knowledge level of the students[4]. The distribution of the taskts by level of complexity by teacher makes a process of estimating the knowledge of trainees subjective, because it is not any task is easy for a teacher is easy for students. Thus, developing the technology for distribution of tasks into the levels complexity is still being[1].

Problem investigation.

Objective: develop a system of testing students' knowledge in the field of computer technology; implement, test and optimize the software and technology of the adaptive computer-based testing.

Object of study: training students in computer technologies; creating adaptive test items system.

Subject of research: creation and use of adaptive technology computer-based testing in the training of students in computer technologies.

The decision of the problem and research results. To create a test sequence tasks of the different type should be included. To calculate the objective evaluation of tests of different types for each of them specialized approach is used. For this we introduce the estimation coefficient r_i. Here are the types of test questions and evaluation system.

1. One-alternative tests, sequences, open test tasks

   r_i = 1 [the correct answer / sequence]

   r_i = 0 [wrong answer / sequence]

2. Multi-alternative test:

   , where Q₁ – the set of all correct answers in the task, Q₂ – the number of correct answers selected by the student, Q₃ – number of incorrect answers selected by the student.

3. Tests for establishing a match between the elements

   , where Q₁ – number of pairs for comparison, Q₂ – number of correct pairings.

4. Open test tasks (tableы)

   , where Q₁ – number of cells in the table, Q₂ – number of cells, which the student has filled correctly.

5. Multistep test tasks (one-alternative, sequences)

   , where i – step number, m_i – the number of errors on the i-th step, n – number of steps

6. Multistep test tasks (correspondence)

   , where j – number of attempts to pass a step, if it was a mistake, Q_2ij – the number of pairs for compilation on the i-th step in the j-th attempt, Q_1ij – the number of correct pairings on the i-th step in the j-th attempt.

7. Multistep test tasks (multialternative)

   , where Q_1ij – the number of correct answers on the i-th step when j-th attempt, Q_2ij – amount of correct answers selected by the student in the i-th step, Q_3ij – number of wrong answers chosen on the i-th step in the j-th attempt

Final performance of the test R, consisting of a set of test items containing Z levels of difficulty, determined by the formula:

, where B – numper of the points at the system, N – number of tests included in the task, z_i – difficulty level i-th test, – the total complexity of the test, which is determined by the formula:

IRT(Item Response Theory) – the mathematical theory of assessing the quality of tests and calculations of the knowledge level of the students. The focus of the study IRT are not tests as system tasks, but individual tests.

The basic mathematical model of IRT is a one-parameter logistic function Rash. This model introduces the concept of probability P, to do correctly the j-th test by the i-th student. It depends of student previous answer and task properties. Thus the function of the success is the probability P depended by measured properties of the subject and the difficulty level of the test. [1] In the one-parameter logistic Rash's model this probability is determined by the difference in levels of student knowlege and dificulty level of the test и определяется по формуле:

Later it was proved that the tests have different differentiating ability. Then, in accordance with the Birnbaum logistic model the probability of success is , where d_j – test's differentiating ability. xclude from the test sequence of tasks with low differentiating ability.

It is also the logistic three-parameter model with the function of success (three-parameter Birnbaum model) . Parameters of the three-parameter functions can be found using the maximum likelihood method through a series of iterations, at the same time as the initial approximation is advisable to use values and , obtained from the solutions of the equations for the one-parameter model, and as d_i и d_j – values ≈1,41, which gives the value at which the Birnbaum model degenerates into Rash model.

For the distribution of tasks in difficulty levels a modification of the modern theory test IRT is presented. Modification of IRT is the following:

• in a classic case the IRT is proposed for calculating the knowledge level of the students. In this modification to determine the difficulty levels of tests IRT and genetic algorithms will be used.

• usage of IRT models for the continuous knowledge assessment system;

• one-, two-and three-parameter model will be applied not separately but in combination, depending on the type of test tasks.

The problem of distribution of tasks in the complexity levels is reduced to determining the complexity of tests, from the experimental pre-test data. The disadvantage of the IRT is the presence of three models, each of which is separately applied for the test tasks specific type. [2] Thus, when attempting to use the model to different types of tests is reduced accuracy of calculation of parameters.

The simultaneous use of one-, two-and three-parameter model reduces to the integrated functional model, calculated on the evaluation of tests on a continuous scale evaluation of knowledge, including different types of test tasks:

, where – parameter, determines the complexity of the j-th test task, – modified one-, two-and three-parameter model, built on the continuous system, – parameter that determines the level of knowledge of i-th student, res_ij – variable, which corresponds to the result of the test task and taking values on [0,1], corresponding to a continuous scale evaluation of knowledge, a_j – parameter characteristics of the differentiating ability jobs, c_j – parameter characterizing the possibility of the correct answer to the j-th task in the event that this response was guessed.[1]

The advantage of an integrated functional model is its ability to simultaneously analyze all the test tasks discussed above types. The technology of integrated functional model is as follows. First determines the type of test task T_i. Then, depending on the type, estimates are for the answers res_j, and for the closed test tasks type additionally calculates the probability of guessing the correct answer c_j. [1]

Then collect statistics on the results of testing. The initial level of their knowledge , the initial difficulty level of test tasks calculated after a preliminary test of the students. Then calculated the differentiating ability of the test tasks[4]:

, , , where p_i – percentage of correct answers received from the i-th student, q_i – proportion of incorrect answers received from the i-th student, p_j – percentage of correct answers received for the implementation of the j-th test task, q_j – percentage of incorrect answers received for the execution of the j-th test task, r_b – biserial correlation coefficient.

Then tasks are distributed over the complexity levels based on latency analysis. Depending on the type of test the distribution of responses to test tasks is performed and the initial values for the complexity levels for the three components (P₁, P₂, P₃).

The first component P1 "1-parameter model" is used to check the sign of the homogeneity difficulty level of tests on a homogeneous group of subjects. The group is a homogeneous, if most of the values located on a narrow range of the axis of latent variable . Homogeneity test is a system of tasks increasing difficulties specific form and specific content. Such a test sequence created for the purpose of an objective, qualitative and efficient method for assessing the structure and measuring the preparedness level of students for one academic discipline.[1]

In the case of a heterogeneous sample, the knowledge parameter constraints should cover a greater range of the axis , and characteristic curves of tasks can be placed far enough from each other.[1]

Parameter characteristics differentiating ability of the task а_j is introduced to increase the accuracy of measuring the difficulty level of tests. This parameter is related to the steep curve of jobs at its inflection. For values а_j, are close to zero, tests shall cease to separate students by level of complexity. The number of tasks in the test should be reduced primarily due to the removal of such assignments, which increases the reliability and validity of the test.

Component P₂ "2-parameter model" is used to determine the level of complexity а_j of tests of different types, excluding close one. For tests with tasks of the closed type there is substantial deviation of the empirical data from the theoretical curve, which predicts the probability of correct assignment for different values of the variable To solve this problem we introduce a parameter c_j,which characterizes the opportunity of the properly respond to the task j in the case the answer is guessed. [3]

Ratings res_j for the tasks of the closed type transmitted to the third component P₃ "3-parameter model"[4]. Fisher method of maximum likelihood determinates the stable values of the complexity levels of test tasks. It's adapted to the estimation of the student knowledge for a continuous system:

, where L_i – probabilistic model of performance tests for the i-th student, – vector characterizing the result of the i-th student for n test tasks, P_ij – the probability of correct execution the j-th test by the i-th student, Q_ij – the probability of incorrect execution the j-th test by the i-th student, Q_ij = 1 – P_ij.

Thus, the output parameters of the integrated functional model is a set of parameters , which correspond to the estimated sustainable level of difficulty of test tasks.[1]

Conclusions. The main goals of adaptive testing are:

1. striving to improve test measurements, which includes reducing the number of tasks, time, cost of testing

2. to improve the accuracy of the results

3. to increase motivation for testing for weak and strong students

That is what formed the basis of advantages over fixed-length tests. The researchers saw the possibility of increasing the efficiency in the adaptation of tests, which difficulty takes into account a range of readiness testing. In computer adaptive testing test tasks are formed individually for each student in the light of his answers to previous questions. Types of tasks, their number and sequence are individual.

Thus, adaptive testing:

1. gives a more objective assessment of knowledge and skills of trainees;

2. allows you to identify what knowledge is erroneous or incomplete;

3. allows you to make recommendations for further construction of the educational process.

With the development of the theory of adaptive testing has become possible not only to adapt test tasks, but testing systems designed to:

1. adaptation to the subject area selected for testing;

2. adaptation to the current needs of a particular subject;

3. adaptation to the current state of a particular student.

References

1. М.Ф.Бондаренко, В.В.Семенец, Н.В.Белоус, И.В.Куцевич, И.А.Белоус «Оценивание тестовых заданий разных типов и определение их уровня сложности», 2009, «Искусственный интеллект» / Национальная библиотека Украины им. В.И.Вернадского [Electronic resource] – Access mode: http://www.nbuv.gov.ua/portal/natural/ii/2009_4/7%5C00_Bondarenko_Semenets_Belous_Kutsevich_Belous.pdf

2. В.В. Кромер, «Об одной трехпараметрической модели тестирования» / Cornell University Library [Electronic resource] – Access mode: http://arxiv1.library.cornell.edu/ftp/cs/papers/0506/0506057.pdf

3. Использование Теории тестовых заданий (Item Response Theory) в адаптивном тестировании. / Викизнание [Electronic resource] – Access mode: http://www.wikiznanie.ru/ru-wz/index.php/Использование_Теории_тестовых_заданий_(Item_Response_Theory)_в_адаптивном_тестировании

4. А.П.Попов «Критический анализ параметрических моделей Раша и Бирнбаума» / Ростовский государственный педагогический университет [Electronic resource] – Access mode:http://www.ast-centre.ru/books/favorits/273/

5. А.А. Проскурнин «Автоматизированная система контроля знаний» / Ростовский государственный педагогический университет [Electronic resource] – Access mode: http://www.philippovich.ru/Projects/RFFI/almanah/Proskurnin1.pdf

6. П.І.Федорук, «Використання адаптивних тестів в інтелектуальних системах контролю знань», 2008, «Искусственный интеллект» Национальная библиотека Украины им. В.И.Вернадского [Electronic resource] – Access mode: http://www.nbuv.gov.ua/portal/natural/II/2008_3/JournalAI_2008_3/Razdel5/03_Fedoruk.pdf

7. Л.В.Сметанюк, Г.М.Кравцов «К теории и практике использования адаптивных тестов» [Electronic resource] – Access mode: http://ite.ksu.ks.ua/?q=en/node/393

Important. When writing this abstract the master’s qualification work is not completed. Date of final completion of work: December, 1, 2010. Full text of the work and materials on a work theme can be received from the author or his scientific supervisor after that date.

• Autobiography
• Abstract
• Individual task

• DonNTU
• Master’s Portal
• ACS Department