Synopsis

Content

INTRODUCTION

1 HOT TOPICS

2 RELATIONSHIP WORK WITH SCIENTIFIC PROGRAMS, PLANS, THEMES

3 PURPOSE AND OBJECTIVES RAZRABOKI AND RESEARCH

4 SCIENTIFIC NOVELTY

5 PRACTICAL SIGNIFICANCE OF RESULTS

6 REVIEW OF RESEARCH AND DEVELOPMENT RELATED

7 MATHEMATICAL FORMULATION

CONCLUSION

REFERENCES

INTRODUCTION

Currently, assessment of knowledge is most often used standard scheme an individual evaluation by peer review, in which, assessment conducted by the expert or group of experts.

This approach has serious drawbacks:

1. subjectivism, which consists in the fact that different teachers may have different abilities to evaluate the same student;
2. lack of broad-scale assessment;
3. "locality" assessment, which makes sense only within a small group evaluated;
4. the complexity of mass testing;
5. task, as a rule, does not cover the whole subject, that does not assess the actual knowledge of the subject.

In connection with this is a very urgent task of evaluating the results of testing as a method of objective assessment.

One advantage of testing is the high formalism of this method, and, consequently, the possibility of automation to reduce labor intensity and improve the quality of knowledge evaluation. There are adaptive and non-adaptive methods of control knowledge [1]. In non-adaptive methods in the verification process, all students are the same, predetermined sequence of frames of verification tasks, which does not depend on the actions of the trainee during the control.

Adaptive methods of maximum use data from a model student (eg, the level of preparedness of the student, the level of anxiety, alarm, correct answer, etc.) and / or model of learning material (eg, the relationship between verifiable concepts).

Tests by its form may be of several types:

• choice;
• for compliance;
• the ranking;
• for design;
• situational.

Despite this diversity of forms of the test tasks in automated testing systems often use simple algorithms for formation of the final grade: additive algorithms and additive algorithms with penalty points [7]. In this regard, I consider it necessary to consider the possibility of applying fuzzy logikidlya implementation of expert system assessment.

1 RELEVANCE

Monitoring of students' knowledge can be implemented using different methods of formation evaluation. There are methods for assessing knowledge using models that take into account only the correct responses of students, and models that take into account the parameters of tasks and level of learning.

Despite the fairly large number of papers on the topic [1, 4, 5, 7], we can identify some common drawbacks of modern methods of automatic knowledge assessment on the results of testing:

• use of a methodology for the test, which reduces the possibility of testing;
• rigid procedures of calculation of the final grade, due to the use of methods that use algorithm of accumulation points, ranking methods, methods of promoting and fines;
• gh complexity of building high-performance tests, or the reduction procedure of forming the dough to a random selection of questions;
• use of pre-generated tests, which precludes the use of adaptive methods of testing or making them flexible enough.

2 RELATIONSHIP WORK WITH SCIENTIFIC PROGRAMS, PLANS, THEMES

This work was carried out during 2009-2010. in accordance with the scientific direction of the Department of Automated Control Systems of Donetsk National Technical University.

3 PURPOSE AND OBJECTIVES RAZRABOKI AND RESEARCH

The purpose of the system under development is to reduce the complexity of compiling adaptive tests by introducing a system of automatic test generation, raising the quality of estimation; reduce complexity by automating the process of checking test results and evaluation of nominations.

Purpose of the system - adequate evaluation expertise person undergoing testing in an adaptive complex testing on a given subject area.

Objectives:

• develop an expert system knowledge assessment, which will allow you to automatically generate tests;
• testing;
• exhibit an adequate assessment of the high degree of accuracy;
• generate explanations exhibited assessment and recommendations on deepening knowledge in a particular field;
• make recommendations to the developers of tests on the quality of tests and test a whole [2].

4 SCIENTIFIC NOVELTY

Scientific novelty of the research is to use fuzzy logikidlya implementation of expert system assessment. A new approach to creating adaptive tests and evaluation of exhibiting a high degree of accuracy.

5 PRACTICAL SIGNIFICANCE OF RESULTS

The practical significance of the research is determined by the fact that an improved method of creating adaptive tests and evaluation of exhibiting a high degree of accuracy is of interest for expert systems courses.

6 REVIEW OF RESEARCH AND DEVELOPMENT RELATED

Developed in the West, the theory of creating tests (Item Response Theory - IRT [3, 5]) designed to assess the latent (hidden) parameters of the test and test questions. The basic principle of IRT is to establish a plausible link between the observed test scores and latent subject parameters and test questions [4]. This relationship is expressed as:

(1)
where xij - element of the matrix of responses equal to 1 if the response from the i-th subject at the j-th job right, 0 - otherwise;
?i - level of the i-th subject, i = 1 .. N;
?i - the difficulty of the j-th job, j = 1 .. n;
f - the logistic function, which depends on the chosen model IRT (see below).

Among the IRT models are distinguished one-parameter Rasch model, two-parameter model of Birnbaum, the three-parameter model of Birnbaum. As a model for the processing of the results of my work was chosen as the basic three-parameter model of Birnbaum, since it apart from all of the above, also reflects the probability of guessing the answer to the test task. This ratio shows how easily subjects can guess the correct answer, based on the wording of the job, not having the necessary knowledge. Such a situation may arise, for example, with illiterate selection of distracters (answers) on the job closed. Dependence obeys the following formula:

(2)
where cj - the probability of guessing.

Regardless of the choice model, are also taught the principles of interaction of different parameters with each other. Therefore, I consider it necessary to create an expert system that will calculate various parameters and to give clear guidance on the quality of the test.

7 MATHEMATICAL FORMULATION

The verification system based on the knowledge test is implemented using fuzzy logic to assess the knowledge test.

The basis of the expert system assessment of knowledge, is a subsystem of intellectual assessment of knowledge based on fuzzy logic. The system is implemented in two levels. The first level evaluates the student's knowledge on each subject separately, on the second level is formed by the final score. Consider in detail the implementation of this subsystem.

Table 1 – Factors assessment of knowledge

№	Name	Description
1	Level of practical knowledge	In fact, the level of training the student on the basis of laboratory work. Put up a teacher (expert).
2	Attendance	Is determined on the basis of the log of visits.
3	Knowledge topics	Integral assessment of knowledge of the subject on a particular topic. Is the output linguistic variable of the first level of expert system assessment.

* For each topic the course created their own linguistic variable.

Linguistic variable "score" - an output variable EC.

Linguistic variable is given a term-set T, the universal set X, the set of syntactic modifiers of G, the membership function F. The set of terms for linguistic variable "level of practical knowledge" is given in Table 2.

Table 2 – The terms for linguistic variable «level of practical knowledge»

№	Name	Criterion
Low	0 .. 3	When the delivery of laboratory work the student performed the necessary minimum requirements.
Average	3 .. 6	When the delivery of laboratory work the student performed the minimum required, good answer questions.
High	6 .. 9	In preparation for laboratory work, students spend an analysis of the considered problem. Answers to additional questions were concise and precise.
Elevated	9 .. 12	In preparation for laboratory work, students spend an analysis of the considered problem. The solution of the tasks was not only loyal, but also original.

The set of terms for linguistic variable "Traffic" is given in Table 3.

Table 3 – The terms for linguistic variable «Traffic»

Level	Scale	Criterion
Bad	0 .. 4	Missing more than 30% of classes.
Average	4 .. 8	Absences ranged from 10 to 30%.
Good	8 .. 12	Missing at least 10% of classes.

The set of terms for linguistic variable "knowledge issues" is given in Table 4.

Table 4 – The terms for linguistic variable «Knowledge of the theme»

Level	Scale	Criterion
Bad	1 .. 2	Значение лингвистических переменных этого типа формируется на первом уровне экспертной системы.
Average	2 .. 3
Good	3 .. 4
Excellent	4 .. 5

To generate fuzzy variables on the basis of linguistic variables using triangular membership function whose value at point X is calculated by formula (2).

(2)
where a, b, and c - left boundary, the maximum point and the right border of the membership function, respectively.

To form the final evaluation, which is the output linguistic variable, applies a knowledge base, which is represented as a set of products and formed a teacher (expert) for each discipline. With the mechanism of production, which is part of the fuzzy inference, the teacher is able to specify the relationship between the studied subjects, the differential approach to the evaluation of attendance, specify the significance level of each topic of the course.

For the final formation evaluation algorithm used fuzzy inference Mamdani.

As mentioned above, the system allows you to generate test tasks of varying complexity. For automatic generation of tests used genetic algorithm presented in Figure 2.

Figure 1. Genetic algoritm which forms the tests. Animation consists of 18 frames with a delay of 50ms between frames; the delay for the replay is 1s, the number of cycles of reproduction is not limited, the volume 47,5kB.

As the fitness function of genetic algorithm was chosen three-parameter model of Birnbaum, since it apart from all of the above, also takes into account the probability of guessing the answer, based on the wording of the job. Such a situation may arise, for example, illiterate selection of distracters (answers) on the job closed. Dependence obeys the following formula:

(3)
where cj - the probability of guessing.

Regardless of the choice model, we study the principles of interaction of different parameters with each other.

In the three-parameter model of Birnbaum probability of correct (resp. incorrect (5) solutions of test tasks are:

(4)

(5)

Let the test contains n jobs. We assume known not only difficult tasks ?1, ?2 ,..., ?n, and differentiating features of all jobs ?1, ?2 ,..., ?n. Keeping the previous notation for the characteristic function. Then the logarithmic likelihood function is:

(6)

A necessary condition for a maximum of function (6) leads to the equation:

(7)
of which must be determined by the level of preparedness of the subject.

The model is estimated Birnbaum quality and complexity of the test.

CONCLUSION

Developed system is designed to improve the quality of evaluation of students' knowledge, to formalize and automate the method of formation of adaptive tests with related tests, to evaluate the quality of the generated system tests, provide an opportunity for knowledge assessment in general, the discipline, as well as on selected topics in particular.

The analysis of existing methods to solve this problem the decision to use adaptive testing as the most accurate and comprehensive evaluation focused on the knowledge test in conjunction with an expert system based on the theory of fuzzy logic.

For automatic generation of adaptive tests is proposed to use the apparatus of genetic algorithms Expert system for knowledge on the test results will reduce the complexity of the current and final control of knowledge among students. Allow for multivariate analysis of students' progress on various topics of the course. Will provide an opportunity to identify topics, the assimilation of which caused the greatest difficulties.

REFERENCES

1. Модели и методы адаптивного контроля знаний [Электронный ресурс] / Л. В.Зайцева, Н. О. Прокофьева // Educational Technology & Society – 2004. – № 7 (4). [Электронный ресурс]. Режим доступа: URL: http://ifets.ieee.org/russian/depository/v7_i4/html/1.html.
2. Минин М.Г. Диагностика качества знаний и компьютерные технологии обучения. Томск: Изд-во ТГПУ,2000. 216 с.
3. Челышкова М.Б. Теория и практика конструирования педагогических тестов: Учебное пособие. – М.: Логос, 2002. – 432 с.: ил.
4. Rash G. Probabilistick Models for Some Intelligence and Attainment Tests, 1960, Copenhagen, Denmark: Danish Institute for Educational Re-search.
5. Wright B.D., Stone M.H. Best Test Design. Mesa Press. 1979
6. Экспертные системы: структура и классификация. [Электронный ресурс]. Режим доступа: URL: http://www.ssti.ru
7. Мухамедиев Р.И., Ограниченность одноуровневых аддитивных моделей оценивания, «Вестник СумДУ», №4(88) 2006, 17-23с.
8. Аванесов В.С. Математические модели педагогического измерения. – М.: Б.и., 1994. – 26с.
9. Нейман Ю.М., Хлебников В.А. Введение в теорию моделирования и параметризации педагогических тестов. – М : Прометей, 2000. – 169 с.
10. Челышкова М.Б. Применение математических моделей для разработки педагогических тестов. – Учебное пособие. – М.: Исследовательский центр, 1995. – 48 c.
11. Батыршин И.З. Основные операции нечеткой логики и их обобщения. – Казань: Отечество, 2001. – 100 с.
12. Леоненков А.В. Нечеткое моделирование в среде MATLAB fuzzyTECH. – СПб.: БХВ-Петербург, 2005. – 736 с.: ил.
13. Скобцов Ю.А. Основы эволюционных вычислений: учебное пособие для вузов/ Ю.А. Скобцов. – Донецкий нац. техн. ун-т. – Донецк: ДонНТУ, 2008.
14. Рутковская Д. Нейронные сети, генетические алгоритмы и нечеткие системы / Д. Рутковская, М. Пилиньский, Л. Рутковский; пер. с польск. И.Д. Рудинский – М.: Горячая линия-Телеком, 2006. – 452 с.
15. Кофман А. Введение в теорию нечетких множеств: Пер с франц. – М.: Радио и связь, 1982. – 432 с., ил.