Русский
ДонНТУ   Master's portal

Content

Introduction

Improving the quality of management of the educational process is directly related to the introduction of new educational and information technologies, which increases the need for automated systems, including automated knowledge control systems. They allow you to quickly assess the knowledge of students (students, schoolchildren or even workers). Today, testing as one of the most effective and objective forms of knowledge testing is widely used both in the learning process and in systems for monitoring and assessing the quality of education in many countries of the world. It is also important to note that, since 2019, humanity has faced a global problem pushing the use of distance learning systems. This problem is the COVID-19 pandemic and the accompanying quarantines and self-isolation.

1. Relevance of the topic

Analysis of works in the field of creating testing systems (both theoretical aspects and practical implementations) shows that the automation of education has significantly improved test technologies for knowledge control and raised them to a qualitatively new level. The use of new information technologies, firstly, made it possible to automate the processing of information obtained as a result of testing, which made it possible to massively test and collect statistics, and secondly, led to the creation of automated knowledge assessment systems. Such systems make it possible not only to digitize traditional blank tests by automating the processes of forming a test and presenting test items, but also provide new opportunities for representing the subject area by adding expanded type items to tests.

2. Purpose and objectives of the study, planned results

The purpose of the master's thesis is to automate the formation of tasks in the testing system

The main objectives of the study:

  1. Analysis of existing testing systems.
  2. Investigation of ways to generate test items.
  3. Development of a subsystem for automating the formation of tasks for the testing system.

Object of study: knowledge testing systems.

Subject of study: formation of tasks for the testing system.

The scientific significance of the work consists in substantiating the advantages of the developed subsystem for automating the formation of tasks in the testing system.

To evaluate the theoretical results obtained, during experiments, and create the foundation for the following studies, as practical results, it is planned to develop a customizable and functional system for automating the formation of tasks with the following properties:

  1. Availability of a database (DB) storing the format of tests and their results.
  2. Teacher Mode
    • The ability to manage database data, namely tests, assessments.
    • The ability to form a set principle of random generation of tasks for tests with subsequent automatic evaluation.
    • Collection and delivery of detailed statistics.
  3. "Student" mode - test emulation to check the work of the task formation system.

3. Research and development overview

Given the importance of knowledge testing, many articles and works are devoted to highlighting the problems of this topic.

3.1 Review of international sources

The validity of educational tests when used as critical indicators of educational outcomes in a dynamic system is discussed by Allano Collins and John R.  Frederiksen [1]. They also develop a set of design principles for creating test systems that are systemically valid. Methods and devices for computerized assessment of a subject's performance against selected comparative norms include a home testing computer that transmits the test-taker's test results to a central computer that evaluates the performance of the test-taker. [2]. Performance assessment can be standardized or individual, and relative or absolute. In addition, the transmitted test scores are configured to reliably link the student to their test scores using encryption, user authentication, or confirmation methods to prevent fraud. [2]. Concrete examples of ways to measure student learning, and describes how to design, evaluate and interpret tests to ensure they are valid, reliable, and fair for all children reviewed [3]. In terms of self-assessment, the computerized adaptive test (CAT) is the right way to give students the opportunity to assess themselves. In CAT, students are assessed through a process that uses Question Answer Theory (QAT), a well-founded psychometric theory. In addition, testing requires a large bank of elements, but when the CAT system has a large bank of elements, the selection of test elements in the TOV becomes more tedious. In addition to a large bank of items, the mechanism of opening items is also important for the testing system. However, all TOV are deprived of the points mentioned above. These reasons prompted authors Aya-Min Huang, Yen-Ting Lin, Shu-Chen Cheng to conduct a study of CAT. [4]. After the system was officially posted online, several surveys and experiments were conducted to assess the usability, accuracy and efficiency of the system. The results of these evaluations showed that the system provides adaptive testing for different devices and supports universal evaluation functions. A data distribution system for use with a computerized testing system that is guaranteed to deliver at least one computerized test to at least one test subject and processing information related to each mentioned delivery, as well as the purpose of standardized testing, are given (with examples) by inventors Roger K. Kershaw By Frank J. Romano, Leonard S. Swanson, and William C. Ward Jr. in their work [5].

3.2 Review of local sources

At the Donetsk National Technical University (Department of Computer Engineering), knowledge assessment systems, as well as approaches to them, are widely studied. Analyzes of the most relevant and widely used automated testing systems such as PikaTest, UniTest, Indigo and Moodle were carried out by MA Bulana and SA Tsololo. The purpose of their work was to select several existing testing systems and conduct a brief analysis, trace the history of the transition of interactive learning from offline to online, choose the best [6]. A similar review was carried out by me, together with my supervisor Cherednikova O.Yu. within the collection [7]. In the course of this review, the main basis for the formation of the LMS was formed, and for example, in 2018, a program was developed, written in the Java programming language, to test knowledge in a certain area on Android devices [4].

4. Implementation approach [7]

Usually a testing system performs a lot of functions and therefore its structure is usually divided into subsystems( Picture 1).

General structure of the testing system(7 frames,106 kb)

Picture 1 - General structure of the testing system

Test generation subsystem

Formation of tests is one of the most important functions of a computer testing system. This subsystem should have the following properties [3]:

  1. provide the ability to create and maintain a database of tasks; the task editor must allow working with a sufficiently large range of multimedia objects;
  2. specify for each task at least the following set of characteristics: topic; section; paragraph; difficulty; labor intensity; estimated time to complete the task;
  3. it should be possible to generate tasks and create sets of tasks of the same type;
  4. the range of types of tasks in the database should be quite diverse, and include the following types:
    • tasks to insert missing text;
    • tasks to insert a missing number or expression; the difference from the text here is that the student can enter a formula, and the system must be able to analyze it;
    • selection of one or more screen objects (creation of tasks with the selection of one or several answers);
    • dragging objects to the desired area (matching tasks).
  5. provide access to statistical information on each of the tasks (which answers were chosen and how many times, how often each of the options was chosen, which incorrect answers were entered, etc.). Only under this condition is it possible to develop tests that meet the requirements of pedagogical diagnostics.

Subsystem Teacher

In this block, the testing system should provide the teacher with the following opportunities:

  1. Authentication with teacher-specific roles;
  2. Set scripts and formulas for the test. A promising direction of the test is adaptive testing, which involves the generation of questions taking into account the previously received answers of the test taker. [10]. Adaptive tests reduce the number of questions, and hence the time spent on testing, for strong learners who give the correct answers to questions at the beginning of testing. Testing an excellent student may be limited to offering him 10-15 difficult questions, while an incorrectly answered student with poor knowledge of the material will be asked a larger number of easier questions to determine whether he can claim a minimum positive mark or not. Adaptive tests require a lot of effort from developers, since it is necessary to determine weights for questions of different difficulty levels, decide how the testing system will determine the initial level of the test taker, and how it will choose subsequent questions for the test taker, taking into account his answers to the questions already asked to him;
  3. Generate test cases;
  4. View detailed statistics with the results of the work of each group, and in the flesh to each student separately. Data should be formed in tabular and graphical form (Picture 2).
An example of registration of statistical data

Picture 2 – An example of registration of statistical data.

Moodle statistics block

Picture 3 - Moodle statistics block

Subsystem Student

It is worth paying due attention to this subsystem, because it is no less important and should be as accessible and convenient as possible for the student.

The student's system must include:

  1. Reliable authentication subsystem.
  2. The ability to conduct testing at any time, but with each choice of the same scenario, receive different, but the same type of tests;
  3. The most comfortable interface for students.
  4. The most comprehensive information on the test results (which tasks were completed correctly and which were incorrect, which mistakes were made, which sections were mastered by the student, which were not). The level of detail of the information provided is determined by the teacher during the scenario assignment.

Conclusion

Testing systems help speed up the process of testing students' knowledge both in schools and in higher education institutions. The introduction of automated testing systems further improves the productivity of both students and teachers. The availability of statistics allows you to tightly control every change in student performance and timely adjust the curriculum. Also, such systems allow students to get access to studies in different parts of the city, country or world. Automation of the formation of test tasks allows you to diversify test options, which simplifies the teacher's work and gives a more objective assessment when passing the test.

List of sources

  1. Frederiksen JR, Collins A. A Systems Approach to Educational Testing. Educational Researcher. 1989;18(9):27-32.
  2. Джей С. Уокер, Брюс Шнайер, Джеймс А. Джораш Метод и аппарат компьютерного обучающего тестирования // 1996.
  3. Джей С. Уокер, Брюс Шнайер, Джеймс А. Джораш Метод и аппарат учебного тестирования // 2009.
  4. Ая-Мин Хуан, Yen-Тин Лин, Шу-Чен Ченг Адаптивная система тестирования для поддержки универсальной образовательной оценки // Компьютеры и образование. - Тайвань: январь 2009.
  5. Система и методы компьютерного тестирования // scholar.google.ru/ Интернет-ресурс. - Режим доступа: Cсылка (дата обращения: 20.11.2020).
  6. Цурган. В.А., Чередникова О.Ю. Разработка системы тестирования знаний студента // Сборник материалов международной конференции Информатика, управляющие системы, математическое и компьютерное моделирование (ИУСМКМ-2020).. - Донецк: - С. 454-458. Интернет-ресурс. - Режим доступа: Cсылка
  7. Прокофьева Н. О. Стратегии и методы проведения контроля знаний в компьютерном обучении // Educational Technology & Society — 13 (1). 2010 — pp. 378-392 ISSN 1436-4522.
  8. Brusilovsky, P. and Miller, P., Web-based testing for distance education. In: P. DeBra and J. Leggett (eds.) Proceedings of WebNet'99, World Conference of the WWW and Internet, Honolulu, HI, Oct. 24-30, 1999, AACE, pp. 149-154.
  9. Аванесов В.С. Композиция тестовых заданий. Учебная книга для преподавателей вузов, учителей школ, аспирантов и студентов пед. вузов. 2 изд., испр. и доп.М.: Адепт.1998-217с.
  10. Ермышкин В.О., Чередникова О.Ю. Разработка программы тестирования знаний в определенной области под Аndroid // Современные тенденции развития и перспективы внедрения инновационных технологий в машиностроении, образовании и экономике. Т.4. № 1 (4). - 2018. Азов, - С.96-100.
  11. Система дистанционного обучения Moodle - Мясникова Т.С., Мясников С.А. Система дистанционного обучения MOODLE.-Харьков, 2008.-232 с.
  12. Работа в системе дистанционного обучения Moodle. - Анисимов А.М., Учебное пособие.2-е изд. испр. и дополн.– Харьков, ХНАГХ, 2009. - 292 стр.
  13. Lomas, Natasha. Finnish Startup Gets Seed Backing For Its Online Education Platform Play, Eliademy. "TechCrunch" April 24, 2014.
  14. Сатунина А. Е. Электронное обучение: плюсы и минусы // Современные проблемы науки и образования : журнал. — 2006. — № 1. — С. 89—90.
  15. Болкунов И.а. Электронное обучение: проблемы, перспективы, задачи // Таврический научный обозреватель. — 2016. — Вып. 11-1 (16).
  16. Udaya Sri K., Vamsi Krishna T. V. E-Learning :Technological Development in Teaching for school kids // International Journal of Computer Science and Information Technologies. — 2014. — P. 6124—6126.
  17. Хамутников В.А., Давыдов А.Н., Самоделов С.К., Макуха Л.В., Сидоров А.Ю. расширение возможностей электронных образовательных систем на платформе moodle // Фундаментальные исследования. – 2017. – № 1. – С. 117-123;