ERI: IoT-Enabled Smart Learning Environment for Ambient Assessment of Socio-Technical Skills in Engineering Students

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2)

Today’s engineers need to attain both technical mastery of their field and socio-technical skills that significantly impact their professional performance and the solutions they produce. Examples of these skills include the ability to make ethically informed judgments, the ability to function effectively in a team, and an ingrained observance of safety procedures.

While technical knowledge and skills can be easily assessed by instructors through traditional assessment methods such as exams and project demonstrations, existing assessment methods for socio-technical learning objectives can be challenging to design, time-consuming to administer, inconsistent in its subjectivity and slow in returning feedback to students. This project will design and develop a novel automated Internet of Things (IoT)-based assessment tool for evaluating student attainment of socio-technical learning objectives in engineering courses that can potentially address these issues.

The Smart Learning Environment (SLE) Ambient Assessment Framework (SAAF) proposed in this project will be able to detect student and instructor actions within a normally conducted class setting such as lab activity or group discussion, reason the context of those actions, and produce a quantitative classification of student achievement level with respect to the learning objectives of the course or activity. This could transform the way engineers are trained in socio-technical areas as ambient observation could capture behaviors that more closely resemble how students would behave in the workplace as professional engineers.

This is in line with NSF’s Research in the Formation of Engineers (RFE) program as this project will develop a tool that can potentially efficiently and effectively help develop professional competencies of engineers.

The SAAF’s overall goal is to perform automated assessment of student performance with respect to a course’s socio-technical learning objectives. SAAF will apply knowledge-based activity recognition of classroom occupants (students and instructors) during a regular class activity to detect actions and reason the context of those actions. A knowledge-based approach to activity recognition will be employed so as not to be restrictive and prescriptive in the way instruction and assessments should be carried out as machine learning-based approaches would.

The detected events will be analyzed against classroom activity and course content ontologies developed through the project. The project will then design and implement a proof of concept for the SAAF by employing video, audio and sensor detection of events within a laboratory classroom and reasoning activities with context, such as “student A put on antistatic wrist band before instructor prompted,” or “in student B and C’s team, B did all the work”.

With this capability developed, the project will then investigate the quality and impact of this system through the investigation of two research questions – (1) How valid, reliable, convenient, and expedient is the SAAF as an automated tool for assessing socio-technical skills of engineering students? and (2) What are the major barriers to SAAF adoption as a primary classroom assessment tool for socio-technical learning objectives, and how can those barriers be addressed to increase buy-in? To answer these questions, student and instructor participants will be asked to conduct regular class laboratory activities in an SAAF-enabled classroom and compare SAAF assessment results with those of comparable traditional assessment methods.

Participant surveys and interviews will also be conducted to analyze user experience. Therefore, the project will provide the following major contributions, each addressing a current gap in engineering education: (1) shareable and extendable classroom activity and assessment ontologies, (2) a proof-of-concept for an automated ambient assessment technology, and (3) data analyzing the reliability, impact, and acceptability of such a system.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.