Niulanshan First Secondary School Classroom Care System opacity

Occurred: September 2018

Can you improve this page?
Share your insights with us

The movements and behaviour of students at Niulanshan First Secondary School in Beijing are being constantly monitored and scored by a system about which they have been no given information nor to which they have provided their consent. 

Launched in 2017, Hanwang Education's Class Care System (CCS) uses facial recognition, emotion recognition and deep learning algorithms to identify students' faces and analyse and classify their behaviour into one of five categories: listening, answering questions, writing, interacting with other students, or sleeping.

Each student is automatically scored weekly and their information shared via a mobile app with their teachers and parents. 

Hanwang argues the scores are important for identifying which students require support. But critics point out that the system is not only highly intrusive, it can also be inaccurate and unfair. 

As Rest of World describes, in one example 'a student who had answered just a single question in his English class was called out for low participation — despite the app recording him as 'focused' 94% of the time.'

Research studies regularly conclude that emotion recognition technologies are based on the fundamentally flawed premise that algorithms can analyse a person’s facial expressions and accurately infer their inner state or mood.

Operator: Niulanshan First Secondary School
Developer: Hanwang Technology

Country: China

Sector: Education

Purpose: Assess and rank student behaviour 

Technology: Facial recognition; Emotion recognition; Deep learning; Neural network; Machine learning
Issue: Accuracy/reliability; Privacy; Surveillance

Transparency: Governance; Marketing; Privacy