A research team led by Professor Yeung Siu Sze Susanna at The Education University of Hong Kong (EdUHK) has developed an award-winning interactive learning social robot, Joey, which received the Gold Medal and Special Award at the Silicon Valley International Invention Festival 2024, the Gold Medal and Jury’s Choice Award at the 9th International Invention Innovation Competition in Canada. This Social Robot built for ages 4 to 9 represents a significant advancement in how children learn English. Focusing on storytelling, interactivity, and emotional engagement, Joey offers a personalized learning experience that benefits both typical learners and those with special educational needs. The Standard had the privilege of learning more during an exclusive interview.
Reimagining language learning with Joey
Prof Yeung and her team have been researching English language learning for many years, focusing on the cognitive, social, and family factors that influence this process. The idea for Joey arose during the COVID-19 pandemic when face-to-face teaching became limited. Prof Yeung was inspired by her visit to a robotics company, where she saw robots used in hospitals. She realized similar technology could support language learners, especially with the growing role of AI in education.
“I wanted to use technology to bridge gaps in learning,” Prof Yeung explains, and Joey became her solution—a tool that offers children a fun, interactive, and personalized way to learn English in a classroom setting.
Joey’s key features: Listen, Co-tell, and Re-tell
According to Prof Yeung, Joey’s design centers around three main functions: ‘Listen,’ ‘Co-tell,’ and ‘Re-tell.’
For the ‘Listen’ function, Joey narrates engaging stories, paired with colorful cartoon visuals. This function exposes children to spoken English, essential for language development. “Listening is crucial,” says Prof Yeung. “Joey helps children develop comprehension skills by immersing them in stories.”
While the ‘Co-tell’ function enables children to participate in storytelling alongside Joey. During this process, children can verbally respond to questions posed by Joey as the story unfolds. “After listening, they can express their understanding and respond to questions, which helps them practice their language skills. Our robot also incorporates typical educational technology features, such as personalization and feedback. For example, it asks comprehension questions after a story, providing immediate feedback that aids learning,” she adds.
Finally, the ‘Re-tell’ function encourages children, possibly with a teacher’s assistance, to verbally recount the entire story to Joey. If they struggle to do so, Joey will ask guiding questions to help them complete the task. This reinforces memory and helps build children’s confidence in speaking.
Another intriguing feature that makes Joey more engaging is its expressive face, which displays emotions without movement. This design draws inspiration from teachers who use expressive gestures and facial expressions when telling stories. We believe that synchronizing these elements enhances engagement, as the emotional delivery of a story is as important as its content. “Currently, we have a total of 7 kinds of expressions, each with three intensities, such as happy, very happy, and extremely happy.”
Joey also features gesture detection functionality. A camera positioned just below its head recognizes specific gestures made by children, enabling even shy children to engage non-verbally. For instance, instead of verbally selecting the correct answer prompted by Joey, children can raise their fingers to indicate their choices, using the number of fingers raised to correspond to different answers.
Additionally, the Social Robot includes a backend system that generates audio, video, and comprehension questions, primarily to alleviate teachers’ workloads in creating teaching materials. This system allows users to create customized content tailored to specific topics and needs. Prof Yeung explains, “We understand that content creation can be time-consuming and often doesn’t fit all classrooms, so we provide training to help teachers effectively use the system.”
Joey is particularly suited for students with special educational needs (SEN), as its tailored functions provide individualized support that teachers may not always be able to offer. It can also assist children with autism by presenting social stories to teach essential social skills.
Positive feedback and future development
Discussing the insights gathered from children and educators who have used the Social Robot in educational settings, Prof Yeung notes that evaluation data confirms a positive impression. “Children particularly enjoy when Joey reads stories to them. We also compared Joey to tablets like iPads and found that Joey can maintain children’s attention for longer periods during similar tasks.”
According to feedback, children especially enjoy the robot’s moving arms, which make interactions more engaging by providing interactive responses. “They also love the pictures generated by the robot,” Prof Yeung adds. “The visual and artistic style of the AI-generated images appeals to them, resembling the pixelated look of the movies they watch.”
Looking ahead, Prof Yeung states that the team is working on research and development for the next-generation Joey, incorporating feedback to enhance features such as adding head movement for a more lifelike appearance.
Plans for future collaboration include sending Joey overseas to Finland, the Philippines, and potentially mainland China, as well as partnering with schools and community centers for after-school programs.
Prof Yeung hopes that Joey can be part of the next exciting chapter in educational technology, offering children an emotionally engaging, interactive way to learn English. “With its AI-driven features and ability to support a range of learning needs, the goal is that Joey can become a useful tool for teachers and a beloved companion for young learners,” she says.
