Top / Laboratory / Research Topics

---

Human-Machine Symbiosis Research
- Exploring the Symbiosis and Evolution of Humans and Machines -

---

Keywords: Human-Machine Symbiotic Engineering, Cognitive Development Systems, Healthcare and Welfare Support
We explore methodologies for symbiotic human-machine behavior and development, where artificial entities such as machines, systems, and robots interact with people and environments to evolve, while also nurturing the minds and capabilities of those who engage with them. Why can humans speak without knowing the principles of language, or walk without understanding the mechanics of gait? What kind of intelligence or mind emerges when machines possess a body? How do human minds and bodies develop and evolve through the intervention of artifacts? We will validate these questions in real-world settings across healthcare, nursing, education, manufacturing, and service industries.

---

Care and Medical Support Robotics

---

Keywords: Robotics, Assistive Devices, Product Design We conduct research and development on robots based on the daily living support needs of the elderly and people with disabilities. We develop robots based on interviews conducted at nursing facilities and hospitals, then test and evaluate them in real-world settings. To date, we have collaborated with nursing facilities, manufacturing companies, and local governments to develop monitoring robots that reduce the workload of staff members.

---

Body Augmentation Interface

---

Keywords: Human Interface, Universal Design, Ergonomics We conduct research and development on interfaces worn on the body to assist in maintaining and restoring motor function. By measuring movement in specific areas such as fingers or the oral cavity to operate external devices, and stimulating the user's senses based on the device's measurement data, we extend bodily functions. This includes linking oral sensory-motor activity with robotic arms or transcribing finger movements for motor training.

---

Machine Learning and Cognitive Development Systems

---

Keywords: Artificial Intelligence, Database, Image Processing We conduct research and development on learning algorithms that recognize bodies and objects through action. Robots autonomously generate movements to learn correlations and causality with sensory input, enabling them to recognize their own bodies and the environment. They exploreively manipulate objects to grasp their shape, weight, and friction, and learn the effects of object manipulation to discover their functionalities as tools.

---

Sensory-Motor Integration and Sensory Information Processing

---

Keywords: Multimedia, Virtual Reality, Computer Graphics We conduct research and development on systems that enhance human perception and cognition by providing complex sensory stimulation. By measuring whole-body movements while delivering combined visual, auditory, and tactile stimuli, we assist in posture and movement perception, enhance target recognition, and guide movement. By intervening in the human sensory-motor loop, we improve sensory information integration and the precision of movement control.

---

Health Coexistence Project

---

In Japan, where the population is aging and birth rates are declining, efforts are underway to build a society that extends the healthy life expectancy of its citizens by promoting measures for preventive care and health management. However, the methods and environmental conditions needed to elicit health-conscious awareness and actions from people in their daily lives remain poorly understood. This research introduces machine learning robot AI into living spaces. By intervening in situations such as walking, conversation, and meals to encourage independent actions and cognition, it creates opportunities to extend our healthy life expectancy. (From University General Guide 2021)

Lucia	healthcare robot that performs mobile patrol monitoring, wandering detection notifications, telepresence for people with disabilities, physiological measurement, and gait measurement.
Spina	A modular robot with an engineered reconstruction of the spinal structure. Each module communicates via CAN bus to perform coordinated and compensatory movements.
Tatto	A soft, skin-like device made of silicone that detects contact. It promotes wakefulness when touching a person and measures vital signs.
Cibo	A system that recognizes eating actions and table conditions. It detects chewing and prevents aspiration through voice prompts. It provides audio guidance for meals to visually impaired individuals.
Camina	A system for measuring physical therapy gait tests and evaluating gait motion. It categorizes elemental movements to visualize, assess, and predict gait.
Occhi	A fixation training system that restores vision impaired by eye strain and prevents diminished depth perception. Stretches the eye muscles through fixation movement.
Orale	An interface that operates devices using the tongue. It measures the state of the tongue as an optical image. Precise manipulation is possible at the center of the device.
Mano	Power transmission via concentric chain joints and push-pull cables assists finger flexion and extension. Enables mirror therapy.
Memoria	Answer quizzes generated from photos and reflections of daily events to preserve memories. Anticipate future events to boost vitality.
Parla	A system that provides explanations and answers questions while ensuring audiovisual information from presentation materials containing explanatory text. It generates explanatory videos.
Mani	A robot that transports food and beverages to the user's mouth or hand. It uses cutlery to carry food, determines the food's shape and volume, and serves it.

---

Nursing Robots and Environmental Sensor Networks

---

We are developing a mobile robot Terapio that supports medical doctors while doing the rounds in hospitals. Terapio has a function to deliver medical tools and information by autonomous tracking and power assisted locomotion. The robot is able to provide access to clinical information and audio-video data stored in previous treatments and recode new information on-line while giving medical cares. We are developing environmental sensor networks that accommodates total nursing with Terapio. The network system connects mobile nursing robots and sensors embedded in the environment. The system then recognizes activities of patients and caregivers to support them with nursing robots. A data server manages sensor information of the position, motion, and identity of people and materials as well as sensory-motor information of nursing robots in order to coordinate between people, robot and environment.
(Contributers: Terashima, K., Tasaki, R., Miura, J., Kitazaki, M.)

---

Developmental action perception and sensory-motor integration

---

We propose a cognitive system that allows an anthropomorphic robot to observe object operation performed by humans and reproduce the similar operation. The robot characterizes a self-generated action based on the action effect for objects. This action coding enables the robot to identify actions that share the common goal but performed by different agents (e.g. the self and other). Moreover, the system allows cross-sensory association that can reconstruct sense of touch from visual observation. The system does not require either body kinematics nor sensory-motor configuration for this learning, since the robot autonomously differentiates the own body from the environment and then learns primitive actions of object operation through stochastic motor exploration.
(Contributers: Saegusa, R., Natale, L., Metta, G., Sandini, G.)

Body schema development based on visuo-motor correlation

This work proposes a plausible approach for a humanoid robot to define its own body based on visuomotor correlation. The high correlation of motion between vision and proprioception informs the robot that a visually moving object is related to the motor function of its own body. When the robot finds a motor-correlated object during motor exploration, visuomotor cues such as body posture and the visual features of the object are stored in visuomotor memory. Then, the robot developmentally defines its own body without prior knowledge on body appearances and kinematics. Body definition is also adaptable for an extended body such as a tool that the robot is grasping. The body movements are generated in the manner of stochastic motor babbling, whereas visuomotor memory biases the babbling to keep the body parts in sight. This ego-attracted bias helps the robot explore the joint space more efficiently. After motor exploration, visuomotor memory allows the robot to anticipate a visual image of its own body from a motor command. The proposed approach was experimentally evaluated with humanoid robot iCub.
(Contributers: Saegusa, R., Metta, G., Sandini, G.)

---

Self-protective whole body movements and motor instruction

---

We developed an intelligence model that allows a humanoid robot to protect the self by whole body movements that the robot learned. The model consolidates local motor reflexes driven by force sensing with global adaptive reactions based on visual and vestibular stimuli. The robot therefore reduces physical load from the environment. We think that these self-protective behaviors are an important factor to realize robot's long-term motor learning. We are extending the model to allow an humanoid robot to learn whole body movements by humans' kinesthetic teaching and its reconstruction. We first teach the robot movements (e.g. walking and sitting-up) with supporting the robot. The robot extracts motor features, and use them in reinforcement learning of the movements without the support. The recycle of encoded motor knowledge enhances the learning speed in the different condition.
(Contributers: Shimizu, T., Saegusa, R., Ikemoto, S., Ishiguro, H., Metta, G.)

---

Rhythm associator to create music with humans and drawing effect

---

We developed a rhythm associating system. We first demonstrate an ensemble of rhythm performances for the system. The system then extracts couples of rhythm sequences and trains the rhythm associator. The similarity of rhythms is computed based on the hierarchical analysis, that provides the natural association in termes of human sense. After the leaning, the system plays rhythm sequences by listening the human rhythm performance. The behavior of the system is original in accepting the human player's suggestion to change the rhythm, and proposing a more suitable rhythm to the human player. This coordination create an music ensemble in which both the man and system can take leadership.
(Contributers: Kasahara, S., Saegusa, R., Hashimoto, S.)

---

Nonlinear data compression and pattern recognition

---

We developed a neuromorphic model of nonlinear principal component analysis that compresses high-dimensional data. Principal component analysis (PCA) is a standard linear method in data analysis. We proposed nonlinearization of the PCA. Non-parametric nonliear functions that play the role of data encoding or decoding are constructed with a hierarchical constrain. The hierarchically structured functions represent the data efficiently with less principal components than the linear PCA. We evaluated efficiency of data compression and decompression with large sets of image data. This approach is applicable for subspace-based pattern recognition. The set of functions forms manifolds (nonlinear principal axes) in the data space, and the manifolds totally represent subspace of a categorical data.
(Contributers: Saegusa, R., Sakano, H.,　Hashimoto, S.)