RESEARCH HIGHLIGHTS
Large Language Models in Conversational Robots for Older Adults
Large Language Models in Conversational Robots for Older Adults
Personalisation in Long-Term Human-Robot Interaction
Personalisation in Long-Term Human-Robot Interaction
Personalisation in Manipulation and Placement Planning
Personalisation in Manipulation and Placement Planning
POSTDOCTORAL RESEARCH
Large Language Models in Conversational Robots for Older Adults
Under the supervision of Prof. Gabriel Skantze and Assist. Prof. Sanna Kuoppamäki,
KTH ROYAL INSTITUTE OF TECHNOLOGY
Designing Conversational Companion Robots with Older Adults through Foundation Models
We conducted two participatory design studies with 34 older adults in Sweden, to understand their expectations towards companion robots in their daily lives. Based on this, I developed design recommendations for companion robots using foundation models.
Challenges and Recommendations for Applying Large Language Models to Conversational Robots
In collaboration with Aida Hosseini
I developed an autonomous conversational robot using GPT-3.5. 34 older adults talked to the robot in casual conversations during the participatory design workshops. Analyzing those conversations, I identified the challenges of applying large language models to conversational robots, and developed design recommendations for overcoming them.
Detecting User Enjoyment with Large Language Models
In collaboration with Dr. Andre Pereira, Jura Miniota, Sofia Thunberg, Erik Lagerstedt, Lubos Marcinek, Prof. Joakim Gustafsson
I coordinated a team of researchers to create a user enjoyment scale for analyzing conversations with robots,
and develop a multimodal enjoyment detector using large language models for real-time dialogue adaptation.
PHD THESIS
MULTI-MODAL PERSONALISATION IN LONG-TERM HUMAN-ROBOT INTERACTION
UNDER THE SUPERVISION OF PROF. TONY BELPAEME, DR. NATALIA LYUBOVA AND DR. MICHAEL GARCIA ORTIZ,
UNIVERSITY OF PLYMOUTH AND SOFTBANK ROBOTICS EUROPE
user recognition
MULTI-MODAL INCREMENTAL BAYESIAN NETWORK WITH ONLINE LEARNING
First step towards personalisation
I created the first open world user recognition method that can recognise known users, detect new people and learn them online and incrementally from a single picture without preliminary training. It is also the first method in combining soft biometrics (e.g., gender, age, height and time of interaction) with a primary biometric (e.g., face recognition) for user identification in real-time in human-robot interaction in the real world. I also created a Multi-modal Long-Term User Recognition Dataset to simulate user recognition in long-term human-robot interaction scenarios.
PERSONALISATION IN SERVICE ROBOTICS
PERSONAL BARISTA ROBOT
IN COLLABORATION WITH MEHDI HELLOU AND ALEXANDRE MAZEL
The barista that remembers you and your favourite order
I created the Barista Datasets for generic and personalised interactions in a coffee shop with simulated real-world problems (e.g., recognition errors, incorrect recalls, out-of-vocabulary words), which is the first dataset for exploring user-specific personalisation in task-oriented long-term interactions. Based on these datasets, we designed generic and personalised barista robots with rule-based dialogue managers, online automatic speech recognition, knowledge-base and a multi-modal user recognition method to recall the most frequent or recent order of the user. We conducted the first real-world study that explores fully autonomous personalisation in dialogue for long-term human-robot interactions. The study showed that rule-based approaches are inflexible to the variations in the user responses, but personalisation can mitigate the negative user experiences.
To improve flexibility in dialogue, I evaluated data-driven approaches, such as Memory Networks and Sequence-to-Sequence models on the Barista Datasets, and found that they are suitable for generic task-based dialogue and real-time interactions, but they cannot perform sufficiently well to be deployed in personalised long-term interactions, due to their inability to learn and use new entities, and poor recall of personal data.
To improve flexibility in dialogue, I evaluated data-driven approaches, such as Memory Networks and Sequence-to-Sequence models on the Barista Datasets, and found that they are suitable for generic task-based dialogue and real-time interactions, but they cannot perform sufficiently well to be deployed in personalised long-term interactions, due to their inability to learn and use new entities, and poor recall of personal data.
PERSONALISATION IN SOCIALLY ASSISTIVE ROBOTICS
PERSONAL SOCIALLY ASSISTIVE ROBOT FOR CARDIAC REHABILITATION
IN COLLABORATION WITH EMMANUEL SENFT, COLOMBIAN SCHOOL OF ENGINEERING and FUNDACIÓN CARDIOINFANTIL INSTITUTO DE CARDIOLOGIA
IMMEDIATE PERSONAL FEEDBACK AND MOTIVATION
We designed a social and personal assistive robots for 18 weeks cardiac rehabilitation therapy to improve patient motivation, engagement and adherence in the therapy. The robot provides immediate and personalised motivation and feedback within and throughout the therapy based on the progress of the patient, and alerts the medical team when necessary. The robots were deployed in a hospital in Bogota for 2.5 years, and significantly improved health and motivation of 17 patients.
MASTER THESIS
USER-CENTRED TASK PLANNING AND MANIPULATION
FOR A DISHWASHER LOADING ROBOT
UNDER THE SUPERVISION OF PROF. ALBERT ALİ SALAH AND PROF. H. LEVENT AKIN, BOĞAZİÇİ UNIVERSITY
PERSONAL DOMESTIC ROBOT
Loading a dishwasher is a tedious task that we have to repeat every few days. While this repetitive task may seem generic, each person has different preferences in placement, based on their arrangement within the machine or the time and energy it takes. Thus, I designed a model for placing mugs on the counter-top to the dishwasher tray with a robot arm that optimises the manipulation and placement planning based on a personalised cost function and object recognition, within simulation and physical world.
ROBOCUP
MOTION AND BEHAVIOUR PLANNING IN ROBOT FOOTBALL
UNDER THE SUPERVISION OF PROF. H. LEVENT AKIN, BOĞAZİÇİ UNIVERSITY
GET UP, KICK AND SAVE THE MATCH
As a member of the Cerberus team in the RoboCup Standard Platform League (SPL), I was in charge of motion planning for getting up on falls, forward, diagonal and side kicks and goalie motions for saving the ball (in 2014, Brasil), behaviour planning for technical challenges, such as whistle detection, realistic environments, intelligent ball search, obstacle avoidance and passing between teammates (2015, China).
SENIOR YEAR PROJECT
DESIGNING AND BUILDING MOBILE EDUCATIONAL ROBOTS
UNDER THE SUPERVISION OF ASSOC. PROF. ÇETİN YILMAZ AND PROF. H. IŞIL BOZMA, BOĞAZİÇİ UNIVERSITY
SMALL AND ROBUST EDUCATIONAL ROBOTS
We designed and built the mechanical parts and the chassis of five Minik-II educational robots for the
Intelligent Systems Laboratory in Bogazici University, which are continuing to be used at the university for course projects. Speed, acceleration, linearity and rotation tests were conducted to ensure the accuracy and robustness of the robots. The robots are still being used in the Robot Control course of the university after 12 years.
OTHER PROJECTS AND WORKSHOPS
IN COLLABORATION WITH ANIKA NARAYANAN AND DR. JAMES KENNEDY, DISNEY RESEARCH
A VIRTUAL CHARACTER WITH PERSONALITY AND EMOTIONS THAT CAN ADAPTS ITS EMOTIONS ACCORDING TO YOURS
In order to achieve more believable interactions with artificial agents, there is a need to produce dialogue that is not only relevant, but also emotionally appropriate and consistent. This work presents a comprehensive system that models the emotional state of users and an agent to dynamically adapt dialogue utterance selection. A Partially Observable Markov Decision Process (POMDP) with an online solver is used to model user reactions in real-time. The model decides the emotional content of the next utterance based on the rewards from the users and the agent. The previous approaches are extended through jointly modeling the user and agent emotions, maintaining this model over time with a memory, and enabling interactions with multiple users. A proof of concept user study is used to demonstrate that the system can deliver and maintain distinct agent personalities during multiparty interactions.
PERSONAL ROBOT NARRATOR
IN COLLABORATION WITH MINA MARMPENA
PERSONALISED NARRATION AND NON-VERBAL COMMUNICATION
We designed a personalised robot narrator with non-verbal emotional communication that identifies users, retrieves their profile, recommends content according to their interests, and adapts its emotional body language according to the user's age and the narration.
TRACKED ROBOTS
UNDER THE SUPERVISION OF PROF. H. IŞIL BOZMA, BOĞAZİÇİ UNIVERSITY
PROVISIONS AND RESEARCH ON TRACKED ROBOTS
My summer research in 2011 focused on modelling and suggesting provisions for the tracked EDAR G3T robot in the Intelligent Systems Laboratory, and conducting an in-depth study of the state of art in tracked robots working in rough terrains and climates, which enabled receiving funding from the scientific and technological research council of Turkey (TÜBİTAK) for buying new tracked robots (Dr. Robot Jaguar robots) for the laboratory.
OPEN-SOURCE PROJECTS FOR ROBOTS
OBJECT DETECTION ON NAO ROBOT
Séverin Lemaignan and I implemented Chilitags (by Bonnard et al., 2013), 2D fiducial markers, on the NAO robot for on-board detection of objects and for pointing at them.
ROBUST POINTING WITH NAO ROBOT
I created a service for pointing and looking at objects with Chilitags or to a world or tablet coordinate. It was used as part of L2TOR project for teaching children objects in a scene.
USER RECOGNITION FOR ROBOTS
I developed a multi-modal user recognition model, Multi-modal Incremental Bayesian Network, that is suitable for any robot, with which you can recognise previous users, detect new people and learn them incrementally and online.
I extended OpenFace (by Amos et al., 2016) face recognition software with deep neural networks, to compare people to celebrities or other known users on the Pepper and NAO robots.
WORKSHOP ON LIFELONG LEARNING AND PERSONALIZATION IN
LONG-TERM HUMAN-ROBOT INTERACTION (LEAP-HRI)
IN COLLABORATION WITH DR. NIKHIL CHURAMANI, DR. ANDREEA BOBU, ASSIST. PROF. MARIACARLA STAFFA,
PREVIOUS CO-ORGANIZERS: DR. ADITI RAMACHANDRAN, PROF. HATICE GUNES, SAMUEL SPAULDING, PROF. SINAN KALKAN, DR. GERMAN I. PARISI
LEAP TOWARDS ADAPTIVE ROBOTS OF THE FUTURE!
I coordinate and co-organise the Workshop on Lifelong Learning and Personalization in Long-Term Human-Robot Interaction (LEAP-HRI) as part of the ACM/IEEE International Conference on Human-Robot Interaction (HRI) since 2021 (4 years in a row). The workshop focuses on studies on lifelong learning and adaptivity to users, context, environment, and tasks in long-term interactions in a variety of fields (e.g., education, rehabilitation, elderly care, collaborative tasks, customer-oriented service and companion robots). We have a brilliant turnout each year, with 101 attendees in 2021, 70 attendees in 2022, and over 50 attendees joining us in person and online in 2023!
PERSONALIZATION IN LONG-TERM HUMAN-ROBOT INTERACTION WORKSHOP (PLOT-HRI)
IN COLLABORATION WITH ADITI RAMACHANDRAN, SAMUEL SPAULDING, DR. DYLAN F. GLAS, assoc. prof. IOLANDA LEITE, dr. KHENG LEE KOAY
MEDIUM FOR RESEARCHERS TO SHARE THEIR WORK ON PERSONALISATION
I coordinated and co-organised the Personalization in Long-Term Human-Robot Interaction (PLOT-HRI) full-day workshop at the 14th Annual ACM/IEEE International Conference on Human Robot Interaction (HRI 2019). The workshop focused on studies on adaptivity to users, context, environment, and tasks in long-term interactions in a variety of fields (e.g. companion robots, collaborative tasks, education, rehabilitation, elderly care). The workshop consisted of Adriana Tapus, Hae Won Park, Takayuki Kanda and Tony Belpaeme as keynotes, 5 paper presentations (proceedings are available online) and brainstorming sessions.