RESEARCH HIGHLIGHTS
Personal Companion Robot for Open-Domain Dialogue in Elderly Care
Personal Companion Robot for Open-Domain Dialogue in Elderly Care 
Personalisation in Long-Term Human-Robot Interaction
Personalisation in Long-Term Human-Robot Interaction 
Personalisation in Manipulation and Placement Planning
Personalisation in Manipulation and Placement Planning 
POSTDOCTORAL RESEARCH
PERSONAL COMPANION ROBOT FOR OPEN-DOMAIN DIALOGUE
IN LONG-TERM ELDERLY CARE
Lifelong Learning for Large Language Models To Personalize Support to the Elderly
Large language models, such as GPT-3 or BLOOM, are often pretrained on English data from young adults and focus on single interactions. This leads to superficial, repetitive, and interrogative small talks in open-domain conversations, in addition to posing a language barrier with non-native speakers. However, to have robots at the houses of elderly people or senior care centers in the future, we need personal robots that can converse in the native language of the person and continuously learn from these conversations to provide personalized assistance and support over long-term interactions. This project aims to address these challenges through participatory design with older adults and lifelong learning in language models.
PHD THESIS
MULTI-MODAL PERSONALISATION IN LONG-TERM HUMAN-ROBOT INTERACTION
UNDER THE SUPERVISION OF PROF. TONY BELPAEME, DR. NATALIA LYUBOVA AND DR. MICHAEL GARCIA ORTIZ,
UNIVERSITY OF PLYMOUTH AND SOFTBANK ROBOTICS EUROPE
user recognition
MULTI-MODAL INCREMENTAL BAYESIAN NETWORK WITH ONLINE LEARNING
First step towards personalisation
I created the first open world user recognition method that can recognise known users, detect new people and learn them online and incrementally from a single picture without preliminary training. It is also the first method in combining soft biometrics (e.g., gender, age, height and time of interaction) with a primary biometric (e.g., face recognition) for user identification in real-time in human-robot interaction in the real world. I also created a Multi-modal Long-Term User Recognition Dataset to simulate user recognition in long-term human-robot interaction scenarios. Both are available online.
PERSONALISATION IN SERVICE ROBOTICS
PERSONAL BARISTA ROBOT
IN COLLABORATION WITH MEHDI HELLOU AND ALEXANDRE MAZEL
The barista that remembers you and your favourite order
I created the Barista Datasets for generic and personalised interactions in a coffee shop with simulated real-world problems (e.g., recognition errors, incorrect recalls, out-of-vocabulary words), which is the first dataset for exploring user-specific personalisation in task-oriented long-term interactions. Based on these datasets, we designed generic and personalised barista robots with rule-based dialogue managers, online automatic speech recognition, knowledge-base and a multi-modal user recognition method to recall the most frequent or recent order of the user. We conducted the first real-world study that explores fully autonomous personalisation in dialogue for long-term human-robot interactions. While rule-based approaches are inflexible to the variations in the user responses, however, personalisation can mitigate the negative user experiences. On the other hand, data-driven approaches are suitable for generic task-based dialogue and real-time interactions, but they cannot perform sufficiently well to be deployed in personalised long-term interactions, due to their inability to learn and use new entities, and poor recall of personal data.
PERSONALISATION IN SOCIALLY ASSISTIVE ROBOTICS
PERSONAL SOCIALLY ASSISTIVE ROBOT FOR CARDIAC REHABILITATION
IN COLLABORATION WITH EMMANUEL SENFT, COLOMBIAN SCHOOL OF ENGINEERING and FUNDACIÓN CARDIOINFANTIL INSTITUTO DE CARDIOLOGIA
IMMEDIATE PERSONAL FEEDBACK AND MOTIVATION
We designed a personalised socially assistive robot for 18 weeks cardiac rehabilitation therapy to improve patient motivation, engagement and adherence in the therapy. The robot provides immediate and personalised motivation and feedback within and throughout the therapy based on the progress of the patient, and alerts the medical team when necessary.
MASTER THESIS
USER-CENTRED TASK PLANNING AND MANIPULATION
FOR A DISHWASHER LOADING ROBOT
UNDER THE SUPERVISION OF PROF. ALBERT ALİ SALAH AND PROF. H. LEVENT AKIN, BOĞAZİÇİ UNIVERSITY
PERSONAL DOMESTIC ROBOT
Loading a dishwasher is a tedious task that we have to repeat every few days. While this repetitive task may seem generic, each person has different preferences in placement, based on their arrangement within the machine or the time and energy it takes. Thus, I designed a model for placing mugs on the counter-top to the dishwasher tray with a robot arm that optimises the manipulation and placement planning based on a personalised cost function and object recognition, within simulation and physical world.
ROBOCUP
MOTION AND BEHAVIOUR PLANNING IN ROBOT FOOTBALL
UNDER THE SUPERVISION OF PROF. H. LEVENT AKIN, BOĞAZİÇİ UNIVERSITY
GET UP, KICK AND SAVE THE MATCH
As a member of the Cerberus team in the RoboCup Standard Platform League (SPL), I was in charge of motion planning for getting up on falls, forward, diagonal and side kicks and goalie motions for saving the ball (in 2014, Brasil), behaviour planning for technical challenges, such as whistle detection, realistic environments, intelligent ball search, obstacle avoidance and passing between teammates (2015, China).
SENIOR YEAR PROJECT
DESIGNING AND BUILDING MOBILE EDUCATIONAL ROBOTS
UNDER THE SUPERVISION OF ASSOC. PROF. ÇETİN YILMAZ AND PROF. H. IŞIL BOZMA, BOĞAZİÇİ UNIVERSITY
SMALL AND ROBUST EDUCATIONAL ROBOTS
We designed and built the mechanical parts and the chassis of five Minik-II educational robots for the
Intelligent Systems Laboratory in Bogazici University, which are continuing to be used at the university for course projects. Speed, acceleration, linearity and rotation tests were conducted to ensure the accuracy and robustness of the robots.
OTHER PROJECTS AND WORKSHOPS
IN COLLABORATION WITH ANIKA NARAYANAN AND DR. JAMES KENNEDY, DISNEY RESEARCH
A VIRTUAL CHARACTER WITH PERSONALITY AND EMOTIONS THAT CAN ADAPTS ITS EMOTIONS ACCORDING TO YOURS
In order to achieve more believable interactions with artificial agents, there is a need to produce dialogue that is not only relevant, but also emotionally appropriate and consistent. This work presents a comprehensive system that models the emotional state of users and an agent to dynamically adapt dialogue utterance selection. A Partially Observable Markov Decision Process (POMDP) with an online solver is used to model user reactions in real-time. The model decides the emotional content of the next utterance based on the rewards from the users and the agent. The previous approaches are extended through jointly modeling the user and agent emotions, maintaining this model over time with a memory, and enabling interactions with multiple users. A proof of concept user study is used to demonstrate that the system can deliver and maintain distinct agent personalities during multiparty interactions.
PERSONAL ROBOT NARRATOR
IN COLLABORATION WITH MINA MARMPENA
PERSONALISED NARRATION AND NON-VERBAL COMMUNICATION
We designed a personalised robot narrator with non-verbal emotional communication that identifies users, retrieves their profile, recommends content according to their interests, and adapts its emotional body language according to the user's age and the narration.
TRACKED ROBOTS
UNDER THE SUPERVISION OF PROF. H. IŞIL BOZMA, BOĞAZİÇİ UNIVERSITY
PROVISIONS AND RESEARCH ON TRACKED ROBOTS
My summer research in 2011 focused on modelling and suggesting provisions for the tracked EDAR G3T robot in the Intelligent Systems Laboratory, and conducting an in-depth study of the state of art in tracked robots working in rough terrains and climates, which enabled receiving funding from the scientific and technological research council of Turkey (TÜBİTAK) for buying new tracked robots (Dr. Robot Jaguar robots) for the laboratory.
OPEN-SOURCE PROJECTS FOR ROBOTS
OBJECT DETECTION ON NAO ROBOT
Séverin Lemaignan and I implemented Chilitags (by Bonnard et al., 2013), 2D fiducial markers, on the NAO robot for on-board detection of objects and for pointing at them.
ROBUST POINTING WITH NAO ROBOT
I created a service for pointing and looking at objects with Chilitags or to a world or tablet coordinate. It was used as part of L2TOR project for teaching children objects in a scene.
USER RECOGNITION FOR ROBOTS
I developed a multi-modal user recognition model, Multi-modal Incremental Bayesian Network, that is suitable for any robot, with which you can recognise previous users, detect new people and learn them incrementally and online.
I extended OpenFace (by Amos et al., 2016) face recognition software with deep neural networks, to compare people to celebrities or other known users on the Pepper and NAO robots.
WORKSHOP ON LIFELONG LEARNING AND PERSONALIZATION IN
LONG-TERM HUMAN-ROBOT INTERACTION (LEAP-HRI)
IN COLLABORATION WITH DR. ADITI RAMACHANDRAN, ASSIST. PROF. MARIACARLA STAFFA, PROF. HATICE GUNES
PREVIOUS CO-ORGANIZERS: SAMUEL SPAULDING, PROF. SINAN KALKAN, DR. GERMAN I. PARISI
LEAP TOWARDS ADAPTIVE ROBOTS OF THE FUTURE!
I coordinate and co-organise the Workshop on Lifelong Learning and Personalization in Long-Term Human-Robot Interaction (LEAP-HRI) as part of the ACM/IEEE International Conference on Human-Robot Interaction (HRI) since 2021 (3 years in a row). The workshop focuses on studies on lifelong learning and adaptivity to users, context, environment, and tasks in long-term interactions in a variety of fields (e.g., education, rehabilitation, elderly care, collaborative tasks, customer-oriented service and companion robots). We have a brilliant turnout each year, with 101 attendees in 2021, 70 attendees in 2022, and over 50 attendees joining us in person and online in 2023!
PERSONALIZATION IN LONG-TERM HUMAN-ROBOT INTERACTION WORKSHOP (PLOT-HRI)
IN COLLABORATION WITH ADITI RAMACHANDRAN, SAMUEL SPAULDING, DR. DYLAN F. GLAS, assoc. prof. IOLANDA LEITE, dr. KHENG LEE KOAY
MEDIUM FOR RESEARCHERS TO SHARE THEIR WORK ON PERSONALISATION
I coordinated and co-organised the Personalization in Long-Term Human-Robot Interaction (PLOT-HRI) full-day workshop at the 14th Annual ACM/IEEE International Conference on Human Robot Interaction (HRI 2019). The workshop focused on studies on adaptivity to users, context, environment, and tasks in long-term interactions in a variety of fields (e.g. companion robots, collaborative tasks, education, rehabilitation, elderly care). The workshop consisted of Adriana Tapus, Hae Won Park, Takayuki Kanda and Tony Belpaeme as keynotes, 5 paper presentations (proceedings are available online) and brainstorming sessions.