Human-Agent Experience Sharing

Intelligent technology is ever more pervasive in modern societies, requiring people to increasingly collaborate with intelligent agents, such as avatars, robots, smart home systems, and other intelligent user interfaces. In order for intelligent agents to provide meaningful support and establish long-lasting personal, well-embedded, confidential, and respectful relationships with people, they need to understand the user and his/her context, i.e. personalisation and context-sensitivity. The research conducted within the ReJAM project aims to realize the potential of human-agent experience sharing by addressing the challenge of ensuring a mutual understanding between the agent and the user. This requires not only an expressive knowledge representation, but also an intuitive design that is informative of the agent’s affordance, and an effective feedback loop that can be used by the agent to personalize its interaction to the needs and desires of the user.

ReJAM constructs models of its users and analyses the social context (e.g. people in the room and their relation to the user) to support two forms of human-agent experience sharing:

a) sharing past experiences: ReJAM uses patients' favourite music to initiate past experience sharing: we use music as a stimulus because the context of the application is always music-related. The framework uses an expressive domain-specific knowledge representation, i.e. an ontology, to engage in conversations about the music and to gradually construct a user model containing personal information about the user.  The ontology describes, for instance, that music can trigger memories, and so when a song is played - especially in the context of the reminiscence activity - the agent can infer that it is reasonable to ask whether the music evokes any special memories. If the user confirms, the agent can use the information about memories available in the ontology to ask follow up questions, e.g. about the associated location or people involved. The user's answers to these questions are stored in the ontology for future reference, resulting in the construction of a model of the user's life story. The agent uses its knowledge about the user - as available in the ontology - to determine the intimacy of the relationship. The agent reciprocates by discussing things deemed appropriate for that level of intimacy.

b) sharing new experiences: ReJAM aims to motivate people with dementia to engage in activities with their social environment, by proposing music-related activities, e.g. playing a game with other patients, reminiscing with their spouse, or singing a song with their grandchildren. All activities are designed to scale from individual settings to group settings, making ReJAM well-suited for a broad range of social contexts. Through engaging in joint activities the agent as well as the social environment of the user share and create new experiences with the user. The agent continues to personalise its behaviour using a feedback loop (e.g. sensing the users' emotional responses). This feedback loop enables the agent to personalise its interaction to the needs and desires of the users, and their social context.