This project aims to build novel tools that increase the capacity of arts therapists to engage cognitively disabled older people in artistic activities. Engagement with the arts is becoming widely accepted as a method for promoting quality of life. However, persons with Alzheimer's disease have difficulty engaging in a creative activity. Art therapists engage older adults in artistic activities that promote autonomy and independence, thereby possibly increasing the older adult's ability to engage with the world. Developed in conjunction with users, the tools are customizable touch-screen interface creative arts devices. A therapist's customized application can track a user's activities and can respond to the user using powerful artificial intelligence (AI) and computer vision algorithms.

Multi-touch tables are becoming increasingly popular and much research is dedicated to developing suitable interaction paradigms. There also exist multiple techniques aimed at extending interactions into the hoverspace--the space directly above a multi-touch table. We propose a novel hoverspace method that does not require any additional hardware or modification of existing vision-based multi-touch tables. Our prototype system was developed on a Diffused Surface Illumination (DSI) vision-based multi-touch set up, and uses light reflected from a person's palm to estimate its position in 3D space above the table. Read more here.

This project is to design and build real-time systems to assist persons with dementia during handwashing. Assistance is given in the form of verbal and/or visual prompts, or through the enlistment of a human caregiver's help. The system uses only video inputs, and combines a Bayesian sequential estimation framework for tracking hands and towel, with a decision theoretic framework for computing policies of action -- specifically a partially observable Markov decision process (POMDP). A key element of the system is the ability to estimate and adapt to user states, such as awareness, responsiveness and overall dementia level.

SNAP: SyNdetic Assistance Processes for Rapid Development of Technologies for Wellness. This project aims to develop a methodology for rapid prototyping of POMDP models for assistive technology. A POMDP for assistance of persons with dementia has system actions as the various prompts or memory aids the system can use to help a user remember things in the task. For example, shining a light on the kettle may remind them of the kettle's purpose. A key problem is the initial specification of the model for a particular task. We accomplish this by qualitative and quantitative task analysis using Syndetic modeling applied to a small number of example videos. The result is a proces for the initial specification of a full working assistive technology for a particular task. In collaboration with Andrew Monk and Patrick Olivier.

This project, joint work with Jesus Favela, CICSE, Ensenada, Mexico, is to build activity recognition systems into smart phones for elderly persons with a cognitive disability. The activity recognition provides a stream of data on the mobile device that can be used to provide the user with situational awareness cues and to update their status on a protected social networking site (SNS), for the information of relevant stakeholders such as family members, or health professionals. Awareness information that can be obtained from sensors in the mobile device includes location, presence, or gestures. Family and friends can use the SNS to be aware of the elders whereabouts and decide when is a good moment to call, assist her remotely when at risk, add or modify entries to her schedule, or send affective messages to the mobile and/or ambient displays. Deciding what information to share with the different stakeholders is an important issue, and we propose a decision theoretic approach to determine what to information to share with whom. Our initial work on this topic is to build a Location And Context Aware Safety Assistant (LACASA).

This project, joint work with Johanna Moore, is to investigate integrating spoken dialogue into cognitive assistants, such as the COACH. Although these congnitive assistants provide verbal prompts, they cannot react to verbal user feedback. This is a major limitation, and we seek to enable these systems to understand feedback from a user and to respond appropriately. We are designing a POMDP-based assistance system for a calendar application using SNAP, and plan to deploy it in a wizard-of-Oz experiment to test it and to gather a corpus of data for building more effective spoken dialogue systems.

This project is developing a real-time system that guides stroke patients during upper extremity rehabilitation. The system automatically modifies exercise parameters to account for the specific needs and abilities of different individuals. The system uses a partially observable Markov decision (POMDP) model of a rehabilitation exercise that can capture this form of customization.

I work on the SIDE Hub, and specifically on Machine Learning for situated prompting systems in the home. This project uses the Ambient Kitchen at Culture Lab, Newcastle University, run by Patrick Olivier.

SPUDD stands for Stochastic Planning Using Decision Diagrams. SPUDD implements a value iteration algorithm for MDPs and POMDPs that uses algebraic decision diagrams (ADDs) to represent value functions and policies.

The DyNaMo project investigates probabilistic graphical models (PGMs) and their applications. PGMs allow the representation of probability distributions with many variables in a compact form, also they help to make probabilistic inference (estimate the probability of certain variables given other known) efficiently. PGMs include: Bayesian classifiers, Bayesian networks, Markov random fields, and have many applications in medicine, expert systems, industrial diagnosis, image analysis, robotics and many others.

This project uses stereo vision to detect card counters in the game of Blackjack. The system tracks the game as it progresses, monitors the cards played, and tracks the player's betting patterns. The correlation between the player's betting patterns and the game card count is analysed to determine the likelihood that a player is card counting, and the system alerts the Casino staff upon positive identification of a card counter. The Blackjack tracking system also has the ability to detect dealer errors, by monitoring dealers' actions made during the game and ascertaining whether or not the correct action was taken.

This project aims to develop realistic large-scale SSDS with an accurate, extended representation of user goals, and to use new Automatic Belief Compression (ABC) techniques to plan over the large state spaces thus generated. Techniques such as Value-Directed Compression demonstrate that compressible structure can be found automatically in the SSDS domain. Joint work with Oliver Lemon and Paul Crook.