Intelligent assistant carer for active aging
© The Author(s) 2017
Received: 22 May 2017
Accepted: 19 October 2017
Published: 2 November 2017
We present the concept of an Intelligent Assistant Carer system for the elderly, designed to help with active aging and to facilitate the interactions with carers. The system is modular, allowing the users to choose the appropriate functions according to their needs, and is built on an open platform in order to make it compatible with third-party products and services. Currently, the system consists of a wearable device (a smartwatch) and an internet portal that manages the data and takes care of the interactions between the user, the carers, and the support services. We present in detail one of the modules, i.e., fall detection, and the results of a pilot study for the system on 150 users over the course of 3 months.
Advances in medicine and living conditions are significantly increasing average life expectancy, while at the same time the natality rate in developed countries is falling. This creates a distorted demographic pyramid where there are more elderly people over 65 than there are children under 15 . It has been estimated by Eurostat that the working population (15–64 years) will decrease to 56% of total population by the year 2050 . This could lead to a number of problems for society, such as insufficient numbers of qualified personnel to care for the elderly, rising costs for individuals, families, and the public purse, and consequently a potential decrease in the perceived quality of life for individuals. At the same time, the WHO is promoting the concept of “active aging”, a process of optimizing opportunities for health, participation and security, in order to enhance the quality of life as people age . We face two main challenges—first, to enable the elderly to live a high-quality independent life at home for as long as possible and, second, to facilitate the care provided in the nursing homes.
Comparison of a professional personal assistant and a virtual one
Check on vital signs
Here, we present the concept of an Intelligent Assistant Carer (IAC), a system that is aimed to address both issues. The system is designed to be modular, allowing the users or their carers to activate individual modules when required. Younger elderly users who live independently are already used to some ICT technologies; therefore, it is easier for them to start using the system. Initially, they might use it for leisure activities, such as an exercise tracker or as a means to stay connected with friends. Through time, the system “grows” with the users, meaning that the system learns about their habits and needs, and the users adopt the system as a part of their daily life. Therefore, the introduction of additional modules is a smooth process. In addition, the IAC is based on an open platform, which, first, allows quick modifications when new, improved hardware becomes available and, second, allows developers to ensure connectivity to other systems. This approach is in line with the emerging European platform for elderly care, within the framework of the EU H2020 IN LIFE project .
In its present incarnation, the IAC consists of a modular wearable device (a wristwatch, we tested a series of commercial products) and an online portal that acts as a central hub for communications between the user, the formal and informal carers, and the support services, such as a 24/7 call center. The IAC system acts as a tool that both takes care of the user in some situations and assists the carer by providing relevant information when required. In this paper, we present the system’s architecture and a series of currently functional modules. We focus especially on the fall-detection module, which is described in a special section. This module is specifically intended to improve the user’s sense of security and thus prolong their independent life—one of the main reasons people decide to enter nursing homes is a fear that they will be unable to call for help in the case of an accident or a medical problem . Approximately 30% of people over the age of 65 experience a fall each year and 20% of those who fall require medical help afterwards . A particularly dangerous scenario is the “long lie”, where a person is unable to get up for several hours . We highlight how this approach allows for a rapid adaptation to new versions of hardware, including additional in-built sensors that may be employed to improve the module’s accuracy. We report on the results of a pilot study of the system, again with the focus on the fall detection and related security modules and on the user experience. The pilot study was carried out on 150 elderly users who were each using the system for 3 months. Finally, we discuss the system’s prospects and further steps.
2 System architecture
The elderly are a vastly heterogeneous group, with various levels of mental capabilities, familiarity with modern technology, and different needs. It is thus important to simplify the user experience to as few different devices as possible, but at the same time allow flexibility to incorporate numerous different applications that fit a specific user the best. We are proposing an agent-based architecture. Here, each module, performing its task, acts as an agent, while the core of the application communicates with all the agents and performs an informative decision based on the inputs from them.
The main advantage of the system is its modularity. There are currently four module groups: Security, Health, Telecare, and Accessibility. Each of these groups includes a series of modules that the user or the carer can choose from to best suit the elderly person ’s needs. Of course, these needs are expected to change over time. Initially, health-module components (e.g., a fitness tracker) may be of interest, while with advanced age, accompanied perhaps with emerging dementia or other cognitive impairments, security and telecare modules become more prominent. In particular when considering a diminishing mental capacity, it is important that the user is already familiar with the IAC, as it is easier to adapt to new modules on familiar devices than to a completely new hardware.
Security modules are intended to boost the user’s sense of security or safety in order to prolong their independent life. The fall-detection module detects a fall or similar dangerous situations and automatically calls for help. The user can also call for help manually, using the SOS button, by pressing it for a couple of seconds. In both cases, the localization module can retrieve the user’s location, either outdoor via GPS or indoor using triangulation with previously mapped Wi-Fi routers or Bluetooth beacons. For the elderly with advanced stages of dementia who may wander away and get lost, carers can set the geo-fencing module. If the user leaves a pre-defined area, the carer is notified.
Most of the users who are still active are more interested in the health modules than in the security ones. General activity monitoring can be used as a sort of fitness tracker. As the system is built on an open platform, it is possible to connect it to external devices, such as a blood-pressure or ECG monitors, and interpret the results. Some smartwatches already include in-built heart-rate and sweating sensors, which can be used for stress detection . Some modules are designed to assist with the management of chronic diseases, such as diabetes  or chronic heart failure (CHF) [14, 15], by monitoring a user’s activities and issuing suggestions . Some of these modules are still in a development phase, while others are already being tested.
Telecare modules are mostly intended to facilitate the interaction between the users, the carers, and the support services, such as a call center. Examples of telecare modules include reminders (e.g., “Take pills”) and scheduling carer tasks. This group also includes modules for the management of settings, modules, phone numbers, etc. Carers have a complete overview of the devices’ logs for alarms and other signals.
These modules facilitate the use of the system for the elderly. The display of a smartwatch is relatively small and some people find it difficult to interact with. The voice interaction module allows the user to control the watch with a small number of simple voice commands. It also converts the watch’s feedback into voice/sound alerts. Another simplification is the NFC album, which allows triggering of a predetermined action by approaching a chosen NFC tag in the album.
3 Fall detection
As discussed above, a system that allows the user to call for help in case of emergency, or even calls automatically, is highly desired by the elderly users to increase their sense of security. Currently, there are two approaches when looking at fall detection and monitoring systems—external and body-worn [17, 18]. External systems typically rely on cameras or similar sensors installed in the living environment . Since they are stationary, they are easy to charge and maintain, but they do not work outdoors and are typically perceived as invading the user’s privacy. Several contemporary systems include a manually triggered SOS button—but such systems fail if the user cannot reach them or is unable to press the button. On the other hand, body-worn sensors can be carried around everywhere. They do not rely on image processing but detect movement based on acceleration. This makes them more acceptable. The downsides include more frequent charging of the battery, and there is also the minimum requirement that the user actually wears the sensor. Another issue could be the unwillingness of the user to wear an unusual device, or even perceive it as a personal stigma—indicating to friends and family members that they require help. This is especially an issue with several contemporary solutions that take the shape of a pendant or a belt-worn sensor. While these locations on the body have proven to be beneficial for fall recognition , they are less easily embraced by the users. On the other hand, as people are used to wearing wristwatches , it is reasonable to integrate the system into a smartwatch—as in the case of the IAC. Here, we demonstrate the development of the algorithm we used in several steps. Most of the commercially available devices (at least those where the algorithm is disclosed) employ some sort of threshold-based method. We further expand this with a ML approach and in combination with additional sensors.
Most contemporary body-worn systems for fall detection are acceleration-based. During a fall, the person is subjected to large and characteristic accelerations that can be recognized and classified with an appropriate method. In this regard, wrist-worn devices are notoriously poor at fall recognition [22, 23] as people perform numerous activities with their hands; many of those can, to some degree, resemble a fall if looked at solely from the point of view of the acceleration patterns.
This approach does, however, not detect falls when the person continues to make substantial movements afterwards, but cannot get up. In this case, we assume that the person can trigger the SOS button manually—thus, the automatic call takes place only when the person is unable to move after the fall.
Due to the limitations of processing power and more importantly battery life on a smartwatch, we did not run ML methods continuously. First, a threshold based method 3.1.1 was tuned to have a sensitivity of 1 in order to detect all possible falls. When such an event was detected, a ML method was run. In this way, ML methods are calculated less frequently, while the accuracy stays at least the same as a continuous run because of the high sensitivity.
3.3 Laboratory testing
In order to determine the best method for fall recognition, we devised a laboratory test where we recorded an elaborate scenario of everyday activities and possible falls on six users (three males and three females, all healthy young adults for safety reasons), equipped with multiple sensors (on both hands, waist, ankle, and a pendant) .
We devised 26 non-fall and 15 fall scenarios, where some fall scenarios have several variants (e.g., fall backward or forward). The test subjects were not instructed on how to perform the falls exactly and were told to just fall as they think they would in a real-life situation. Scenarios included most of the common ways a person can fall: tripping or stumbling, slipping, fainting (no hands), collapse (drop attack), arising from a bed and falling, arising from a chair and falling (including grabbing a table or chair to try and prevent the fall), missing the bed and falling on the ground, sitting on the edge of a chair and falling down, hitting a closet and falling, rolling out of bed, walking and losing balance, dizziness, and falling when picking up something. Due to safety reasons, we did not record more dangerous scenarios, such as falling down the stairs.
Non-fall activities were devised in multiple subgroups, i.e., normal everyday activities like walking, collapsing into a chair, standing up and lying on a bed, and activities that might resemble falling, i.e., stumbling but catching balance, quickly lying on a bed, and quickly changing position in bed. We also included some exercise activities and normal everyday activities such as cooking, talking, working on a PC, reading, playing cards, and similar.
In total, there were 48 non-fall recordings and 83 falls. In some of the fall scenarios, the subjects were trying to recover after the fall, while in others they were lying still, unable to move. With this approach, we attempted to recreate real-life falls as closely as possible.
We did some light pre-processing on the data in order to unify the input for all the methods tested and to eliminate some of the noise caused by the sensor error. First, we applied a low-pass filter to smooth out the signal. After that, we split the recordings into segments containing only one of the events. We classified this segments binary as fall or non-fall.
For testing, we used the leave-one-person-out approach, where we trained the model on five people and then tested it on the remaining one.
Results of laboratory tests for a simple threshold-based model
Comparison of different ML methods for fall detection
DNN (4 layers)
We noticed that the tested methods typically perform better for the left hand as opposed to the right hand. The reason behind this is the fact that all the volunteers in the test were right-handed. As they are more active with their dominant right hand, there is less noise on the left-hand data. In addition, all the ML-based methods outperform the simple threshold-based approach; however, they are also significantly more computationally intensive. Of the tested ML-based methods, the best average accuracy is achieved for kNN, with k=5. The kNN method is easier to implement on a smartwatch (compared to DNN, for example) and works sufficiently quickly for our purposes.
In the next step, we added the barometer data, which further increased the accuracy of the kNN method by 1.9%. We experimented with continuous learning for both the single- and multiple-users-based (collaborative) approach. The problem with this approach is that any mis-labeling by the user can greatly reduce the accuracy of the model in the future. A sufficiently large number of users could even allow us to create various user types with different habits, presenting an improvement over the learning on the entire set.
4 Evaluation on real users and discussion
For pilot testing of the system, we recruited 150 volunteers to test our system in real-life conditions for at least 3 months each. We performed the tests in two consecutive rounds—this approach allowed us to work with a manageable number of users, lowered the hardware costs, and allowed us to enroll minor fixes to the bugs encountered in the first round before starting with the second round.
4.1 Experimental setting
During the pilot test, each user received one model of the watch (out of three models, the only difference being the hardware (size, battery, charger)). At the beginning, the type was distributed randomly, but towards the end, more weight was put on the type that was best received by the users.
By default most of the modules from the security and telecare group were turned on. Users had the option to turn on/off additional modules, but in general never chose to do so. For the fall detection, we used both threshold-based and kNN methods. We chose kNN since it gave the best results in a laboratory test and was best at classifying unseen events. Because of battery constraints, we also used threshold-based (uses much less battery than kNN). It later turned out that an increase in battery life is preferred by the users, compared to better accuracy of the kNN. Only one method was active at any given time.
The users and their carers were instructed in how to use the system beforehand, and the carers were encouraged to help charge the watch or to remind users to wear them in case they forgot. This was especially emphasized in homes for the elderly, for users suffering from dementia. Other than that users were not instructed to behave any differently because of the system. The main goal of the pilot test was to see how well the system performs in a real-life setting with currently available technology (both hardware and software).
4.2 User profile
All the users in our tests were over 65 years, with the average age being roughly 78. We decided to include different profiles of users: users that live independently at home (without help), users in sheltered housing, and users in nursing homes. Around 50 of the users were in nursing homes under the constant care of caregivers; the rest were living independently, either alone or with their spouses or family. The majority of the users had mild cognitive impairment (mild dementia), while some showed no signs of it at all. Only two had advanced stages of dementia. The majority of the users were able to move without help. On average, the users who lived at home seemed to have been in better physical and mental health and were also slightly younger.
4.2.1 User feedback
After 3 months of testing, the feedback from the users was generally positive. Acceptance of the systems was around 60%, we elaborate on this in Section 4.2.2.
The analysis of usage patterns revealed that the users living in nursing homes and being taken care of (food, hygiene, living space) already felt safe while having nursing staff around them. While agreeing to participate in the project, they in most cases stored the watch in a drawer and forgot it existed shortly after. When asked about it, they said they did not feel the need for wearing such a system, knowing that the carers would take care of them if anything happened.
On the other hand, most users living at home (alone or with their spouses) continued to use the system throughout the whole testing phase. When asked about it, they confirmed they felt safer while using the system. About half of this group only wore the watch when going outside (and alone). Most of the users actually started going out more, as they felt no danger in doing so. For example, a 77-year-old user started riding his motorcycle again, while another 80-year-old resumed with some light chores in a nearby forest. Both users agreed that they could do this because they felt that someone would always find them in case anything happened to them.
We asked the users for technical feedback and one of the most frequent requests was a desire for a longer battery life (which, with the current hardware, lasted about a day). The users needed to remember to charge their devices daily, which turned out to be more difficult to remember for patients with dementia. Other users had some trouble with the volume of the device or with the size of the screen being too small to see without glasses. Surprisingly, there were no complaints regarding the false alarms. Users said they understood that the system cannot be perfect and that they would rather have occasional false alarms than the other way around.
4.2.2 Usage analysis
We deployed the watches to the users in multiple waves over the first few weeks. After approximately 3 months of using the system, 65% of the users used the system at least once in the previous 5 days. It is interesting to observe that if the user used the system continuously for the first 5 days, he/she was almost four times as likely to continue to use the system in the future—as opposed to those who used it only once or twice during the first week.
Total events recorded in the pilot test and their distribution
Total events recorded
Manual alarms (SOS button)
Watch not worn
“No-movement” events relate to the general activity monitoring module, where the system recognizes that the user is less active than in the previous days—which could be a possible concern. In most cases, those events were false positives, as the watch was not able to determine that it was not on the user’s hand. There were two cases when the person was feeling ill and was lying in bed more than usual.
The most frequent notification was that the user was not wearing the watch. Our system was designed to be worn 24 h a day—otherwise it sends a reminder to the carer.
We offered the system to different groups of elderly, which is clearly reflected in the system usage. The system was best accepted by the active (generally younger) user group, some of whom actively contacted us in advance in order to test the system. We believe that such systems can greatly benefit the users both physically (monitoring) and psychologically (sense of security). However, the introduction of such bleeding-edge technology to the elderly may prove problematic because the users can have problems learning to use it. Therefore, as discussed at the beginning, we believe that the solution lies in a multi-modal system that can be introduced to the users early, with different solutions that can be enabled by users’ needs as they age.
We noticed some problems with the hardware device, the main problem being the battery life. Ideally, the system would function for months with a single charging, while it currently works only for about a day. We implemented several solutions to increase the battery life by accurately determining user actions and, for example, lowered the sampling frequency, disconnected the device from the network, and put the device into a sleep mode while the user was sleeping. As soon as the user started to move around, the device went into normal operating mode again. With this approach, we more than doubled the battery life, which is progress. There were also some problems with the device’s size and weight, mostly from women with generally smaller hands. But these issues can be solved by creating a custom device with a focus on these aspects.
ICT solutions are promising for making aging a better experience, by improving the user’s quality of life, lowering the costs, and making the interactions with all involved easier. In this paper, we present the concept of a system that assists in all three fields—the intelligent assistant carer takes care of the user in a variety of situations, 24/7, and helps the human carer when needed.
We discuss the two features of the system—being modular, the IAC allows personalization by enabling custom modules (for example, fall detection within the group of security-related modules, running on a smartwatch), while being built on an open platform allows for quick modifications for improvements (such as adding a barometer to the accelerometer sensor in order to improve the fall-detection algorithms). We briefly present a couple of approaches with various algorithms we used based on the data obtained in laboratory conditions (the best being kNN) and during the pilot testing of the system.
The results of the pilot tests are in agreement with our premise that it is beneficial to introduce the IAC to the users as early as possible, since it is more difficult for people with various levels of cognitive impairments to learn to use such a system from scratch. Users suffering from dementia had a poor acceptance rate of the system, while the active users showed an increase in their physical activities, which has a beneficial effect on the overall health of the user. While the users agreed that the system has numerous beneficial effects on their lifestyle (security, health, socialization through alarm/SOS calls), they also pointed out some shortcomings, mainly with respect to the hardware. The battery life is one of the key aspects that needs to improve, followed by the size and ergonomics of the watch. While we are currently using commercial devices, building custom designs should help us to avoid several of these shortcomings and improve the user experience.
1 Earth’s gravity causes all stationary objects to experience 1 G. Because of this, the fall threshold is reached when the person is close to free falling (0 G).
The research leading to these results has received funding from the EU H2020 project IN LIFE (grant agreement no. 643442).
All authors contributed equally. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- W Lutz, W Sanderson, S Scherbov, The coming acceleration of global population ageing. Nature. 451(7179), 716–719 (2008).View ArticleGoogle Scholar
- Statistics on Regional Population Projections. http://ec.europa.eu/eurostat/statistics-explained/index.php/Statistics_on_regional_population_projections. Accessed 31 July 2017.
- What Is Active Ageing? http://www.who.int/ageing/active_ageing/en. Accessed 31 July 2017.
- AutoAlert Automatic Fall Detection. https://www.lifeline.philips.com/automatic-fall-detection.html. Accessed 31 July 2017.
- Medical Guardian - Medical Alert System. https://www.medicalguardian.com/product/classic-guardian. Accessed 31 July 2017.
- Omate Seniors Smartwatch. https://www.omate.com/s3/. Accessed 31 July 2017.
- MJ Deen, Information and communications technologies for elderly ubiquitous healthcare in a smart home. Pers. Ubiquit. Comput.19(3), 573–599 (2015). doi:10.1007/s00779-015-0856-x.View ArticleGoogle Scholar
- IN LIFE Project. http://www.inlife-project.eu/index.php. Accessed 31 July 2017.
- ME Tinetti, CS Williams, Falls, injuries due to falls, and the risk of admission to a nursing home. N. Engl. J. Med. 337(18), 1279–1284 (1997).View ArticleGoogle Scholar
- LD Gillespie, WJ Gillespie, MC Robertson, SE Lamb, RG Cumming, BH Rowe, et al., Interventions for preventing falls in elderly people. Cochrane Database Syst. Rev. 4: (2003).Google Scholar
- D Wild, U Nayak, B Isaacs, How dangerous are falls in old people at home?Br. Med. J. (Clin. Res. Ed.)282(6260), 266–268 (1981).View ArticleGoogle Scholar
- M Gjoreski, H Gjoreski, M Luštrek, M Gams, in Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. Continuous stress detection using a wrist device: in laboratory and real life (ACM, 2016), pp. 1185–1193.Google Scholar
- B Cvetković, R Milić, M Luštrek, Estimating energy expenditure with multiple models using different wearable sensors. IEEE J. Biomed. Health Inform.20(4), 1081–1087 (2016).View ArticleGoogle Scholar
- Cvetkovic, B́, A Vodopija, D Rudel, Z Balorda, M Luštrek, in IJCAI 2016 – Workshop on Knowledge Discovery in Healthcare Data, 71. Hospitalisation prediction from telemonitoring data in congestive heart failure patients, (2016), pp. 9–4.Google Scholar
- Heart Man. http://heartman-project.eu/content/what-heartman. Accessed 31 July 2017.
- V Janko, B Cvetković, A Gradišek, M Luštrek, Štrumbelj B, T Kajtna, in Journal of Ambient Intelligence and Smart Environments, vol. 9. E-gibalec: Mobile application to monitor and encourage physical activity in schoolchildren (IOS Press, 2017), pp. 595–609.Google Scholar
- M Mubashir, L Shao, L Seed, A survey on fall detection: Principles and approaches. Neurocomputing. 100:, 144–152 (2013).View ArticleGoogle Scholar
- X Yu, in E-health Networking, Applications and Services, 2008. HealthCom 2008. 10th International Conference On. Approaches and principles of fall detection for elderly and patient (IEEE, 2008), pp. 42–47.Google Scholar
- E Auvinet, F Multon, A Saint-Arnaud, J Rousseau, J Meunier, Fall detection with multiple cameras: An occlusion-resistant method based on 3-d silhouette vertical distribution. IEEE Trans. Inf. Technol. Biomed.15(2), 290–300 (2011).View ArticleGoogle Scholar
- A Clark, Recommended Medical Alert Systems. https://www.theseniorlist.com/2015/01/2015-recommended-medical-alert-systems. Accessed 31 July 2017.
- JH Bergmann, V Chandaria, A McGregor, Wearable and implantable sensors: the patient’s perspective. Sensors. 12(12), 16695–16709 (2012).View ArticleGoogle Scholar
- M Kangas, A Konttila, P Lindgren, I Winblad, T Jämsä, Comparison of low-complexity fall detection algorithms for body attached accelerometers. Gait Posture. 28(2), 285–291 (2008).View ArticleGoogle Scholar
- M Gjoreski, H Gjoreski, M Luštrek, M Gams, How accurately can your wrist device recognize daily activities and detect falls?. Sensors. 16(6), 800 (2016).View ArticleGoogle Scholar
- Y Ge, B Xu, Detecting falls using accelerometers by adaptive thresholds in mobile devices. JCP. 9(7), 1553–1559 (2014).View ArticleMathSciNetGoogle Scholar
- M Luštrek, H Gjoreski, S Kozina, B Cvetković, V Mirchevska, M Gams, in Proceedings of the Twenty-Third Innovative Applications of Artificial Intelligence Conference. Detecting falls with location sensors and accelerometers, (2011), pp. 1662–1667.Google Scholar
- M Bochanovski, H Gjoreski, J Bizjak, Matjaz, Ǧ, in Proceedings 19th International Multiconference IS 2016, ed. by M Bajec. Smartwatch fall detection (Jožef Stefan Institute, 2016), pp. 17–20.Google Scholar
- AmI Repository. https://dis.ijs.si/ami-repository/. Accessed 31 July 2017.