Document Type : Research Articles

Authors

1 Faculty of Electrical and Robotic Engineering, Shahrood University of Technology

2 Shahrood University of Technology, Shahrood, Iran

3 Electrical department, Faculty of Engineering, Ferdowsi University of Mashhad

Abstract

In this paper, the main focus is on blood glucose level control and the possible sensor and actuator faults which can be observed in a given system. To this aim, the eligibility traces algorithm (a Reinforcement Learning method) and its combination with sliding mode controllers is used to determine the injection dosage. Through this method, the optimal dosage will be determined to be injected to the patient in order to decrease the side effects of the drug. To detect the fault in the system, residual calculation techniques are utilized. To calculate the residual, it is required to predict states of the normal system at each time step, for which, the Radial Basis Function neural network is used. The proposed method is compared with another reinforcement learning method (Actor-Critic method) with its combination with the sliding mode controller. Finally, both RL-based methods are compared with a combinatory method, Neural network and sliding mode control. Simulation results have revealed that the eligibility traces algorithm and actor-critic method can control the blood glucose concentration and the desired value can be reached, in the presence of the fault. However, in addition to the reduced injected dosage, the eligibility traces algorithm can provide lower variations about the desired value. The reduced injected dosage will result in the mitigated side effects, which will have considerable advantages for diabetic patients.

Keywords

Main Subjects

[1] N. C. Van Der Ven et al., "The confidence in diabetes
self-care scale: psychometric properties of a new
measure of diabetes-specific self-efficacy in Dutch and
US patients with type 1 diabetes," Diabetes care, vol.
26, no. 3, pp. 713-718, 2003.

[2] A. A. Sharief and A. Sheta, "Developing a mathematical
model to detect diabetes using multigene genetic
programming," IJARAI) International Journal of
Advanced Research in Artificial Intelligence, vol. 3, no.
10, 2014.

[3] M. O. M. Javad, S. Agboola, K. Jethwani, I. Zeid, and
S. Kamarthi, "Reinforcement Learning Algorithm for
Blood Glucose Control in Diabetic Patients," in ASME
2015 International Mechanical Engineering Congres
and Exposition, 2015: American Society of Mechanical
Engineers, pp. V014T06A009-V014T06A009.

[4] I. Hochberg, G. Feraru, M. Kozdoba, S. Mannor, M.
Tennenholtz, and E. Yom-Tov, "A reinforcement
learning system to encourage physical activity in
diabetes patients," arXiv preprint arXiv:1605.04070,
2016.

[5] A. Noori and M. A. Sadrnia, "Glucose level control
using Temporal Difference methods," in 2017 Iranian
Conference on Electrical Engineering (ICEE), 2017:
IEEE, pp. 895-900.

[6] W.-H. Weng, M. Gao, Z. He, S. Yan, and P. Szolovits,
"Representation and reinforcement learning for
personalized glycemic control in septic patients," arXiv
preprint arXiv:1712.00654, 2017.
[7] P. D. Ngo, S. Wei, A. Holubová, J. Muzik, and F.
Godtliebsen, "Control of Blood Glucose for Type-1
Diabetes by Using Reinforcement Learning with
Feedforward Algorithm," Computational and
mathematical methods in medicine, vol. 2018, 2018.

[8] J. Skach, I. Punčochář, and F. L. Lewis, "Temporal-
difference Q-learning in active fault diagnosis," in 2016
3rd Conference on Control and Fault-Tolerant Systems
(SysTol), 2016: IEEE, pp. 287-292.

[9] J. Cao, "Using reinforcement learning for agent-based
network fault diagnosis system," in 2011 IEEE
International Conference on Information and
Automation, 2011: IEEE, pp. 750-754.

[10] J. Škach and I. Punčochář, "Input design for fault
detection using extended kalman filter and
reinforcement learning," IFAC-PapersOnLine, vol. 50,
no. 1, pp. 7302-7307, 2017.

[11] F. Farivar and M. N. Ahmadabadi, "Continuous
reinforcement learning to robust fault tolerant control
for a class of unknown nonlinear systems," Applied Soft
Computing, vol. 37, pp. 702-714, 2015.

[12] K.-Z. Han, J. Feng, and X. Cui, "Fault-tolerant
optimised tracking control for unknown discrete-time
linear systems using a combined reinforcement learning
and residual compensation methodology," International
Journal of Systems Science, vol. 48, no. 13, pp. 2811-
2825, 2017.

[13] D. Zhang, Z. Lin, and Z. Gao, "Reinforcement-learning
based fault-tolerant control," in 2017 IEEE 15th
International Conference on Industrial Informatics
(INDIN), 2017: IEEE, pp. 671-676.

[14] H. H. Afshari, D. Al-Ani, and S. Habibi, "Fault
Prognosis of Roller Bearings Using the Adaptive Auto-
Step Reinforcement Learning Technique," in ASME
2014 Dynamic Systems and Control Conference, 2014:
American Society of Mechanical Engineers Digital
Collection.

[15] P. Herrero et al., "Robust fault detection system for
insulin pump therapy using continuous glucose
monitoring," Journal of diabetes science and
technology, vol. 6, no. 5, pp. 1131-1141, 2012.

[16] Z. Mahmoudi, K. Nørgaard, N. K. Poulsen, H. Madsen,
and J. B. Jørgensen, "Fault and meal detection by
redundant continuous glucose monitors and the
unscented Kalman filter," Biomedical Signal Processing
and Control, vol. 38, pp. 86-99, 2017.

[17] K. Turksoy, I. Hajizadeh, E. Littlejohn, and A. Cinar,
"Multivariate statistical monitoring of sensor faults of a
multivariable artificial pancreas," IFAC-PapersOnLine,
vol. 50, no. 1, pp. 10998-11004, 2017.

[18] K. Kölle, A. L. Fougner, K. A. F. Unstad, and Ø.
Stavdahl, "Fault detection in glucose control: Is it time
to move beyond CGM data?," IFAC-PapersOnLine, vol.
51, no. 27, pp. 180-185, 2018.

[19] X. Yu et al., "Fault Detection in Continuous Glucose
Monitoring Sensors for Artificial Pancreas Systems,"
IFAC-PapersOnLine, vol. 51, no. 18, pp. 714-719,
2018.

[20] I. Contreras and J. Vehi, "Artificial intelligence for
diabetes management and decision support: literature
review," Journal of medical Internet research, vol. 20,
no. 5, p. e10775, 2018.

[21] A. Khajeh and Z. Shabani, "Adaptive Gain Scheduling
Control of Doubly Fed Induction Generator Based Wind
Turbines to Improve Fault Ride Through Performance,"
International Journal of Industrial Electronics, Control
and Optimization, vol. 1, no. 1, pp. 61-70, 2018.
[22] R. Sedaghati and M. R. Shakarami, "A New Sliding
Mode-based Power Sharing Control Method for
Multiple Energy Sources in the Microgrid under
Different Conditions," International Journal of
Industrial Electronics, Control and Optimization, vol. 2,
no. 1, pp. 25-38, 2019.

[23] S. M. E. Oliaee, "Fault Detection and Identification of
High Dimension System by GLOLIMOT,"
International Journal of Industrial Electronics, Control
and Optimization, vol. 2, no. 4, pp. 331-342, 2019.

[24] S. Baniardalani, "Fault Diagnosis of Discrete-Time
Linear Systems Using Continuous Time Delay Petri
Nets," International Journal of Industrial Electronics,
Control and Optimization, vol. 3, no. 1, pp. 81-90, 2020.

[25] A. Roy and R. S. Parker, "Dynamic modeling of
exercise effects on plasma glucose and insulin levels,"
ed: SAGE Publications, 2007.

[26] P. Magni and R. Bellazzi, "A stochastic model to assess
the variability of blood glucose time series in diabetic
patients self-monitoring," IEEE Transactions on
biomedical engineering, vol. 53, no. 6, pp. 977-985,
2006.

[27] A. Makroglou, J. Li, and Y. Kuang, "Mathematical
models and software tools for the glucose-insulin
regulatory system and diabetes: an overview," Applied
numerical mathematics, vol. 56, no. 3-4, pp. 559-573,
2006.

[28] Y. C. Kueh, T. Morris, E. Borkoles, and H. Shee,
"Modelling of diabetes knowledge, attitudes, self-
management, and quality of life: a cross-sectional study
with an Australian sample," Health and quality of life
outcomes, vol. 13, no. 1, p. 129, 2015.

[29] F. Nani and M. Jin, "Mathematical modeling and
simulations of the pathophysiology of Type-2 Diabetes
Mellitus," in 2015 8th International Conference on
Biomedical Engineering and Informatics (BMEI), 2015:
IEEE, pp. 296-300.

[30] J. R. Moore and F. Adler, "Mathematical modeling of
type 1 diabetes in the NOD mouse: separating incidence
and age of onset," arXiv preprint arXiv:1412.6566,
2014.

[31] A. Mahata, S. P. Mondal, S. Alam, and B. Roy,
"Mathematical model of glucose-insulin regulatory
system on diabetes mellitus in fuzzy and crisp
environment," Ecological Genetics and Genomics, vol.
2, pp. 25-34, 2017.

[32] E. Lehmann and T. Deutsch, "A physiological model of
glucose-insulin interaction in type 1 diabetes mellitus,"
Journal of biomedical engineering, vol. 14, no. 3, pp.
235-242, 1992.

[33] A. Onvlee, H. Blauw, N. Middelhuis, and H. Zwart, "In
silico modeling of patients with type 1 diabetes
mellitus," MS thesis, Dep. Tech. Med., Univ. Twente,
Enschede, Netherlands, 2016.

[34] P. Palumbo, S. Panunzi, and A. De Gaetano,
"Qualitative behavior of a family of delay-differential
models of the glucose-insulin system," Discrete and
Continuous Dynamical Systems Series B, vol. 7, no. 2,
p. 399, 2007.

[35] R. S. Sutton and A. G. Barto, Reinforcement learning:
An introduction. MIT press, 2018.

[36] J.-J. E. Slotine and W. Li, Applied nonlinear control (no.
1). Prentice hall Englewood Cliffs, NJ, 1991.