Evolving connectionist systems for on-line, knowledge-based learning: Principles and applications

dc.contributor.author	Kasabov, Nikola	en_NZ
dc.date.copyright	1999-03	en_NZ
dc.identifier.citation	Kasabov, N. (1999). Evolving connectionist systems for on-line, knowledge-based learning: Principles and applications (Information Science Discussion Papers Series No. 99/02). University of Otago. Retrieved from http://hdl.handle.net/10523/1122	en
dc.identifier.uri	http://hdl.handle.net/10523/1122
dc.description.abstract	The paper introduces evolving connectionist systems (ECOS) as an effective approach to building on-line, adaptive intelligent systems. ECOS evolve through incremental, hybrid (supervised/unsupervised), on-line learning. They can accommodate new input data, including new features, new classes, etc. through local element tuning. New connections and new neurons are created during the operation of the system. The ECOS framework is presented and illustrated on a particular type of evolving neural networks---evolving fuzzy neural networks (EFuNNs). EFuNNs can learn spatial-temporal sequences in an adaptive way, through one pass learning. Rules can be inserted and extracted at any time of the system operation. The characteristics of ECOS and EFuNNs are illustrated on several case studies that include: adaptive pattern classification; adaptive, phoneme-based spoken language recognition; adaptive dynamic time-series prediction; intelligent agents.	en_NZ
dc.format.mimetype	application/pdf
dc.publisher	University of Otago	en_NZ
dc.relation.ispartofseries	Information Science Discussion Papers Series	en_NZ
dc.subject	evolving connectionist systems	en_NZ
dc.subject	on-line learning	en_NZ
dc.subject	spatial-temporal adaptation	en_NZ
dc.subject	adaptive speech recognition	en_NZ
dc.subject	Evolving Fuzzy Neural Networks	en_NZ
dc.subject.lcsh	QA76 Computer software	en_NZ
dc.title	Evolving connectionist systems for on-line, knowledge-based learning: Principles and applications	en_NZ
dc.type	Discussion Paper	en_NZ
dc.description.version	Unpublished	en_NZ
otago.bitstream.pages	33	en_NZ
otago.date.accession	2010-11-10 20:38:10	en_NZ
otago.school	Information Science	en_NZ
otago.openaccess	Open
otago.place.publication	Dunedin, New Zealand	en_NZ
dc.identifier.eprints	986	en_NZ
otago.school.eprints	Knowledge Engineering Laboratory	en_NZ
otago.school.eprints	Information Science	en_NZ
dc.description.references	1. Albus, J.S., A new approach to manipulator control: The cerebellar model articulation controller (CMAC), Tarns. of the ASME: Journal of Dynamic Systems, Measurement, and Control, pp.220:227, Sept. (1975) 2. Amari, S. and Kasabov, N. eds, “Brain-like Computing and Intelligent Information Systems”, Springer Verlag,1998. 3. Amari, S., Mathematical foundations of neuro-computing, Proc. of IEEE, 78 (9), Sept. (1990) 4. Arbib, M. (ed) The Handbook of Brain Theory and Neural Networks,The MIT Press, 1995. 5. Bollacker, K., S.Lawrence and L.Giles, CiteSeer: An autonomous Web agent for automatic retrieval and identification of interesting publications, 2nd International ACM conference on autonomous agents, ACM Press, 1998, 116-123 6. Bottu and Vapnik, “Local learning computation”, Neural Computation, 4, 888-900 (1992) 7. Carpenter, G. and Grossberg S., Pattern recognition by self-organizing neural networks , The MIT Press, Cambridge, Massachusetts (1991) 8. Carpenter, G. and S. Grossberg, “ART3: Hierarchical search using chemical transmitters in self-organising pattern-recognition architectures”, Neural Networks, 3(2) 129-152(1990). 9. Carpenter, G. S. Grossberg, N. Markuzon, J.H. Reynolds, D.B. Rosen, “FuzzyARTMAP: A neural network architecture for incremental supervised learning of analog multi-dimensional maps,” IEEE Transactions of Neural Networks , vol.3, No.5, 698-713 (1991). 10. Cybenko, G., Approximation by super-positions of sigmoidal function, Mathematics of Control, Signals and Systems, 2, 303-314 (1989) 11. DeGaris, H., “Circuits of Production Rule - GenNets – The genetic programming of nervous systems”, in: Albrecht, R., Reeves, C. and Steele, N. (eds) Artificial Neural Networks and Genetic Algorithms, Springer Verlag (1993) 12. Duda and Hart, “Pattern classification and scene analysis”, New York: Willey (1973) 13. Edelman, G., Neuronal Darwinism: The theory of neuronal group selection, Basic Books (1992). 14. Elman, J., E.Bates, M.Johnson, A.Karmiloff-Smith, D.Parisi and K.Plunkett, Rethinking Innateness (A Connectionist Perspective of Development), The MIT Press, 1997 15. Fahlman, C., and C. Lebiere, "The Cascade-Correlation Learning Architecture", in: Turetzky, D (ed) Advances in Neural Information Processing Systems, vol.2, Morgan Kaufmann, 524-532 (1990). 16. Farmer, J.D., and Sidorowitch, Predicting chaotic time series, Physical Review Letters, 59, 845 (1987) 17. Freeman, J., D. Saad, “On-line learning in radial basis function networks”, Neural Computation vol. 9, No.7 (1997). 18. French, “Semi-destructive representations and catastrophic forgetting in connectionist networks, Connection Science, 1, 365-377 (1992) 19. Fritzke, B. “A growing neural gas network learns topologies”, Advances in Neural Information Processing Systems, vol.7 (1995). 20. Fukuda, T., Y. Komata, and T. Arakawa, "Recurrent Neural Networks with Self-Adaptive GAs for Biped Locomotion Robot", In: Proceedings of the International Conference on Neural Networks ICNN'97, IEEE Press (1997) 21. Funuhashi, K., On the approximate realization of continuous mappings by neural networks, Neural Networks, 2, 183-192 (1989) 22. Gaussier, T., and S. Zrehen, “A topological neural map for on-line learning: Emergence of obstacle avoidance in a mobile robot”, In: From Animals to Animats No.3, 282-290, (1994). 23. Goldberg, D.E., Genetic Algorithms in Search, Optimisation and Machine Learning, Addison-Wesley (1989) 24. Goodman, R., C.M. Higgins, J.W. Miller, P.Smyth, "Rule-based neural networks for classification and probability estimation", Neural Computation, 14, 781-804 (1992). 25. Hashiyama, T., T. Furuhashi, Y Uchikawa,. “A Decision Making Model Using a Fuzzy Neural Network”, in: Proceedings of the 2nd International Conference on Fuzzy Logic & Neural Networks, Iizuka, Japan, 1057-1060, (1992). 26. Hassibi and Stork, “Second order derivatives for network pruning: Optimal Brain Surgeon,” in: Advances in Neural Information Processing Systems, 4, 164-171, (1992). 27. Hech-Nielsen, R. “Counter-propagation networks”, IEEE First int. conference on neural networks, San Diego, vol.2, pp.19-31 (1987) 28. Heskes, T.M., B. Kappen, “On-line learning processes in artificial neural networks”, in: Math. foundations of neural networks, Elsevier, Amsterdam, 199-233, (1993). 29. Ishikawa, M., "Structural Learning with Forgetting", Neural Networks 9, 501-521, (1996). 30. Kasabov, N. "Adaptable connectionist production systems”. Neurocomputing, 13 (2-4) 95-117, (1996). 31. Kasabov, N. The ECOS Framework and the ECO Learning Method for Evolving Connectionist Systems, Journal of Advanced Computational Intelligence, 2 (6) 1998, 1-8 32. Kasabov, N., "Investigating the adaptation and forgetting in fuzzy neural networks by using the method of training and zeroing", Proceedings of the International Conference on Neural Networks ICNN'96, Plenary, Panel and Special Sessions volume, 118-123 (1996). 33. Kasabov, N., "Learning fuzzy rules and approximate reasoning in fuzzy neural networks and hybrid systems", Fuzzy Sets and Systems 82 (2) 2-20 (1996). 34. Kasabov, N., “A framework for intelligent conscious machines utilising fuzzy neural networks and spatial temporal maps and a case study of multilingual speech recognition", in: Amari, S. and Kasabov, N. (eds) Brain-like computing and intelligent information systems, Springer Verlag, 106-126 (1998) 35. Kasabov, N., “ECOS: A framework for evolving connectionist systems and the ECO learning paradigm”, Proc. of ICONIP'98, Kitakyushu, Japan, Oct. 1998, IOS Press, 1222-1235 36. Kasabov, N., “Evolving Fuzzy Neural Networks - Algorithms, Applications and Biological Motivation”, in: in: Yamakawa and Matsumoto (eds), Methodologies for the Conception, design and Application of Soft Computing, World Scientific, 1998, 271-274 37. Kasabov, N., E. Postma, and J. Van den Herik, “AVIS: A Connectionist-based Framework for Integrated Audio and Visual Information Processing”, in Proc. of Iizuka'98, Iizuka, Japan, Oct.1998. 38. Kasabov, N., Foundations of Neural Networks, Fuzzy Systems and Knowledge Engineering, The MIT Press, CA, MA, (1996). 39. Kasabov, N., J. S Kim, M. Watts, A. Gray, “FuNN/2- A Fuzzy Neural Network Architecture for Adaptive Learning and Knowledge Acquisition”, Information Sciences - Applications, 101(3-4): 155-175 (1997) 40. Kasabov, N., M. Watts, “Genetic algorithms for structural optimisation, dynamic adaptation and automated design of fuzzy neural networks”, in: Proceedings of the International Conference on Neural Networks ICNN'97, IEEE Press, Houston (1997). 41. Kasabov, N., R. Kozma, R. Kilgour, M. Laws, J. Taylor, M. Watts, and A. Gray, “A Methodology for Speech Data Analysis and a Framework for Adaptive Speech Recognition Using Fuzzy Neural Networks and Self Organising Maps”, in: Kasabov and Kozma (eds) Neuro-fuzzy techniques for intelligent information systems, Physica Verlag (Springer Verlag) 1999 42. Kasabov, N., Song, Q. “Dynamic, evolving fuzzy neural networks with ‘m-out-of-n’ activation nodes for on-line adaptive systems” , TR 99/04, Department of Information Science, University of Otago (1999) 43. Kasabov, N., Watts, M. Spatial-temporal evolving fuzzy neural networks STEFuNNs and applications for adaptive phoneme recognition, TR 99/03 Department of Information Science, University of Otago (1999) 44. Kasabov, N., Woodford, B. Rule Insertion and Rule Extraction from Evolving Fuzzy Neural Networks: Algorithms and Applications for Building Adaptive, Intelligent Expert Systems, in Proc. of Int. Conf. FUZZ-IEEE, Seoul, August 1999 (1999) 45. Kasabov,N. "Learning fuzzy rules and approximate reasoning in fuzzy neural networks and hybrid systems", Fuzzy Sets and Systems 82 (2) 2-20 (1996). 46. Kawahara, S., Saito, T. “On a novel adaptive self-organising network”, Cellular Neural Networks and Their Applications, 41-46 (1996) 47. Kohonen, T., “The Self-Organizing Map”, Proceedings of the IEEE, vol.78, N-9, pp.1464-1497, (1990). 48. Kohonen, T., Self-Organizing Maps, second edition, Springer Verlag, 1997 49. Krogh, A., and J.A. Hertz, “A simple weight decay can improve generalisation”, Advances in Neural Information Processing Systems, 4 951-957, (1992). 50. Le Cun, Y., J.S. Denker and S.A. Solla, “Optimal Brain Damage”, in: Touretzky, D.S., ed., Advances in Neural Information Processing Systems, Morgan Kaufmann, 2, 598-605 (1990). 51. Lin, C.T. and C.S. G. Lee, Neuro Fuzzy Systems, Prentice Hall (1996). 52. Maeda, M., Miyajima, H. and Murashima, S., “A self organizing neural network with creatinga nd deleting methods, Nonlinear theory and its applications, 1, 397-400 (1996) 53. Mandziuk, J., Shastri, L. “Incremental class learning approach and its application to hand-written digit recognition, Proc. of the fifth int. conf. on neuro-information processing, Kitakyushu, Japan, Oct. 21-23, 1998 54. Massaro, D., and M.Cohen, "Integration of visual and auditory information in speech perception", Journal of Experimental Psychology: Human Perception and Performance, Vol 9, pp.753-771, (1983). 55. McClelland, J., B.L. McNaughton, and R.C. Reilly "Why there are Complementary Learning Systems in the Hippocampus and Neo-cortex: Insights from the Successes and Failures of Connectionist Models of Learning and Memory", CMU TR PDP.CNS.94.1, March, (1994). 56. Miller, D.J., Zurada and J.H. Lilly, "Pruning via Dynamic Adaptation of the Forgetting Rate in Structural Learning," Proc. IEEE ICNN'96, Vol.1, p.448 (1996). 57. Mitchell, M.T., "Machine Learning", MacGraw-Hill (1997) 58. Moody, J., Darken, C., Fast learning in networks of locally-tuned processing units, Neural Computation, 1, 281-294 (1989) 59. Mozer, M., and P. Smolensky, “A technique for trimming the fat from a network via relevance assessment”, in: D. Touretzky (ed) Advances in Neural Information Processing Systems, vol.2, Morgan Kaufmann, 598-605 (1989). 60. Murphy, P. and Aha, D. “UCI Repository of machine learning databases, Irvin, CA: University of California, Department of Information and Computer Science (1994), (http://www.ics.uci.edu/~mlearn/MLRepository.html) 61. Port, R., and T.van Gelder (eds) Mind as motion (Explorations in the Dynamics of Cognition) , The MIT Press, 1995 62. Quartz, S.R., and T.J. Sejnowski, “The neural basis of cognitive development: a constructivist manifesto”, Behavioral and Brain Science, to appear. 63. R. Jang, “ANFIS: adaptive network-based fuzzy inference system”, IEEE Trans. on Syst., Man, Cybernetics, 23(3), May-June, 665-685, (1993). 64. Reed, R., “Pruning algorithms - a survey”, IEEE Trans. Neural Networks, 4 (5) 740-747, (1993). 65. Robins, A. and Frean, M. “Local learning algorithms for sequential learning tasks in neural networks, Journal of Advanced Computational Intelligence, vol.2, 6 (1998) 66. Robins, A., “Consolidation in neural networks and the sleeping brain, Connection Science”, 8, 2, 259-275, (1996). 67. Rummery, G.A., and M. Niranjan, “On-line Q-learning using connectionist systems”, Cambridge University Engineering Department, CUED/F-INENG/TR 166 (1994) 68. S.R.H. Joseph, “Theories of adaptive neural growth”, PhD Thesis, University of Edinburgh, 1998 69. Saad, D. (ed) On-line learning in neural networks, Cambridge University Press, 1999 70. Sankar, A., and R.J. Mammone, “Growing and Pruning Neural Tree Networks”, IEEE Trans. Comput. 42(3) 291-299 (1993). 71. Schiffman, W., M. Joost, and R. Werner, “Application of Genetic Algorithms to the Construction of Topologies for Multilayer Perceptrons” In: Albrecht, R.F., Reeves, Segalowitz, S.J. Language functions and brain organization, Academic Press, 1983 72. Segev, R. and E.Ben-Jacob, From neurons to brain: Adaptive self-wiring of neurons, TR /98 Faculty of Exact Sciences, Tel-Aviv University (1998) 73. Selverston, A. (ed) Model neural networks and behaviour, Plenum Press, 1985 74. Sinclair, S., and C. Watson, “The Development of the Otago Speech Database”, In Kasabov, N. and Coghill, G. (Eds.), Proceedings of ANNES ’95, Los Alamitos, CA, IEEE Computer Society Press (1995). 75. Towel, G., J. Shavlik, and M. Noordewier, "Refinement of approximate domain theories by knowledge-based neural networks", Proc. of the 8th National Conf. on Artificial Intelligence AAAI'90, Morgan Kaufmann, 861-866 (1990). 76. Van Ooyen, and J. Van Pelt, “Activity-dependent outgrowth of neurons and overshoot phenomena in developing neural networks”, Journal Theoretical Biology, 167, 27-43 (1994). 77. Waibel, A., M.Vo, P.Duchnovski, S.Manke, "Multimodal Interfaces", Artificial Intelligence Review, 1997. 78. Watts, M., and N. Kasabov, “Genetic algorithms for the design of fuzzy neural networks”, in Proc. of ICONIP'98, Kitakyushu, Oct. 1998. 79. Whitley, D., and C. Bogart, The evolution of connectivity: Pruning neural networks using genetic algorithms. Proc. Int. Joint Conf. Neural Networks, No. 1, 17-22. (1990). 80. Woldrige, M., and N. Jennings, “Intelligent agents: Theory and practice”, The Knowledge Engineering review (10) 1995. 81. Wong, R.O. “Use, disuse, and growth of the brain”, Proc. Nat. Acad. Sci. USA, 92 (6) 1797-99, (1995). 82. Yamakawa, T., H. Kusanagi, E. Uchino and T. Miki, "A new Effective Algorithm for Neo Fuzzy Neuron Model", in: Proceedings of Fifth IFSA World Congress, 1017-1020, (1993).	en_NZ
otago.relation.number	99/02	en_NZ