I am attempting to develop a practical connectionist mechanism for compositional and analogical memory. This is concerned with the ability to recognise and predict situations and objects in terms of the pattern of relationships between their component parts independently of the precise identity of the component parts. Current standard connectionist techniques have limited capacity to deal with patterns of relationships and consequently have difficulty recognising novel configurations of familiar components or recognising familiar patterns of relationship when the components being related are novel. Human analogical reasoning is the extreme example of the neural functionality I am striving for. I believe that cognition is based on sensorimotor planning and that analogical retrieval is needed to allow the application of sensorimotor schemas to novel and abstract situations. I argue that high-level perception and all cognition are fundamentally the same process and are implemented by this neural mechanism of compositional, analogical memory.
The memory mechanisms I am investigating are based on Vector Symbolic Architectures (VSAs), Predictive State Representations (PSRs), and Map-Seeking Circuits (MSCs). VSAs are systems of distributed representations and operations on them in high-dimensional vector spaces (http://cogprints.org/3983/). They enable the principled representation, storage, and manipulation of structured data (e.g. trees and graphs) in vector spaces of fixed dimension. Because VSAs are based on simple vector operations they can be readily mapped to a neural implementation. (See Pentti Kanerva (http://www.springerlink.com/content/966151841g415165/), and Tony Plate (http://citeseer.ist.psu.edu/32374.html) for related research on VSAs.)
Given VSA as the mechanism for representation and processing of representations, what should be represented? PSRs represent an agent’s state entirely in terms of observable experience (the temporal stream of the agent’s actions and observations), e.g. see Richard Sutton (http://www.cs.ualberta.ca/~sutton/Talks/McGill_2005.pdf) or Michael James (http://ijcai.org/papers/1621.pdf). A PSR represents the current state of the agent as a set of predictions of future experience (observations conditional on the agent’s actions). This makes PSRs directly relevant to sensorimotor integration and planning. Current PSR research has noted the desirability of composing multiple PSRs to allow the agent to exploit its knowledge in novel situations.
David Arathorn has proposed MSCs (http://books.nips.cc/papers/files/nips18/NIPS2005_0176.pdf) as a neurobiologically inspired mechanism for discovering/generating the composition of transformations that maximise the similarity between a cue and an item in memory. An MSC is a recurrent network that searches simultaneously over a space of transformations to apply to the cue and a space of memory items to be recalled in response to the transformed cue. The MSC is similar to attentional mechanisms that have been proposed for the construction of invariant perceptual representations (e.g. Bruno Olshausen (http://redwood.berkeley.edu/bruno/papers/jneurosci93.pdf)).
The VSAs are the basic processing units, connected in recurrent circuits to form an MSC, with PSRs as the information processed by the system. My work can be construed as a generalisation of MSCs. Whereas Arathorn’s MSCs use localist representations and a fixed palette of geometric transformations, my objective is to use distributed connectionist representations and arbitrary systematic substitutions as the transformations. Using VSAs as the primitives from which the MSC is constructed allows the distributed connectionist representation and manipulation of composite structures (such as PSRs). One extremely useful feature of VSAs is that the representations can be applied as substitution operators to transform other representations. This opens the way for transformations to be composed on the fly by the MSC. I am also attempting to generalise the MSC by allowing it to recall multiple items simultaneously (rather than only one) and to find transformations between recalled items (rather than only between a cue and a recalled item). This would allow the recognition of novel composite objects and situations in terms of familiar components and the discovered relationships between them.
When I get time I will add some more strands to this overview, in particular:
(Google Scholar citation profile)
My ORCID ID is 0000-0003-4679-585X
Gayler, R.W., Levy, S.D. (Eds.) (2011). Compositional connectionism in cognitive science II: the localist/distributed dimension. Connection Science, 23(2), 85-89.
Gayler, R.W., Levy, S.D., & Bod, R. (2010). Explanatory aspirations and the scandal of cognitive neuroscience. In A.Samsonovich, K. Johannsdottir, A. Chella, & B. Goertzel (Eds.), Proceedings of Biologically Inspired Cognitive Architectures 2010 - BICA-2010 (pp. 42-51), Washington DC, USA. [PDF]
Phua, C., Smith-Miles, K., Lee, V., & Gayler, R.W. (2010). Resilient identity crime detection. IEEE Transactions on Knowledge and Data Engineering. doi:10.1109/TKDE.2010.262 [PDF]
Christen, P., Gayler, R., & Hawking, D. (2009). Similarity-aware indexing for real-time entity
resolution. In Proceedings of the
ACM Conference on Information and Knowledge
Management (CIKM), Hong Kong, November 2009. doi:10.1145/1645953.1646173 The extended paper (10 pages) is published as ANU Computer Science
technical report TR-CS-09-01. [PDF]
Gayler, R.W., & Levy, S.D. (2009). A distributed basis for analogical mapping. In B. Kokinov, K. Holyoak, & D. Gentner (Eds.), New frontiers in analogy research; Proceedings of the Second International Analogy Conference - Analogy 09 (pp. 165-174. Sofia, Bulgaria: New Bulgarian University Press). [PDF]
Levy, S.D., & Gayler, R.W. (2009). "Lateral inhibition" in a fully distributed connectionist architecture. In A. Howes, D. Peebles, & R. Cooper (Eds.), 9th International Conference on Cognitive Modeling - ICCM 2009 (pp.318-323). Manchester, UK. [PDF]
Phua, C., Gayler, R., Lee, V., & Smith-Miles, K. (2009). On the communal analysis suspicion scoring for identity crime in streaming credit applications. European Journal of Operational Research, 195, 595-612, doi:10.1016/j.ejor.2008.02.015 . [ScienceDirect][PDF]
Christen, P. & Gayler, R. (2008). Towards scalable real-time entity resolution using a similarity-aware inverted index approach . In Proceedings of the Seventh Australasian Data Mining Conference (AusDM 2008), Glenelg, Adelaide, November 2008. [PDF]
Levy, S.D., & Gayler, R.W. (2008). Vector Symbolic Architectures: A new building material for Artificial General Intelligence. Proceedings of the First Conference on Artificial General Intelligence (AGI-08). IOS Press. [AGI-08][PDF][video]
Phua, C., Lee, V., Smith-Miles, K., & Gayler, R. (2007). Adaptive Communal Detection in Search of Adversarial Identity Crime. Proc. of SIGKDD07 Workshop on Domain-Driven Data Mining, pp. 1-10. [pdf]
Phua, C., Smith-Miles, K., Lee, V., & Gayler, R. (2007). Adaptive Spike Detection for Resilient Data Stream Mining. Data Mining and Analytics 2007. Proceedings of the Sixth Australasian Data Mining Conference (AusDM 2007), pp. 181-188. [pdf]
Gayler, R.W. (2006). Comment: Classifier technology and the illusion of progress - Credit scoring. Statistical Science, 21(1), 19-23. Commentary on: Hand, D.J. (2006) "Classifier technology and the illusion of progress", Statistical Science, 21(1), 1-15. [arXiv][PDF]
Gayler, R.W. (2006). Vector Symbolic Architectures are a viable alternative for Jackendoff’s challenges. Behavioral and Brain Sciences, 29(1), 78-79, doi:10.1017/S0140525X06309028. Commentary on: van der Velde, F. & de Kamps, M. (2006) "Neural blackboard architectures of combinatorial structures in cognition", Behavioral and Brain Sciences, 29(1), 37-70. [prepublication PDF]
Phua, C., Gayler, R., Smith-Miles, K., & Lee, V. (2006). Communal Detection of Implicit Personal Identity Streams. Proc. of ICDM06 Workshop on Mining Evolving and Streaming Data, pp. 620-625. [pdf]
Phua, C., Lee, V., Gayler, R., & Smith-Miles, K. (2006). Temporal Representation in Spike Detection of Sparse Personal Identity Streams. Proc. of PAKDD06 Workshop on Intelligence and Security Informatics, pp. 115-126. [pdf]
Gayler, R.W. (2003). Vector Symbolic Architectures answer Jackendoff’s challenges for cognitive neuroscience. In Peter Slezak (Ed.), ICCS/ASCS International Conference on Cognitive Science (pp. 133-138). Sydney, Australia: University of New South Wales. [CogPrints][PDF]
Gayler, R.W. (1999). Holographic networks are hiking the foothills of analogy. In Arun Jagota, Tony Plate, Lokendra Shastri, & Ron Sun (Eds.), Connectionist symbol processing: Dead or alive? Neural Computing Surveys, 2, 1-40.[PDF]
Gayler, R.W. (1998). Multiplicative binding, representation operators, and analogy. In Keith Holyoak, Dedre Gentner, & Boicho Kokinov (Eds.), Advances in analogy research: Integration of theory and data from the cognitive, computational, and neural sciences (p. 405). Sofia, Bulgaria: New Bulgarian University.[CogPrints][PDF]
Gayler, R.W., & Wales, R. (1998). Connections, binding, unification, and analogical promiscuity. In Keith Holyoak, Dedre Gentner, & Boicho Kokinov (Eds.), Advances in analogy research: Integration of theory and data from the cognitive, computational, and neural sciences (pp. 181-190). Sofia, Bulgaria: New Bulgarian University. [CogPrints][PDF]
Halford, G.S., Wilson, W.H., Guo, J., Gayler, R.W., Wiles, J., & Stewart, J.E.M. (1994). Connectionist implications for processing capacity limitations in analogies. In K.J. Holyoak & J. Barnden (Eds.), Advances in connectionist and neural computation theory, Vol. 2: Analogical connections (pp. 363-415). Norwood, NJ: Ablex. [HTML]
Gayler, R. W. (1988). Development of a methodology and theoretical framework for melodic discrimination. Doctoral dissertation, University of Queensland, Brisbane, Australia. [ProQuest, Publication Number: AAT 8904966, ProQuest document ID: 745800051] (This PDF is a searchable image. The pages are scanned and linked to OCR text - so the searchable terms will have occasional errors. I have also attached the Examiners' comments and my response, because it is useful to see the inner workings occasionally.)
Reject inference with nested conditional models based on joint risk and fraud scores Presentation given at the Credit Scoring and Credit Control XIII conference, Edinburgh, Scotland, 28th August, 2013.
Credit scoring & R: Reject inference, nested conditional models, and joint scores Presentation given to the Melbourne Users of R Network, Melbourne, 14th August, 2013 [Video 44min.]
VSA: Vector Symbolic Architectures for Cognitive Computing in Neural Networks Presentation given at Redwood Center for Theoretical Neuroscience, UC Berkeley on Friday, 14th June, 2013 [Video 98min.]
Vector Symbolic Architectures: Computing with discrete compositional data structures in analogue computers Presentation given at Monash University, Faculty of Information Technology on Wednesday, 8th May, 2013 [Video - Part 1] [Video - Part 2] [Video - Part 3]
Vector Symbolic Architectures: Analogue computation on discrete data structures Presentation given at ANU Computer Science, 9 February, 2012.
The design of software tools for continuous characteristic analysis Presentation given at Credit Scoring and Credit Control XII, Edinburgh, 25 August, 2011.
How to be smooth Presentation given to the Melbourne Users of R Network (MelbURN), Melbourne, 19 April, 2011.
Income verification under NCCP Presentation given at the Australasian Retail Credit Risk Conference, Kingscliff NSW, 5 April, 2011.
Analogical mapping with Vector Symbolic Architectures Presentation given at the Australasian Mathematical Psychology Conference, Melbourne, 19 February, 2011.The craft of credit scoring: Data mining applied to retail finance Keynote presentation given to the International Workshop on Data Mining Application in Government and Industry (DMAGI10) as part of the 10th IEEE International Conference on Data Mining (ICDM2010), Sydney, 14 December, 2010.
Analogical mapping with Vector Symbolic Architectures Presentation given to the Melbourne Cognitive Science Forum, Melbourne, 16 October, 2009.
The practicalities of scoring with continuous predictors Presentation given at Credit Scoring and Credit Control XI, Edinburgh, 28 August, 2009.
VSA demonstration: The shifter problem Presentation given at Institut für Neuroinformatik, Bochum, 27 February, 2007.
Development of an application fraud scorecard: A case study of design Presentation given at Credit Scoring and Credit Control IX, Edinburgh, 9 September, 2005.
Vector Symbolic Architectures Answer Jackendoff’s Challenges Seminar given at Redwood Neuroscience Institute, Menlo Park, 15 October 2004.
Compositional memory for recognition of complex objects: A proposal Seminar given at Redwood Neuroscience Institute, Menlo Park, 22 October, 2003.
Multiplicative binding, representation operators and analogical inference Presentation given at 5th Australasian Cognitive Science Conference, Melbourne, 1 February, 2000.
Signal Detection for credit scoring practitioners Presentation given at Credit Scoring and Credit Control VI, Edinburgh, 30 September, 1999.
Committee member of the Melbourne Risk Analytics Group (October 2010 - present).
Co-chair (with Simon Levy) of the workshop "Compositional Connectionism in Cognitive Science II: The localist/dsitributed dimension", CogSci 2010, Portland, Oregon, USA.
Co-chair (with Simon Levy) of the symposium "Compositional Connectionism in Cognitive Science", American Association for Artificial Intelligence, Fall Symposium Series 2004, Washington D.C., USA.
My 1977 Honours thesis show the direction I was heading when I started my doctoral research. It also shows my unfortunate tendency to build all the required infrastructure from scratch.
This is the write-up of my entry in the Science Teachers Association of Queensland 1973 science contest. The work was done without any guidance, so the writing does not adhere to the minutiae of any scientific style guide. Sadly, the robot I built is long gone and there are no photographs or circuit diagrams. The only useful conclusion that can be drawn from this work is that I have been interested in cognitive science since before the field was named.
These two documents are searchable PDF image files. The pages are scanned and linked to OCR text - so the searchable terms will have occasional OCR errors.
(Up to date, but much in need of correction of the metadata)
I will be repeating my talk "Reject inference with nested conditional models based on joint risk and fraud scores" to the Melbourne Risk Analytics Group on Friday, 27th September, 2013.
I gave the talk "Reject inference with nested conditional models based on joint risk and fraud scores" at the Credit Scoring and Credit Control XIII conference in Edinburgh, Scotland on 28th August, 2013. The slides of the presentation are available from this page in the "Recent & Selected Talks" section.
I gave the talk "Credit scoring & R: Reject inference, nested conditional models, and joint scores" to the MelbURN group on 14th August, 2013. The slides and video recording of the presentation are available from this page in the "Recent & Selected Talks" section.
I am third author on a position paper/poster for RepLearn 2013, 15th July, 2013, Bellevue WA, USA.
I gave the talk "VSA: Vector Symbolic Architectures for Cognitive Computing in Neural Networks" at the Redwood Center for Theoretical Neuroscience, UC Berkeley on Friday, 14th June, 2013. The slides and video recording of the presentation are available from this page in the "Recent & Selected Talks" section.
I presented the improbably named talk "Vector
Symbolic Architectures: Computing with discrete compositional data
structures in analog computers OR Cognitive computation in neural
networks with a fixed hardware budget" as a seminar at Monash University, Faculty of Information Technology on Wednesday, 8th May, 2013. The slides and video recording of the presentation are available from this page in the "Recent & Selected Talks" section.
The paper "Analogical mapping and inference with binary spatter codes and sparse distributed memory", written with Blerim Emruli and Fredrik Sandin of Luleå University of Technology, has been accepted for presentation at the 2013 International Joint Conference on Neural Networks and for publication in the conference proceedings published by IEEE.
Issue 8 of the Credit Risk Analytics Occasional Newsletter published 18 March, 2013.
Between 25 June and 7 July 2012 I visited Kai-Uwe Kuehnberger and Helmar Gust at the University of Osnabrueck, Herbert Jaeger and Peter beim Graben at Jacobs University, Bremen, Tony Bellotti and David Hand at Imperial College London, Jonathan Crook at Edinburgh University, Simon Levy at Washington & Lee University, and Pentti Kanerva at Stanford University for research and collaboration discussions.
I visited Lyn Thomas and Christophe Mues at the University of Southampton on 26-27 March 2012 and attended the CONAS 2012 workshop on "Cognitive dynamics in neural systems" in Lyon, 29-30 March.
I gave a seminar at ANU Computer Science Dept. on Thursday, 9th February, 2012 on "Vector Symbolic Architectures: Analogue computation on discrete data structures"
I presented a talk on "Design of software tools for continuous characteristic analysis" at Credit Scoring and Credit Control XII in Edinburgh on 25th August, 2011.
On 19/5/2011 I gave an introductory talk on cognitive science to the Victorian Chapter of the Independent Scholars Association of Australiaia in the Melbourne CBD.
On 19/4/2011 I spoke on scatterplot smoothers in R to MelbURN (Melbourne Users of R Network) in the Melbourne CBD.
I spoke about "Income verification under NCCP" on 5/4/2011 at the Australasian Retail Credit Risk Conference at Kingscliff, NSW.
I spoke at the the Australasian Mathematical Psychology Conference on Saturday 19/2/2011 at the University of Melbourne - "Analogical mapping with Vector Symbolic Architectures". (Here is the Stewart and Eliasmith paper on estimates of the number of neurons implied by several connectionist models that I mentioned during questions following my talk.)
I gave the keynote presentation at the International Workshop on Data Mining Application in Government and Industry on 14 December, 2010. This was part of the IEEE International Conference on Data Mining, held in Sydney.
On 8 October, 2010 I presented a brief talk on Signal Detection and Reject Inference to the initial meeting of the Melbourne Risk Analytics Group.
Our "scandal of cognitive neuroscience" paper has been accepted for BICA-2010.
The Compositional Connectionism II workshop was held on August 11, 2010 in Portland, Oregon.
On 16 October 2009 I gave a presentation to the Melbourne Cognitive Science Forum on analogical mapping with Vector Symbolic Architectures.
Research done jointly with Peter Christen and David Hawking on similarity-aware indexing for real-time entity resolution has been accepted for poster presentation at the CIKM 2009 conference in Hong Kong, 2-6 November, 2009.
I spoke on the practicalities of scoring with continuous predictors at Credit Scoring and Credit Control XI in Edinburgh, Scotland (26-28 August, 2009).
I attended the Analogy 09 conference in Sofia, Bulgaria, to participate in the symposium on formal models of analogy and present a paper on distributed connectionist analogical mapping written with Simon Levy.
Simon Levy and I wrote a paper on distributed connectionist lateral inhibition for the ICCM 2009 conference in Manchester, UK. Chris Eliasmith kindly presented the paper for us.
I visited Simon Levy at Washington & Lee University from 10th to 15th November 2008 for collaborative work on a VSA-based circuit for discovery of graph isomorphism. On the 17th to 19th of November I visited the Artificial Intelligence group of the Institute of Cognitive Science at the University of Osnabrueck for discussions on analogy and VSA. I also took advantage of the travel to meet with credit scoring colleagues in London and Zurich.
Monday 1st October 2007, I gave a 10 minute talk on Predictive State Representations and Vector Symbolic Architectures to the seminar on Representation and Meaning held by the Cognitive Science group at the University of Melbourne. Brevity is very challenging for me! Here are the slides and presentation.
I attended the ASCS 2007 cognitive science conference in Adelaide (9th to 11th, July 2007).
I visited the Institute of Cognitive Science, University of Osnabrueck for discussions with the Analogy Research Group (19th February to 23rd February, 2007); the Institut für Neuroinformatik, Ruhr-Universität Bochum to give a seminar and hold discussions on Dynamic Field Theory (27th February); and the Croatian National Bank, Zagreb for credit scoring discussions (1st March to 2nd March). See the photographs here.
I gave seminars on the role of design expertise in the development of a fraud detection model at Imperial College London (12th September, 2006), University of Edinburgh (22nd September), and University of Southampton (27th September).
I was a member of the discussion panel at the Basel II & Credit Risk Modelling in Consumer Lending conference (8th September, 2006).
Last update: 12 October, 2014
Created: 23 March, 2006