Embedded Files
My research focuses on making language models more concept-aware. Concepts play a pivotal role in various cognitive abilities such as categorization, learning, communicating, planning, and decision-making.
My research focuses on making language models more concept-aware. Concepts play a pivotal role in various cognitive abilities such as categorization, learning, communicating, planning, and decision-making.
Concepts are the glue that holds our mental model of the world together. They can be thought of as hierarchical categories, used to comprehend the world (similar to knowledge graphs). Concepts can be concrete (“soup”) or abstract (“tasty”). They can also be complex, e.g., “lactose-free and gluten-free chocolate soufflé”.
Concepts are the glue that holds our mental model of the world together. They can be thought of as hierarchical categories, used to comprehend the world (similar to knowledge graphs). Concepts can be concrete (“soup”) or abstract (“tasty”). They can also be complex, e.g., “lactose-free and gluten-free chocolate soufflé”.
I wish to automatically generate flexible, context-dependent concepts in different levels of abstraction, similar to human thinking. I believe that the ability to extract context-dependent representations is crucial for artificial agents and that it will have an overarching impact across many areas. Thus, I wish to create models that capture the abstraction into concepts humans perform when analyzing text. This would resemble human intuition better since we often ignore superficial signals (e.g., style) and distill the message the text aims to convey.
I wish to automatically generate flexible, context-dependent concepts in different levels of abstraction, similar to human thinking. I believe that the ability to extract context-dependent representations is crucial for artificial agents and that it will have an overarching impact across many areas. Thus, I wish to create models that capture the abstraction into concepts humans perform when analyzing text. This would resemble human intuition better since we often ignore superficial signals (e.g., style) and distill the message the text aims to convey.
One of the main tasks language models are used for is text completion: given a sequence with missing words, the objective is to find the most likely completion. However, their performance is far from perfect, where one notable problem is surface form competition: different surface forms compete for probability mass, even if they represent the same underlying concept in a given context, e.g., “computer” and “PC” (or even more trivial differences such as capitalization). This splits the probability mass between variations of the same concept, distorting the ranking.
One of the main tasks language models are used for is text completion: given a sequence with missing words, the objective is to find the most likely completion. However, their performance is far from perfect, where one notable problem is surface form competition: different surface forms compete for probability mass, even if they represent the same underlying concept in a given context, e.g., “computer” and “PC” (or even more trivial differences such as capitalization). This splits the probability mass between variations of the same concept, distorting the ranking.
This problem emphasizes the need for concepts to create more robust language models that can truly understand human language by abstracting the superficial words to the concepts they convey. Moreover, I believe that shifting language models from the token- to the concept level will enhance their performance on downstream NLP tasks.
This problem emphasizes the need for concepts to create more robust language models that can truly understand human language by abstracting the superficial words to the concepts they convey. Moreover, I believe that shifting language models from the token- to the concept level will enhance their performance on downstream NLP tasks.
However, context-dependent concept generation is a challenging, ambitious goal. In papers such as
However, context-dependent concept generation is a challenging, ambitious goal. In papers such as
I am working on optimizing concept-aware language models in various ways and phases of the pipeline. If you find this direction interesting and wish to hear more about it -- let's talk!
I am working on optimizing concept-aware language models in various ways and phases of the pipeline. If you find this direction interesting and wish to hear more about it -- let's talk!
Page updated
Report abuse