Natural language processing: What and how

Introduction

Laymen explanation

Is a machine capable of understanding a text as we humans do? I know, it seems hard to believe, and for years it seemed like magic to me, until I discovered Natural Language Processing (NLP), a field that deals with this kind of problem.

Yes, by combining the power of artificial intelligence, computational linguistics, and computer science, NLP allows a machine to understand natural language. A task that, up until now, only humans could perform.

Technical explanation

Its computer technology which is capable of "understanding" the contents of human made documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves.

Context challenge

• Consider some of the meanings, in English, of the word “big”. When used as a Comparative, as in “That is a big tree,” a likely inference of the intent of the author is that the author is using the word “big” to imply a statement about the tree being ”physically large” in comparison to other trees or the authors experience. When used as a Stative verb, as in ”Tomorrow is a big day”, a likely inference of the author’s intent it that ”big” is being used to imply ”importance”.

Areas which covers NLP

• speech recognition, 

• natural language understanding

• natural-language generation.

NLP approaches

Rule based NLP vs Machine learning based NLP

 machine-learning algorithms have many advantages over hand-produced rules: 

• • The learning procedures used during machine learning automatically focus on the most common cases, whereas when writing rules by hand it is often not at all obvious where the effort should be directed.

  • Automatic learning procedures can make use of statistical inference algorithms to produce models that are robust to unfamiliar input (e.g. containing words or structures that have not been seen before) and to erroneous input (e.g. with misspelled words or words accidentally omitted). Generally, handling such input gracefully with handwritten rules, or, more generally, creating systems of handwritten rules that make soft decisions, is extremely difficult, error-prone and time-consuming.

  • Systems based on automatically learning the rules can be made more accurate simply by supplying more input data. However, systems based on handwritten rules can only be made more accurate by increasing the complexity of the rules, which is a much more difficult task.

Statistical methods vs Neural networks

• A major drawback of statistical methods is that they require elaborate feature engineering.

Some of the earliest-used machine learning algorithms, such as decision trees, produced systems of hard if-then rules similar to existing hand-written rules. However, part-of-speech tagging introduced the use of hidden Markov models to natural language processing, and increasingly, research has focused on statistical models, which make soft, probabilistic decisions based on attaching real-valued weights to the features making up the input data. The cache language models upon which many speech recognition systems now rely are examples of such statistical models. Such models are generally more robust when given unfamiliar input, especially input that contains errors (as is very common for real-world data), and produce more reliable results when integrated into a larger system comprising multiple subtasks.

Since the neural turn, statistical methods in NLP research have been largely replaced by neural networks. However, they continue to be relevant for contexts in which statistical interpretability and transparency is required.

Text and speech processing

• Optical character recognition

  • Determine text

• Speech recognition

  • Determine words considering ascent

• Text-to-speech

  • Form speech from text

• Word segmentation

  • Extracts words from text

Reference

https://en.wikipedia.org/wiki/Natural_language_processing

https://monkeylearn.com/blog/definitive-guide-natural-language-processing/