Chatbots need to effectively process natural language. Since starting around the 1960’s, natural language processing (NLP) research has had slow progress until the 1990’s (Lester, Branting, & Mott, 2004). NLP research has been greatly enhanced by development of large corpora of tagged text, as well as by development of better statistical machine learning and other empirical techniques of extracting information from these corpora (Lester et al., 2004). Accordingly, chatbots are now able to more effectively process natural language.
The following are three general steps of chatbot Natural Language Processing according to (Lester et al., 2004):
- The First step entails breaking down the utterances (statement). This syntactic analysis separates the statement into parts of speech (noun, verb,etc) using tagging and parsing to define the different parts of the text. There are different approaches to syntactic analysis, and vary greatly according to the type of language and corpora used.
- The second step entails reference resolution which involves a discourse analysis of the utterance to create meaningful relationships. For example, this step would resolve the equivalent relationship of the word I” with the word ME”. After relationships are created, a new string would be crafted from the resolution process.
- The third step happens when the result of the string is passed to a dialog manager for further processing. An example would be comparing the string (pattern matching) and determine if it is equal or not to another string.