Conflict Searches - Technical Details and Settings

This document provides information on how the methods and processes LegalServer uses when a user initiates a conflict search. 

How Searches in LegalServer Work

Note that there are two types of searches in LegalServer: "conflict" searches and "people finder" searches. People finder searches are performed when using the red Search link or Search menu, and can be performed anywhere on the site. Conflict searches are initiated as a result of form configuration by site administrators, for example during an intake process. Distinctions between the searches are noted where relevant.

There are three parts to the search process:

Source - the types of people/organizations to search for. This will include Aliases, Clients, Contacts, Family Members, and Adverse Parties.

Requirements that must be met in order to appear in the results

For a person search where both first and last name are present there must be either:

• First and Last Name Match: (one of the following combinations must be met)

  • First name is an exact match, OR first name sounds similar, OR first name ‘nickname’ matches, OR first name “begins with” (if enabled; only applies to people finder searches), AND

  • Last name is an exact match, OR last name sounds similar, OR last name “begins with” (if enabled; only applies to people finder searches).

• DOB matching can be configured for one of the following:

  • An exact match on DOB alone will return a result (this is the default setting).

  • An exact match on DOB alone will not return a result. Other search terms must also match to return a result.

  • An exact match on DOB will return a result unless the search includes Approximate DOB or the potential match has Approximate DOB set to Yes.

• SSN searching can be configured for one of the following:

  • An exact match on SSN alone will return a result (this is the default setting). If SSNs are set to use 4 digits, this can produce many results. A search on a 4 digit SSN will not match a 9 digit SSN (for example, if a site was collecting 9 digit SSNs, then later switched to 4 digit SSNs).

  • An exact SSN match on its own will not cause a result to be included. Other search terms must match for a result to appear.

  • The last 4 digits of the SSN are matched. This allows 9 digit SSNs to match a 4 digit SSN search if the last 4 digits match. A match of a 4 to 4 digit or 9 to 9 digit SSN will rank higher than a match of a 9 to 4 or 4 to 9. This option is the slowest of the 3 and is primarily useful for searching when a system contains mixed 9 and 4 digit SSNs due to imported data or a change in policy.

For an organization search the only requirement is an inclusive full text index match on the organization name.

Indexing

• Full text indexing allows the text in the relevant fields to be preprocessed and an index saved for later rapid searching.

• Initially the text t is parsed or ‘deconstructed’ into ‘tokens’. Tokens are usually a predefined set of classes e.g. numbers, complex words, or e-mail addresses.

• These ‘tokens’ are then converted into lexemes, roughly construed as a set of words that are different forms of the same word e.g. run, runs, ran, and running are all forms of the same lexeme.

• During preprocessing these lexemes are normalized which makes different forms of the same word alike. The result of normalization means that when looking for the search term banks, it will match bank and therefore pull it into the results, walking will match walk and so on.

• By using this process, variant forms of the same word can be found without the need to enter all of the possible variants, thereby saving time and effort.

For a people finder search without either an organization name or a first and a last name, it will require matches on all of the fields filled out.

Scores 

Search results are scored to determine their presentation order. Scores can be positive or negative, e.g. a DOB mismatch results in a negative score. The results are reflected in the results window by a column with Score stars. By holding the mouse cursor over the Score stars a small window will popup showing what matched for each result. (Scores and the results window are covered in greater detail below).

Which Fields are Searched

LegalServer searches the following fields (although not all of the fields must be filled in):

• DOB

• First name

• Last name

• Middle initial

• Middle name

• Nickname (Administrators can add Nicknames in Admin / Nicknames)

• Organization name

• Business, mobile, home, fax, or other phone number

• Social Security Number

• Suffix or Second Last Name

• Visa Number/Alien Registration Number

Name Comparison Algorithms

LegalServer uses 2 name comparison algorithms, ‘sounds similar’ and ‘looks similar’, to determine if a record will be considered a match and score enough points to be included in search results. 

These algorithms are implemented as functions in the database. Exact representations of the process, as opposed to the results, would run many pages. An analogy is spell check and grammar check; it is easy to see the results, but a step by step analysis of exactly how those functions are implemented in Word, WordPerfect, and other programs is not practical. Links for further reading are provided.

Looks Similar (Levenshtein Distance)

The algorithm for this rule is the Levenshtein distance, also called the Edit Distance algorithm. When presented with two names (or ‘strings’) the algorithm calculates the least number of edits needed to modify or transform one string into another. Taking the first names of Jan (the search term) and Janet (the potential result) as an example, the algorithm would calculate how many edits would be required to transform Jan into Janet. The answer in this case is 2, while to transform John into James the answer is 4. How the algorithm actually calculates the number is beyond the scope of this document, however, the greater the Levenshtein distance, the number of edits required, the greater the difference between the two names.

In LegalServer the result only qualifies as ‘looks similar’ if the Levenshtein distance is 2 or less. Therefore, in the example of Jan and Janet where the score is 2, Janet would show in the results. However, if the search term was John, the score would be 4, and James would not show in the results.

An an example of the grid method employed by Levenshtein can be found on this page(external link).

Sounds Similar (Double Metaphone)

The algorithm employed for this rule is a phonetic algorithm called Double Metaphone. In an attempt to detect phonetic relationships between words, phonetic algorithms convert words to encoded representations of their pronunciation. Double Metaphone is the second generation of the Metaphone algorithm. It groups words not only by their spelling, but also by different pronunciations, attempting to compensate for irregularities in English of names Spanish, German, Slavic, and other origins. An an example of applying metaphone ("sounds like") and additional background material can be found on this page(external link).

Trigram

The Trigram algorithm is used when searching for people. it is enabled by default when searching organization names, but can be disabled on the Admin > Top Level Navigation/Search page by setting "Organization Search Includes "Similar" results" to No.

Trigram breaks a name into a list of trigrams, first prepending two spaces to the word and adding one space to the end — this effectively weights the beginning of the word higher than the end of the word. The similarity between two words is the number of shared trigrams divided by the number of unique trigrams. To be considered a match the similarity needs to be greater than 0.3. Examples:

• Jon broken up into trigrams is {" j"," jo",jon,"on "} and John is {" j"," jo","hn ",joh,ohn}. They share two trigrams, " j" and " jo" out of a set of seven unique trigrams so their similarity is rated as 2/7, which is less than 3/10 so it is not a match.

• Kathy {" k"," ka",ath,"hy ",kat,thy} and Kathleen {" k"," ka",ath,een,"en ",hle,kat,lee,thl} share four trigrams {" k"," ka",ath,kat} out of 11 unique, which is > 3/10 so they are considered a match.

• Bug {" b"," bu",bug,"ug "} and Bugz {" b"," bu",bug,"gz ",ugz} share three {" b"," bu",bug} out of six unique so they are considered a match.

Scores and Ranking

The table below gives a breakdown of the various fields and scores. 

The Data Element is the field where the user inputs the search term. Remember that not all fields have to be filled out, but IF the search term is present, and LegalServer finds a potential result, it runs the algorithms, determines if there is a match, and applies the pertinent score. The text column shows the text the user will see when they hold the mouse cursor over the Score stars in the results window.