Making a Custom Dictionary

Fluent supports the use of custom dictionaries made by users. 

This page explains the technical details of how a custom dictionary is structured. This information is likely to be of interest to only programmers since it requires a higher level of technical experience.

Design Goals

Fluent Dictionary has the following design goals at it's core

• Highly optimized to leverage the features of MacOS and iOS on the iPhone and iPad.

  • Split screen viewing and standard gesture navigation within the UI.

• Speed, speed, speed - dictionary lookup should always be fast.

• Extensibility: from day one we wanted the application to be extendable by its. user community.

In harmony with the goal of extensibility, this document outlines the information needed for a programmer to assemble a custom dictionary

Technical Overview

Fluent uses the SQLite database engine under the hood. All custom dictionaries are SQLite databases that conform o the following requirement:

• The file is a valid SQLite file.

• Ends with the extension .fluent. For example, if you make a custom dictionary named EnglishToRussian, the file would be named EnglishToRussian.fluent.

• The file contains all the required tables as defined in the section of this page Table Structures.

Table Structures

This section defines the required tables needed in the custom dictionary, along with all field definitions.

SQLite schema structure for a custom dictionary:

CREATE TABLE DictionaryInformation (

   DictionaryID TEXT PRIMARY KEY, 

   Name TEXT NOT NULL, 

   Information TEXT NOT NULL

);

CREATE TABLE DictionaryItems (

   ItemID INTEGER PRIMARY KEY AUTOINCREMENT,

   DictionaryID TEXT NOT NULL,

   DisplayTerm TEXT NOT NULL,

   DisplayDefinition TEXT NOT NULL,

   SearchTerm TEXT NOT NULL,

   SortTerm TEXT NOT NULL

);

CREATE TABLE DictionaryResources (

   ItemID INTEGER PRIMARY KEY AUTOINCREMENT,

   DictionaryID TEXT NOT NULL, 

   Name TEXT NOT NULL, 

   URL TEXT NOT NULL

);

CREATE TABLE CharacterMapping (

   ItemID INTEGER PRIMARY KEY AUTOINCREMENT,

   DictionaryID TEXT NOT NULL, 

   OriginalCharacter TEXT NOT NULL, 

   MappedCharacter TEXT NOT NULL

);

TABLE: DictionaryInformation

Purpose: General information about your custom dictionary.

Table Fields:

• DictionaryID:  Unique ID for your custom dictionary. Can be any number of characters, but should be kept to 3 to 4 characters. 

• Name: Name of the database, for example WordNet Open Source English Dictionary. 

• Information: Provide additional useful information about the database, such as copyright or other information describing the source of your custom dictionary. 

TABLE: DictionaryItems

Purpose: Individual dictionary terms and their definitions. Each row represents a term in your dictionary. When the user searches, this is the table that contains all the entries. 

Table Fields:

• DictionaryID:  The DictionaryID is defined in the DictionaryInformation table.

• DisplayTerm: Word or phrase displayed to the user.

• DisplayDefinition: Detailed explanation or definition of the term.

• SearchTerm: The actual word or phrase that is used in the search lookup. This is useful for optimizing indexing and the type of search phrase a user will provide. For example, they could type a word like ćevapčići, which has special characters, or just cevapcici which is easier to type. Basically, this allows for transliterated searches.

  • General best practices:

    • the term must be lowercase

    • remove punctuation

    • when the user types in the search field in Fluent, the SearchTerm field is what is searched against.  Internally Fluent is executing this query for each search:

      • SELECT * FROM DictionaryItems
        WHERE SearchTerm LIKE 'word%'
        ORDER BY SortTerm
        LIMIT 100

• SortTerm: When results are returned from a search against the database, they will be sorted by this field

• ItemID: Autoincrementing ID field, this field is auto-populated by SQLite and thus should not be modified by a developer.

Notes:

• It might seem that the following three fields are redundant: DisplayTerm, SearchTerm, SortTerm. However, while they often contain the same string of characters, as a developer you can tweak them for your specific needs. DisplayTerm is what the user sees, SearchTerm is what is used for the actual search comparison and SortTerm is what used for sorting allowing the developer to customize the resulting sort order.

TABLE: DictionaryResources

Purpose: Each dictionary can provide a set of unique URLs for resources related to this dictionary. This allows the developer to linked terms found in a search in Fluent to additional resources online, such as Wikipedia, Forvo and so on.

Table Fields:

• DictionaryID:  The DictionaryID as is defined in the DictionaryInformation table. 

• Name: Name of the resource (examples: Wikipedia, Forvo),

• URL: The URL to the path. The URL should contain a {TERM} token to identify where the matching Term will be inserted. Example URLs:

  • https://forvo.com/search/{TERM}/en/

  • https://wiktionary.org/wiki/{TERM}

  • https://glosbe.com/en/ru/{TERM}

• ItemID: Autoincrementing ID field, this field is auto-populated by SQLite and thus should not be modified by a developer.

TABLE: CharacterMapping

Purpose: Each database can map a specific character from one to another character. These character maps are used to modify the search before running against SearchTerm. For example, imagine the letter č is used in the language of your custom dictionary, but in the SearchTerm field, you prefer the user uses c for a search since it can be easier to type on a mobile device. However if the user types č and this is not used in SearchTerm, it will not find a matching word. CharacterMapping helps fluent to convert these unique language characters to another character that is compatible with the SearchTerm.

The CharacterMapping table can be left empty if you don't want to use this feature.

Table Fields:

• DictionaryID:  The DictionaryID is defined in the DictionaryInformation table. 

• OriginalCharacter: The character from the language.

• MappedCharacter: What the original character should be mapped to.

• ItemID: Autoincrementing ID field, this field is auto-populated by SQLite and thus should not be modified by a developer.