Smart Rule Probability Configuration

While the standard explicit way to set rule probabilities in GIGL requires the probabilities for all rules expanded from the same nonterminal be explicitly specified and adding up to one, in many cases this is not the most direct expression of designer intention and look too verbose. For mitigating this issue, GIGL provides several shorthanded syntax features and cross-referring features for minimize encoding effort and encouraging focusing on the most semantically significant parts. We use the item grammar in the Quiz demo to illustrate this feature, but disregard the custom (non rule probability) configure parameters on the rule intExpr and disregard the issue of controlling expansion depth.

The standard explicit way for set a configuration for the nonterminal type Expr look like this:

Expr := mulExpr @ {0.0} | addExpr @ {0.5} | intExpr @ {0.5} (denoted as configuration #1)

or this:

Expr := mulExpr @ {0.2} | addExpr @ {0.4} | intExpr @ {0.4} (denoted as configuration #2)

We introduce those smart configuration features with different variation of expression those configurations below. Some of these are already mentioned briefly at [Here]. For exact syntax, please refer to [Here].

  • Expr := addExpr @ {0.5} | intExpr @ {0.5}

  • has an effect equivalent to #1 as GIGL consider a rule is set with zero probability if the rule (even its name) does not appear at all in the configuration.

  • Expr := mulExpr | addExpr @ {0.4} | intExpr @ {0.4}

    • or

    • Expr := mulExpr @ {0.2} | addExpr | intExpr

    • has an effect equivalent to #2 as GIGL distribute the remaining probabilities (from 1.0) evenly to rules without a probability field (however the rule itself needs to appear, otherwise it would consider zero probability for it instead). Combining the two features, we can also simplify #1 to:

    • Expr := addExpr | intExpr

    • which can serve as what the first thing a designer may do, i.e. without worrying about the probabilities but only specifying the possible types, which is also what appeared in the very first HelloWorld demo.

  • Expr := mulExpr @ {2} | addExpr @ {4} | intExpr @ {4}

    • has an effect equivalent to #2 under default item type option setting, as by default a normalization to (a sum of) 1.0 is performed before the rule selection. This is useful when the user only wants to specify a ratio, which may or may not be normalized probabilities.

  • Expr := mulExpr @ {-0.1} | addExpr @ {0.5} | intExpr @ {0.5}

    • or

    • Expr := mulExpr @ {-1} | addExpr @ {5} | intExpr @ {5}

    • has an effect equivalent to #1 under default setting, as by default negative probabilities are clamped to zero before the rule selection (and before normalization). It may seem meaningless to allow writing negative probabilities like the examples above, however, this can be much more meaningful when the probabilities are set with expressions containing variables, such as

    • Expr := mulExpr @ {0.3 * diffic} | addExpr @ {0.2 * diffic} | intExpr @ {1.0 - 0.5 * diffic}

    • or equivalently (but neither equivalent to #1 or #2)

    • Expr := mulExpr @ {0.3 * diffic} | addExpr @ {0.2 * diffic} | intExpr

    • Here, if the probabilities depends on the value of the "diffic", the implicit clamping to zero can guard against the possible negative probability for the rule intExpr, without explicitly encoding the conditional check.

  • Expr := mulExpr @ {0.2} | addExpr @ {2 * probof[0]} | intExpr @ {probof(addExpr)}

    • has an effect equivalent to #2. This example uses the features for cross-referencing rule probabilities. Here it means the probability of the rule addExpr is double of that of the first rule (mulExpr), and the probability of the rule intExpr is the same as the probability of the rule addExpr. The referencing syntax uses the same convention as the referencing of rule probabilities in the pre-selector blocks (the exact implementation might be slightly different, as it uses the substitution mechanism mentioned below), which allows both referencing by index and referencing by name.

  • Expr := mulExpr @ {0.2} | addExpr @ {2 * specified} | intExpr @ {2 * specified}

    • has an effect equivalent to #2. The keyword 'specified' refers to the the sum of probabilities from the same nonterminal type that are set with expressions not containing any'specified' or relevant variants ('remain', 'remainof', 'specifiedremain', 'specifiedremainof' etc. or probability field not existing), which approximately means those explicit specified.

  • Expr := mulExpr @ {remain} | addExpr @ {0.4} | intExpr @ {0.4}

    • has an effect equivalent to #2. The term 'remain' means what is remaining after taking out certain set of probabilities. The keyword 'remain' is an alias of 'specifiedremain', which refers to the remaining probability after taking out the 'specified' probabilities (i.e. those set with expressions not containing any'specified' or relevant variants), which basically subtract those probabilities from 1.0. The 'remainof' keyword, an alias of 'specifiedremainof', allows to change the total probability to be subtracted from to some arbitrary number or expression other than 1.0, which can be used for probability configuration that are not normalized in a standard way, such as:

    • Expr := mulExpr @ {remainof(10)} | addExpr @ {4} | intExpr @ {4}

  • Expr := addExpr @ {0.4} | intExpr @ {0.4} | mulExpr @ {0.25 * prev}

    • or

    • Expr := mulExpr @ {0.25 * post} | addExpr @ {0.4} | intExpr @ {0.4}

    • has an effect equivalent to #2. The keyword 'prev' refers to the sum of rule probabilities from the same nonterminal type that are configured before the current rule (exclusive), while the keyword 'post' refers to the sum of rule probabilities configured after the current rule (exclusive). This set of series is different from the 'specified' series in two main aspects (apart from their apparent semantics). First, 'prev' (and 'post') series does not skip probability configurations containing 'prev' (and 'post') series expressions, while the sums for the 'specified' series skips those containing 'specified' series expressions. Second, 'prev' (and 'post') series is sensitive on the order rules, while the'specified' series is not.

  • Expr := addExpr @ {0.4} | intExpr @ {0.4} | mulExpr @ {prevremain}

    • or

    • Expr := mulExpr @ {postremainof(10)} | addExpr @ {4} | intExpr @ {4}

    • has an effect equivalent to #2. The semantics of the "remain" (subtracted from 1.0) and "remainof" (subtracted from a specified expression) suffixes apply analogously to the 'prev' and 'post' series as the do for the 'specified' series.

Almost all of these features (expect the default normalization and negative probability clamping) involves the referencing of probability of one rule in configuring another one, explicitly or implicitly (e.g. the implicit even distribution of remaining probabilities to those without a probability field). In the implementation of these cross referencing, there are several points to note:

    • GIGL implements these cross referencing with substitution method, i.e. substitute with the referenced target (recursively if needed) until all references are resolved, i.e. no reference remaining. The examples includes:

    • Expr := mulExpr @ {0.2} | addExpr @ {2 * probof[0]} | intExpr @ {probof(addExpr)} will be resolved into Expr := mulExpr @ {0.2} | addExpr @ {2 * 0.2} | intExpr @ {2 * 0.2}

    • Expr := mulExpr @ {remainof(10)} | addExpr @ {4} | intExpr @ {4} will be resolved into Expr := mulExpr{10 - (4 + 4)} | addExpr @ {4} | intExpr @ {4} (before normalization etc.)

    • Expr := mulExpr @ {0.2} | addExpr | intExpr will be resolved into Expr := mulExpr @ {0.2} | addExpr @ {(1.0 - 0.2)/2.0} | intExpr @ {(1.0 - 0.2)/2.0}

  • Circular reference will cause the compiler to stuck, because it cannot be fully resolved and the substitution process does not halt until the system breaks. The examples includes:

  • Expr := addExpr @ {probof(intExpr)} | intExpr @ {probof(addExpr)}

  • Expr := mulExpr @ {0.2} | addExpr @ {post} | intExpr @ {prev * 0.5}

    • The user of GIGL should avoid circular references and they does not represent the semantics from any reasonable design.

    • Due to the substitution implementation, expression containing side-effects can be of potential threat to the semantics. For example,

    • Expr := mulExpr @ {i = 1} | addExpr @ {i++} | intExpr @ {probof(addExpr)}

    • may be intended to mean the probability of the rule intExpr should be equal to the one for the rule addExpr, however, it will not be the case with the current implementation, because it gets resolved into something like

    • Expr := mulExpr @ {i = 1} | addExpr @ {i++} | intExpr @ {i++}

    • where the variable i gets incremented between configuring the two rule probabilities. In addition, the impact from these side-effect may even be even less clear with the default generation-time evaluation scheme with the lambda configuration feature. In generation time, i.e. when generating the nodes in the item trees, the order of evaluating these rule probabilities follows the order of the declaration of the rules in the item type, not the order of the rules in the configuration, in other words, if the rule intExpr were declared earlier than the rule addExpr in the item type, then addExpr would actually get a larger probability than intExpr for each of the rule selections. The suggestion for this part is, avoid using expressions with side effects (unless being absolutely sure the side effects does not negatively impact the semantics).

  • Currently, the referencing to rule probabilities with many of these features may be used in the expressions to set custom configure parameters (i.e. those not for setting rule probabilities). While the implementation is very natural this way, it is unclear if this as more advantage or disadvantages. Currently we expect the need for this type of usage is very uncommon if needed, but does not identify clear harm of having it either. If clear pros (or cons) of it is identified in the future, it may gets strengthened (or restricted).