Readings:

Chang, F. (2002). Symbolically speaking: A connectionist model of sentence production. Cognitive Science, 26(5), 609-651.

You will need to save the day2.tar.gz file and decompress it as before (tar -zxvf day2.tar.gz).

Here are the files:

prodSRN.in     # model file for production SRN

dualpath.in       # model file for dual-path model

train.ex            # same training and test files as before, but with messages.

test.ex         

decode9.perl     # translates model's activations into word sequences

syncode.perl     # codes word sequences with syntactic categories

arcthis.perl        # saves everything in a separate diretory, cleans up main directory

Before starting, we need to add the local directory to the path (notice the period at the end) so that the program can find the perl files.  Type this in the xterm: PATH=$PATH:.

Here we will be examining ways to add messages to a simple recurrent network model.  We will talk about two ways to do this.  The prodSRN model uses a binding-by-space message.  That means that different units represent the same concept in different roles.  So the agent DOG (e.g., "the dog chases the man") is different from a patient DOG (e.g., "the man chases the dog").  You can confirm this by starting lens (lens -c prodSRN.in) and opening the unit viewer.  Then click through the patterns, and look at how the message layer representations change.  You should see different semantic units for the nouns and articles in different roles (agent/patient/goal).

Now we will train the model on the input and save results along the way.

type: lens -b prodSRN.in 'trainSave;exit' &

The model will start and you should see something like this.

Performing 2000 updates using Doug's Momentum...

__Update____Error___UnitCost__Wgt.Cost__Grad.Lin__TimeUsed__TimeLeft__

    500)   29.4636   0.00000   2454.34  -0.05084        5s       17s

   1000)   31.7288   0.00000   2790.08  -0.15305       11s       11s

   1500)   8.16277   0.00000   3113.09  -0.04456       17s        5s

   2000)   12.9550   0.00000   3404.09  -0.03212       23s        0s

Test results on 1000 examples, 9472 ticks:

                     Network     

Error total:         8686.99

Error per example:   8.68699

Error per tick:      0.91712

Unit cost per tick:  0.00000

Output unit criterion reached on 0 examples (0.00%)

Every 2000 epochs, the model tests the training and testing set.  The Output unit criterion gives a rough measure of how many utterances were correctly produced.  To see the actual utterances that were produced, examine the sum40000train file by typing "less sum40000train".  You can scroll down by pressing the space bar, and "q" to quit.

You should see something like this.  Each name refers to a pattern in the appropriate set (here the training set).   tar: precedes the target in the environment file, and act shows the model's output.  On the next two lines, the tar and act output has been tagged with syntactic categories.  If it says "gram" at the end, then the sequence of syntactic categories is grammatical.

name:{ a girl was present -ing a sister a stick . . }

tar: a girl was present -ing a sister a stick . . 

act: a girl was present -ing a sister a stick . . 

syntar: DET NOUN AUX VERB ING DET NOUN DET NOUN PER PER 

synact: DET NOUN AUX VERB ING DET NOUN DET NOUN PER PER = gram

At the bottom of each file (type ">" in less to skip to the bottom), you will see some general statistics.  Here we see that 99% of the sequences were grammatical (gram), 76% were the exact target sentence (sent), and 96% of the word prediction were correct (word).

##results 

##gram=_ corr c 995 t 1000 perc 99%

##sent=_ corr c 765 t 1000 perc 76%

##word=_ corr c 9173 t 9472 perc 96%

To look at how sentence accuracy changes over development, type this:  grep "sent=" sum*train

You should see that the model develops quickly.

Dog-goal test:

  Now training set is similar to the training set in the Chang 2002 paper in that dogs were not allowed to be goals.  So now we can do the dog-goal test with this model.  The test set is made up of only sentences with dogs as goals.  So by looking at the sum*test files, we can see how well the model generalizes outside of its training space.  Take a look at the overall accuracy and compare those measures with the training set.

To examine the model using the gui, type: lens -c prodSRN.in 

Then "Load Weights" and click on comp40000.wt.gz.  The model should now be set to the final set of weights.

Look at the unit viewer and the link viewer and try to figure out why the model cannot produce dog-goal sentences.

Dual-path Model

Now we will do the same test with the Dual-path model.  

Use arcthis.perl to save the prodSRN results by typing: arcthis.perl prodsrn

Now train and save results for the Dual-path model by typing: lens -b dualpath.in 'trainSave;exit' &

Take a look at the sum*train and sum*test files.  See if the model can produce the dog-goal sentences.

Then examine the model using the gui: lens -c dualpath.in 

Load the weights from the final epoch and see if you understand how it works.