Chang, F. (2002). Symbolically speaking: A connectionist model of sentence production. Cognitive Science, 26(5), 609-651.
You will need to save the day2.tar.gz file and decompress it as before (tar -zxvf day2.tar.gz).
Here are the files:
prodSRN.in # model file for production SRN
dualpath.in # model file for dual-path model
train.ex # same training and test files as before, but with messages.
test.ex
decode9.perl # translates model's activations into word sequences
syncode.perl # codes word sequences with syntactic categories
arcthis.perl # saves everything in a separate diretory, cleans up main directory
Before starting, we need to add the local directory to the path (notice the period at the end) so that the program can find the perl files. Type this in the xterm: PATH=$PATH:.
Here we will be examining ways to add messages to a simple recurrent network model. We will talk about two ways to do this. The prodSRN model uses a binding-by-space message. That means that different units represent the same concept in different roles. So the agent DOG (e.g., "the dog chases the man") is different from a patient DOG (e.g., "the man chases the dog"). You can confirm this by starting lens (lens -c prodSRN.in) and opening the unit viewer. Then click through the patterns, and look at how the message layer representations change. You should see different semantic units for the nouns and articles in different roles (agent/patient/goal).
Now we will train the model on the input and save results along the way.
type: lens -b prodSRN.in 'trainSave;exit' &
The model will start and you should see something like this.
Performing 2000 updates using Doug's Momentum...
__Update____Error___UnitCost__Wgt.Cost__Grad.Lin__TimeUsed__TimeLeft__
500) 29.4636 0.00000 2454.34 -0.05084 5s 17s
1000) 31.7288 0.00000 2790.08 -0.15305 11s 11s
1500) 8.16277 0.00000 3113.09 -0.04456 17s 5s
2000) 12.9550 0.00000 3404.09 -0.03212 23s 0s
Test results on 1000 examples, 9472 ticks:
Network
Error total: 8686.99
Error per example: 8.68699
Error per tick: 0.91712
Unit cost per tick: 0.00000
Output unit criterion reached on 0 examples (0.00%)
Every 2000 epochs, the model tests the training and testing set. The Output unit criterion gives a rough measure of how many utterances were correctly produced. To see the actual utterances that were produced, examine the sum40000train file by typing "less sum40000train". You can scroll down by pressing the space bar, and "q" to quit.
You should see something like this. Each name refers to a pattern in the appropriate set (here the training set). tar: precedes the target in the environment file, and act shows the model's output. On the next two lines, the tar and act output has been tagged with syntactic categories. If it says "gram" at the end, then the sequence of syntactic categories is grammatical.
name:{ a girl was present -ing a sister a stick . . }
tar: a girl was present -ing a sister a stick . .
act: a girl was present -ing a sister a stick . .
syntar: DET NOUN AUX VERB ING DET NOUN DET NOUN PER PER
synact: DET NOUN AUX VERB ING DET NOUN DET NOUN PER PER = gram
At the bottom of each file (type ">" in less to skip to the bottom), you will see some general statistics. Here we see that 99% of the sequences were grammatical (gram), 76% were the exact target sentence (sent), and 96% of the word prediction were correct (word).
##results
##gram=_ corr c 995 t 1000 perc 99%
##sent=_ corr c 765 t 1000 perc 76%
##word=_ corr c 9173 t 9472 perc 96%
To look at how sentence accuracy changes over development, type this: grep "sent=" sum*train
You should see that the model develops quickly.
Dog-goal test:
Now training set is similar to the training set in the Chang 2002 paper in that dogs were not allowed to be goals. So now we can do the dog-goal test with this model. The test set is made up of only sentences with dogs as goals. So by looking at the sum*test files, we can see how well the model generalizes outside of its training space. Take a look at the overall accuracy and compare those measures with the training set.
To examine the model using the gui, type: lens -c prodSRN.in
Then "Load Weights" and click on comp40000.wt.gz. The model should now be set to the final set of weights.
Look at the unit viewer and the link viewer and try to figure out why the model cannot produce dog-goal sentences.
Dual-path Model
Now we will do the same test with the Dual-path model.
Use arcthis.perl to save the prodSRN results by typing: arcthis.perl prodsrn
Now train and save results for the Dual-path model by typing: lens -b dualpath.in 'trainSave;exit' &
Take a look at the sum*train and sum*test files. See if the model can produce the dog-goal sentences.
Then examine the model using the gui: lens -c dualpath.in
Load the weights from the final epoch and see if you understand how it works.