The files used to restart an optimization (i.e. continue it from where it finished off when the job timed out) are optimize.nsvd and optimize.rstrt. These files contain 1) the number of saved function evaluations and 2) the function and gradient values at each function evaluation. When a restart is done, SNOPT is essentially started from scratch and whenever SNOPT makes a call to usrfun() to do a function evaluation it checks if that function evaluation has already been computed and stored in optimize.rstrt and if so it reads it instead of computing it. It does this until the files are exhausted and then it starts evaluating new functions. This restart procedure makes any restarts invisible to SNOPT and hence the optimization deterministic. Some times it may be beneficial to "trim" off some of those saved evaluations and restart from earlier in the optimization history. For example let's say that at function call 103 it started experiencing multiple flow solver failures and then SNOPT went on to fail. If you want to modify a flowsolver parameter to help make the solver more robust, you would want to "trim" off the saved evaluations where it previously had touble and try them again with the improved parameters. The trimOptFiles script allows you to do this.
To "trim" the optimization requires removing some data from these files so when the restart is done it picks up from a few iterations earlier. There is a script in /home/z/zingg/treist/Scripts/trimOptFiles which does this. To use it, do the following:
Make a backup copy of the optimization directory (I'll use opt_dir as an example) just in case something goes wrong (you won't always need to do this, but just until you get familiar with the scripts)
>> cp -r opt_dir opt_dir_bkup
When you run the script it will ask for two inputs: 1) how many function calls you want to trim off from the restart file, 2) the size of the gradient array. For the number of function evaluations just select how many evaluations you want to trim from the end, i.e. how far back you want to rewind the optimization. For example, this could be until just before it started experiencing solver failures. The size of the gradient array is given by the value "nnzGrad" in results.scr.
In the opt_dir, run the trimOptFiles script after copying it to your ~/bin.
>> trimOptFiles
and enter the above values when prompted
The script will produce two files called optimize.nsvd.trim and optimize.rstrt.trim. Rename these files to optimize.nsvd and optimize.rstrt
>> mv optimize.nsvd.trim optimize.nsvd
>> mv optimize.rstrt.trim optimize.rstrt
Submit the case (with jtstrm%restart=.true.) the same way you would for a regular restart. Note that after it runs the old optimize.his and optimize.info will correspond to the "new" optimization run since the trimming is invisible to SNOPT, but the the results.ohis file will contain two entries for any function evaluation that was trimmed off and then run again after the trimmed restart.