Evaluating the performance of your model
Evaluating the performance of your model
Evaluating how well your NNN (and any other neural network framework) performs has to be done in two stages. First, you would need to test the predictions of the NNNs on datasets they weren't exposed to during the training process. Then, you need to compare these predictions to the presently used methods.
So how do I test the NNN predictions?
Go to the script runNNNsOnTestMesa80.py, found in
$NuclearNeuralNetworks/python_scripts_for_analysis/TestNNNs/
This script allows you to use your NNN trained models to predict the composition and energy terms of datasets the NNNs didn't see during the training process. You can generate these datasets following the procedure described in the training sets section for smaller parameter grids, or use the test datasets found in
$NuclearNeuralNetworks/test_datasets/mesa_${net}/${net}_output_files
where net=mesa_80 or net=meas_151.
runNNNsOnTestMesa80.py, goes over all the bbq test sets, extracts the temperature and density from the name of the output files and the composition in the index initialAgeIndex, and gives them as input for the NNNs. It uses the trained NNN models trainedModel to predict the composition and energy terms in finalAgeIndex. When using this script, you should make sure that timeStepIndex corresponds to the age on which trainedModel was trained, and that you re-multiply the energy terms by the normalization factor you divided the training sets (done in allNNNepsSingleFile). You could produce you own NNN trained models following the instructions in the training section, or use the NNN trained models found in
$NuclearNeuralNetworks/trained_NNN_models/mesa_${net}/
This script produces the files AverageParamsNNN.cvs which contains all the average electron faction and energy terms inferred from NNN predictions, AverageLossesNNN.csv which contains the average errors in NNN predictions compared to the target values from bbq, and NNNcomps.npz which contains the compositions predicted by NNNs for all files in the test database. You can find the results files for our NNN trained models in
$NuclearNeuralNetworks/python_scripts_for_analysis/CreateFigures/Figure4_main_results/${net}_results
And what should I compare them to?
Best practice should be to compare your NNN predictions to the currently used methods and see if there is an improvement in the results. We compare the parameters our NNNs predict to the results obtained by using a small nuclear reaction network composed of 22 isotopes. To do that yourself, you first need to computer the error that results from using this small nuclear reaction network. You can use the script compareApprox21andMesa80.py, found in
$NuclearNeuralNetworks/python_scripts_for_analysis/CompareLargeAndSmallNets/
Even though mesa_80 is in the name, it can be used for mesa_151 as well. This script compares between the electron fraction and energy terms of bbq files produced with the 22 isotopes network and with one of the large networks. You should give it the same indices as in the file runNNNsOnTestMesa80.py. The bbq sets generated with this 22 nuclear reaction networks are found in
$NuclearNeuralNetworks/test_datasets/mesa_${net}/$approx21_output_files
where net=mesa_80 or net=meas_151.
This script produces the files AverageParams21to80.csv that contains the average electron faction and energy terms across all test sets for small and large nuclear reaction network, and AverageLosses21to80.csv that contains the average errors in the results of the small net compared to the large net.
How can I make sure I am comparing apples to apples?
In order to make a meaningful comparison, the test sets of NNNs need to contain the same initial conditions as the small network. For the temperatures and densities this is trivial, but designing similar initial compositions requires a bit more work. To do that, you should run the scripts prepareTestDirsApprox21mesa80sameInitialComp.py or prepareTestDirsApprox21mesa151sameInitialComp.py found in
$NuclearNeuralNetworks/python_scripts_for_analysis/CompareLargeAndSmallNets/
on the 22 isotopes test sets you generated using the instruction in the training sets sections (see more details in the documentation of the script). The test sets provided abobe were prepared using this file.