Results

We are happy to announce the results of the NLI Shared Task 2013. Congratulations to all, and thank you for making this an exciting and interesting shared task!
 
Closed
 
  L1 F-Score
Team Name Submission Number Sub-task Overall Accuracy ARA CHI FRE GER HIN ITA JPN KOR SPA TEL TUR
Jarvis 2 closed 0.836 0.785 0.856 0.860 0.893 0.775 0.905 0.854 0.813 0.798 0.802 0.854
Oslo NLI 2 closed 0.834 0.816 0.850 0.874 0.912 0.792 0.873 0.828 0.806 0.783 0.792 0.840
Unibuc 5 closed 0.827 0.840 0.866 0.853 0.931 0.736 0.873 0.851 0.812 0.779 0.760 0.796
MITRE ``Carnie'' 2 closed 0.826 0.859 0.847 0.810 0.921 0.762 0.877 0.825 0.827 0.768 0.802 0.790
Tuebingen 1 closed 0.822 0.810 0.853 0.806 0.897 0.768 0.883 0.842 0.776 0.772 0.824 0.812
NRC 4 closed 0.818 0.804 0.845 0.848 0.916 0.745 0.903 0.818 0.790 0.788 0.755 0.790
CMU-Haifa 1 closed 0.815 0.804 0.842 0.835 0.903 0.759 0.845 0.825 0.806 0.776 0.789 0.784
Cologne-Nijmegen 2 closed 0.814 0.778 0.845 0.848 0.882 0.744 0.857 0.812 0.779 0.787 0.784 0.827
NAIST 1 closed 0.811 0.814 0.829 0.828 0.876 0.755 0.864 0.806 0.789 0.757 0.793 0.802
UTD 2 closed 0.809 0.778 0.846 0.832 0.892 0.731 0.866 0.846 0.819 0.715 0.784 0.784
Ualberta 3 closed 0.803 0.820 0.804 0.822 0.905 0.724 0.850 0.811 0.736 0.777 0.792 0.786
Toronto 1 closed 0.802 0.754 0.827 0.827 0.878 0.722 0.850 0.820 0.808 0.747 0.784 0.798
MQ 4 closed 0.801 0.800 0.828 0.789 0.885 0.738 0.863 0.826 0.780 0.703 0.782 0.802
cywu 1 closed 0.797 0.769 0.839 0.782 0.833 0.755 0.842 0.815 0.770 0.741 0.828 0.788
dartmouth 2 closed 0.781 0.761 0.806 0.812 0.870 0.706 0.846 0.788 0.776 0.730 0.723 0.767
ItaliaNLP 1 closed 0.779 0.738 0.775 0.832 0.873 0.711 0.860 0.788 0.742 0.708 0.762 0.780
Chonger 1 closed 0.775 0.764 0.835 0.798 0.888 0.721 0.816 0.783 0.670 0.688 0.786 0.758
HAUTCS 1 closed 0.773 0.731 0.820 0.806 0.897 0.686 0.830 0.832 0.763 0.703 0.702 0.736
LIMSI 4 closed 0.756 0.737 0.760 0.788 0.886 0.654 0.808 0.775 0.756 0.712 0.701 0.745
CoRAL Lab @ UAB 5 closed 0.748 0.704 0.806 0.783 0.898 0.670 0.738 0.794 0.739 0.616 0.730 0.741
LTRC IIIT Hyderabad 1 closed 0.744 0.680 0.778 0.748 0.839 0.693 0.788 0.781 0.735 0.613 0.770 0.754
CUNI (Charles University) 1 closed 0.725 0.696 0.743 0.737 0.830 0.714 0.838 0.676 0.670 0.680 0.697 0.684
UNT 3 closed 0.645 0.667 0.682 0.635 0.746 0.558 0.687 0.676 0.620 0.539 0.667 0.609
Bobicev 4 closed 0.625 0.513 0.684 0.638 0.751 0.612 0.706 0.647 0.549 0.495 0.621 0.608
kyle, crossley, dai, mcnamara 1 closed 0.590 0.589 0.603 0.643 0.634 0.554 0.663 0.627 0.569 0.450 0.649 0.507
UKP 2 closed 0.583 0.592 0.560 0.624 0.653 0.558 0.616 0.631 0.565 0.456 0.656 0.489
Michigan 3 closed 0.430 0.419 0.386 0.411 0.519 0.407 0.488 0.422 0.384 0.400 0.500 0.396
eurac 1 closed 0.386 0.500 0.390 0.277 0.379 0.487 0.522 0.441 0.352 0.281 0.438 0.261
VTEX 5 closed 0.319 0.367 0.298 0.179 0.297 0.159 0.435 0.340 0.370 0.201 0.410 0.230
 
Open-1
 
  L1 F-Score
Team Name Submission Number Sub-task Overall Accuracy ARA CHI FRE GER HIN ITA JPN KOR SPA TEL TUR
Toronto 5 open1 0.565 0.410 0.776 0.692 0.754 0.277 0.680 0.660 0.650 0.653 0.190 0.468
Tuebingen 2 open1 0.385 0.114 0.502 0.420 0.430 0.167 0.611 0.485 0.348 0.385 0.236 0.314
NAIST 2 open1 0.356 0.329 0.450 0.331 0.423 0.066 0.511 0.426 0.481 0.314 0.000 0.207
 
Open-2
 
  L1 F-Score
Team Name Submission Number Sub-task Overall Accuracy ARA CHI FRE GER HIN ITA JPN KOR SPA TEL TUR
Tuebingen 1 open2 0.835 0.798 0.876 0.844 0.883 0.777 0.883 0.836 0.794 0.846 0.826 0.818
Toronto 4 open2 0.816 0.770 0.861 0.840 0.900 0.704 0.860 0.834 0.800 0.816 0.804 0.790
LTRC IIIT Hyderabad 1 open2 0.741 0.677 0.782 0.755 0.829 0.693 0.784 0.777 0.728 0.613 0.766 0.744
NAIST 3 open2 0.703 0.676 0.695 0.708 0.846 0.618 0.830 0.677 0.610 0.663 0.726 0.688
Comments