The main idea...
Null hypothesis
There is no difference between the means of two or more groups of (ranked) dissimilarities.
The ANalysis Of SIMilarity (ANOSIM) test has some similarity to an ANOVA-like hypothesis test, however, it is used to evaluate a dissimilarity matrix rather than raw data (Clarke, 1993). Further, raw (dis)similarities are often ranked prior to performing an ANOSIM. Ranking dissimilarities aligns ANOSIM to the non-metric multidimensional scaling (NMDS) procedure. Together, the dimension reduction and visualisation capacities of NMDS and the hypothesis testing offered by ANOSIM are complementary approaches in evaluating nonparametric multivariate data. Readers should also consider the later technique of non-parametric MANOVA (NPMANOVA).
In addition to the one-way ANOSIM shown in Figure 1, two-way ANOSIM as well as crossed and nested designs are possible.
Figure 1: Grouping in a dissimilarity matrix. Given a dissimilarity matrix with defined groups (a) a ranked dissimilarity matrix may be calculated (b) from which ANOSIM may compare the mean rank within groups (blue triangles) to the mean rank between groups (orange rectangle). Note that ANOSIM does not strictly require ranked data; however, if ANOSIM is to be coupled with NMDS, this is recommended. For more than two groups, the mean ranks of all within- and between-group sub-matrices are considered simultaneously when computing the ANOSIM R statistic.
The ANOSIM R
The ANOSIM statistic compares the mean of ranked dissimilarities between groups to the mean of ranked dissimilarities within groups. An R value close to "1.0" suggests dissimilarity between groups while an R value close to "0" suggests an even distribution of high and low ranks within and between groups. R values below "0" suggest that dissimilarities are greater within groups than between groups. See Clarke and Gorley (2001) for a guide to interpreting ANOSIM R values.
Significance of the R statistic is determined by permuting group membership a large number of times to obtain the null distribution of the R statistic. Comparing the position of the observed R value to the null distribution allows an assessment of statistical significance.
The ranges of (ranked) dissimilarities within groups are equal, or at least very similar.
Do not assign group membership based on the results of clustering (or a similar exploratory method) applied to the same data set and then treat a significant ANOSIM result as meaningful. This is an example of data dredging.
Running ANOSIM on groups with very different dispersions can lead to unreliable results. Groups with very different dispersions may produce high R values, even if there's no real difference in their centroids. If differences in group dispersion are as meaningful to your analysis as differences in group centre, this may not be an issue.
Criticism of this and other (dis)similarity-based methods should be considered (e.g. Warton et al., 2012).
MASAME ANOSIM app
Clarke KR (1988) Detecting change in benthic community structure. 131-142 in R. Oger [ed.] Proceedings of invited papers, 14th international biometric conference, Namour, Belgium.
Clarke KR (1993) Non-parametric multivariate analyses of changes in community structure. Aust J Ecol 18:117-43.
Clarke KR, Gorley RN (2001). Primer v5: user manual/tutorial. Primer-E Ltd: Plymouth.
Warton DI, Wright ST, Wang Y (2012). Distance-based Multivariate Analyses Confound Location and Dispersion Effects. Methods Ecol Evol. 3(1):89–101.