SwitchNet

Viewpoint 1

The output of a neuron in an ordinary neural networks connects to n weights in the next layer and the pattern in those weights is projected with intensity x (the output value of the single neuron) into the next layer. 

In a fast transform if a single input is made non-zero the basis vector it corresponds to is projected onto the output with intensity x, x being the value of the single non-zero input. And that pattern is typically a highly ordered sine wave or cosine wave or something like that.  The parametric activation function then controls the intensity the highly ordered pattern is projected with.

Viewpoint 2:

A random projection maps an input vector to a vector pointing in a different (random) direction.

If you make the random projection adjustable then you can start to map an input set of vectors to a wanted set of output vectors. 

If you include adjustable non-linear functions you can perform a sequence of adjustable mapping using the random projections and there is no real difference with a normal neural network.

Using randomly assigned random sign flips before a fast Walsh Hadamard transform results in a random projection.

Altering those sign flips makes the (random) projection adjustable. Since multiplying by a randomly assigned pattern of +1's and -1's is one method of applying sign flips you can make the system more adjustable by multiplying by any numbers between say -1 and +1 for example.

You can introduce non-linear behavior by switching in different "flip" values depending on some condition.  This allows you form a sequence of adjustable projections using switching and the fast Walsh Hadamard transform culminating in a neural network.