SOMALib - Library of Activation Functions

This library is part of paper "SOMALib : Library of Exact and Approximate Activation Functions for Hardware-efficient Neural Network Accelerators" in 40th ICCD 2022.

Approximate computing along with quantized lowprecision computing has gained significant interest in today’s neural network (NN) implementation. This paper proposes a library of VLSI implementations of different activation functions, aimed towards designing hardware-efficient NN accelerators. Cartesian genetic programming (CGP), an evolutionary algorithm was employed to generate gate-level designs of approximate and exact representations of activation functions. We open-source the hardware library of 9444 circuits containing a majority of the activation functions employed in NN architectures, including Sigmoid, Hyperbolic-Tangent, Gaussian, ReLU, GeLU, Softplus, and Binary-Step. The library also presents the error characteristics and hardware metrics of the designs which will aid in the usage of the library in future research. Additionally a hardware comparison of the proposed circuits against existing implementations including piecewise-linear (PWL), memory-based, hls4ml, DNNweaver implementations to realize activation functions on FPGA and ASIC flow is presented. The CGP evolved hardware library shows minimal silicon space requirement, least power consumption when investigated for ASIC flow, and the least LUT utilization’s in FPGA flow. Besides, SOMALib designs are purely combinatorial, allowing various synthesis stage optimizations towards the target Power-Performance-Area budget, which is not possible in standard memory block implementations.

This library contains hardware-efficient implementations of Activation functions that can be used in NN, CNN applications.

The activation functions Binary step (Bstep), Sigmoid, Gaussian, Hyperbolic-tangent(tanh), Rectified Linear Unit (ReLU), Gaussian Error Linear Unit (GeLU), Softplus are implemented :

The library contains a total of 9444 circuits :

The variation of circuits hardware-resource in the library is represented in the following hardware resource vs MAE plots :

Data format

All the functions contain circuits implementing quantized activation function of word length 4bit to 8bit.

Round-to-floor quantization is used to uniformly quantize all continuous functions to the fixed point data.

An example of the quantization scheme :

The input data format has Four different fixed point configurations with different range-precision tradeoffs :

The range of input data possible for the configurations :

The output data format for Sigmoid, Hyperbolic-tangent and Gaussian activation function was configured similar to configuration 2 of Gaussian function since it offers enough integer bits to represent full range of output values between -1 to 1.

ReLU, GeLU and Softplus circuits have the same output configuration as their input configuration.

Hardware Characterization

All circuits in the library are characterized for Area, Power and Delay using Cadence Genus Synthesis software. PDK(Process Design Kit) used is the Cadence-gpdk45, a 45nm technology library. The circuits are characterized as purely combinational circuits.

The reports are for the standard cell ASIC synthesis, and similar trend can be expected if a different library is used.

Error Characterization

All the circuits are characterized for the following error metrics : Error Count (EC),Error Probability (EP), Absolute Error Maximum value(AEmax), Absolute Error Minimum value (AEmin), Absolute Error Standard Deviation (AESTD), Hamming Distance(HD) between expected and circuit output, Mean Absolute Error (MAE), Mean Square Error (MSE), Mean Relative Error (MRE), Median and Mode of Absolute Error

A error plot is also provided to visualize the circuit output, location of erroneous output and the histogram of error values.

The Circuit Photos of all the activation functions are included to understand the structure of combinatorial circuits.