Auto-tests are custom-built, bash-written in order to support testing of some very specific alert processing features: test results can be evaluated from multiple sources, timing of test execution can be steered to test time-sensitive test.
Every test works by emitting a stored event (sometimes with some minor modifications), and evaluating results from many angles: checking processing result which contains extensive context information, checking resulting profiles, checking existence of alerts in Elastic-Search, and API checks.
system: docker
CLI: bc jq uuidgen python3 mktemp curl awk cut tr grep egrep
servers: redis-stack, elastic-search
npm packages: retrieved upon test start when a container is built (Dockerfile.nodejs.test)
To set the system environment up, modify environment variables in ./cloudmon/.env.test. Normally, only REDIS and Elastic Search addresses need to be changed, other parameters shall be left untouched.
Further setting are in ./test/alerts/config.sh -- these should not be touched either.
Test-cases are grouped in directories ./test/alerts/<testgroup> . Each of the directories contains an alert configuration (config.json), execution script (script.sh), optionally variable configuration (vars.json) and predefined events. To start execution of a test group change directory to ./test/alerts and call:
./v2at -O -t <testgroup>
To start all test-cases as defined in ./testsuite.sh, call v2at without the -t option:
./v2at -O
Step-wise execution can be enabled using the "-s" option.
./v2at -O -s
The first test is supposed to send an unrelated event (auth-failed) which shall be filtered out and not effect the profile.
The singletest test command is followed by the following parameters:
resetenv: specifies if configuration shall be set up and all related profiles shall be cleared before test execution begins. This typically makes sense for the first test in a test group to ensure the test will not be affected by the content of profiles from previous tests and is specified using the yes option. The yes option clears all related profiles and also merges a default configuration with the requested configuration -- this allows to test for possible interference of the tested configuration with other alerts. If the test configuration shall be limited exactly to the specified configuration without merging with defaults, the option 'resetwodefaults' must be used instead. Subsequent tests in a group typically preserve the profiles updated by previous tests: no option.
callid: specifies if CallID shall be modified. Use CREATE unless you have a reason to special-case. CREATE creates a new CAll-ID in the test-events dynamically -- this is particularly important when 1) call timers play role and repeated test execution is supposed to create new calls with their own timers 2) elastic criteria is used to evaluate a test-case because the resulting alert is queries in Elastic by call-id. KEEP recycles call ids created in previous test-case -- this helps to address the cases when there are multiple events relating to the same call. PRESERVE leaves it as in the original JSON document.
event filename . The file is expected to be a JSON event. It may also come in AWS load-balancer "wrapping".
test criteria in form (can be repeated):
source. It can be
a profile (e.g. 'customprofile:.attrs.from-ua') where the profile identified by attribute name and attribute's value as in the test event. Joint keys can be formed using the "+" operator like in customprofile:.attrs.from+.attrs.source. Multi-keys can be formed like in customprofile:.attrs.to:attrs.from ... then the value is taken from test-event using the second attribute name. A key referring to all events owned by a tenant is defined by attribute name omission ('customprofile:').
HTTP reply returned in response to submitting an event ('answer')
presence of the same-callid event in elastic-search ('elastic').
value reference to look at in jq syntax (e.g. '.Item["CTHR_CTHR.agg.sum"]')
expected value as returned by jq (e.g. 15 or null). if enclosed in [], it means the value is expected to be element of a list. if prefixed with tilda ~, the value is expected to be contained in a string. If the value __ignore is used, the result will not affect the testing score.
singletest yes CREATE filtered_custom_rate/auth-failed.json \
'customprofile:.attrs.from-ua' ".Item[\"CTHR_CTHR.agg.sum\"]" 'null' \
'answer' '.assessment.evilScore' '0'
The next case uses configuration and profiles from the execution of the previous test-cases, creates its own call-id and injects a call-end event. It expects the UA's profile to include now 3000 second as in the events and the score to be neutral because 3000 is bellow threshold.
singletest no CREATE filtered_custom_rate/call-end.json \
'customprofile:.attrs.from-ua' ".Item[\"CTHR_CTHR.agg.sum\"]" '3000' \
'answer' '.assessment.evilScore' '0'
When too many minutes accumulate above the threshold, the event score shall become high and result in propagation of an alert in elastic search
singletest no CREATE filtered_custom_rate/call-end.json \
'customprofile:.attrs.from-ua' ".Item[\"CTHR_CTHR.agg.sum\"]" '6000' \
'answer' '.assessment.evilScore' '100' \
elastic 1 what
API calls can be also evaluated using the "testapi" command followed by the following parameters:
API name: alertapi or bwapi
test criteria:
attribute name in response body, for example .statusCode
expected value, for example 200. _ignore can be used as a special value, the result is ignored and neither positive nor negative result is reported. Useful when some preparatory API calls are made.
parameters (repeated), that are turned into URI parameters or body
name. if "--bodyfile", the value is taken from content of filename in the next parameter and placed in request body. Otherwise it is used as URL parameter name.
value. URL parameter value, unless a special-cased name is used.
FILE=expimp/strconfig.json
testapi alertapi .statusCode 200 addalert "--bodyfile" "$FILE"
Results from the past tests can be reused in the next ones. The variable SELECTVAL contains the most recent tested value.
# create a shortlink
testapi bwapi ".key" __ignore createshortlink --bodyfile $URLFILE
# retrieve it from the tested value
SHORTLINK=`echo -n $SELECTVAL| jq -r .`
# test if the shortlink value returned in previous test expands
testapi bwapi .redirect.longurl "\"$MYURL\"" getshortlink short $SHORTLINK
In some advanced cases, additional API calls need to be made to make sure that the results of previous tests don't interfere. In this case, `__ignore` is used as expected value: the result will not count against the testing score.
a URI profile can be deleted:
testapi bwapi .statusCode __ignore delete key "sip:foo@bar.com" list uriprofile hmac plain
all blacklist entries can be deleted (stringmatch testcase):
testapi bwapi .statusCode 200 wipelist list ipblack hmac plain
all timers can be deleted (regcount testcase):
testapi alertapi .statusCode "__ignore" rmallprofiles list "timer-MPX#168000" hmac timer-MPX