Watchlists serve the purpose of tracking the values of an attribute encountered in incoming events. The lists are global for a tenant. When an alert during processing a key spots a new element, it places it on the list. The element stays there, regardless of which alert(s) placed it there during the processing of some keys. The element disappears only if its time-to-live expires or is administratively removed using the API. Third-party tools can synchronize the list content at any point in time using the API.
A typical use case is blacklisting or whitelisting an IP address. Therefore the notion of watchlist is used interchangingly with whitelists and blacklists, even though other uses may appear in the future. The watchlists in use are specified in alertTypes.js and currently include blacklists and whitelists for source IP and From URI addresses. For example, an alert detecting too many failed authentications for a user (the key is the user's URI) may choose to blacklist one or more IP addresses the failures came from.
The lifecycle of a watchlist element looks like this:
An event is received that meets the following conditions:
An alert's processing score for an event is> 0, for example, when authentication repeatedly fails for the same URI.
IP blacklisting is activated in the alert's configuration
If the conditions are met, the IP address is placed on a blacklist. An additional possibility for placement exists, see "hysterezy" bellow. The placement is time-limited and will expire. The timestamp of blacklisting is recorded in URI's profile for informational purposes.
A third-party tool finds in the next iteration of an API polling loop that a new IP address was added and places the IP address in its packet-filtering policy. It also notices that the IP address comes with a time-to-live and shall expire at a point of time.
An administrator chooses to delete the IP address from the blacklist via API. This makes the address delete-marked so that polling API users can detect the deletion.
The third-party tool finds out that since the last polling time, the item has been delete-marked and remove the IP address from its policy.
Hysteresis. This feature allows administrators to decide that for a key, any new element will be immediately placed on a list. For example an alert may reveal that security of a user identified by his URI is entirely compromised. If the admin activates hysteresis for this alert and key value (in our example: authentication failing for user foo@bar.com), any future events relating to the same key and processable by the alert (e.g., not filtered out by matching expression) will cause their IP addresses to be blacklisted even if their score is zero. Again: this is manually activated using the API and remains in force until deactivated.
Blocking. Some lists may be configured in such a way that if an element is listed, alerts are no longer processed for events with this element. This is practical, for example, if an IP address is blocked, and a probe sitting before a firewall keeps sending events on timed-out traffic: this way, the timed-out traffic is not processed anymore and does not affect other alerts. Blocking applies for a period of time set in limits.DONT_RELIST_PERIOD (300 sec by default) since the last listing. Blocked events appear in stats under the "fwalled" tag. Some lists may be configured so that events with listed elements are processed, but alerts will not be raised during this period. Events whose alerts were blocked this way appear in the stats as "blisted."
Mutual Exclusion. It may make little sense to both whitelist or blacklist an IP address. In general it is possible that an element is scored to appear in multiple list, which are mutually exclusive. Then some may be configured to take precedence over the others. In this example, if an IP address shall be both blacklisted, say because of geo-fencing, and whitelisting because of 200 response-code, blacklist will prevail and whitelisting will not happen. TBD: CLARIFY IF THIS REFERS TO CURRENT SCORING OR INCLUDES LI:STING HISTORY!
Expiry. TBD Whats is TTL
not relisting. (efficiency) TBD
API can be used for inspecting the watchlist elements. The most interesting capability is that of retrieving watchlist updates: a third-party app with proper credentials may periodically retrieve updates to a watchlist. For example, an IP firewall may keep asking if some IP addresses have been added to or removed from IP blacklist, and adjust its packet filtering policy accordingly.
TBD: reference to API, make sure the following is there:
remove TBD
poll TBD
blockmall
Taken from alertTypes.js:
const WBLISTS = {
// name of the "profile" for a listed value (part of the secondary
// key in DB
ipblack: {
// name of the attribute in event
path: "attrs.source",
// condition : prefix of score name in testContext (must be >0)
// the value must be "evil" or "legit"
scorename: "evil",
// minimum score to be present (0 by default)
minscore: 10,
// condition: a config param must must be turned on ...
config: constants.IPBLACKLIST_PAR,
// ... or a bit in profile
profile: utils.LISTMALL,
// don't process incoming event if the watched element recently listed
// (meaning no alert processing, no profile updates, no alerts...)
// if undefined/false, everything will be processed even for listed eleemnts;
// if true, everything will be blocked if a listed element is present;
// if a filter is there, all events with a listed element will be blocked if
// the filter matches
dontProcessFilter: "attrs.sip-code~'408|487'",
// don't raise alerts if the watched element recently listed
supressAlerts: true,
// relative time-to-live
ttl: limits.DELETE_EXPIRES_IN_SEC
},
ipwhite: {
path: "attrs.source",
scorename: "legit",
config: constants.IPWHITELIST_PAR,
// don't list if element listed on some higher-precedence list
// MUST refer to some other watchlists
precludeif: ["ipblack"]
},
uriblack: {
path: "attrs.from",
scorename: "evil",
config: constants.URIBLACKLIST_PAR
},
uriwhite: {
path: "attrs.from",
scorename: "legit",
config: constants.URIWHITELIST_PAR,
precludeif: ["uriblack"]
}
};
TBD
Working out other use cases: Watchlists may constitute an alternative to associative memory alert types. This use would scale better for long lists but make it more difficult to present the lists as part of profiles. (Unless profiles are more virtualized and put together from various pieces, like watchlists and series to present a single view for a key.) One could for example generated an inventory of spotted MAC addresses, device types, etc this way and keep polling it.
Customization. The watchlist specification could be in config (making it more complicated though). Particularly needed if the use-cases shall be extended.
Greylisting. It may make sense to place a new element on probation.
Write the list specification down for every alert type (now a default list is used for all of them). When there are more of them with the same name, make sure their settings don't conflict, and neither do names with reserved profile names (`varconfig`, ... , `customprofile`)