SNFServer/readme.md

SNFServer is a basic Message Sniffer “service” that provides scanning via the XCI protocol. The most common way to use it is with SNFClient, but you can also build your own software to call the XCI endpoint directly. Generally, the client makes a tcp connection, tells SNFServer where the message file is, and gets back a scan result… all in simple XML.

Changelog / Journal

20200623_M [Server 3.2.2, Engine 3.2.2]

• Removed saccades algorithm to provide more comprehensive scanning. This will allow SNF to operate as a feature extractor for machine learning algorithms by matching “above-band” rules that do not compete with normal “in-band” rules.

The growing use of above-band rule groups like “Experimental Bulk/Noisy” and additional groups for machine learning feature extraction has changed the paradigm for heuristic competition in the SNF world. Where previously it was important to optimize scanning performance for low-powered hardware and heuristic competition could add pressure to select for more efficient rules; the new paradigm requires that any available patterns will match (at least once) and hardware constraints are no longer a serious concernt. For example, SNF is easily able to operate at scanning rates that are 3 orders of magnitude higher than most deployments require on modern equipment.

This isn’t to say that heuristic efficiency optimization will be going away -- but rather that the mechanisms for optimizing that efficiecncy can be moved more toward the back-end so that the front-end scanners can concentrate on making all available matches available for analysis and even more sophisticated learning algorithms.

Saccades was fun, and effective, but it’s time has passed.

20200622_M

• encapsulated all codedweller in namespace codedweller
• removed all using namespace std
• got a clean build (saccades still commented out, but not cleanly removed)

20200618_M

• Cleaned up all warnings in the build with the latest g++
• Added SOP for installing the latest g++ in ubuntu
• Added SOP for upgrading libstdc++6 on target ubuntus

Additional note: Using RESTsnf as a test jig measured throughput at 8064/minute. However, this was only a single data point so only gives us a ballpark, and most systems operate at least 2 orders of magnitude below this message rate.

Using the same test jig measured 7960/minute throughput with saccades off. Allowed the torture test to run so newer data would be in play and after 5 minutes measured 7806/minute. Here are a few more numbers:

2113 7279.7 2114 6084.89

Conclusion is that performance penalty for disabling saccades is not significant.

20200617_M

• Set up the readme.md file
• Set up basic build structures for “the new way” of making all things SNF.