git-svn-id: https://svn.microneil.com/svn/SNFMulti/trunk@47 dc71a809-1921-45c4-985c-09c81d0142d9

11 years ago · 0e4a12fa3e
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,405 +0,0 @@
 2009-05-27  Alban Deniz  <adeniz@skidmark.localdomain>

 	* Makefile.am (EXTRA_DIST): Include Makefile.am and ChangeLog in
 	the distribution.

 2009-05-23  Alban Deniz  <adeniz@skidmark.localdomain>

 	* Makefile.am (noinst_HEADERS): Removed tcp_watchdog.hpp and
 	mangler.hpp.
 	(libSNFMulti_a_SOURCES): Removed tcp_watchdog.cpp and mangler.cpp.

 2009-02-06  Alban Deniz  <adeniz@skidmark.localdomain>

 	* SNFMulti.cpp: Replaced with file from Pete, Jan 29, 2009.

 2008-12-09  Alban Deniz  <adeniz@skidmark.localdomain>

 	* SNFMulti.cpp (enum PatternResultTypes): Remove 'typedef'.

 	(snf_EngineHandler::scanMessageFile): Make XHDRInjState const char *.
 	
 	(snf_EngineHandler::scanMessage): Make DebugInfo const char *.

 	* GBUdb.cpp (GBUdbAlert::toXML): Make FlagName a pointer to const
 	char *.

 	* FilterChain.hpp (class FilterChainHeaderAnalysis): Pass const
 	char * to SetFollowPattern.  Make MatchPattern a pointer to const
 	char *.

 SNF Command Line & SNFMulti Engine / Client Change Log
 ------------------------------------------------------------------------------

 20080710 - Version 3.0.1

 Minor change to SNFServer main.cpp:59 - removed cast to (int) which caused
 a precision loss error when compiling on 64 bit systems. This changes the
 thread pointer info in debug mode slightly (better).

 20080626 - Version 3.0, It's official.

 Changed build information.
 Removed extraneous comments from configuration file.

 20080524 - Version V2-9rc2.25.7

 Optimized networking library for additional speed & stability by moving
 receive buffer allocation from heap to stack (automatic).

 Optimized timing parameters in SNFClient for improved speed. Polling dealys
 are now reduced to 10ms from 30ms.

 Removed speed-bug in SNFClient, 100ms guard time between retries was always
 executed after an attempt (even a successful attempt). The guard time is now
 condition and only fires on unsuccessful attempts.

 Updated XCI server logic to ensure non-blocking sockets for clients in all
 socket implementations.

 20080424 - Version V2-9rc2.24.6

 Refactored snfScanData.clear() to reduce heap work and fragments.

 Added mutex to scanMessageFile() entry point just in case some app attempts to
 put multiple threads through a single engine handler. scanMessage() is already
 protected and fully wraped by the new scanMessageFile() mutex.

 Added non-specific runtime exception handling to XHDR injection code.

 Added 2 retries w/ 300ms delay to remove original message in XHDR inject code.
 If remove fails after 3 attempts the injector throws.

 Added 2 retries w/ 300ms delay to rename temp file to msg in XHDR inject code.
 If rename fails after 3 attempts the injector throws.

 20080416 - Version V2-9rc2.23.6

 Fixed bug where SNCY open() would fail on some Win* platforms with
 WSAEINVAL instead of the standard EINPROGRESS or EALREADY which were expected.
 Also added WSAEWOULDBLOCK to cover other "ambiguities" in windows sockets
 implementations. InProgress() on Win* now test for any of:

 WSAEINPROGRESS, WSAEALREADY, WSAEWOULDBLOCK, WSAEINVAL

 20080413 - Version V2-9rc2.22.6

 Fixed bug in TCPHost.open() where EALREADY was not counted as a version of
 EINPROGRESS. This would cause open() to throw an unnecessary exception when
 an open() required extra time.

 20080413 - Version V2-9rc2.21.6

 Extended timeout for SYNC session open() to the full session length. This way
 if a session takes a long time to open it still has a shot at success.

 20080411 - Version V2-9rc2.20.6

 Adjusted snfNETmgr to use non-blocking open in SYNC sessions. Open timeout
 is 1/3 of the session timeout. Session timeout is 2 * Session pacing. Open
 polling uses golden spiral delay from 10ms to 340ms.

 20080410 - Version V2-9rc2.19.6

 Adjusted XCI manager to use new snfCFGPacket paradigm in checkCFG().

 Adjusted snf_RulebaseHandler::addRulePanic() to use MyMutex and eliminated
 the AutoPanicMutex and waiting scheme.

 Refactored scanMessage() to use a ScopeMutex() rather than lock()/unlock().

 Refactored scanMessage() to use MyCFGPacket.isRulePanic() test.

 Redesigned snfCFGPacket handling to automate grab() / drop() functions.

 Fixed lock-up bug: Redesigned AutoPanic posting and checking mechanisms to 
 eliminate potential dead-lock condition. Under some conditions a precisely
 timed auto-panic posting could cause the RulebaseHandler mutex and the 
 AutoPanicMutex to become intertwined leading to a cascading deadlock. When
 this occurred all XCI processing threads and eventually the XCI listener
 thread would become blocked waiting to get the current configuration.

 20080409 - Version V2-9rc2.18.6

 Enhanced XCI exception handling and logging to provide additional detail.

 Added code to explicitely check for zero length files in scanMessagFile().
 Previously a zero length file would cause the CBFR module of the filter
 chain to throw an invalid buffer exception. Now if the message file is empty
 scanMessageFile() will throw a FileError stating FileEmpty!.

 20080407 - Version V2-9rc2.17.6

 Enhanced exception reporting in snfXCImrg


 20080405 - SNFServer V2-9rc2.16.6

 Reduced safetly limits on status reports to 100K for status reports and 100K
 for samples. Previous values were 10M. Most full sessions from the busiest
 systems are < 50K total.

 Recoded sendDataTimeout() to break uploads into 512 byte chunks and insert
 delays only when a chunk is fragmented. This methodology improves reliability
 on Win* systems without any significant penalty on systems that don't need
 socket sends() to be in smaller chunks.

 Fixed TCPClient::transmit() and TCPHost::transmit() bug where returned byte
 count might be -1. Now returned byte counts can only be 0 or more.

 20080403 - SNFServer V2-9rc2.15.5

 Minor modifications to networking module to better support non-blocking open()

 Updated SNFClient with new timing and non-blocking open(). Worst case return
 time from SNFClient estimated at 200 seconds (theoretically impossible). No-
 connection return time from SNFClient estimated at 20 seconds.

 20080326 - SNFServer V2-9rc2.15.4

 Refactored snfNETmgr::sync() to consolidate non-blocking io routines.

 Added detailed thread status data to XCI listener thread.

 Fixed minor bug in main (not changing revision), Debug flag for internal use
 was left on in the last build cycle. It is commented out now.

 20080325 - SNFServer V2-9rc2.14.4

 Updated snfNETmgr with comprehensive thread status data.

 Refactored snfNETmgr::sync() to check a Timeout, removed TCPWatchdog.

 20080325 - SNFServer V2-9rc2.13.4

 Upgraded TCPWatcher code to use new threading features (type, status).

 20080324 - SNFServer v2-9rc2.12.4

 Added a "Rulebase Getter" feature as part of the snf_Reloader. When enabled
 the Rulebase Getter will launch a user defineable system() call whenever a
 new rulebase file is available. The call will be repeated until the condition
 is cleared by a successful update of the rulebase file. The Rulebase Getter
 will wait a configurable "guard time" between attempts. The default system()
 call is "getRulebase" with a guard time of 3 minutes. In most cases this will
 launch the provided getRulebase script which should be present in the start
 location of SNFServer on most systems. Best practice is to configure the full
 path to the update script. The system() call is made in a separate thread so
 that if the system() call hangs for some reason only the Rulebase Getter is
 stuck.

 Built thread monitoring function for SNFServer.exe (Full status report / sec).
 The thread monitoring report is turned on when the program is renamed to
 SNFDebugServer.exe or if "debug" appears in the file path to the program.

 Refactored XCI channels to leverage new thread monitoring.

 Refactored Threading to eliminate inline code.

 Improved exception handling/reporting in scanMessageFile().

 Updated scanMessagFile() header injection code to accommodate messages with
 no body. Previous version would throw an exception when it could not find an
 injection point. The new version makes the injection point byte 0 and puts
 the injected headers at the top of the message using it's best guess about the
 type of line endings (CRLF or LF) to use.

 Updated Threading library to include high level thread state tracking and
 naming. Also creates a global Threads object that can produce a real-time
 status report on all threads.

 Updated Networking library to use SO_REUSEADDR by default on listeners.

 20080318 - SNF2-9rc1.11.exe Consolidated several mods/fixes

 Corrected scan error logging bug. Was posting <s/> now posts <e/>.

 Updated scan error logging to be more uniform with non-scan errors.

 Developed various script prototypes for postfix integration & automated
 updates on win* systems using the new UpdateReady.txt file mechanism.

 Fixed a bug in scanMessageFile() where an \n\n style insertion point
 would never be detected.

 Modified scanMessageFile() header injection to strip <CR> from line ends
 when the message file provided does not use them. The line-end style of
 the message file is detected while locating the insertion point. If the
 insertion point (first blank line) does not use <CR><LF> then the SNF
 generated X-Headers are stripped of <CR> in a tight loop before injection.

 Enhanced error and exception reporting in SNFMulti.cpp scanMessageFile().

 Enhanced exception handling in networking module. All exceptions now
 throw descriptive runtime_error exceptions.

 20080306 - SNF2-9rc1.8.exe (FIRST RELEASE CANDIDATE for VERSION 3!)

 Added Drilldown Header Directive Functions - When the candidate source IP
 comes from a header matching a drilldown directive the IP is marked "Ignore"
 in GBUdb and the candidate is no longer eligible to be the source for that
 message. This allows SNF to follow the trusted chain of devices (by IP) down
 to the actual source of the message. It is handy for ignoring net blocks
 because it can match partial IPs but it is designed to allow SNF to learn
 it's way through the servers at large ISPs so that the original source for
 each message can be evaluated directly.

 Added Source Header Directive Functions - This feature allows SNF to acquire
 the source IP for a message from a specific header rather than searching
 through the Received headers in the message. This is useful when the original
 source for a message is not represented in Received headers. For example:
 Hotmail places the originating source IP in a special header and does not
 provide a Received header for that IP. This feature is protected from abuse
 by a "Context" feature which only activates the source header directive when
 specific content is found in a specific received header. Using the above
 example, this feature can be configured so that a Hotmail source header would
 only be read if the top Recieved header contained "hotmail.com [" indicating
 that the ptr lookup for the header matched the hotmail domain. Note: When a
 source is pulled from a header directive that source is put into a synthetic
 Received header and injected into the scanning stream (not the message) as
 the first Received header.

 Added forced source IP to XCI - It is now possible to "inject" or "force"
 the source IP for any message by providing that IP in the XCI request or
 directly in a scan...() function call. This allows the calling application
 to provide the source IP for a message ahead of any Received headers that
 might be in the message. This is useful when the calling application knows
 the original source IP for the message but that IP is not represented in
 the Received headers and it is not desireable to use the Source Header
 Directive mechanism.

 Added forced source IP mode to SNFClient - It is now possible to call the
 SNFClient utility with an IP4Address using the syntax:

 SNFClient -source=12.34.56.78

 The -source mode of SNFClient exercises the forced source IP feature in
 the XCI (see above)

 Added Status Report features to SNFClient and XCI - It is now possible to
 request the latest status.second, status.minute, or status.hour data via
 the XCI and SNFClient. The syntax for requesting a status report using the
 SNFClient is:

 SNFClient -status.second
 SNFClient -status.minute
 SNFClient -status.hour

 In addition to providing status reports the SNFClient in this mode will
 return a nonzero value (usually 99) if it is unable to get a status report
 from SNFServer. This feature can be used to verify that SNFServer is up
 and responding. If SNFServer is OK then the result code returned is 0.

 Added result codes to SNFClient -test and XCI IP test functions - The XCI
 engine has been upgraded to provide the range value for the IP under test
 as well as the symbolic result code associated with that range. This allows
 the -test function to provide results that are consistent with the GBUdb
 configuration without additional processing: For example, if the IP falls
 in the Caution range then the Caution result code will be returned just
 as if a message had been scanned with the same IP and no pattern match
 occurred. The same is true for Truncate and Black range hits.

 Added Timestamp and Command Line Parameter data to SNFClient.exe.err - When
 an error occurs with SNFClient that may not appear in the SNFServer logs an
 entry is appended to the SNFClient.exe.err file. That in itself is not new.
 The new feature is that the entries added to the SNFClient.exe.err file now
 include timestamp and command line data to aid in debugging.

 Added BIG-ENDIAN Conversion - When the SNFServer program is compiled on a
 system that uses a BIG-ENDIAN processor (such as a power-mac) the rulebase
 load process now includes a routine to convert the token matrix from it's
 native LITTLE-ENDIAN format to a BIG-ENDIAN format. This solves a bug where
 Power-Mac (and presumably other BIG-ENDIAN systems) could compile and run
 the SNF* software but were unable to capture spam because the token matrix
 in the rulebase file was misinterpreted.

 Note: The BIG-ENDIAN Conversion feature is still considered experimental
 because it has not yet been thoroughly tested.

 Updated the Configuration Log to include all of the current configuration
 features and to improve it's readability.


 20080207 - SNF2-9b1.7.exe

 SYNC Timeout now 2x SYNC Schedule

 SNFServer now produces an UpdateReady.txt file when the UTC timestamp on
 the SYNC server is newer than the UTC timestamp of the active rulebase. It
 is presumed that a suitable update script or program will run periodically
 and download a fresh rulebase file if the UpdateReady.txt file is present.
 The update script should remove the UpdateReady.txt file when it completes
 a successful download of the new rulebase file.

 Added available rulebase UTC in status reports <udate utc.../>

 Added Automatic path fixup for ending / or \

 Added option to use local time in log rotation <rotation localtime='no'/>
 The default is still utc.

 20071102 - SNF2-9b1.6.exe

 Increased MAX_EVALS from 1024 to 2048.

 Adjusted defult range envelopes in snf_engine.xml to be more conservative.

 20071017 - SNF2-9b1.5.exe

 Added a missing #include directive to the networking.hpp file. The
 missing #include was not a factor on Linux and Windows systems but
 caused compiler errors on BSD systems.

 Corrected a bug in the GBUdb White Range code where any message with a
 white range source IP was being forced to the white result code. The
 engine now (correctly) only forces the result and records the event when
 a black pattern rule was matched and the White Range IP causes that
 scan result to be overturned. If the scan result was not a black pattern
 match then the original scan result is allowed to pass through.

 Corrected a bug in the Header Analysis filter chain module that would
 cause the first header in the message to be ignored in some cases.

 Corrected an XML log format problem so that <s/> elements are correctly
 open ended <s ....> or closed (empty) <s..../> according to whether they
 have subordinate elements.

 Adjusted the GBUdb header info format. The order of the Confidence
 figure and Probabilty figure is now the same as in the XML log files
 (C then P). The confidence and probability figures are now preceeded
 with c= and p= respectively so that it's easy to tell which is which.

 20071009 - SNF2-9b1.4.exe

 Tightened up the XCI handler code and removed the watchdog. The watchdog
 would restart the listener if there were no connections in 5 minutes. It
 was originally added to provide additional stability, however in practice
 there have been no "stalled listeners". Also, a stalled listener would
 likely be a sign of a different problem that the watchdog would tend to
 hide.

 Modified and refactored the XCI configuration management code. All XCI config
 changes and up-down operations are now handled in a single function except
 upon exit from the main XCI thread where XCI_shutdown() is always called.

 Added some more detailed exception handling code to the XCI component so that
 more data will be logged in the event of an error.


 20071008 - SNF2-9b1.2.exe

 Added support for passing Communigate Message Files directly. Communigate adds
 data to the top of the message file. That data stops at the first blank line and
 the rfc822 message begins. The SNFServer engine can now be told to ignore this
 extra data using the following option:

 <msg-file type='cgp'/> <!-- type='cgp'  for communigate message files -->

 If the msg-file type is anything other than 'cgp' then it will treat the message
 file as a standard rfc822 message in the usual way. The default setting is

 <msg-file type='rfc822'/>




--- a/Makefile.am
+++ b/Makefile.am
@@ -1,56 +0,0 @@
 ## Process this file with automake to produce Makefile.in
 ##
 ## $Id$
 ##
 ##
 ## Author:  Alban Deniz
 ##
 ## Copyright (C) 2008 by MicroNeil Corporation. All rights reserved.
 ##

 CXXFLAGS = $(SNF_CXXFLAGS) -I@top_srcdir@/SNFMulti -I@top_srcdir@/CodeDweller

 noinst_LIBRARIES = 					\
 	libSNFMulti.a

 libSNFMulti_a_SOURCES =        			 	\
 	@top_srcdir@/SNFMulti/FilterChain.cpp		\
 	@top_srcdir@/SNFMulti/GBUdb.cpp			\
 	@top_srcdir@/SNFMulti/scanner.cpp		\
 	@top_srcdir@/SNFMulti/snfCFGmgr.cpp		\
 	@top_srcdir@/SNFMulti/snf_engine.cpp		\
 	@top_srcdir@/SNFMulti/snfGBUdbmgr.cpp		\
 	@top_srcdir@/SNFMulti/snf_HeaderFinder.cpp	\
 	@top_srcdir@/SNFMulti/snfLOGmgr.cpp		\
 	@top_srcdir@/SNFMulti/SNFMulti.cpp		\
 	@top_srcdir@/SNFMulti/snfNETmgr.cpp		\
 	@top_srcdir@/SNFMulti/snf_sync.cpp		\
 	@top_srcdir@/SNFMulti/snf_xci.cpp		\
 	@top_srcdir@/SNFMulti/snfXCImgr.cpp

 noinst_HEADERS =                    				\
 	@top_srcdir@/SNFMulti/FilterChain.hpp			\
 	@top_srcdir@/SNFMulti/GBUdb.hpp				\
 	@top_srcdir@/SNFMulti/GBUdb.inline.hpp			\
 	@top_srcdir@/SNFMulti/scanner.hpp			\
 	@top_srcdir@/SNFMulti/snfCFGmgr.hpp			\
 	@top_srcdir@/SNFMulti/snfCFGmgr.inline.hpp		\
 	@top_srcdir@/SNFMulti/snf_engine.hpp			\
 	@top_srcdir@/SNFMulti/snfGBUdbmgr.hpp			\
 	@top_srcdir@/SNFMulti/snf_HeaderFinder.hpp		\
 	@top_srcdir@/SNFMulti/snf_HeaderFinder.inline.hpp	\
 	@top_srcdir@/SNFMulti/snfLOGmgr.hpp			\
 	@top_srcdir@/SNFMulti/snfLOGmgr.inline.hpp		\
 	@top_srcdir@/SNFMulti/SNFMulti.hpp			\
 	@top_srcdir@/SNFMulti/snfNETmgr.hpp			\
 	@top_srcdir@/SNFMulti/snf_sync.hpp			\
 	@top_srcdir@/SNFMulti/snf_xci.hpp			\
 	@top_srcdir@/SNFMulti/snfXCImgr.hpp			\
 	@top_srcdir@/SNFMulti/snf_match.h

 EXTRA_DIST = 						\
 	Makefile.am					\
 	ChangeLog

 clean-local:
 	rm -f *.gcno *.gcov *.gcda *~