Updated Makefile.am: Added mangler.cpp and mangler.hpp, removed

tcp_watchdog.cpp and tcp_watchdog.hpp, added faults.hpp.  Added
ChangeLog for logging changes.


git-svn-id: https://svn.microneil.com/svn/CodeDweller/trunk@9 d34b734f-a00e-4b39-a726-e4eeb87269ab

ChangeLog

@@ -0,0 +1,381 @@
+2009-05-23  Alban Deniz  <adeniz@skidmark.localdomain>
+
+	* CodeDweller/Makefile.am (noinst_HEADERS): Added faults.hpp and
+	mangler.hpp.
+	(libCodeDweller_a_SOURCES): Added mangler.cpp.
+
+SNF Command Line & SNFMulti Engine / Client Change Log
+------------------------------------------------------------------------------
+
+20080710 - Version 3.0.1
+
+Minor change to SNFServer main.cpp:59 - removed cast to (int) which caused
+a precision loss error when compiling on 64 bit systems. This changes the
+thread pointer info in debug mode slightly (better).
+
+20080626 - Version 3.0, It's official.
+
+Changed build information.
+Removed extraneous comments from configuration file.
+
+20080524 - Version V2-9rc2.25.7
+
+Optimized networking library for additional speed & stability by moving
+receive buffer allocation from heap to stack (automatic).
+
+Optimized timing parameters in SNFClient for improved speed. Polling dealys
+are now reduced to 10ms from 30ms.
+
+Removed speed-bug in SNFClient, 100ms guard time between retries was always
+executed after an attempt (even a successful attempt). The guard time is now
+condition and only fires on unsuccessful attempts.
+
+Updated XCI server logic to ensure non-blocking sockets for clients in all
+socket implementations.
+
+20080424 - Version V2-9rc2.24.6
+
+Refactored snfScanData.clear() to reduce heap work and fragments.
+
+Added mutex to scanMessageFile() entry point just in case some app attempts to
+put multiple threads through a single engine handler. scanMessage() is already
+protected and fully wraped by the new scanMessageFile() mutex.
+
+Added non-specific runtime exception handling to XHDR injection code.
+
+Added 2 retries w/ 300ms delay to remove original message in XHDR inject code.
+If remove fails after 3 attempts the injector throws.
+
+Added 2 retries w/ 300ms delay to rename temp file to msg in XHDR inject code.
+If rename fails after 3 attempts the injector throws.
+
+20080416 - Version V2-9rc2.23.6
+
+Fixed bug where SNCY open() would fail on some Win* platforms with
+WSAEINVAL instead of the standard EINPROGRESS or EALREADY which were expected.
+Also added WSAEWOULDBLOCK to cover other "ambiguities" in windows sockets
+implementations. InProgress() on Win* now test for any of:
+
+WSAEINPROGRESS, WSAEALREADY, WSAEWOULDBLOCK, WSAEINVAL
+
+20080413 - Version V2-9rc2.22.6
+
+Fixed bug in TCPHost.open() where EALREADY was not counted as a version of
+EINPROGRESS. This would cause open() to throw an unnecessary exception when
+an open() required extra time.
+
+20080413 - Version V2-9rc2.21.6
+
+Extended timeout for SYNC session open() to the full session length. This way
+if a session takes a long time to open it still has a shot at success.
+
+20080411 - Version V2-9rc2.20.6
+
+Adjusted snfNETmgr to use non-blocking open in SYNC sessions. Open timeout
+is 1/3 of the session timeout. Session timeout is 2 * Session pacing. Open
+polling uses golden spiral delay from 10ms to 340ms.
+
+20080410 - Version V2-9rc2.19.6
+
+Adjusted XCI manager to use new snfCFGPacket paradigm in checkCFG().
+
+Adjusted snf_RulebaseHandler::addRulePanic() to use MyMutex and eliminated
+the AutoPanicMutex and waiting scheme.
+
+Refactored scanMessage() to use a ScopeMutex() rather than lock()/unlock().
+
+Refactored scanMessage() to use MyCFGPacket.isRulePanic() test.
+
+Redesigned snfCFGPacket handling to automate grab() / drop() functions.
+
+Fixed lock-up bug: Redesigned AutoPanic posting and checking mechanisms to 
+eliminate potential dead-lock condition. Under some conditions a precisely
+timed auto-panic posting could cause the RulebaseHandler mutex and the 
+AutoPanicMutex to become intertwined leading to a cascading deadlock. When
+this occurred all XCI processing threads and eventually the XCI listener
+thread would become blocked waiting to get the current configuration.
+
+20080409 - Version V2-9rc2.18.6
+
+Enhanced XCI exception handling and logging to provide additional detail.
+
+Added code to explicitely check for zero length files in scanMessagFile().
+Previously a zero length file would cause the CBFR module of the filter
+chain to throw an invalid buffer exception. Now if the message file is empty
+scanMessageFile() will throw a FileError stating FileEmpty!.
+
+20080407 - Version V2-9rc2.17.6
+
+Enhanced exception reporting in snfXCImrg
+
+
+20080405 - SNFServer V2-9rc2.16.6
+
+Reduced safetly limits on status reports to 100K for status reports and 100K
+for samples. Previous values were 10M. Most full sessions from the busiest
+systems are < 50K total.
+
+Recoded sendDataTimeout() to break uploads into 512 byte chunks and insert
+delays only when a chunk is fragmented. This methodology improves reliability
+on Win* systems without any significant penalty on systems that don't need
+socket sends() to be in smaller chunks.
+
+Fixed TCPClient::transmit() and TCPHost::transmit() bug where returned byte
+count might be -1. Now returned byte counts can only be 0 or more.
+
+20080403 - SNFServer V2-9rc2.15.5
+
+Minor modifications to networking module to better support non-blocking open()
+
+Updated SNFClient with new timing and non-blocking open(). Worst case return
+time from SNFClient estimated at 200 seconds (theoretically impossible). No-
+connection return time from SNFClient estimated at 20 seconds.
+
+20080326 - SNFServer V2-9rc2.15.4
+
+Refactored snfNETmgr::sync() to consolidate non-blocking io routines.
+
+Added detailed thread status data to XCI listener thread.
+
+Fixed minor bug in main (not changing revision), Debug flag for internal use
+was left on in the last build cycle. It is commented out now.
+
+20080325 - SNFServer V2-9rc2.14.4
+
+Updated snfNETmgr with comprehensive thread status data.
+
+Refactored snfNETmgr::sync() to check a Timeout, removed TCPWatchdog.
+
+20080325 - SNFServer V2-9rc2.13.4
+
+Upgraded TCPWatcher code to use new threading features (type, status).
+
+20080324 - SNFServer v2-9rc2.12.4
+
+Added a "Rulebase Getter" feature as part of the snf_Reloader. When enabled
+the Rulebase Getter will launch a user defineable system() call whenever a
+new rulebase file is available. The call will be repeated until the condition
+is cleared by a successful update of the rulebase file. The Rulebase Getter
+will wait a configurable "guard time" between attempts. The default system()
+call is "getRulebase" with a guard time of 3 minutes. In most cases this will
+launch the provided getRulebase script which should be present in the start
+location of SNFServer on most systems. Best practice is to configure the full
+path to the update script. The system() call is made in a separate thread so
+that if the system() call hangs for some reason only the Rulebase Getter is
+stuck.
+
+Built thread monitoring function for SNFServer.exe (Full status report / sec).
+The thread monitoring report is turned on when the program is renamed to
+SNFDebugServer.exe or if "debug" appears in the file path to the program.
+
+Refactored XCI channels to leverage new thread monitoring.
+
+Refactored Threading to eliminate inline code.
+
+Improved exception handling/reporting in scanMessageFile().
+
+Updated scanMessagFile() header injection code to accommodate messages with
+no body. Previous version would throw an exception when it could not find an
+injection point. The new version makes the injection point byte 0 and puts
+the injected headers at the top of the message using it's best guess about the
+type of line endings (CRLF or LF) to use.
+
+Updated Threading library to include high level thread state tracking and
+naming. Also creates a global Threads object that can produce a real-time
+status report on all threads.
+
+Updated Networking library to use SO_REUSEADDR by default on listeners.
+
+20080318 - SNF2-9rc1.11.exe Consolidated several mods/fixes
+
+Corrected scan error logging bug. Was posting <s/> now posts <e/>.
+
+Updated scan error logging to be more uniform with non-scan errors.
+
+Developed various script prototypes for postfix integration & automated
+updates on win* systems using the new UpdateReady.txt file mechanism.
+
+Fixed a bug in scanMessageFile() where an \n\n style insertion point
+would never be detected.
+
+Modified scanMessageFile() header injection to strip <CR> from line ends
+when the message file provided does not use them. The line-end style of
+the message file is detected while locating the insertion point. If the
+insertion point (first blank line) does not use <CR><LF> then the SNF
+generated X-Headers are stripped of <CR> in a tight loop before injection.
+
+Enhanced error and exception reporting in SNFMulti.cpp scanMessageFile().
+
+Enhanced exception handling in networking module. All exceptions now
+throw descriptive runtime_error exceptions.
+
+20080306 - SNF2-9rc1.8.exe (FIRST RELEASE CANDIDATE for VERSION 3!)
+
+Added Drilldown Header Directive Functions - When the candidate source IP
+comes from a header matching a drilldown directive the IP is marked "Ignore"
+in GBUdb and the candidate is no longer eligible to be the source for that
+message. This allows SNF to follow the trusted chain of devices (by IP) down
+to the actual source of the message. It is handy for ignoring net blocks
+because it can match partial IPs but it is designed to allow SNF to learn
+it's way through the servers at large ISPs so that the original source for
+each message can be evaluated directly.
+
+Added Source Header Directive Functions - This feature allows SNF to acquire
+the source IP for a message from a specific header rather than searching
+through the Received headers in the message. This is useful when the original
+source for a message is not represented in Received headers. For example:
+Hotmail places the originating source IP in a special header and does not
+provide a Received header for that IP. This feature is protected from abuse
+by a "Context" feature which only activates the source header directive when
+specific content is found in a specific received header. Using the above
+example, this feature can be configured so that a Hotmail source header would
+only be read if the top Recieved header contained "hotmail.com [" indicating
+that the ptr lookup for the header matched the hotmail domain. Note: When a
+source is pulled from a header directive that source is put into a synthetic
+Received header and injected into the scanning stream (not the message) as
+the first Received header.
+
+Added forced source IP to XCI - It is now possible to "inject" or "force"
+the source IP for any message by providing that IP in the XCI request or
+directly in a scan...() function call. This allows the calling application
+to provide the source IP for a message ahead of any Received headers that
+might be in the message. This is useful when the calling application knows
+the original source IP for the message but that IP is not represented in
+the Received headers and it is not desireable to use the Source Header
+Directive mechanism.
+
+Added forced source IP mode to SNFClient - It is now possible to call the
+SNFClient utility with an IP4Address using the syntax:
+
+SNFClient -source=12.34.56.78
+
+The -source mode of SNFClient exercises the forced source IP feature in
+the XCI (see above)
+
+Added Status Report features to SNFClient and XCI - It is now possible to
+request the latest status.second, status.minute, or status.hour data via
+the XCI and SNFClient. The syntax for requesting a status report using the
+SNFClient is:
+
+SNFClient -status.second
+SNFClient -status.minute
+SNFClient -status.hour
+
+In addition to providing status reports the SNFClient in this mode will
+return a nonzero value (usually 99) if it is unable to get a status report
+from SNFServer. This feature can be used to verify that SNFServer is up
+and responding. If SNFServer is OK then the result code returned is 0.
+
+Added result codes to SNFClient -test and XCI IP test functions - The XCI
+engine has been upgraded to provide the range value for the IP under test
+as well as the symbolic result code associated with that range. This allows
+the -test function to provide results that are consistent with the GBUdb
+configuration without additional processing: For example, if the IP falls
+in the Caution range then the Caution result code will be returned just
+as if a message had been scanned with the same IP and no pattern match
+occurred. The same is true for Truncate and Black range hits.
+
+Added Timestamp and Command Line Parameter data to SNFClient.exe.err - When
+an error occurs with SNFClient that may not appear in the SNFServer logs an
+entry is appended to the SNFClient.exe.err file. That in itself is not new.
+The new feature is that the entries added to the SNFClient.exe.err file now
+include timestamp and command line data to aid in debugging.
+
+Added BIG-ENDIAN Conversion - When the SNFServer program is compiled on a
+system that uses a BIG-ENDIAN processor (such as a power-mac) the rulebase
+load process now includes a routine to convert the token matrix from it's
+native LITTLE-ENDIAN format to a BIG-ENDIAN format. This solves a bug where
+Power-Mac (and presumably other BIG-ENDIAN systems) could compile and run
+the SNF* software but were unable to capture spam because the token matrix
+in the rulebase file was misinterpreted.
+
+Note: The BIG-ENDIAN Conversion feature is still considered experimental
+because it has not yet been thoroughly tested.
+
+Updated the Configuration Log to include all of the current configuration
+features and to improve it's readability.
+
+
+20080207 - SNF2-9b1.7.exe
+
+SYNC Timeout now 2x SYNC Schedule
+
+SNFServer now produces an UpdateReady.txt file when the UTC timestamp on
+the SYNC server is newer than the UTC timestamp of the active rulebase. It
+is presumed that a suitable update script or program will run periodically
+and download a fresh rulebase file if the UpdateReady.txt file is present.
+The update script should remove the UpdateReady.txt file when it completes
+a successful download of the new rulebase file.
+
+Added available rulebase UTC in status reports <udate utc.../>
+
+Added Automatic path fixup for ending / or \
+
+Added option to use local time in log rotation <rotation localtime='no'/>
+The default is still utc.
+
+20071102 - SNF2-9b1.6.exe
+
+Increased MAX_EVALS from 1024 to 2048.
+
+Adjusted defult range envelopes in snf_engine.xml to be more conservative.
+
+20071017 - SNF2-9b1.5.exe
+
+Added a missing #include directive to the networking.hpp file. The
+missing #include was not a factor on Linux and Windows systems but
+caused compiler errors on BSD systems.
+
+Corrected a bug in the GBUdb White Range code where any message with a
+white range source IP was being forced to the white result code. The
+engine now (correctly) only forces the result and records the event when
+a black pattern rule was matched and the White Range IP causes that
+scan result to be overturned. If the scan result was not a black pattern
+match then the original scan result is allowed to pass through.
+
+Corrected a bug in the Header Analysis filter chain module that would
+cause the first header in the message to be ignored in some cases.
+
+Corrected an XML log format problem so that <s/> elements are correctly
+open ended <s ....> or closed (empty) <s..../> according to whether they
+have subordinate elements.
+
+Adjusted the GBUdb header info format. The order of the Confidence
+figure and Probabilty figure is now the same as in the XML log files
+(C then P). The confidence and probability figures are now preceeded
+with c= and p= respectively so that it's easy to tell which is which.
+
+20071009 - SNF2-9b1.4.exe
+
+Tightened up the XCI handler code and removed the watchdog. The watchdog
+would restart the listener if there were no connections in 5 minutes. It
+was originally added to provide additional stability, however in practice
+there have been no "stalled listeners". Also, a stalled listener would
+likely be a sign of a different problem that the watchdog would tend to
+hide.
+
+Modified and refactored the XCI configuration management code. All XCI config
+changes and up-down operations are now handled in a single function except
+upon exit from the main XCI thread where XCI_shutdown() is always called.
+
+Added some more detailed exception handling code to the XCI component so that
+more data will be logged in the event of an error.
+
+
+20071008 - SNF2-9b1.2.exe
+
+Added support for passing Communigate Message Files directly. Communigate adds
+data to the top of the message file. That data stops at the first blank line and
+the rfc822 message begins. The SNFServer engine can now be told to ignore this
+extra data using the following option:
+
+<msg-file type='cgp'/> <!-- type='cgp'  for communigate message files -->
+
+If the msg-file type is anything other than 'cgp' then it will treat the message
+file as a standard rfc822 message in the usual way. The default setting is
+
+<msg-file type='rfc822'/>
+
+
+
+

Makefile.am

@@ -18,6 +18,7 @@ libCodeDweller_a_SOURCES =        			\
 	@top_srcdir@/CodeDweller/configuration.cpp	\
 	@top_srcdir@/CodeDweller/networking.cpp		\
 	@top_srcdir@/CodeDweller/threading.cpp		\
+	@top_srcdir@/CodeDweller/mangler.cpp		\
 	@top_srcdir@/CodeDweller/timing.cpp
 
 noinst_HEADERS =                    				\
@@ -28,6 +29,8 @@ noinst_HEADERS =                    				\
 	@top_srcdir@/CodeDweller/networking.hpp			\
 	@top_srcdir@/CodeDweller/networking.inline.hpp		\
 	@top_srcdir@/CodeDweller/threading.hpp			\
+	@top_srcdir@/CodeDweller/faults.hpp			\
+	@top_srcdir@/CodeDweller/mangler.hpp			\
 	@top_srcdir@/CodeDweller/timing.hpp
 
 clean-local: