123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449 |
- SNF MDaemon Plugin Change Log...
- ------------------------------------------------------------------------------
-
- 20080626 - Version 3.0, It's official.
-
- Changed build information.
- Removed extraneous comments from configuration file.
-
- 20080524 - Version V2-9rc6.25.7
-
- Optimized networking library for additional speed & stability by moving
- receive buffer allocation from heap to stack (automatic).
-
- Optimized timing parameters in SNFClient for improved speed. Polling dealys
- are now reduced to 10ms from 30ms.
-
- Removed speed-bug in SNFClient, 100ms guard time between retries was always
- executed after an attempt (even a successful attempt). The guard time is now
- condition and only fires on unsuccessful attempts.
-
- Updated XCI server logic to ensure non-blocking sockets for clients in all
- socket implementations.
-
- 20080424 - Version V2-9rc6.24.6
-
- Refactored snfScanData.clear() to reduce heap work and fragments.
-
- Added mutex to scanMessageFile() entry point just in case some app attempts to
- put multiple threads through a single engine handler. scanMessage() is already
- protected and fully wraped by the new scanMessageFile() mutex.
-
- Added non-specific runtime exception handling to XHDR injection code.
-
- Added 2 retries w/ 300ms delay to remove original message in XHDR inject code.
- If remove fails after 3 attempts the injector throws.
-
- Added 2 retries w/ 300ms delay to rename temp file to msg in XHDR inject code.
- If rename fails after 3 attempts the injector throws.
-
- Added IPTest logging.
-
- 20080416 - Version V2-9rc5.23.6
-
- Fixed bug where SNCY open() would fail on some Win* platforms with
- WSAEINVAL instead of the standard EINPROGRESS or EALREADY which were expected.
- Also added WSAEWOULDBLOCK to cover other "ambiguities" in windows sockets
- implementations. InProgress() on Win* now test for any of:
-
- WSAEINPROGRESS, WSAEALREADY, WSAEWOULDBLOCK, WSAEINVAL
-
- 20080413 - Version V2-9rc5.22.6
-
- Fixed bug in TCPHost.open() where EALREADY was not counted as a version of
- EINPROGRESS. This would cause open() to throw an unnecessary exception when
- an open() required extra time.
-
- 20080413 - Version V2-9rc5.21.6
-
- Extended timeout for SYNC session open() to the full session length. This way
- if a session takes a long time to open it still has a shot at success.
-
- 20080411 - Version V2-9rc5.20.6
-
- Adjusted snfNETmgr to use non-blocking open in SYNC sessions. Open timeout
- is 1/3 of the session timeout. Session timeout is 2 * Session pacing. Open
- polling uses golden spiral delay from 10ms to 340ms.
-
- 20080410 - Version V2-9rc5.19.6
-
- Adjusted XCI manager to use new snfCFGPacket paradigm in checkCFG().
-
- Adjusted snf_RulebaseHandler::addRulePanic() to use MyMutex and eliminated
- the AutoPanicMutex and waiting scheme.
-
- Refactored scanMessage() to use a ScopeMutex() rather than lock()/unlock().
-
- Refactored scanMessage() to use MyCFGPacket.isRulePanic() test.
-
- Redesigned snfCFGPacket handling to automate grab() / drop() functions.
-
- Fixed lock-up bug: Redesigned AutoPanic posting and checking mechanisms to
- eliminate potential dead-lock condition. Under some conditions a precisely
- timed auto-panic posting could cause the RulebaesHandler mutex and the
- AutoPanicMutex to become intertwined leading to a cascading deadlock. When
- this occurred all XCI processing threads and eventually the XCI listener
- thread would become blocked waiting to get the current configuration.
-
- 20080409 - Version V2-9rc5.18.6
-
- Enhanced XCI exception handling and logging to provide additional detail.
-
- Added code to explicitely check for zero length files in scanMessagFile().
- Previously a zero length file would cause the CBFR module of the filter
- chain to throw an invalid buffer exception. Now if the message file is empty
- scanMessageFile() will throw a FileError stating FileEmpty!.
-
- 20080407 - Version V2-9rc5.17.6
-
- Enhanced exception reporting in snfXCImrg
-
- 20080405 - Version V2-9rc5.16.6
-
- Reduced safetly limits on status reports to 100K for status reports and 100K
- for samples. Previous values were 10M. Most full sessions from the busiest
- systems are < 50K total.
-
- Recoded sendDataTimeout() to break uploads into 512 byte chunks and insert
- delays only when a chunk is fragmented. This methodology improves reliability
- on Win* systems without any significant penalty on systems that don't need
- socket sends() to be in smaller chunks.
-
- Fixed TCPClient::transmit() and TCPHost::transmit() bug where returned byte
- count might be -1. Now returned byte counts can only be 0 or more.
-
- 20080403 - Version SNF2-9vr5.15.5
-
- Minor modifications to networking module to better support non-blocking open()
-
- Updated SNFClient with new timing and non-blocking open(). Worst case return
- time from SNFClient estimated at 200 seconds (theoretically impossible). No-
- connection return time from SNFClient estimated at 20 seconds.
-
- 20080326 - Version SNF2-9rc4.15.4
-
- Refactored snfNETmgr::sync() to consolidate non-blocking io routines.
-
- Added detailed thread status data to XCI listener thread.
-
- Refactored snfNETmgr::sync() to check a Timeout, removed TCPWatchdog.
-
- 20080325 - Version SNF2-9rc4.12.4
-
- Added a "Rulebase Getter" feature as part of the snf_Reloader. When enabled
- the Rulebase Getter will launch a user defineable system() call whenever a
- new rulebase file is available. The call will be repeated until the condition
- is cleared by a successful update of the rulebase file. The Rulebase Getter
- will wait a configurable "guard time" between attempts. The default system()
- call is "getRulebase" with a guard time of 3 minutes. In most cases this will
- launch the provided getRulebase script which should be present in the start
- location of SNFServer on most systems. Best practice is to configure the full
- path to the update script. The system() call is made in a separate thread so
- that if the system() call hangs for some reason only the Rulebase Getter is
- stuck.
-
- Improved exception handling/reporting in scanMessageFile().
-
- Updated scanMessagFile() header injection code to accommodate messages with
- no body. Previous version would throw an exception when it could not find an
- injection point. The new version makes the injection point byte 0 and puts
- the injected headers at the top of the message using it's best guess about the
- type of line endings (CRLF or LF) to use.
-
- Updated Networking library to use SO_REUSEADDR by default on listeners.
-
- 20080319 - Version SNF2-9rc4.11
-
- Added IPScan on-off to snfmdplugin.xml. This allows users to turn off the
- IPScan feature without editing the Plugins.dat file as was previously
- required. The feature can now be enabled or disabled at will by editing the
- configuration file.
-
- Added Configuration editor options to snfmdplugin.xml. Previously the built-
- in configuration function was hard coded to start notepad with the config
- file. Now the system() call made by the ConfigFunc() can be edited in the
- configuration file. The configuration file name can be appended to the
- command optionally. The default is still to start notepad and append the
- configuration file path so that it is loaded automatically. It is hoped that
- GUI based configuration editors for the SNF plugin will be built by third
- parties and in the mean time folks can now configure their favorite XML file
- editor to modify their SNF plugin configuration.
-
- Modified API use fixed shutdown bug - The plugin used to initialize the SNF
- scanning engine when the DLL was loaded and would shut it down when the DLL
- was unloaded. Now the Startup and Shutdown functions in the MDaemon plugin
- API. This ensures that the engine components are started and shutdown in the
- proper sequence.
-
- Included new SNFEngine core (excerpts from that change log included).
-
- 20080318 - SNF2-9rc1.11.exe Consolidated several mods/fixes
-
- Corrected scan error logging bug. Was posting <s/> now posts <e/>.
-
- Updated scan error logging to be more uniform with non-scan errors.
-
- Developed various script prototypes for postfix integration & automated
- updates on win* systems using the new UpdateReady.txt file mechanism.
-
- Fixed a bug in scanMessageFile() where an \n\n style insertion point
- would never be detected.
-
- Modified scanMessageFile() header injection to strip <CR> from line ends
- when the message file provided does not use them. The line-end style of
- the message file is detected while locating the insertion point. If the
- insertion point (first blank line) does not use <CR><LF> then the SNF
- generated X-Headers are stripped of <CR> in a tight loop before injection.
-
- Enhanced error and exception reporting in SNFMulti.cpp scanMessageFile().
-
- Enhanced exception handling in networking module. All exceptions now
- throw descriptive runtime_error exceptions.
-
- 20080306 - SNF2-9rc1.8.exe (FIRST RELEASE CANDIDATE for VERSION 3!)
-
- Added Drilldown Header Directive Functions - When the candidate source IP
- comes from a header matching a drilldown directive the IP is marked "Ignore"
- in GBUdb and the candidate is no longer eligible to be the source for that
- message. This allows SNF to follow the trusted chain of devices (by IP) down
- to the actual source of the message. It is handy for ignoring net blocks
- because it can match partial IPs but it is designed to allow SNF to learn
- it's way through the servers at large ISPs so that the original source for
- each message can be evaluated directly.
-
- Added Source Header Directive Functions - This feature allows SNF to acquire
- the source IP for a message from a specific header rather than searching
- through the Received headers in the message. This is useful when the original
- source for a message is not represented in Received headers. For example:
- Hotmail places the originating source IP in a special header and does not
- provide a Received header for that IP. This feature is protected from abuse
- by a "Context" feature which only activates the source header directive when
- specific content is found in a specific received header. Using the above
- example, this feature can be configured so that a Hotmail source header would
- only be read if the top Recieved header contained "hotmail.com [" indicating
- that the ptr lookup for the header matched the hotmail domain. Note: When a
- source is pulled from a header directive that source is put into a synthetic
- Received header and injected into the scanning stream (not the message) as
- the first Received header.
-
- Added forced source IP to XCI - It is now possible to "inject" or "force"
- the source IP for any message by providing that IP in the XCI request or
- directly in a scan...() function call. This allows the calling application
- to provide the source IP for a message ahead of any Received headers that
- might be in the message. This is useful when the calling application knows
- the original source IP for the message but that IP is not represented in
- the Received headers and it is not desireable to use the Source Header
- Directive mechanism.
-
- Added forced source IP mode to SNFClient - It is now possible to call the
- SNFClient utility with an IP4Address using the syntax:
-
- SNFClient -source=12.34.56.78
-
- The -source mode of SNFClient exercises the forced source IP feature in
- the XCI (see above)
-
- Added Status Report features to SNFClient and XCI - It is now possible to
- request the latest status.second, status.minute, or status.hour data via
- the XCI and SNFClient. The syntax for requesting a status report using the
- SNFClient is:
-
- SNFClient -status.second
- SNFClient -status.minute
- SNFClient -status.hour
-
- In addition to providing status reports the SNFClient in this mode will
- return a nonzero value (usually 99) if it is unable to get a status report
- from SNFServer. This feature can be used to verify that SNFServer is up
- and responding. If SNFServer is OK then the result code returned is 0.
-
- Added result codes to SNFClient -test and XCI IP test functions - The XCI
- engine has been upgraded to provide the range value for the IP under test
- as well as the symbolic result code associated with that range. This allows
- the -test function to provide results that are consistent with the GBUdb
- configuration without additional processing: For example, if the IP falls
- in the Caution range then the Caution result code will be returned just
- as if a message had been scanned with the same IP and no pattern match
- occurred. The same is true for Truncate and Black range hits.
-
- Added Timestamp and Command Line Parameter data to SNFClient.exe.err - When
- an error occurs with SNFClient that may not appear in the SNFServer logs an
- entry is appended to the SNFClient.exe.err file. That in itself is not new.
- The new feature is that the entries added to the SNFClient.exe.err file now
- include timestamp and command line data to aid in debugging.
-
- Updated the Configuration Log to include all of the current configuration
- features and to improve it's readability.
-
- 20080207 - SNF2-9b1.7.exe
-
- SYNC Timeout now 2x SYNC Schedule
-
- SNFServer now produces an UpdateReady.txt file when the UTC timestamp on
- the SYNC server is newer than the UTC timestamp of the active rulebase. It
- is presumed that a suitable update script or program will run periodically
- and download a fresh rulebase file if the UpdateReady.txt file is present.
- The update script should remove the UpdateReady.txt file when it completes
- a successful download of the new rulebase file.
-
- Added available rulebase UTC in status reports <udate utc.../>
-
- Added Automatic path fixup for ending / or \
-
- Added option to use local time in log rotation <rotation localtime='no'/>
- The default is still utc.
-
- 20071102 - SNF2-9b1.6.exe
-
- Increased MAX_EVALS from 1024 to 2048.
-
- Adjusted defult range envelopes in snf_engine.xml to be more conservative.
-
-
- 20071017 - Version SNF2-9b1.5
-
- Added a missing #include directive to the networking.hpp file. The
- missing #include was not a factor on Linux and Windows systems but
- caused compiler errors on BSD systems.
-
- Corrected a bug in the GBUdb White Range code where any message with a
- white range source IP was being forced to the white result code. The
- engine now (correctly) only forces the result and records the event when
- a black pattern rule was matched and the White Range IP causes that
- scan result to be overturned. If the scan result was not a black pattern
- match then the original scan result is allowed to pass through.
-
- Corrected a bug in the Header Analysis filter chain module that would
- cause the first header in the message to be ignored in some cases.
-
- Corrected an XML log format problem so that <s/> elements are correctly
- open ended <s ....> or closed (empty) <s..../> according to whether they
- have subordinate elements.
-
- Adjusted the GBUdb header info format. The order of the Confidence
- figure and Probabilty figure is now the same as in the XML log files
- (C then P). The confidence and probability figures are now preceeded
- with c= and p= respectively so that it's easy to tell which is which.
-
- 20071009 Version 2-9b1.4
-
- Tightened up the XCI handler code and removed the watchdog. The watchdog
- would restart the listener if there were no connections in 5 minutes. It
- was originally added to provide additional stability, however in practice
- there have been no "stalled listeners". Also, a stalled listener would
- likely be a sign of a different problem that the watchdog would tend to
- hide.
-
- Modified and refactored the XCI configuration management code. All XCI config
- changes and up-down operations are now handled in a single function except
- upon exit from the main XCI thread where XCI_shutdown() is always called.
-
- Added some more detailed exception handling code to the XCI component so that
- more data will be logged in the event of an error.
-
- Reviewed and modified the InstallInstructions.txt file. Removed this log
- to this separate file.
-
- Modified the snfmdplugin.xml file to properly configure the new features in
- the engine.
-
- * Header training directives and new <training/> section.
-
- * XCI interface configuration.
-
- * Tweaks to GBUdb ranges.
-
- * msg-file type configuration (not used in MDaemon, but configured anyway)
-
- ----
-
- Version 2-9a11 (engine a53)
-
- * Enhanced IP extraction from Received headers so that any unexpected bytes
- between the [ and ] will force the attempt to be aborted.
-
- * Fix the IP test code so that the IP 0.0.0.0 cannot be the source IP and
- cannot be tested.
-
- Version 2-9a11 (engine a52)
-
- * Corrected plug-in log entry logic. Allowed/Rejected tag now comes directly
- from the message rejection logic and is accurate in all cases.
-
- Version 2-9a10 (engine a52)
-
- * Corrected a bug in the MessageIPFunc where Ignore flagged IPs would still
- cause rejected messages if the statistics were in the Truncate range. Now
- messages are rejected in only two cases:
-
- The Flag is _Ugly_ and the rating is _Truncate_ or, the Flag is _Bad_.
-
- Version 2-9a9 (engine a52)
-
- * Adjusted IPtest module in HeaderAnalysis to handle TooManyIPs exception
- locally and silently.
-
- * Increased HeaderAnalysis IP limit from 20 to 50.
-
- Version 2-9a9 (engine a51)
-
- * Corrected possible heap corruption bug in EvaulationMatrix Destructors.
-
- * Added trace strings to scanMessage() for tighter panic reporting.
-
- * Added caching to snf_engine Evaluator allocation scheme.
-
- * Added optimizations to snf_engine Evaluator safety checks.
-
- Version 2-9a8
-
- * Added deep exception handling to Token Matrix objects.
-
- Version 2-9a7
-
- * Exception handling throughout the engine has been refactored to use std:exception
- and to provide additional detail via e.what()
-
- * The plug-in log will now show e.what() data as SNF Debug: whenever an exception
- is thrown during a message scan.
-
- Version 2-9a6
-
- * Adjusted .ctl file path converter to accept either .msg or .tmp paths.
-
- Version 2-9a5
-
- A lot of new things were learned, updated, and corrected.
-
- * Fixed the "lockup" when the plugin failed to start successfully. The cause of this
- appears to be a threading issue associated with DLLs that are being initialized.
- If threads are created during the initialization of a DLL, the DLL must succeed!
- The threads that are created do not get any cycles until after the DLL is loaded
- successfully. As a result, if the initialization process attempts to join() these
- threads a deadlock is created. The fix was to allow the SNF plugin initialization
- process to succeed in all cases while setting a flag that forces the engine to
- be inert if the initialization was not successful. When the DLL is later unloaded
- the threads are already running so the join() calls that are part of the engine
- cleanup code are able to complete without incident.
-
- * Installed detailed exception handling for the start-up sequence. The plugin can
- now report on very specific reasons for failing to initialize properly.
-
- * Fixed a bug in the GBUdbIgnoreList processor where long lines would cause the
- remainder of the file not to be read. The line length limit still exists, but
- it is now 255 characters which is unlikely to occur and would be considered
- incorrect formatting.
-
- * The threading library now includes top-level exception handling to trap any
- exception that was not handled by myTask(). Along with this two flags were
- added to thread objects: isRunning() and isBad(). isRunning() is true when a
- thread object is still active. isBad() is true if the thread failed to start or
- an exception escaped myTask().
-
- * At least one GBUdbIgnoreList entry is now REQUIRED. If the count of IPs from the
- GBUdbIgnoreList.txt file is less than 1 (or the file is missing) then the plug-in
- will complain and fail to start.
-
- * snf2check.exe has been removed from the distribution for the time being since it
- causes some systems to strip the attachment or block the email. This is the same
- program that is already on existing SNF systems.
|