SIEM Best Practices: Evaluation Criteria

SIEM Best Practices: Evaluation Criteria


Decurity often has the opportunity to our customers find the right Log Management and/or SIEM solution. We are honored that our customers trust us with that very important question so we wanted to take a moment and explain our requirements gathering/documentation process for vendor selection and hope that our explanation helps a few of more folks out there! We also get asked by Vendors on how they can improve their products, but that’s a entirely different blog post.

In March of 2008 I authored a couple of posts related to SIEM pre-requisites:

SIEM Best Practices: Very Basic SIEM Implementation Success Criteria and
SIEM Best Practices: Before you buy.

In those posts I tried to create a baseline of information customers looking to purchase and implement a SIEM solution should have before engaging the vendors. The point of those posts really boiled down to this idea: You must have a strong set requirements defined up front for the vendors to indicate how they meet that requirement. Allowing the vendors to "work their magic" and define your problems is roughly equivalent to handing them a blank check. Along those lines I wanted to highlight a strategy we employ when helping company’s to define their SIEM requirements by presenting a sample of the categories of questions we ask of the customer and the vendors during evaluation process.



A couple of quick notes before we begin with the listings:
1.) All of this assumes you’ve answered the initial “Key Problems we are trying to solve” question and the answer is something more tangible than to meet PCI,SOX, Audit requirements.
1a.) If events per second (EPS) is your key measurement you are looking at the wrong product set - seek out Log Management Tools first.
2.) It is also important to note that when we perform the evaluation each of these stated technical requirement categories breaks down into a dozen or more actual testing criteria that is prioritized according to your requirements The "Sample Questions" presented are only a very quick overview of the types of questions that fall into that category.
3.) This post is simply highlighting the fact that significant thought should be given to this decision. Don’t worry if you need help – we’re here.


Sample Categories of Requirements to consider:
Common Requirement Categories
Category (Sample Questions)
Access Control (Application, User, flexibility, inherited controls, etc)
Authentication (LDAP, SSO, AD, Internal, other)
Architecture (Reliable and Scalable)
Event Sources (Supported Technologies and versions, Connection Methods for each, Data Parsing Errors, Normalization Data Loss, Categorization Correctness, Structured/Unstructured Data Handling)
Log Management (Is the Integration Bi-Directional, easy to implement, etc)
Event Forwarding (Security, Methods, Low-Bandwidth options, etc)
Overall Security (System and the data)
External Integrations: (Tool Integration, Ticketing System Integrations, etc)
Storage Requirements (Compression, Costs, Management)
Storage Flexibility (NAS, SAN, Internal, Offline/Online, Tiered Storage)
Data Processing (Internally how does the system handle new event sources with uncommon field requirements "unstructured data")
Installation (Does the solution match our standards?)
Patching and Upgrade (Level of effort required for Minor and Major Versions)
Overall User Experience (Can I see what is important quickly and easily? Can I drill down quickly and intuitively?)
Standard Reporting (Easy, Flexible, Exportable)


Advanced Usage Requirement Categories
Category (Sample Questions)
Basic Alerting Criteria (Pattern Matching or Aggregation/Counting)
Basic Correlation (IF ,THEN, ELSE, AND, NOT, OR type Statements)
Advanced Correlation (Meta Analysis of enriched and/or raw data across technologies, time and result sets in real time.)
Statistical Analysis (Flexible event statistics that can be used in alerting or to enrich data sets for correlation)
Custom Reporting (Can I create my favorite report or extend it)
Data Mining (Can I easily look for patterns across the entire DB?)
Data Visualization (Can data viz be integrated and does it matter for me?)
Vulnerability Integration (Is the correlation useful for our environment and is the reporting useful?)
Network Modeling (How hard is to model our environment and what value is lost/gained?)
Asset Modeling (Can I easily assign systems to relevant categories and assign priorities, can I update them easily, etc)
User/Activity Modeling (Can we realistically “profile” users or activities and alert on deviations?)
External Threat Feeds (Does the vendor or a partner provide daily updates for Hotlists?)
Built in Mgmt Tools (Does the vendor provide a way of measuring the health of the system?)


Other Important Criterion
Category (Sample Questions)
Company Performance (This is becoming more and more a key decision factor.)
Support (What can I escalate, response times, expertise, RMA)
Thought Leadership (What is the vision for the technology?)
Training (Do I need 4 weeks of training to use the product? If so how many types of training opportunities are available?)
Services Support (Do I need 12 weeks of Services? How can I guarantee I don’t get the new guy? Is the team compensated on billability or Customer Success?)
Content Updates (How often can I receive content updates? Do I need constant "workshops" to move forward, Are there external providers that can help?)
Licensing Model (Price can be greatly affected by various pricing models, make sure you understand the total cost of all phases of your deployment before you begin).

This post is a mirror of my personal post on http://blog.decurity.com
http://blog.decurity.com/index.php/dec_template/more/siem_best_practices_evaluation_criteria/