Topics > Intrusion Detection Systems

IT Security











Introduction Limitations to NIDS Things to Consider When Choosing a NIDS Host-Based Intrustion Detection Conclusion

Introduction to Network-Based Intrusion Detection Systems (NIDS) provides a helpful definition of an intrusion detection system (IDS). "An intrusion detection system (IDS) inspects all inbound and outbound network activity and identifies suspicious patterns that may indicate a network or system attack from someone attempting to break into or compromise a system."


Some IDSs function from a dedicated (black box) appliance, meaning that there is no need for the customer to load the operating system, install the application software, and harden the operating system separately. Others are software based and have to be installed on top of a supported platform and operating system.



IDSs generally can be broken into two components: the sensor and the console. The sensor sits upon the network and acts as a sniffer, listening to network traffic in promiscuous mode. The console is the point of central management for an IDS system. By using the console, an administrator may take notice of any current attack alerts. In many cases, the console may be used to customize certain preferences for the IDS.


Signature Detection or Anomaly Detection

Most IDSs function by means of a built-in attack signatures database. If the IDS detects a match between current network activity and an attack in the signatures database, the IDS will document the attempted attack in a log. In many cases the IDS sensor will also send an alert to the console regarding the attack. Other IDSs function based upon anomaly detection. This approach is more statistical, because the IDS compares all network traffic to whatever is considered a "normal" load for a particular network. The IDS analyzes packet sizes, protocols, and traffic load in this comparison process. Therefore, if a particular transaction is atypical to a certain predefined extent, it is designated an attack by the IDS system.


Placement on the Network

IDS can be set up either inside or outside of a firewall, depending on the needs of an organization. An external IDS monitors attacks that occur on a firewall that are not allowed into a network; therefore potential attacks are discovered, but internal threats go undetected. Internal IDS configurations do not see attacks that are repelled by the firewall, but monitor attacks that penetrate the firewall as well as internal attacks.


Network-Based and Host-Based

There are two types of IDS. One is network-based and real-time, while the other is host-based and examines log file data. Network-based systems examine all traffic on a system, while host-based systems examine traffic only on that specific system. This page is primarily devoted to network based intrusion detection systems (NIDS), however, there is a brief overview of Host-based IDSs at the end of this article.



When searching for a NIDS, one of the first aspects to consider it the type of attacks detected by the IDS. It is not sufficient to merely rely on the number of attack signatures in the database. It is better to ensure that the particular IDS has signatures for a wide variety of attack types, such as buffer overflows, stealth port scans, CGI attacks, SMB probes, NMAP probes, fragment attacks, and OS fingerprinting attempts. An effective IDS should also perform protocol analysis, detecting protocols such as TCP/IP, ICMP, UDP, FTP, SMTP, HTTP, DNS, RPC, NetBIOS, NNTP, SNMP, and Telnet. More advanced NIDS can actually display these protocol transactions in real time. One such product is Netprowler.

Some vendors have attempted to integrate network and host-based intrusion detection into a single product. ISS's RealSecure is the strongest example of such a product. The combined ability to watch network-based attacks (including port scans and remote buffer overflow-based attacks) with system-level events (such as failed login attempts and modified registry keys) in one interface is incredibly powerful. Additionally some IDSs can be integrated with firewalls and scanners in an attempt to increase security.


Limitations to NIDS:

False Positives

One important limitation to NIDSs is the frequency of false positives. No current IDS can completely eliminate the possibility of a false positive. However, most NIDS may be reconfigured so that a particular false positive does not continue to register. This reconfiguration is usually done via the attack signatures database. Many NIDS products have customizable attack signatures and also allow for the creation of new signatures. The programming of these signatures varies from product to product. For instance, Network Flight Recorder uses a proprietary N-code programming language to create new signatures. Other NIDS, such as ISS's RealSecure, have customizable functions that allow one to determine how the IDS should respond when it detects suspicious activity. These functions may somewhat alleviate the occurrence of false positives. However, it may be difficult to get full information on the hundreds of signatures that are built-in to a particular IDS, making customization more difficult. In addition, depending upon the product, the use of custom signatures may slow the performance of the IDS.


TCP Stream Reassembly/IP Defragmentation

Attacks involving TCP and IP packets are the cause for special concern. In order to monitor a TCP/IP connection, the target network must keep track of all of the individual TCP or IP packets. Though a set of TCP packets may arrive out of order, the receiving network may reorder the packets by using the packet sequence numbers. Many attacks exist that attempt to "confuse" the process of stream reassembly. For example, a teardrop attack causes a buffer overflow through the use of malformed data packets. The danger lies in the fact that the first packet looks no different than an ordinary data packet, so the IDS does not immediately detect the attack. In some cases, depending upon the operating system, it only takes one bad packet to crash the IDS. Once the IDS fails, most NIDS tend to fail open, so that once an attacker has crashed the IDS, s/he has access to the network. Although a report was issued in 1998 arguing that novice attacks using fragmented packets could elude all commercial NIDS, as of 2000 most NIDS were still unable to cope fully with this possibility. A few companies have added reassembly capabilities into their IDS products, such as Cisco and NFR. Other products can recognize fragmented packets, but are unable to perform TCP reassembly.


Switched Networks

Special limitation and problems emerge if the network is switched. This depends on the type of switches deployed as well as the type of NIDS in use. Most Internet-delivery environments are switched. The switches create a bit of a problem, as the NIDS device needs to see the traffic before inspecting it. The solution here is either to inspect the traffic at certain bottleneck points (such as perimeter firewalls) or to figure out a method of siphoning traffic off the wire onto a private inspection network.


Things to Consider When Choosing a NIDS

Operating System

The types operating systems supported vary greatly among IDS products. OS support is a big concern when considering a software-based NIDS. Some products only support Windows NT, whereas others, such as Snort, can be run on a wide variety of operating systems. Other NIDS will be designed so that the console runs on a Windows machine, while the sensor runs on OpenBSD. Regardless of which operating system is supported, an organization should choose their NIDS carefully when considering the operating system. The administrator of the IDS should be aware of all of the vulnerabilities related to the operating system that the IDS sits upon, so that the IDS may not be compromised.


NICS Supported

A very important consideration for NIDS is the type of Network Interface Cards (NICS) supported. Most prevailing technologies provide support for a wide variety of types, such as Token ring, FDDI, Ethernet, Fast Ethernet, or Gigabit Ethernet. Netprowler, by Symantec/Axent, currently only supports Ethernet or Fast Ethernet. Such details should be taken into account before purchasing and deploying an IDS.


Reactive Versus Passive Systems

One important aspect to consider is the need for a reactive NIDS versus a passive NIDS. A passive NIDS will simply log any suspicious network activity. If a serious attack takes place, the IDS will also send an alert to the console and perhaps by email or pager. A reactive NIDS, on the other hand, will perform those tasks and more. For example, suppose a reactive IDS detects some type of attack from a particular IP address. The reactive IDS may be programmed to automatically rewrite the rules of the network's firewall in order to deny future traffic from the attacking IP address-all of this taking place without human intervention. Certain reactive NIDS may have the following features:

  • Setting SNMP traps
  • Disconnecting and capturing sessions
  • Killing processes
  • Disabling user accounts
  • Launching program commands
  • Shunning attacker IP addresses

There are advantages and disadvantages for both types of NIDS. When considering a reactive NIDS, a network administrator may be assured of a timely response in the event of an attack. However, this response can backfire, depending upon the actual circumstances. For instance, in our example above, suppose the attacking IP address had been spoofed by a hacker. As a result, the legitimate network, which was spoofed, would be restricted access to the network by the reactive NIDS. Therefore a hacker could use the reactive features of an IDS to cause a denial of service attack.



The alerting features of a NIDS may be in the form of an email, pager, telephone call, or an alarm. Most NIDS include many types of alerting features. Most importantly, alerting should be done via the NIDS console, if no other alerting mechanism is available or enabled.


Logging and Reporting

A competent NIDS should minimally include a logging feature. The log enables an administrator to review any suspicious network traffic. Logs cannot be solely depended upon when deploying a NIDS. A determined hacker may easily flood the network to the extent that the log reaches its capacity and fails. Depending upon the operating system that the IDS sits on, a hacker may also compromise the IDS and easily delete information in the log. On the other hand, all false positives will also be present in the log. If there are an unusually large number of false positives, this can be quite an annoyance to the person who has to review the log. However, it is important to use the IDS log to search for any possible threats. Most network security experts encourage administrators to look at the IDS log at least once a day.



When deploying an IDS, it is important to keep the signature database up to date. Depending upon the particular product, there are various ways to ensure that the NIDS has the latest signature files available. The frequency of updates may vary from company to company. Most commercial vendors offer a download of new signatures from the vendor Web site. Others have automated updating features, though in some cases, the process of updating the signature database may mean the upgrade of the entire NIDS. Open source NIDS, such as Snort, are very flexible concerning signature updates. Often someone in the particular community of open source users can write a signature that is available to others in a short period of time.



Most NIDS include a console that provides various views and controls of the intrusion detection system. The interface of the console will vary greatly depending upon the IDS. Most commercial NIDS include a GUI interface with several possible views of the network. Other NIDS, such as Snort, have a command line interface. Additionally, some consoles may be accessed remotely, depending upon the product. Many consoles will have a hierarchical tree GUI interface. Some interfaces have the ability to sort attack by type, attacker, or target host. These aspects should be considered, as some IDS consoles do not provide as many viewing options as others.

An important question to consider is the communication between the sensors and the central console. It is important that the communication, such as attack alarms, are delivered to the console and to other recipients in a reliable, quick, and secure manner.



Some NIDS will scale more efficiently than others. As a network grows, the traffic may be too much for the IDS to handle. For certain NIDS, this problem may be solved simply by deploying more sensors (or appliances) on the network, in order to keep up with the network load. But this option is not suitable for a company or organization with less monetary resources. Other IDS vendors do not sell the console and sensors separately. Therefore as the network grows, an organization may have to change their IDS if the current one cannot scale.



Good redundancy for the IDS usually depends upon the amount of traffic on the network. As the amount of traffic increases, some NIDS will be less effective in detecting certain threats due to the heavy load. This largely depends upon how the particular product is designed. Netprowler, by Symantec/Axent, provides a unique approach to this concern. Netprowler does not attempt to monitor all network traffic. Instead, it is configured to detect only certain attacks. The configuration is determined by the types of machines sitting on a particular network, so that Netprowler listens for the most relevant threats for the network. For other NIDS, redundancy may be very closely related to scaling concerns-as the network grows, traffic may be too overwhelming for the particular IDS. Therefore it is vital that an organization be fully aware of the size and constitution of its network, so that an effective IDS may be deployed to fit the unique needs of the particular network.


Host-Based Intrusion Detection


Host-based IDS monitors individual systems upon the network. In this case, the sensor of the IDS is located inside of the particular host to monitor system-level behavior. This type of intrusion detection is especially useful for monitoring potentially dangerous user activity within the network. According to Zirkle, there are two types of host-based intrusion detection software: host wrappers (or personal firewalls) and agent-based software. Zirkle describes the host wrappers as tools that "can be configured to look at all network packets, connection attempts, or login attempts to the monitored machine." The agent-based software has the same abilities as the host wrappers, but can also detect changes in system files and changes in user privileges, according to Zirkle. A report by Network Associates makes a good argument for host-based intrusion detection, stating, "Any masking techniques such as insertion, padding, fragmentation, or out-of-sequence delivery, which would evade a network-based IDS can be easily caught by a host-based IDS." Additionally, host-based IDSs can be quite effective in switched environments, whereas network-based IDS systems are less effective in that environment. A switch tends to isolate communications on the network, making it difficult for a network-based IDS to monitor all traffic. However, if the systems on the switched network have host-based IDSs installed, potential attacks may be thwarted. Brenton clarifies this point, stating, "Since it [host-based IDS] runs on the system you wish to protect, it is unaffected by the traffic isolation properties of the switch."



There are some problems with host-based IDSs. According to Chris Brenton, many commercial host-based IDSs can only monitor certain types of systems. Brenton also makes the point that host-based IDSs "do not have access to the core communication functionality of the system. This means that the IDS is incapable of fending off attacks against the protocol stack itself."



Clearly there are benefits and limitations to both network and host-based IDSs. Network-based IDSs cannot be solely relied upon, considering their general lack of performance in TCP stream reassembly and their inadequacy in switched networks. However, until products become more robust in these areas, many host-based IDSs can compensate for these inadequacies. Most security experts will recommend deploying both network and host-based intrusion detection systems to enable security for network transmissions as well as system level events.