This is the fifth and final installment in a series of articles on understanding and developing signatures for network intrusion detection systems. In the previous article, we looked at the topic of protocol analysis, meaning that the intrusion detection system actually understands how various protocols, such as FTP, are supposed to work. We initially looked at protocol analysis as it applied to a single request or response. In this article, we will extend this discussion by looking closely at stateful protocol analysis, which involves performing protocol analysis for an entire connection or session, capturing and storing certain pieces of relevant data seen in the session, and using that data to identify attacks that involve multiple requests and responses.

Stateful Protocol Analysis

The concept of stateful protocol analysis is simple: to add stateful characteristics to regular protocol analysis. When we perform protocol analysis, we examine TCP and UDP payloads, which contain protocols such as DNS, FTP, HTTP and SMTP. IDS sensors that perform protocol analysis understand how each protocol is supposed to work, based on RFCs and on real-world implementations of these protocols. So the IDS sensor can detect many suspicious values within protocol application payloads. Protocol analysis signatures can also be designed to overcome attempts by attackers to obfuscate their exploits. For example, hex encoding can be used to slightly modify URLs so that they appear different to us but have the same meaning to Web servers. Microsoft IIS Web servers perform two hex decoding operations on URLs before processing them: an HTTP protocol analysis signature set looking for IIS attempts should also perform two hex decoding operations, so that it is examining the same URL that the IIS Web server would see.

Although protocol analysis by itself is a very powerful technique, it is limited to examining a single request or response. Of course, many attacks cannot be detected by looking at one request - the attack may involve a series of requests. The best way to detect such attacks is by adding stateful characteristics to protocol analysis. When we perform stateful protocol analysis, we monitor and analyze all of the events within a connection or session. The IDS sensor can “remember” significant events and data for the duration of the session. This allows the sensor to find correlations among different events within a session, identifying attacks with multiple components that cannot be detected otherwise. Without the ability to keep state, we can only examine each packet, request or response on its own, completely independent of the rest of the session.

Pairing Requests and Responses

The concept of stateful protocol analysis is best explained by looking at a few examples that illustrate its power. There are several ways to think of “state” in a connection. One of the simplest forms of state is being able to associate a request with its corresponding response. Because an IDS sensor that performs stateful protocol analysis can monitor every request and response in a session sequentially, it is easily able to match a response with the request that generated it. In many cases, this is extremely valuable, as many responses contain some sort of status indicator that tells us what the result of the request was. For example, when an FTP command is issued, the server’s response will start with a three-digit code. A status code that starts with a “2” indicates that the command was successful, while a “5” indicates failure.

IDS sensors can make use of this information in multiple ways:

• The most obvious way is to tell whether certain attacks succeeded or failed. If an attacker attempts to retrieve a sensitive file, and we see that the status code in the response starts with a “2”, then the attempt was successful.
• Because we can identify each failed request and keep track of how many requests have failed in a session, we can look for large numbers of failures. This is very useful in finding brute force attacks, such as password guessing attacks.
• A more sophisticated use is to identify instances where a request does not have a matching response. Some attacks involve rapidly sending the same command to a server hundreds or thousands of times. In these cases, the IDS sensor may see many requests from the client before seeing a single response from the server.

As you can see, the simple capability to remember the last command and associate it with its response gives us the ability to identify several types of attacks. However, we are not limited to just remembering the previous command; we can remember previous commands and responses as well, if we wish. It’s much more likely that we will just extract the important data from certain commands and responses and save that data for future use.

Saving Data

Here’s an example of when we would save data from a session for the IDS sensor to use during the rest of the session. An FTP user logs in, using a four-step process:

1. The user sends “USER kkent” to the FTP server, where kkent is her username.
2. The FTP server responds with “331 Send password”.
3. The user sends “PASS foobar” to the FTP server.
4. The FTP server confirms that the password is correct and responds with “230 User logged in.”

The IDS sensor detects the USER command and stores “kkent” as the tentative username for this FTP session. It then watches for a PASS command that has a response with a status code that begins with a 2, which indicates that the authentication was successful. (The IDS sensor also watches for another USER command, which indicates that the user is attempting to authenticate again.) Once the successful authentication has been detected, the IDS sensor knows that the session it is monitoring is that of username kkent.

This information is useful to us in two main ways. First, if an attack occurs during this session, the IDS can report that username kkent was used for this session. This information may be extremely helpful when investigating an incident. Second, imagine that a more interesting username than kkent was used – perhaps “root” or “administrator” was used. By statefully analyzing all the authentication-related requests and responses, the IDS sensor can detect attempts to use such accounts, plus it can record whether each was successful or not.

Verifying the Sequence of Commands

Another way to think of state is the phases of a session. Let’s look at the phases of an FTP session to gain a better understanding of the benefits of tracking state.

1. Connection: The FTP client establishes a TCP connection to the server’s FTP port (TCP port 21).
2. Authentication: The user of the FTP client either sends a username and password, or an “anonymous” login, to the FTP server. The user remains in this state until either the authentication is successful, or the connection is terminated.
3. Transaction: Once the user has been authenticated, he or she can issue commands to the FTP server. If any of these commands cause data to be transferred, such as files copied between the client and server or directory listings, a separate short-lived TCP connection is established for each transfer.
4. Disconnection: When the FTP client or server wishes to end the session, the TCP connection is torn down.

Being able to identify which state a session is in is the only way of identifying certain kinds of attacks. For example, some FTP attacks and reconnaissance methods rely on issuing transaction-phase commands before the user has successfully authenticated. The only way we can accurately determine when this has occurred is by monitoring the whole session for a successful authentication, so when a vulnerable transaction-phase command is issued, we can generate an alert if a successful authentication hasn’t yet occurred. This is a special case of a larger goal of stateful protocol analysis, verifying the sequence of commands (we will look at that in more depth shortly).

In the FTP transaction stage, an additional TCP connection is generated every time FTP-related data needs to be transferred between the FTP client and server. Before establishing an FTP data connection, the FTP client and server communicate with each other (through commands and responses) about which IP address and port should be used. If the FTP client wants to initiate the data connections to the server (passive FTP), it will send a command to the FTP server, and it expects the FTP server’s response to contain a port number. The FTP client will then attempt to establish a new TCP connection to that port on the server. An IDS sensor that uses stateful protocol analysis can detect the FTP server responses that tell clients what port numbers to use for data connections. When it sees connections involving the FTP server, it can then confirm whether they are legitimate FTP data connections, or whether an attacker is attempting to connection to the server in order to gain unauthorized access to data. This is a great example of how the tracking of state can identify attacks that otherwise could not be detected.

As mentioned earlier, we can utilize stateful protocol analysis to verify the sequence of commands. We’ve already discussed how this can be useful when identifying the phases of a connection, such as making sure that a user has successfully authenticated before issuing certain commands. We can also use state to check for particular sequences of commands. For example, when you rename a file in FTP, two commands are used. First, a RNFR (rename from) command provides the old name of the file; after the server’s positive response to this command is received, a RNTO (rename to) command should be issued. There are attacks that involve issuing many consecutive RNTO commands. Since a RNTO command should be preceded by a RNFR, we should be suspicious if we see ten consecutive RNTO commands. Advanced signatures based on stateful protocol analysis can count how many consecutive RNTO commands are seen, or they can remember the previous command and when a RNTO is issued, check the previous command.

Conclusion

In this article, we have completed our examination of protocol analysis by studying stateful protocol analysis. Stateful protocol analysis performs regular protocol analysis on each request and response in a session, but adds several benefits because it can track various types of state within each session. By adding the ability to pair requests with their corresponding responses, several more types of network intrusion detection signatures can be created to identify attacks that could not otherwise be found. Because we can extract and “remember” important values within a session, we can correlate various events within a session to identify attacks that involve two or more requests and responses. Another benefit of tracking state is that we can identify the phases of a connection and verify that the commands we see take place in the proper phase. Similarly, we can confirm that commands are issued in the proper sequence. Stateful protocol analysis is the best way, and in many cases the only way, to identify various complex attacks.

Karen Kent Frederick (kkent@bigfoot.com) is a senior security engineer for the Rapid Response Team at NFR Security. She is a graduate of the University of Wisconsin-Parkside and is currently completing her master's in computer science, focusing in network security, through the University of Idaho's Engineering Outreach program. Karen has over 10 years of experience in technical support, system administration and information security. She holds several certifications, including SANS GIAC Certified Intrusion Analyst, GIAC Certified Unix Security Administrator, and GIAC Certified Incident Handler. She is one of the authors of “Intrusion Signatures and Analysis", and she is a contributing author to the "Handbook of Computer Crime Investigation".