Earlier I’ve talked about various ways of observing and debugging SSL connections to the web server, such as acting as a client and acting as a proxy.

Both of these methods have their advantages and disadvantages. Today I’ll review using ssldump, which combines some of the advantages of both of the previous tools: you can observe real requests from a real client and you can also observe the actual application data (HTTP layer). The only drawback is that there is a bit of additonal setup work to do before this will work, but it is definitely worth it.

Unlike ssltap, ssldump observes network traffic directly. So it is not necessary to proxy the client requests through it, just point it at a network interface where the traffic of interest is visible. For the easiest use case you can simply run:

# ssldump -i bge0 -d port 8088
New TCP connection #1: myclient(34286) <-> myserver(8088)
1 1  0.0265 (0.0265)  C>S  Handshake
      ClientHello
        Version 3.0
        resume [32]=
          08 22 e7 bc cf 13 e7 7f 80 0d 62 43 24 4c 65 5b
          1e 19 69 ab 3c 51 0e 95 29 d9 79 9d 9f 79 04 92
        cipher suites
        Unknown value 0x39
        Unknown value 0x38
        Unknown value 0x35
        Unknown value 0x33
        Unknown value 0x32
        SSL_RSA_WITH_RC4_128_MD5
        SSL_RSA_WITH_RC4_128_SHA
        Unknown value 0x2f
        SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        Unknown value 0xfeff
        SSL_RSA_WITH_3DES_EDE_CBC_SHA
        SSL_DHE_DSS_WITH_DES_CBC_SHA
        Unknown value 0xfefe
        compression methods
                  NULL
1 2  0.0273 (0.0007)  S>C  Handshake
      ServerHello
        Version 3.0
        session_id[32]=
          08 22 e7 bc cf 13 e7 7f 80 0d 62 43 24 4c 65 5b
          1e 19 69 ab 3c 51 0e 95 29 d9 79 9d 9f 79 04 92
        cipherSuite         SSL_RSA_WITH_RC4_128_MD5
        compressionMethod                   NULL
1 3  0.0273 (0.0000)  S>C  ChangeCipherSpec
1 4  0.0273 (0.0000)  S>C  Handshake
1 5  0.0635 (0.0362)  C>S  ChangeCipherSpec
1 6  0.0635 (0.0000)  C>S  Handshake
1 7  0.0635 (0.0000)  C>S  application_data
1 8  0.0643 (0.0008)  S>C  application_data
1 9  30.1176 (30.0532)  S>C  Alert

Note that I ran ssldump as root since it needs to listen to the traffic on the network device bge0 (substitute the correct device for your system here – check with ifconfig if in doubt). My SSL-enabled web server listener is on port 8088 for this example, so I limited ssldump to tracking traffic on that port (I could’ve said “dst port 8088” as well), otherwise it would’ve dumped all network traffic on that device, making it difficult to follow the output.

Also check the first column in the output, which above is always “1”. Since ssldump observes the network traffic there may be multiple requests going to the server at once. Above there was only one:

New TCP connection #1: myclient(34286) <-> myserver(8088)

In the case where there are multiple connections observed, each will be numbered. The packets for each connection will most likely be interleaved in the output, but you can track each one by correlating these numbers. That can be quite useful – I’ve used ssldump to diagnose some SSL connectivity problems which were only reproducible when multiple concurrent requests hit the server all at once.

The example above is nice in that I didn’t have to proxy my browser traffic through it, I could just connect to my server as usual. But aside from that it is not so interesting. The output isn’t really any more useful than ssldump.

So, let’s make things more interesting.

If given access to the server private key, ssldump can decrypt the traffic to and from that server on the fly. That’s where it gets really useful. We’ll need to do a bit of prep work to set this up.

First, extract the private key from the server instance into a PKCS#12 format file using pk12util.

• You’ll need to know the nickname of the server keypair/cert (see your server.xml) for the -n parameter.
• I changed to the directory where the NSS *.db files live so I type “-d .”. Alternatively you could run the command from elsewhere by giving the right path.
• Finally, I am runing JES Web Server 7.0 so there is no prefix to the NSS files, but if you are on 6.1 you’ll need to give a -P parameter with the right prefix for that instance.

% pk12util -o myserver.pk12 -n Server-Cert -d . -v
Enter Password or Pin for "NSS Certificate DB":
Enter password for PKCS12 file:
Re-enter password:
pk12util: PKCS12 EXPORT SUCCESSFUL

Perhaps needless to say, but if this was a production server I’d have to be very careful where I store this private key file. Keep that in mind.

Next I’ll just convert this to a format suitable for ssldump using openssl:

% openssl pkcs12 -in myserver.pk12 -out myserverkey
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

Ok.. now lets run ssldump and give it access to the server key material (the “PEM pass phrase” you need to type in below is the same you gave it above):

# ssldump -i bge0 -d -k /tmp/myserverkey port 8088
Enter PEM pass phrase:

Then I connect to my server from a browser:

New TCP connection #1: laptop(39699) <-> myserver(8088)
1 1  0.0853 (0.0853)  C>S SSLv2 compatible client hello
  Version 3.1
  cipher suites
  Unknown value 0x39
  Unknown value 0x38
  Unknown value 0x35
  Unknown value 0x33
  Unknown value 0x32
  TLS_RSA_WITH_RC4_128_MD5
  TLS_RSA_WITH_RC4_128_SHA
  Unknown value 0x2f
  TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
  TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
  Unknown value 0xfeff
  TLS_RSA_WITH_3DES_EDE_CBC_SHA
  TLS_DHE_RSA_WITH_DES_CBC_SHA
  TLS_DHE_DSS_WITH_DES_CBC_SHA
  Unknown value 0xfefe
  TLS_RSA_WITH_DES_CBC_SHA
  TLS_RSA_EXPORT1024_WITH_RC4_56_SHA
  TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA
  TLS_RSA_EXPORT_WITH_RC4_40_MD5
  TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
1 2  0.0856 (0.0002)  S>C  Handshake
      ServerHello
        Version 3.1
        session_id[32]=
          08 22 c2 25 34 4e 85 61 dd 24 ba 9a 59 a2 dc b0
          77 a0 3f b7 ac c9 d3 ce 76 4a b5 42 cc 44 30 fb
        cipherSuite         TLS_RSA_WITH_RC4_128_MD5
        compressionMethod                   NULL
      Certificate
      ServerHelloDone
1 3  6.1870 (6.1013)  C>S  Handshake
      ClientKeyExchange
1 4  6.1870 (0.0000)  C>S  ChangeCipherSpec
1 5  6.1870 (0.0000)  C>S  Handshake
      Finished
1 6  6.1931 (0.0061)  S>C  ChangeCipherSpec
1 7  6.1931 (0.0000)  S>C  Handshake
      Finished
1 8  6.2852 (0.0921)  C>S  application_data
    ---------------------------------------------------------------
    GET / HTTP/1.1
    Host: myserver:8088
    User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.5)
 Gecko/20041217
    Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/
plain;q=0.8,image/png,*/*;q=0.5
    Accept-Language: en-us,en;q=0.5
    Accept-Encoding: gzip,deflate
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    Keep-Alive: 300
    Connection: keep-alive

    ---------------------------------------------------------------
1 9  6.2859 (0.0007)  S>C  application_data
    ---------------------------------------------------------------
    HTTP/1.1 403 Forbidden
    Server: Sun-Java-System-Web-Server/7.0
    Date: Thu, 27 Apr 2006 08:12:15 GMT
    Content-length: 142
    Content-type: text/html

    Forbidden


Forbidden



    Your client is not allowed to access the requested object.
    --------------------------------------------------------------
-
1 10 12.0328 (5.7468)  C>S  Alert
    level           warning
    value           close_notify
1    12.0360 (0.0032)  C>S  TCP FIN
1    12.0361 (0.0000)  S>C  TCP FIN

Nice! Now I can watch all the application data traffic in plaintext. As you can imagine, this can be very useful for diagnosing all kinds of problems with an SSL-enabled server.

As a final note, when building ssldump be sure to build it with openssl support or the traffic decryption will not work. Here’s the configure options I gave it on my Solaris 10 machine:

./configure --with-pcap=/opt/sfw --with-openssl=/usr/sfw

Earlier I talked about issuing test requests to a SSL/TLS-enabled web server.

That works well as long as you can recreate the request contents by hand. But sometimes you really need to diagnose a connection which is being issued by some real client (such as a browser or client app). There are a number of ways to observe such request and response data.

One such tool is ssltap, also part of NSS. ssltap takes the proxy approach – it serves as a simple proxy between the client and the real server and displays information about the connections it forwards (note that you can just as well use ssltap for observing plain HTTP requests or even requests based on other protocols).

Let’s look at an example. I have a JES Web Server with an SSL-enabled listener on port 8088 of my machine, so I issue the following (you’ll always want the -l option so ssltap doesn’t exit after a single request):

% ssltap  -l -s localhost:8088
Looking up "localhost"...
Proxy socket ready and listening

By default ssltap listens on port 1924 (run ssltap without any options to see usage info) which will do fine. I then connect from firefox to ‘https://localhost:1924’ which produces ssltap output such as this:

Connection #1 [Mon Apr 10 15:49:49 2006]
Connected to localhost:8088
--> [
alloclen = 87 bytes
(87 bytes of 87)
 [Mon Apr 10 15:49:49 2006] [ssl2]  ClientHelloV2 {
           version = {0x03, 0x01}
           cipher-specs-length = 60 (0x3c)
           sid-length = 0 (0x00)
           challenge-length = 16 (0x10)
           cipher-suites = {
                (0x000039) TLS/DHE-RSA/AES256-CBC/SHA
                (0x000038) TLS/DHE-DSS/AES256-CBC/SHA
                (0x000035) TLS/RSA/AES256-CBC/SHA
                (0x000033) TLS/DHE-RSA/AES128-CBC/SHA
                (0x000032) TLS/DHE-DSS/AES128-CBC/SHA
                (0x000004) SSL3/RSA/RC4-128/MD5
                (0x000005) SSL3/RSA/RC4-128/SHA
                (0x00002f) TLS/RSA/AES128-CBC/SHA
                (0x000016) SSL3/DHE-RSA/3DES192EDE-CBC/SHA
                (0x000013) SSL3/DHE-DSS/DES192EDE3CBC/SHA
                (0x00feff) SSL3/RSA-FIPS/3DESEDE-CBC/SHA
                (0x00000a) SSL3/RSA/3DES192EDE-CBC/SHA
                (0x000015) SSL3/DHE-RSA/DES56-CBC/SHA
                (0x000012) SSL3/DHE-DSS/DES56-CBC/SHA
                (0x00fefe) SSL3/RSA-FIPS/DES-CBC/SHA
                (0x000009) SSL3/RSA/DES56-CBC/SHA
                (0x000064) TLS/RSA-EXPORT1024/RC4-56/SHA
                (0x000062) TLS/RSA-EXPORT1024/DES56-CBC/SHA
                (0x000003) SSL3/RSA/RC4-40/MD5
                (0x000006) SSL3/RSA/RC2CBC40/MD5
                }
           session-id = { }
           challenge = { 0xdfb5 0x1d22 0x6562 0x34f6 0x95b9 0x668a 0x234e 0x38ea
 }
}
]

This is the SSL client hello being sent from the browser to the server. Of interest, note the list of cipher suites the browser sent. This is the set of cipher suites the browser is configured to handle (note also they are sorted in order of preference). The server will pick one of those for the handshake (if the server is not set up to handle any of these, the connection will then immediately fail). Notice also the session-id is empty, which tells us the browser does not have any cached SSL session (not to be confused with HTTP sessions!) with this particular server.

The server responds with:

<-- [
(1015 bytes of 1010)
SSLRecord { [Mon Apr 10 15:49:49 2006]
   type    = 22 (handshake)
   version = { 3,1 }
   length  = 1010 (0x3f2)
   handshake {
      type = 2 (server_hello)
      length = 70 (0x000046)
         ServerHello {
            server_version = {3, 1}
            random = {...}
            session ID = {
                length = 32
                contents = {..}
            }
            cipher_suite = (0x0035) TLS/RSA/AES256-CBC/SHA
         }
      type = 11 (certificate)
      length = 928 (0x0003a0)
         CertificateChain {
            chainlength = 925 (0x039d)
            Certificate {
               size = 485 (0x01e5)
               data = { saved in file 'cert.001' }
            }
            Certificate {
               size = 434 (0x01b2)
               data = { saved in file 'cert.002' }
            }
         }
      type = 14 (server_hello_done)
      length = 0 (0x000000)
   }
}
]

The server picked TLS/RSA/AES256-CBC/SHA as the cipher suite to use (JES Web Server 7.0 supports AES in addition to ECC).

A ‘session ID’ was sent, which this client will include in subsequent requests. The server also sent its certificate chain for the browser to verify. ssltap saved these certificates in the files noted (‘cert.001’, ‘cert.002’). You can examine these certificates later with any tool which can parse X.509 certificates. For example, using openssl you could do:

% openssl x509 -in cert.001 -text -inform DER

I will skip the rest of the ssltap output as there is a lot of it, but try it and see. You can observe a fair amount of useful information from it such as the number of requests and responses and the size of each.

The major drawback is that the actual session data is of course encrypted between the client and the server so ssltap has no way to decipher it (with the -h flag ssltap will give a hex dump of the raw data). Thus, while this is useful for observing the handshake and request progress, it won’t help if you needed to look at the request or response details.

One option here is to enable a NULL cipher suite on both the client and the server. The NULL suites are SSL/TLS suites which perform all the regular protocol steps with the only difference that the bulk encryption algorithm used to encrypt the request/response data is ‘NULL’ – that is, it goes in plain text.

As noted earlier, the handshake protocol attempts to negotiate the strongest cipher suite that is common to the client and the server, so the NULL suites will never be selected unless it is the only choice. That means that in order to make it work you will have the explicitly disable every cipher suite except a NULL one in either the client or the server.

To show this scenario, I disabled all suites in the JES Web Server 7.0 except SSL_RSA_WITH_NULL_SHA and also enabled the corresponding suite in firefox (unfortunately firefox has no UI to configure these but you can do it by entering ‘about:config’ as the URL; search for the ‘security.ssl3’ options).

With that, the initial client hello containing the list of ciphers shows up through ssltap as:

            cipher_suites[22] = {
                (0x0039) TLS/DHE-RSA/AES256-CBC/SHA
                (0x0038) TLS/DHE-DSS/AES256-CBC/SHA
                (0x0035) TLS/RSA/AES256-CBC/SHA
                (0x0033) TLS/DHE-RSA/AES128-CBC/SHA
                (0x0032) TLS/DHE-DSS/AES128-CBC/SHA
                (0x0004) SSL3/RSA/RC4-128/MD5
                (0x0005) SSL3/RSA/RC4-128/SHA
                (0x002f) TLS/RSA/AES128-CBC/SHA
                (0x0016) SSL3/DHE-RSA/3DES192EDE-CBC/SHA
                (0x0013) SSL3/DHE-DSS/DES192EDE3CBC/SHA
                (0xfeff) SSL3/RSA-FIPS/3DESEDE-CBC/SHA
                (0x000a) SSL3/RSA/3DES192EDE-CBC/SHA
                (0x0015) SSL3/DHE-RSA/DES56-CBC/SHA
                (0x0012) SSL3/DHE-DSS/DES56-CBC/SHA
                (0xfefe) SSL3/RSA-FIPS/DES-CBC/SHA
                (0x0009) SSL3/RSA/DES56-CBC/SHA
                (0x0064) TLS/RSA-EXPORT1024/RC4-56/SHA
                (0x0062) TLS/RSA-EXPORT1024/DES56-CBC/SHA
                (0x0003) SSL3/RSA/RC4-40/MD5
                (0x0006) SSL3/RSA/RC2CBC40/MD5
                (0x0002) SSL3/RSA/NULL/SHA
                (0x0001) SSL3/RSA/NULL/MD5

The NULL ciphers are of course at the very bottom of the list. The server, having nothing else enabled, responds with:

...
           cipher_suite = (0x0002) SSL3/RSA/NULL/SHA
...

And so, the application data of this session is now unencrypted. Without the -h option ssltap will still show the data records as ‘< encrypted >’ because it doesn’t know any better (remember that all the regular SSL protocol encapsulation is in place even though the bulk encryption algorithm happens to be a no-op). If you give the -h option you’ll see the hex dump of what is now the readable request/response data. Of course, being plan text, now you can use your favorite network observation tool instead.

If you do this, be absolutely sure to remember restore the original configuration afterwards! Disable all NULL cipher suites and reenable the ones you need. You never want to have the NULL suites enabled outside of special testing circumstances (JES Web Server 7.0 will log warnings if weak suites are enabled, as a reminder).

A final note about ssltap – it is a single threaded proxy so if you issue multiple requests through it they will get serialized. Usually not a problem, but if you need to analyze some problem with your application which only occurs on concurrent requests through SSL, it will not help (try running multiple ssltap instances).