When working with SSL-enabled web servers it is often useful to create self-signed certificates for testing and development. This is much quicker and more convenient than going through an external CA when all you need to do is run some tests on your development machine. Unfortunately Web Server 6.1 (formally, Sun Java System Web Server 6.1 (which you may also have met under the SunONE or the older iPlanet brand names)) does not support creating self-signed certificates through the admin UI. On the bright side, it is actually quite easy to create these certificates using the NSS tool certutil.

First: Create the NSS databases

You can do this through certutil, but let’s do it through the supported admin UI interface. “Servers” -> “Manage Servers” -> select server, click ‘Manage’. Then click “Security” tab. Enter password twice into the fields and submit. Popup says “Success”! -> Click OK.

Second: Create a local CA

Despite the title of this entry, instead of directly creating a self-signed cert, I’ll first create a local CA for myself and then use it to sign the server cert, so I can demonstrate both possibilities. If you prefer, skip the hierarchy and generate a self-signed server cert directly.

Go to the alias directory under the install root. That is where the NSS database files live in 6.1. At this point you should have at least the files shown below in the alias directory.

Note that boqueron.virkki.com here is the host and domain name, your files will have names corresponding to your installation. For all subsequent commands in this example, substitute the corresponding names for your instance in place of these.

$BASE/alias% ls -1
https-boqueron.virkki.com-boqueron-cert8.db
https-boqueron.virkki.com-boqueron-key3.db
secmod.db

You can use certutil -L to list all the certs in the database. If you just created the database through the UI like I did, it’ll be empty:

$BASE/alias% certutil -L -d . -P "https-boqueron.virkki.com-boqueron-"
certutil -L -d . -P "https-boqueron.virkki.com-boqueron-"

Now, I will create a sample CA with certutil (note this is a single command line, which I’ve split here only for readability). I have ommitted some of the output for brevity:

$BASE/alias% certutil -S  -P "https-boqueron.virkki.com-boqueron-"
   -d . -n SelfCA -s "CN=Self CA,OU=virkki.com,C=US" -x -t "CT,CT,CT"
   -m 101 -v 99 -5

Generating key.  This may take a few moments...

                          0 - SSL Client
                          1 - SSL Server
                          2 - S/MIME
                          3 - Object Signing
                          4 - Reserved for futuer use
                          5 - SSL CA
                          6 - S/MIME CA
                          7 - Object Signing CA
                          Other to finish

Enter 5 since we want a CA.

                          0 - SSL Client
                          1 - SSL Server
                          2 - S/MIME
                          3 - Object Signing
                          4 - Reserved for futuer use
                          5 - SSL CA
                          6 - S/MIME CA
                          7 - Object Signing CA
                          Other to finish

Enter 9 to end.

Is this a critical extension [y/n]?

Enter y.

Third: Use this local CA to sign your server cert

$BASE/alias% certutil -S  -P "https-boqueron.virkki.com-boqueron-"
   -d . -n MyServerCert -s "CN=boqueron.virkki.com,C=US" -c SelfCA -t "u,u,u"
   -m 102 -v 99 -5

Generating key.  This may take a few moments...

                          0 - SSL Client
                          1 - SSL Server
                          2 - S/MIME
                          3 - Object Signing
                          4 - Reserved for futuer use
                          5 - SSL CA
                          6 - S/MIME CA
                          7 - Object Signing CA
                          Other to finish

Enter 1

                          0 - SSL Client
                          1 - SSL Server
                          2 - S/MIME
                          3 - Object Signing
                          4 - Reserved for futuer use
                          5 - SSL CA
                          6 - S/MIME CA
                          7 - Object Signing CA
                          Other to finish

Enter 9 to end.

Is this a critical extension [y/n]?

Enter y.

Try certutil -L again, this time you’ll see both your CA and your server cert:

$BASE/alias% certutil -L -d . -P "https-boqueron.virkki.com-boqueron-"
certutil -L -d . -P "https-boqueron.virkki.com-boqueron-"
MyServerCert                                                 u,u,u
SelfCA                                                       CTu,Cu,Cu

Also try looking at them from the admin UI. Under “Security” ->”Manage Certificates”, you will see these newly created certificates listed.

That’s it! You can now assign the MyServerCert to any of your SSL-enabled listeners.

For example, if you want to change one of your non-SSL listeners to enable SSL and use the new certificate, you can follow this sequence in the admin UI: “Preferences” -> “Edit Listen Socket” -> click on an ls ID to edit. Then check the security box and click OK. Once again, “Edit Listen Socket” -> click on the same ls. This time you see SSL options, change any if desired and then click OK. Finally Apply to apply the changes.

I used quite a few options to certutil which I didn’t describe in any detail to keep this brief. Run certutil -H to read the description of every option I used (and the ones I didn’t) so you can tailor the options to your needs.

A while ago I worked for a bit on some Java servlet code which needed to talk to a MySQL backend database. Diverging a bit from my usual topics I thought I’d write this down in case it is useful for others (or at least it’ll be useful for me next time I need to do the same).

This configuration assumes Web Server 7. In server.xml I configured a JDBC resource for mysql:

  <jdbc-resource>
    <jndi-name>jdbc/mysql</jndi-name>
    <datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource</datasource-class>
    <min-connections>1</min-connections>
    <max-connections>5</max-connections>
    <property>
      <name>password</name>
      <value>password-here</value>
    </property>
    <property>
      <name>user</name>
      <value>jyri</value>
    </property>
    <property>
      <name>url</name>
      <value>jdbc:mysql://boqueron/dbname</value>
    </property>
  </jdbc-resource>

Then I copied mysql-connector-java-3.1.12-bin.jar into $INSTANCE/lib (where $INSTANCE is the top level directory of the specific instance, note that you may need to create the lib directory there if it hasn’t been needed before).

In the web application itself, in web.xml, I configured:

 <resource-ref>
    <description>JDBC Connection Pool</description>
    <res-ref-name>jdbc/mysql</res-ref-name>
    <res-type>javax.sql.DataSource</res-type>
    <res-auth>Container</res-auth>
  </resource-ref>

And in sun-web.xml:

  <resource-ref>
    <res-ref-name>jdbc/mysql</res-ref-name>
    <jndi-name>jdbc/mysql</jndi-name>
  </resource-ref>

That’s it for the configuration. In my servlet code:

    Context initContext = new InitialContext();
    Context webContext = (Context)initContext.lookup("java:/comp/env");

    DataSource ds = (DataSource) webContext.lookup("jdbc/mysql");
    Connection conn = ds.getConnection();

    Statement stmt = conn.createStatement();
     ....

Earlier I have coveredseveral different methods of approaching the problem of diagnosing HTTP (or even other protocol) traffic obscured within SSL/TLS.

There’s another approach that I haven’t mentioned yet which I use every now and then, whenever I want to look at what exactly the browser is sending. Check out tamperdata, a firefox extension which can alter the requests from the browser. While its main purpose is presumably to alter the data (which, as an aside, can be very useful for testing server and server application behavior in many ways) it can just as well be used merely for observing said data.

Of all the methods I’ve discussed this one is the most convenient as there is really nothing to do beyond installing the extension. Whenever you wish to observe the interaction just open the tamperdata window (from Tools menu) and watch away. The limitation, of course, is that it’s only good for watching interaction originating from your single browser session. But often that is all you need.

Every now and then I get requests on how to load one or more CRLs (Certificate Revocation List) into the web server without going through the admin GUI (in 6.1, only the GUI has direct support for updating CRLs). Clearly this is a useful goal since going through the GUI would require manual intervention, hardly practical for most web servers.

It is possible to do this in 6.1 via crlutil. However, there’s a potentially big catch – it is necessary to restart the server instance(s) to pick up the new CRL(s). Depending on how often the CRLs are refreshed, how many CA (Certificate Authority) CRLs need to be refreshed and the site’s uptime requirements, this can be a problem.

One of the many nice features of 7.0 (preview here) is that it allows CRLs to be loaded dynamically at runtime.

Every server instance can define a directory from where CRLs will be loaded, as follows, in server.xml (if you have multiple instances you’ll most likely want to share this location amongst them):

  <pkcs11>
     <crl-path>/export/crls</crl-path>
  </pkcs11>

7.0 also introduces a mechanism for running recurring events. This can be used (among other things – but a full description is a topic for another time) to refresh the CRLs. The following will check every minute (60 seconds) to see if any new CRLs are available:

  <event>
    <interval>60</interval>
    <update-crl/>
  </event>

And that’s all there is to it. You can copy new CRLs as they become available into your crl-path and the server will pick them up automatically. I recommend setting up a simple script which will obtain the new CRLs from your CA periodically and copy them into crl-path; you can call this from cron at desired intervals. Also look into the <time> element within <event>, it can be a better fit if you set up refreshes at fixed times.

As a side note, while experimenting with WS 7.0 configuration it is handy to edit server.xml manually. However, if you end up wanting to automate configuring your instances, always go through wadm instead. You definitely don’t want to do something fragile like writing perl scripts to directly manipulate server.xml – such code is likely to break when you upgrade to a future version of the server. Instead, write scripts around wadm. That way wadm will shield your code from any internal configuration changes. In that spirit, take a look at the following wadm commands: list-crl, install-crl, delete-crl, get-crl-prop, list-events, create-event, delete-event, get-event-prop, enable-event, disable-event.

An exciting week in the land of web server – if you’ve visited our Java One booth or been following the blog you already know that a preview of Sun’s JES Web Server 7.0 is now available for you to download.

7.0 is a new Major release and there is quite a bit new and improved indeed. If you are used to operating through the admin GUI the major change you will notice very quickly is the all-new admin.

On the other hand, if you’re the type to go looking at the actual installation, the major change you’ll notice right away is that the file layout is quite a bit different from previous releases.

Let me digress for a moment… One of my jobs here at Sun is as a member of Sun’s ARC (Architecture Review Committee). John Plocher has written to some length about Sun’s ARC process so I won’t repeat too much of that. One of the many goals of the ARC process is to make sure there is clarity as to which interfaces customers can rely on.

As long as you rely only on public Stable interfaces you will be assured that your code will continue to work in future releases (this is why you can take binaries you might’ve compiled on Solaris over 10 years ago and they still run on Solaris 10!). If, however, you build a dependency on any undocumented (private) interface, you must assume your code will break at any time without warning (usually at the worst possible time). Clearly, not a good idea.

(Please note that the word interface is used in a very broad sense. Anything you might depend on is an interface. APIs are interfaces, but so are CLI names and options, file locations, some output and so forth. Anything you might be able to build a dependency on can be an interface. Note also that Stable interfaces may in some cases also change – but only in Major releases with advance notice, so you’d have plenty of time to adjust.)

Why do I bring this up? Well, I’ll give a quick view of the file layout changes in 7.0 and I’ll also say a bit about where things fall with respect to their interface classification.

The web server file layout up to 6.1 harks back to the old iPlanet days and it had grown a bit messy over the years. In 7.0 things have been reorganized and cleaned up. Also, the layout makes it a bit easier for you to identify what you can rely on and what is off-limits.

If you look at the top-level installation directory here are a few important areas to note:

That’s the quick tour of the 7.0 file layout (I didn’t cover every detail here, but enough to get you started). Download the bits and play with them to see what else is new.

I should point out that while the interface details I’ve given here are unlikely to change significantly from now until the product is released, they may of course change. The current preview build is just a preview, after all, still subject to change. For the same reason, I didn’t write down precise interface levels, just a general description of each area. The formal product documentation which will be available when the production releasecomes out will contain the authoritative interface information.

RFC 4492 has just recently been released, covering ECC support in TLS. A timely coincidence, given today’s announcement of the JES Web Server 7 technology preview availability, containing the ECC support I’ve talked about earlier.

Earlier I’ve talked about various ways of observing and debugging SSL connections to the web server, such as acting as a client and acting as a proxy.

Both of these methods have their advantages and disadvantages. Today I’ll review using ssldump, which combines some of the advantages of both of the previous tools: you can observe real requests from a real client and you can also observe the actual application data (HTTP layer). The only drawback is that there is a bit of additonal setup work to do before this will work, but it is definitely worth it.

Unlike ssltap, ssldump observes network traffic directly. So it is not necessary to proxy the client requests through it, just point it at a network interface where the traffic of interest is visible. For the easiest use case you can simply run:

# ssldump -i bge0 -d port 8088
New TCP connection #1: myclient(34286) <-> myserver(8088)
1 1  0.0265 (0.0265)  C>S  Handshake
      ClientHello
        Version 3.0
        resume [32]=
          08 22 e7 bc cf 13 e7 7f 80 0d 62 43 24 4c 65 5b
          1e 19 69 ab 3c 51 0e 95 29 d9 79 9d 9f 79 04 92
        cipher suites
        Unknown value 0x39
        Unknown value 0x38
        Unknown value 0x35
        Unknown value 0x33
        Unknown value 0x32
        SSL_RSA_WITH_RC4_128_MD5
        SSL_RSA_WITH_RC4_128_SHA
        Unknown value 0x2f
        SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        Unknown value 0xfeff
        SSL_RSA_WITH_3DES_EDE_CBC_SHA
        SSL_DHE_DSS_WITH_DES_CBC_SHA
        Unknown value 0xfefe
        compression methods
                  NULL
1 2  0.0273 (0.0007)  S>C  Handshake
      ServerHello
        Version 3.0
        session_id[32]=
          08 22 e7 bc cf 13 e7 7f 80 0d 62 43 24 4c 65 5b
          1e 19 69 ab 3c 51 0e 95 29 d9 79 9d 9f 79 04 92
        cipherSuite         SSL_RSA_WITH_RC4_128_MD5
        compressionMethod                   NULL
1 3  0.0273 (0.0000)  S>C  ChangeCipherSpec
1 4  0.0273 (0.0000)  S>C  Handshake
1 5  0.0635 (0.0362)  C>S  ChangeCipherSpec
1 6  0.0635 (0.0000)  C>S  Handshake
1 7  0.0635 (0.0000)  C>S  application_data
1 8  0.0643 (0.0008)  S>C  application_data
1 9  30.1176 (30.0532)  S>C  Alert

Note that I ran ssldump as root since it needs to listen to the traffic on the network device bge0 (substitute the correct device for your system here – check with ifconfig if in doubt). My SSL-enabled web server listener is on port 8088 for this example, so I limited ssldump to tracking traffic on that port (I could’ve said “dst port 8088” as well), otherwise it would’ve dumped all network traffic on that device, making it difficult to follow the output.

Also check the first column in the output, which above is always “1”. Since ssldump observes the network traffic there may be multiple requests going to the server at once. Above there was only one:

New TCP connection #1: myclient(34286) <-> myserver(8088)

In the case where there are multiple connections observed, each will be numbered. The packets for each connection will most likely be interleaved in the output, but you can track each one by correlating these numbers. That can be quite useful – I’ve used ssldump to diagnose some SSL connectivity problems which were only reproducible when multiple concurrent requests hit the server all at once.

The example above is nice in that I didn’t have to proxy my browser traffic through it, I could just connect to my server as usual. But aside from that it is not so interesting. The output isn’t really any more useful than ssldump.

So, let’s make things more interesting.

If given access to the server private key, ssldump can decrypt the traffic to and from that server on the fly. That’s where it gets really useful. We’ll need to do a bit of prep work to set this up.

First, extract the private key from the server instance into a PKCS#12 format file using pk12util.

• You’ll need to know the nickname of the server keypair/cert (see your server.xml) for the -n parameter.
• I changed to the directory where the NSS *.db files live so I type “-d .”. Alternatively you could run the command from elsewhere by giving the right path.
• Finally, I am runing JES Web Server 7.0 so there is no prefix to the NSS files, but if you are on 6.1 you’ll need to give a -P parameter with the right prefix for that instance.

% pk12util -o myserver.pk12 -n Server-Cert -d . -v
Enter Password or Pin for "NSS Certificate DB":
Enter password for PKCS12 file:
Re-enter password:
pk12util: PKCS12 EXPORT SUCCESSFUL

Perhaps needless to say, but if this was a production server I’d have to be very careful where I store this private key file. Keep that in mind.

Next I’ll just convert this to a format suitable for ssldump using openssl:

% openssl pkcs12 -in myserver.pk12 -out myserverkey
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

Ok.. now lets run ssldump and give it access to the server key material (the “PEM pass phrase” you need to type in below is the same you gave it above):

# ssldump -i bge0 -d -k /tmp/myserverkey port 8088
Enter PEM pass phrase:

Then I connect to my server from a browser:

New TCP connection #1: laptop(39699) <-> myserver(8088)
1 1  0.0853 (0.0853)  C>S SSLv2 compatible client hello
  Version 3.1
  cipher suites
  Unknown value 0x39
  Unknown value 0x38
  Unknown value 0x35
  Unknown value 0x33
  Unknown value 0x32
  TLS_RSA_WITH_RC4_128_MD5
  TLS_RSA_WITH_RC4_128_SHA
  Unknown value 0x2f
  TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
  TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
  Unknown value 0xfeff
  TLS_RSA_WITH_3DES_EDE_CBC_SHA
  TLS_DHE_RSA_WITH_DES_CBC_SHA
  TLS_DHE_DSS_WITH_DES_CBC_SHA
  Unknown value 0xfefe
  TLS_RSA_WITH_DES_CBC_SHA
  TLS_RSA_EXPORT1024_WITH_RC4_56_SHA
  TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA
  TLS_RSA_EXPORT_WITH_RC4_40_MD5
  TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
1 2  0.0856 (0.0002)  S>C  Handshake
      ServerHello
        Version 3.1
        session_id[32]=
          08 22 c2 25 34 4e 85 61 dd 24 ba 9a 59 a2 dc b0
          77 a0 3f b7 ac c9 d3 ce 76 4a b5 42 cc 44 30 fb
        cipherSuite         TLS_RSA_WITH_RC4_128_MD5
        compressionMethod                   NULL
      Certificate
      ServerHelloDone
1 3  6.1870 (6.1013)  C>S  Handshake
      ClientKeyExchange
1 4  6.1870 (0.0000)  C>S  ChangeCipherSpec
1 5  6.1870 (0.0000)  C>S  Handshake
      Finished
1 6  6.1931 (0.0061)  S>C  ChangeCipherSpec
1 7  6.1931 (0.0000)  S>C  Handshake
      Finished
1 8  6.2852 (0.0921)  C>S  application_data
    ---------------------------------------------------------------
    GET / HTTP/1.1
    Host: myserver:8088
    User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.5)
 Gecko/20041217
    Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/
plain;q=0.8,image/png,*/*;q=0.5
    Accept-Language: en-us,en;q=0.5
    Accept-Encoding: gzip,deflate
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    Keep-Alive: 300
    Connection: keep-alive

    ---------------------------------------------------------------
1 9  6.2859 (0.0007)  S>C  application_data
    ---------------------------------------------------------------
    HTTP/1.1 403 Forbidden
    Server: Sun-Java-System-Web-Server/7.0
    Date: Thu, 27 Apr 2006 08:12:15 GMT
    Content-length: 142
    Content-type: text/html

    Forbidden


Forbidden



    Your client is not allowed to access the requested object.
    --------------------------------------------------------------
-
1 10 12.0328 (5.7468)  C>S  Alert
    level           warning
    value           close_notify
1    12.0360 (0.0032)  C>S  TCP FIN
1    12.0361 (0.0000)  S>C  TCP FIN

Nice! Now I can watch all the application data traffic in plaintext. As you can imagine, this can be very useful for diagnosing all kinds of problems with an SSL-enabled server.

As a final note, when building ssldump be sure to build it with openssl support or the traffic decryption will not work. Here’s the configure options I gave it on my Solaris 10 machine:

./configure --with-pcap=/opt/sfw --with-openssl=/usr/sfw

Earlier I talked about issuing test requests to a SSL/TLS-enabled web server.

That works well as long as you can recreate the request contents by hand. But sometimes you really need to diagnose a connection which is being issued by some real client (such as a browser or client app). There are a number of ways to observe such request and response data.

One such tool is ssltap, also part of NSS. ssltap takes the proxy approach – it serves as a simple proxy between the client and the real server and displays information about the connections it forwards (note that you can just as well use ssltap for observing plain HTTP requests or even requests based on other protocols).

Let’s look at an example. I have a JES Web Server with an SSL-enabled listener on port 8088 of my machine, so I issue the following (you’ll always want the -l option so ssltap doesn’t exit after a single request):

% ssltap  -l -s localhost:8088
Looking up "localhost"...
Proxy socket ready and listening

By default ssltap listens on port 1924 (run ssltap without any options to see usage info) which will do fine. I then connect from firefox to ‘https://localhost:1924’ which produces ssltap output such as this:

Connection #1 [Mon Apr 10 15:49:49 2006]
Connected to localhost:8088
--> [
alloclen = 87 bytes
(87 bytes of 87)
 [Mon Apr 10 15:49:49 2006] [ssl2]  ClientHelloV2 {
           version = {0x03, 0x01}
           cipher-specs-length = 60 (0x3c)
           sid-length = 0 (0x00)
           challenge-length = 16 (0x10)
           cipher-suites = {
                (0x000039) TLS/DHE-RSA/AES256-CBC/SHA
                (0x000038) TLS/DHE-DSS/AES256-CBC/SHA
                (0x000035) TLS/RSA/AES256-CBC/SHA
                (0x000033) TLS/DHE-RSA/AES128-CBC/SHA
                (0x000032) TLS/DHE-DSS/AES128-CBC/SHA
                (0x000004) SSL3/RSA/RC4-128/MD5
                (0x000005) SSL3/RSA/RC4-128/SHA
                (0x00002f) TLS/RSA/AES128-CBC/SHA
                (0x000016) SSL3/DHE-RSA/3DES192EDE-CBC/SHA
                (0x000013) SSL3/DHE-DSS/DES192EDE3CBC/SHA
                (0x00feff) SSL3/RSA-FIPS/3DESEDE-CBC/SHA
                (0x00000a) SSL3/RSA/3DES192EDE-CBC/SHA
                (0x000015) SSL3/DHE-RSA/DES56-CBC/SHA
                (0x000012) SSL3/DHE-DSS/DES56-CBC/SHA
                (0x00fefe) SSL3/RSA-FIPS/DES-CBC/SHA
                (0x000009) SSL3/RSA/DES56-CBC/SHA
                (0x000064) TLS/RSA-EXPORT1024/RC4-56/SHA
                (0x000062) TLS/RSA-EXPORT1024/DES56-CBC/SHA
                (0x000003) SSL3/RSA/RC4-40/MD5
                (0x000006) SSL3/RSA/RC2CBC40/MD5
                }
           session-id = { }
           challenge = { 0xdfb5 0x1d22 0x6562 0x34f6 0x95b9 0x668a 0x234e 0x38ea
 }
}
]

This is the SSL client hello being sent from the browser to the server. Of interest, note the list of cipher suites the browser sent. This is the set of cipher suites the browser is configured to handle (note also they are sorted in order of preference). The server will pick one of those for the handshake (if the server is not set up to handle any of these, the connection will then immediately fail). Notice also the session-id is empty, which tells us the browser does not have any cached SSL session (not to be confused with HTTP sessions!) with this particular server.

The server responds with:

<-- [
(1015 bytes of 1010)
SSLRecord { [Mon Apr 10 15:49:49 2006]
   type    = 22 (handshake)
   version = { 3,1 }
   length  = 1010 (0x3f2)
   handshake {
      type = 2 (server_hello)
      length = 70 (0x000046)
         ServerHello {
            server_version = {3, 1}
            random = {...}
            session ID = {
                length = 32
                contents = {..}
            }
            cipher_suite = (0x0035) TLS/RSA/AES256-CBC/SHA
         }
      type = 11 (certificate)
      length = 928 (0x0003a0)
         CertificateChain {
            chainlength = 925 (0x039d)
            Certificate {
               size = 485 (0x01e5)
               data = { saved in file 'cert.001' }
            }
            Certificate {
               size = 434 (0x01b2)
               data = { saved in file 'cert.002' }
            }
         }
      type = 14 (server_hello_done)
      length = 0 (0x000000)
   }
}
]

The server picked TLS/RSA/AES256-CBC/SHA as the cipher suite to use (JES Web Server 7.0 supports AES in addition to ECC).

A ‘session ID’ was sent, which this client will include in subsequent requests. The server also sent its certificate chain for the browser to verify. ssltap saved these certificates in the files noted (‘cert.001’, ‘cert.002’). You can examine these certificates later with any tool which can parse X.509 certificates. For example, using openssl you could do:

% openssl x509 -in cert.001 -text -inform DER

I will skip the rest of the ssltap output as there is a lot of it, but try it and see. You can observe a fair amount of useful information from it such as the number of requests and responses and the size of each.

The major drawback is that the actual session data is of course encrypted between the client and the server so ssltap has no way to decipher it (with the -h flag ssltap will give a hex dump of the raw data). Thus, while this is useful for observing the handshake and request progress, it won’t help if you needed to look at the request or response details.

One option here is to enable a NULL cipher suite on both the client and the server. The NULL suites are SSL/TLS suites which perform all the regular protocol steps with the only difference that the bulk encryption algorithm used to encrypt the request/response data is ‘NULL’ – that is, it goes in plain text.

As noted earlier, the handshake protocol attempts to negotiate the strongest cipher suite that is common to the client and the server, so the NULL suites will never be selected unless it is the only choice. That means that in order to make it work you will have the explicitly disable every cipher suite except a NULL one in either the client or the server.

To show this scenario, I disabled all suites in the JES Web Server 7.0 except SSL_RSA_WITH_NULL_SHA and also enabled the corresponding suite in firefox (unfortunately firefox has no UI to configure these but you can do it by entering ‘about:config’ as the URL; search for the ‘security.ssl3’ options).

With that, the initial client hello containing the list of ciphers shows up through ssltap as:

            cipher_suites[22] = {
                (0x0039) TLS/DHE-RSA/AES256-CBC/SHA
                (0x0038) TLS/DHE-DSS/AES256-CBC/SHA
                (0x0035) TLS/RSA/AES256-CBC/SHA
                (0x0033) TLS/DHE-RSA/AES128-CBC/SHA
                (0x0032) TLS/DHE-DSS/AES128-CBC/SHA
                (0x0004) SSL3/RSA/RC4-128/MD5
                (0x0005) SSL3/RSA/RC4-128/SHA
                (0x002f) TLS/RSA/AES128-CBC/SHA
                (0x0016) SSL3/DHE-RSA/3DES192EDE-CBC/SHA
                (0x0013) SSL3/DHE-DSS/DES192EDE3CBC/SHA
                (0xfeff) SSL3/RSA-FIPS/3DESEDE-CBC/SHA
                (0x000a) SSL3/RSA/3DES192EDE-CBC/SHA
                (0x0015) SSL3/DHE-RSA/DES56-CBC/SHA
                (0x0012) SSL3/DHE-DSS/DES56-CBC/SHA
                (0xfefe) SSL3/RSA-FIPS/DES-CBC/SHA
                (0x0009) SSL3/RSA/DES56-CBC/SHA
                (0x0064) TLS/RSA-EXPORT1024/RC4-56/SHA
                (0x0062) TLS/RSA-EXPORT1024/DES56-CBC/SHA
                (0x0003) SSL3/RSA/RC4-40/MD5
                (0x0006) SSL3/RSA/RC2CBC40/MD5
                (0x0002) SSL3/RSA/NULL/SHA
                (0x0001) SSL3/RSA/NULL/MD5

The NULL ciphers are of course at the very bottom of the list. The server, having nothing else enabled, responds with:

...
           cipher_suite = (0x0002) SSL3/RSA/NULL/SHA
...

And so, the application data of this session is now unencrypted. Without the -h option ssltap will still show the data records as ‘< encrypted >’ because it doesn’t know any better (remember that all the regular SSL protocol encapsulation is in place even though the bulk encryption algorithm happens to be a no-op). If you give the -h option you’ll see the hex dump of what is now the readable request/response data. Of course, being plan text, now you can use your favorite network observation tool instead.

If you do this, be absolutely sure to remember restore the original configuration afterwards! Disable all NULL cipher suites and reenable the ones you need. You never want to have the NULL suites enabled outside of special testing circumstances (JES Web Server 7.0 will log warnings if weak suites are enabled, as a reminder).

A final note about ssltap – it is a single threaded proxy so if you issue multiple requests through it they will get serialized. Usually not a problem, but if you need to analyze some problem with your application which only occurs on concurrent requests through SSL, it will not help (try running multiple ssltap instances).

This entry gives a short walkthrough of the steps that occur in JES Web Server when it needs to authenticate and authorize a client (say, browser) request using an LDAP server. Understanding what goes on behind the scenes makes it easier to diagnose configuration issues.

In this example the server is configured with an ACL which requires LDAP authentication such as the one below which states that access to /index.html is limited to those users who authenticate correctly and are part of the group webappusers in the ldap directory.

acl "uri=/index.html";
authenticate (user,group) {
  database = "myldap";
  prompt = "ldap authentication";
};
deny (all) (user = "all");
allow (all) (group = "webappusers");

The database ‘myldap’ referenced above is defined in dbswitch.conf where the directory URL and base DN are configured:

directory myldap ldap://ldapserver:389/dc=virkki,dc=com

I turned on verbose logging in my LDAP server. If you have access to configure the LDAP server you’re authenticating against I recommend doing the same as it will make it easier to follow what is going on (and if you don’t have such access, consider setting up a test LDAP server of your own for this purpose).

I access /index.html through a browser. As expected, I’m prompted for authentication. I entered my username ‘jvirkki’ and my password. Here’s what I find in the LDAP server logs afterwards (omitting some repetitive fields to cut the clutter):

op=0 BIND dn="" method=128 version=3
op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn=""
op=1 SRCH base="dc=virkki,dc=com" scope=2 filter="(uid=jvirkki)" attrs="c"
op=1 RESULT err=0 tag=101 nentries=1 etime=0
op=2 BIND dn="uid=jvirkki,ou=People, dc=virkki,dc=com" method=128 version=3
op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=jvirkki,ou=people,dc=virkki,
dc=com"
op=3 BIND dn="" method=128 version=3
op=3 RESULT err=0 tag=97 nentries=0 etime=0 dn=""
op=4 SRCH base="dc=virkki,dc=com" scope=2 filter="(|(&(objectClass=groupofunique
names)(|(uniqueMember=uid=jvirkki,ou=People, dc=virkki,dc=com)))(&(objectClass=g
roup)(|(member=uid=jvirkki,ou=People, dc=virkki,dc=com)))(&(objectClass=groupofn
ames)(|(member=uid=jvirkki,ou=People, dc=virkki,dc=com))))" attrs="cn"
op=4 RESULT err=0 tag=101 nentries=1 etime=0
op=5 SRCH base="dc=virkki,dc=com" scope=2 filter="(&(cn=webappusers)(|(objectCla
ss=groupofuniquenames)(objectClass=groupofnames)(objectClass=groupofurls)(object
Class=groupofcertificates)(objectClass=group)))" attrs="c"
op=5 RESULT err=0 tag=101 nentries=1 etime=0

Now, let’s duplicate the steps done by the web server using ldapsearch(1).

In the interest of space I will only go through the succesful request presented above. However, I encourage you to try several failure cases at every step of the way (wrong user, wrong password, wrong group) both via ldapsearch as well as through your browser. Trying both the success and failure cases is quite useful for becoming familiar with what is going on.

The first thing that happens after the browser transmitted my user name ‘jvirkki’ and my password to the web server is that the web server needs to find if such a user even exists. If the client sent a non-existing user there is no need to do anything else – if the user does not exist it will certainly not be able to authenticate nor will it belong to any group (additionally, in order to bind as this user in the next step the server needs to know the full DN of the LDAP entry corresponding to the user, not just the uid which at this point is all it has).

To do this a search is issued to find ‘uid=jvirkki’, which corresponds with the lines of op=1 in the LDAP logs above. We can do the same from the CLI:

% ldapsearch -h ldapserver -p 389 -b "dc=virkki,dc=com"  "(uid=jvirkki)"
version: 1
dn: uid=jvirkki,ou=People, dc=virkki,dc=com
uid: jvirkki
givenName: jyri
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetorgperson
sn: virkki
cn: jyri virkki

As the output shows, the user named jvirkki was found and the LDAP entry is “uid=jvirkki,ou=People, dc=virkki,dc=com”. By default ldapsearch displayed all the attributes of the entry.

A few notes about the ldapsearch issued: Note that the value of the basedn, given by the -b option, must be the same as the basedn configured for this directory in the corresponding dbswitch.conf entry. The last argument to ldapsearch is the search filter, here, “(uid=jvirkki)”. Notice that this corresponds to the ‘filter’ string found in the LDAP server logs above for op=1.

Minor note: If you look at the LDAP logs at this stage you may notice a small difference, the log entry corresponding to the manual ldapsearch query reads ‘attrs=ALL’ whereas the log entry corresponding to the web server query reads ‘attrs=”c”‘. You could duplicate this
precisely by issuing the following ldapsearch command instead:

% ldapsearch -h ldapserver -p 389 -b "dc=virkki,dc=com"  "(uid=jvirkki)" c
version: 1
dn: uid=jvirkki,ou=People, dc=virkki,dc=com

Ok now that the web server knows the user exists it needs to verify the password. This corresponds to the log entries op=2 seen earlier. You can duplicate this from ldapsearch by giving the -D option which tells ldapsearch to attempt to bind (authenticate) as the given user and then entering the same password you entered into the browser:

% ldapsearch -h ldapserver -p 389 -b "dc=virkki,dc=com" -D "uid=jvirkki,ou=Peopl
e, dc=virkki,dc=com"  "(uid=jvirkki)" c
Enter bind password:
version: 1
dn: uid=jvirkki,ou=People, dc=virkki,dc=com

All right.. so the user exists and the password is correct. If the ACL permitted this user to perform the request, that would be all that is needed. However, the ACL I set earlier requires the user to belong to the group ‘webappusers’. So that’s the next step.

Look at the log entry for op=4 above. The web server queries to see if there is any group where this user is a member and it wants to find the name (cn) of such a group (if any). Copying the filter line from the op=4 log entry to the ldapsearch command:

% ldapsearch -h ldapserver -p 389 -b "dc=virkki,dc=com"  "(|(&(objectClass=group
ofuniquenames)(|(uniqueMember=uid=jvirkki,ou=People, dc=virkki,dc=com)))(&(objec
tClass=group)(|(member=uid=jvirkki,ou=People, dc=virkki,dc=com)))(&(objectClass=
groupofnames)(|(member=uid=jvirkki,ou=People, dc=virkki,dc=com))))" cn
version: 1
dn: cn=webappusers,ou=Groups, dc=virkki,dc=com
cn: webappusers

A group entry ‘cn=webappusers,ou=Groups, dc=virkki,dc=com’ was identified. Next the web server attempts to retrieve the LDAP entry corresponding to this name. I’ll reproduce the query once again by copying the filter line, from op=5, to ldapsearch:

% ldapsearch -h ldapserver -p 389 -b "dc=virkki,dc=com"  "(&(cn=webappusers)(|(o
bjectClass=groupofuniquenames)(objectClass=groupofnames)(objectClass=groupofurls
)(objectClass=groupofcertificates)(objectClass=group)))"
version: 1
dn: cn=webappusers,ou=Groups, dc=virkki,dc=com
description: users who have access to the web applications
objectClass: top
objectClass: groupofuniquenames
cn: webappusers
uniqueMember: uid=jvirkki,ou=People, dc=virkki,dc=com

There’s the group and “uid=jvirkki,ou=People, dc=virkki,dc=com” is a member (see uniqueMember attribute) of it. The ACL requirements have been satisfied and the index.html page is sent to the browser.

If you compare the LDAP server logs between the web server and manual runs (you should), you’ll probably notice a few differences. These don’t change the results we’re interested in that I discussed above. Here’s some of the differences:

• Each ldapsearch CLI invocation needs to connect and bind, run the
  search and then disconnect. So you’ll see a lot more BIND/UNBIND
  events occurring when running ldapsearch. The web server reuses
  connections so it does not need to constantly connect and disconnect.
• ldapsearch is meant to (surprise!) run a search, so some search filter needs to be given. Even when I only care to check the password (in step 2),
  I gave ldapsearch a search filter to run. When the web server checks
  the password it can issue only a BIND so the logs will not show this
  extra search.
• The web server runs all the searches under the identity of the default
  user (if any) given for this LDAP server in dbswitch.conf (see
  binddn/bindpw options in http://docs.sun.com/source/817-6248/crflothr.html#wp1026118). Since the web server reuses connections, after binding as the remote user (‘jvirkki’, in my example) to verify the password, it will re-bind as the binddn user to restore the correct identity for subsequent searches. Since ldapsearch creates a new connect ion every time, this is not needed. Since the myldap directory configuration does not include binddn/bindpw, anonymous bind is used whenever needed (corresponding to the BIND dn=”” entried in the logs).

Hopefully this walk-through helps you visualize what goes on when the web server uses an LDAP directory for authentication and access control. As we’ve seen the steps are quite straightforward and they can easily be reproduced via command line LDAP tools. If you ever need to diagnose such installations it is very helpful to run these steps to observe what is going on:

• Turn on verbose logging, both on the web and LDAP servers.
• In the LDAP server logs, observe what requests are being sent.
• Reproduce the same queries via ldapsearch. If you can not find or
  authenticate or check the group info via ldapsearch, it’s not going to
  work via the web server either.
• If nothing seems to work, run ldapsearch with the search filter
  “(objectclass=*)”. Every LDAP entry has an objectclass attribute and
  by matching all objectclass values, you will get a list of every visible LDAP
  entry (under the given base DN). If that output doesn’t contain the entries
  you were looking for, either the basedn (-b) is wrong, the LDAP server doesn’t
  really contain any of the data you thought it did or maybe the LDAP server
  is not configured to allow access to that data given the user (-D) you’re using.

Often while diagnosing problems between web servers and clients it is useful to observe the requests and responses that take place between the server and some known client (such as web browsers) and/or to directly act as a client and send handcrafted requests for testing.

As long as SSL/TLS is not in use this is quite easy. All you need to act as a client is telnet. For capturing requests & responses between another client and the server you can use any network capture tool such as ethereal wireshark.

When SSL/TLS is used it becomes a bit more difficult since it’s no longer possible to simply telnet to the HTTP port nor to observe the traffic with the most common tools.

The good news is that there are many tools that provide equivalent functionality, it just seems that these are less well known so I’m often asked how to diagnose anything when SSL is in use. I’ll cover a few of the ones I use most in this and later blog entries.

I’ll look at the telnet replacements first. Most often I use openssl s_client
or sometimes the tstclnt from NSS (while most of the NSS tools can be found in the SUNWtlsu package for Solaris, tstclnt is not there).

Starting with openssl, check the available options (showing the path in Solaris here):

% /usr/sfw/bin/openssl s_client -h

If all you need is a SSL-enabled ‘telnet replacement’ and are not looking for info about the SSL connection itself simply run the following and (assuming a successful connection) enter the HTTP request as desired.

% openssl s_client -host localhost -port 8080 -quiet

Using the same command above but omitting the -quiet flag you can see information about the connection, such as the server DN and certificate, negotiated cipher suite and so forth. There are additional flags such as -showcerts, -debug and -msg which will produce additional output.

For testing particular cipher suites, check the -cipher option for tuning the list of ciphers openssl will negotiate with the server. For example:

% openssl s_client -host localhost -port 8080  -cipher DES-CBC-SHA

Be sure to check the manpage for s_client for additional details.

As I mentioned, NSS also has a similar tool, called tstclnt. Run the command with no options for a description of all options.

% tstclnt
Usage:  tstclnt -h host [-p port] [-d certdir] [-n nickname] [-23BTfosvx]
                   [-c ciphers] [-w passwd] [-q]
[.. additional output omitted ..]


% tstclnt -h localhost -p 8080

You may have to provide the -o option in order to be able to connect if the server certificate chain can’t be verified (such as for a self-signed cert). Similar to openssl s_client you can specify which specific cipher suites to negotiate.

So that’s it.. either of these tools will allow you to connect to an SSL/TLS-enabled web server and issue requests as well as obtain diagnostics about the SSL handshake itself.