Observing SSL Requests

Earlier I talked about issuing test requests to a SSL/TLS-enabled web server.

That works well as long as you can recreate the request contents by hand. But sometimes you really need to diagnose a connection which is being issued by some real client (such as a browser or client app). There are a number of ways to observe such request and response data.

One such tool is ssltap, also part of NSS. ssltap takes the proxy approach – it serves as a simple proxy between the client and the real server and displays information about the connections it forwards (note that you can just as well use ssltap for observing plain HTTP requests or even requests based on other protocols).

Let’s look at an example. I have a JES Web Server with an SSL-enabled listener on port 8088 of my machine, so I issue the following (you’ll always want the -l option so ssltap doesn’t exit after a single request):

% ssltap  -l -s localhost:8088
Looking up "localhost"...
Proxy socket ready and listening

By default ssltap listens on port 1924 (run ssltap without any options to see usage info) which will do fine. I then connect from firefox to ‘https://localhost:1924’ which produces ssltap output such as this:

Connection #1 [Mon Apr 10 15:49:49 2006]
Connected to localhost:8088
--> [
alloclen = 87 bytes
(87 bytes of 87)
 [Mon Apr 10 15:49:49 2006] [ssl2]  ClientHelloV2 {
           version = {0x03, 0x01}
           cipher-specs-length = 60 (0x3c)
           sid-length = 0 (0x00)
           challenge-length = 16 (0x10)
           cipher-suites = {
                (0x000039) TLS/DHE-RSA/AES256-CBC/SHA
                (0x000038) TLS/DHE-DSS/AES256-CBC/SHA
                (0x000035) TLS/RSA/AES256-CBC/SHA
                (0x000033) TLS/DHE-RSA/AES128-CBC/SHA
                (0x000032) TLS/DHE-DSS/AES128-CBC/SHA
                (0x000004) SSL3/RSA/RC4-128/MD5
                (0x000005) SSL3/RSA/RC4-128/SHA
                (0x00002f) TLS/RSA/AES128-CBC/SHA
                (0x000016) SSL3/DHE-RSA/3DES192EDE-CBC/SHA
                (0x000013) SSL3/DHE-DSS/DES192EDE3CBC/SHA
                (0x00feff) SSL3/RSA-FIPS/3DESEDE-CBC/SHA
                (0x00000a) SSL3/RSA/3DES192EDE-CBC/SHA
                (0x000015) SSL3/DHE-RSA/DES56-CBC/SHA
                (0x000012) SSL3/DHE-DSS/DES56-CBC/SHA
                (0x00fefe) SSL3/RSA-FIPS/DES-CBC/SHA
                (0x000009) SSL3/RSA/DES56-CBC/SHA
                (0x000064) TLS/RSA-EXPORT1024/RC4-56/SHA
                (0x000062) TLS/RSA-EXPORT1024/DES56-CBC/SHA
                (0x000003) SSL3/RSA/RC4-40/MD5
                (0x000006) SSL3/RSA/RC2CBC40/MD5
                }
           session-id = { }
           challenge = { 0xdfb5 0x1d22 0x6562 0x34f6 0x95b9 0x668a 0x234e 0x38ea
 }
}
]

This is the SSL client hello being sent from the browser to the server. Of interest, note the list of cipher suites the browser sent. This is the set of cipher suites the browser is configured to handle (note also they are sorted in order of preference). The server will pick one of those for the handshake (if the server is not set up to handle any of these, the connection will then immediately fail). Notice also the session-id is empty, which tells us the browser does not have any cached SSL session (not to be confused with HTTP sessions!) with this particular server.

The server responds with:

<-- [
(1015 bytes of 1010)
SSLRecord { [Mon Apr 10 15:49:49 2006]
   type    = 22 (handshake)
   version = { 3,1 }
   length  = 1010 (0x3f2)
   handshake {
      type = 2 (server_hello)
      length = 70 (0x000046)
         ServerHello {
            server_version = {3, 1}
            random = {...}
            session ID = {
                length = 32
                contents = {..}
            }
            cipher_suite = (0x0035) TLS/RSA/AES256-CBC/SHA
         }
      type = 11 (certificate)
      length = 928 (0x0003a0)
         CertificateChain {
            chainlength = 925 (0x039d)
            Certificate {
               size = 485 (0x01e5)
               data = { saved in file 'cert.001' }
            }
            Certificate {
               size = 434 (0x01b2)
               data = { saved in file 'cert.002' }
            }
         }
      type = 14 (server_hello_done)
      length = 0 (0x000000)
   }
}
]

The server picked TLS/RSA/AES256-CBC/SHA as the cipher suite to use (JES Web Server 7.0 supports AES in addition to ECC).

A ‘session ID’ was sent, which this client will include in subsequent requests. The server also sent its certificate chain for the browser to verify. ssltap saved these certificates in the files noted (‘cert.001’, ‘cert.002’). You can examine these certificates later with any tool which can parse X.509 certificates. For example, using openssl you could do:

% openssl x509 -in cert.001 -text -inform DER

I will skip the rest of the ssltap output as there is a lot of it, but try it and see. You can observe a fair amount of useful information from it such as the number of requests and responses and the size of each.

The major drawback is that the actual session data is of course encrypted between the client and the server so ssltap has no way to decipher it (with the -h flag ssltap will give a hex dump of the raw data). Thus, while this is useful for observing the handshake and request progress, it won’t help if you needed to look at the request or response details.

One option here is to enable a NULL cipher suite on both the client and the server. The NULL suites are SSL/TLS suites which perform all the regular protocol steps with the only difference that the bulk encryption algorithm used to encrypt the request/response data is ‘NULL’ – that is, it goes in plain text.

As noted earlier, the handshake protocol attempts to negotiate the strongest cipher suite that is common to the client and the server, so the NULL suites will never be selected unless it is the only choice. That means that in order to make it work you will have the explicitly disable every cipher suite except a NULL one in either the client or the server.

To show this scenario, I disabled all suites in the JES Web Server 7.0 except SSL_RSA_WITH_NULL_SHA and also enabled the corresponding suite in firefox (unfortunately firefox has no UI to configure these but you can do it by entering ‘about:config’ as the URL; search for the ‘security.ssl3’ options).

With that, the initial client hello containing the list of ciphers shows up through ssltap as:

            cipher_suites[22] = {
                (0x0039) TLS/DHE-RSA/AES256-CBC/SHA
                (0x0038) TLS/DHE-DSS/AES256-CBC/SHA
                (0x0035) TLS/RSA/AES256-CBC/SHA
                (0x0033) TLS/DHE-RSA/AES128-CBC/SHA
                (0x0032) TLS/DHE-DSS/AES128-CBC/SHA
                (0x0004) SSL3/RSA/RC4-128/MD5
                (0x0005) SSL3/RSA/RC4-128/SHA
                (0x002f) TLS/RSA/AES128-CBC/SHA
                (0x0016) SSL3/DHE-RSA/3DES192EDE-CBC/SHA
                (0x0013) SSL3/DHE-DSS/DES192EDE3CBC/SHA
                (0xfeff) SSL3/RSA-FIPS/3DESEDE-CBC/SHA
                (0x000a) SSL3/RSA/3DES192EDE-CBC/SHA
                (0x0015) SSL3/DHE-RSA/DES56-CBC/SHA
                (0x0012) SSL3/DHE-DSS/DES56-CBC/SHA
                (0xfefe) SSL3/RSA-FIPS/DES-CBC/SHA
                (0x0009) SSL3/RSA/DES56-CBC/SHA
                (0x0064) TLS/RSA-EXPORT1024/RC4-56/SHA
                (0x0062) TLS/RSA-EXPORT1024/DES56-CBC/SHA
                (0x0003) SSL3/RSA/RC4-40/MD5
                (0x0006) SSL3/RSA/RC2CBC40/MD5
                (0x0002) SSL3/RSA/NULL/SHA
                (0x0001) SSL3/RSA/NULL/MD5

The NULL ciphers are of course at the very bottom of the list. The server, having nothing else enabled, responds with:

...
           cipher_suite = (0x0002) SSL3/RSA/NULL/SHA
...

And so, the application data of this session is now unencrypted. Without the -h option ssltap will still show the data records as ‘< encrypted >’ because it doesn’t know any better (remember that all the regular SSL protocol encapsulation is in place even though the bulk encryption algorithm happens to be a no-op). If you give the -h option you’ll see the hex dump of what is now the readable request/response data. Of course, being plan text, now you can use your favorite network observation tool instead.

If you do this, be absolutely sure to remember restore the original configuration afterwards! Disable all NULL cipher suites and reenable the ones you need. You never want to have the NULL suites enabled outside of special testing circumstances (JES Web Server 7.0 will log warnings if weak suites are enabled, as a reminder).

A final note about ssltap – it is a single threaded proxy so if you issue multiple requests through it they will get serialized. Usually not a problem, but if you need to analyze some problem with your application which only occurs on concurrent requests through SSL, it will not help (try running multiple ssltap instances).

Posted in Sun

Diagnosing LDAP authentication from JES Web Server

This entry gives a short walkthrough of the steps that occur in JES Web Server when it needs to authenticate and authorize a client (say, browser) request using an LDAP server. Understanding what goes on behind the scenes makes it easier to diagnose configuration issues.

In this example the server is configured with an ACL which requires LDAP authentication such as the one below which states that access to /index.html is limited to those users who authenticate correctly and are part of the group webappusers in the ldap directory.

acl "uri=/index.html";
authenticate (user,group) {
  database = "myldap";
  prompt = "ldap authentication";
};
deny (all) (user = "all");
allow (all) (group = "webappusers");

The database ‘myldap’ referenced above is defined in dbswitch.conf where the directory URL and base DN are configured:

directory myldap ldap://ldapserver:389/dc=virkki,dc=com

I turned on verbose logging in my LDAP server. If you have access to configure the LDAP server you’re authenticating against I recommend doing the same as it will make it easier to follow what is going on (and if you don’t have such access, consider setting up a test LDAP server of your own for this purpose).

I access /index.html through a browser. As expected, I’m prompted for authentication. I entered my username ‘jvirkki’ and my password. Here’s what I find in the LDAP server logs afterwards (omitting some repetitive fields to cut the clutter):

op=0 BIND dn="" method=128 version=3
op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn=""
op=1 SRCH base="dc=virkki,dc=com" scope=2 filter="(uid=jvirkki)" attrs="c"
op=1 RESULT err=0 tag=101 nentries=1 etime=0
op=2 BIND dn="uid=jvirkki,ou=People, dc=virkki,dc=com" method=128 version=3
op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=jvirkki,ou=people,dc=virkki,
dc=com"
op=3 BIND dn="" method=128 version=3
op=3 RESULT err=0 tag=97 nentries=0 etime=0 dn=""
op=4 SRCH base="dc=virkki,dc=com" scope=2 filter="(|(&(objectClass=groupofunique
names)(|(uniqueMember=uid=jvirkki,ou=People, dc=virkki,dc=com)))(&(objectClass=g
roup)(|(member=uid=jvirkki,ou=People, dc=virkki,dc=com)))(&(objectClass=groupofn
ames)(|(member=uid=jvirkki,ou=People, dc=virkki,dc=com))))" attrs="cn"
op=4 RESULT err=0 tag=101 nentries=1 etime=0
op=5 SRCH base="dc=virkki,dc=com" scope=2 filter="(&(cn=webappusers)(|(objectCla
ss=groupofuniquenames)(objectClass=groupofnames)(objectClass=groupofurls)(object
Class=groupofcertificates)(objectClass=group)))" attrs="c"
op=5 RESULT err=0 tag=101 nentries=1 etime=0

Now, let’s duplicate the steps done by the web server using ldapsearch(1).

In the interest of space I will only go through the succesful request presented above. However, I encourage you to try several failure cases at every step of the way (wrong user, wrong password, wrong group) both via ldapsearch as well as through your browser. Trying both the success and failure cases is quite useful for becoming familiar with what is going on.

The first thing that happens after the browser transmitted my user name ‘jvirkki’ and my password to the web server is that the web server needs to find if such a user even exists. If the client sent a non-existing user there is no need to do anything else – if the user does not exist it will certainly not be able to authenticate nor will it belong to any group (additionally, in order to bind as this user in the next step the server needs to know the full DN of the LDAP entry corresponding to the user, not just the uid which at this point is all it has).

To do this a search is issued to find ‘uid=jvirkki’, which corresponds with the lines of op=1 in the LDAP logs above. We can do the same from the CLI:

% ldapsearch -h ldapserver -p 389 -b "dc=virkki,dc=com"  "(uid=jvirkki)"
version: 1
dn: uid=jvirkki,ou=People, dc=virkki,dc=com
uid: jvirkki
givenName: jyri
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetorgperson
sn: virkki
cn: jyri virkki

As the output shows, the user named jvirkki was found and the LDAP entry is “uid=jvirkki,ou=People, dc=virkki,dc=com”. By default ldapsearch displayed all the attributes of the entry.

A few notes about the ldapsearch issued: Note that the value of the basedn, given by the -b option, must be the same as the basedn configured for this directory in the corresponding dbswitch.conf entry. The last argument to ldapsearch is the search filter, here, “(uid=jvirkki)”. Notice that this corresponds to the ‘filter’ string found in the LDAP server logs above for op=1.

Minor note: If you look at the LDAP logs at this stage you may notice a small difference, the log entry corresponding to the manual ldapsearch query reads ‘attrs=ALL’ whereas the log entry corresponding to the web server query reads ‘attrs=”c”‘. You could duplicate this
precisely by issuing the following ldapsearch command instead:

% ldapsearch -h ldapserver -p 389 -b "dc=virkki,dc=com"  "(uid=jvirkki)" c
version: 1
dn: uid=jvirkki,ou=People, dc=virkki,dc=com

Ok now that the web server knows the user exists it needs to verify the password. This corresponds to the log entries op=2 seen earlier. You can duplicate this from ldapsearch by giving the -D option which tells ldapsearch to attempt to bind (authenticate) as the given user and then entering the same password you entered into the browser:

% ldapsearch -h ldapserver -p 389 -b "dc=virkki,dc=com" -D "uid=jvirkki,ou=Peopl
e, dc=virkki,dc=com"  "(uid=jvirkki)" c
Enter bind password:
version: 1
dn: uid=jvirkki,ou=People, dc=virkki,dc=com

All right.. so the user exists and the password is correct. If the ACL permitted this user to perform the request, that would be all that is needed. However, the ACL I set earlier requires the user to belong to the group ‘webappusers’. So that’s the next step.

Look at the log entry for op=4 above. The web server queries to see if there is any group where this user is a member and it wants to find the name (cn) of such a group (if any). Copying the filter line from the op=4 log entry to the ldapsearch command:

% ldapsearch -h ldapserver -p 389 -b "dc=virkki,dc=com"  "(|(&(objectClass=group
ofuniquenames)(|(uniqueMember=uid=jvirkki,ou=People, dc=virkki,dc=com)))(&(objec
tClass=group)(|(member=uid=jvirkki,ou=People, dc=virkki,dc=com)))(&(objectClass=
groupofnames)(|(member=uid=jvirkki,ou=People, dc=virkki,dc=com))))" cn
version: 1
dn: cn=webappusers,ou=Groups, dc=virkki,dc=com
cn: webappusers

A group entry ‘cn=webappusers,ou=Groups, dc=virkki,dc=com’ was identified. Next the web server attempts to retrieve the LDAP entry corresponding to this name. I’ll reproduce the query once again by copying the filter line, from op=5, to ldapsearch:

% ldapsearch -h ldapserver -p 389 -b "dc=virkki,dc=com"  "(&(cn=webappusers)(|(o
bjectClass=groupofuniquenames)(objectClass=groupofnames)(objectClass=groupofurls
)(objectClass=groupofcertificates)(objectClass=group)))"
version: 1
dn: cn=webappusers,ou=Groups, dc=virkki,dc=com
description: users who have access to the web applications
objectClass: top
objectClass: groupofuniquenames
cn: webappusers
uniqueMember: uid=jvirkki,ou=People, dc=virkki,dc=com

There’s the group and “uid=jvirkki,ou=People, dc=virkki,dc=com” is a member (see uniqueMember attribute) of it. The ACL requirements have been satisfied and the index.html page is sent to the browser.

If you compare the LDAP server logs between the web server and manual runs (you should), you’ll probably notice a few differences. These don’t change the results we’re interested in that I discussed above. Here’s some of the differences:

  • Each ldapsearch CLI invocation needs to connect and bind, run the
    search and then disconnect. So you’ll see a lot more BIND/UNBIND
    events occurring when running ldapsearch. The web server reuses
    connections so it does not need to constantly connect and disconnect.
  • ldapsearch is meant to (surprise!) run a search, so some search filter needs to be given. Even when I only care to check the password (in step 2),
    I gave ldapsearch a search filter to run. When the web server checks
    the password it can issue only a BIND so the logs will not show this
    extra search.
  • The web server runs all the searches under the identity of the default
    user (if any) given for this LDAP server in dbswitch.conf (see
    binddn/bindpw options in http://docs.sun.com/source/817-6248/crflothr.html#wp1026118). Since the web server reuses connections, after binding as the remote user (‘jvirkki’, in my example) to verify the password, it will re-bind as the binddn user to restore the correct identity for subsequent searches. Since ldapsearch creates a new connect ion every time, this is not needed. Since the myldap directory configuration does not include binddn/bindpw, anonymous bind is used whenever needed (corresponding to the BIND dn=”” entried in the logs).

Hopefully this walk-through helps you visualize what goes on when the web server uses an LDAP directory for authentication and access control. As we’ve seen the steps are quite straightforward and they can easily be reproduced via command line LDAP tools. If you ever need to diagnose such installations it is very helpful to run these steps to observe what is going on:

  • Turn on verbose logging, both on the web and LDAP servers.
  • In the LDAP server logs, observe what requests are being sent.
  • Reproduce the same queries via ldapsearch. If you can not find or
    authenticate or check the group info via ldapsearch, it’s not going to
    work via the web server either.
  • If nothing seems to work, run ldapsearch with the search filter
    “(objectclass=*)”. Every LDAP entry has an objectclass attribute and
    by matching all objectclass values, you will get a list of every visible LDAP
    entry (under the given base DN). If that output doesn’t contain the entries
    you were looking for, either the basedn (-b) is wrong, the LDAP server doesn’t
    really contain any of the data you thought it did or maybe the LDAP server
    is not configured to allow access to that data given the user (-D) you’re using.
Posted in Sun

Issuing test requests to an SSL-enabled web server

Often while diagnosing problems between web servers and clients it is useful to observe the requests and responses that take place between the server and some known client (such as web browsers) and/or to directly act as a client and send handcrafted requests for testing.

As long as SSL/TLS is not in use this is quite easy. All you need to act as a client is telnet. For capturing requests & responses between another client and the server you can use any network capture tool such as ethereal wireshark.

When SSL/TLS is used it becomes a bit more difficult since it’s no longer possible to simply telnet to the HTTP port nor to observe the traffic with the most common tools.

The good news is that there are many tools that provide equivalent functionality, it just seems that these are less well known so I’m often asked how to diagnose anything when SSL is in use. I’ll cover a few of the ones I use most in this and later blog entries.

I’ll look at the telnet replacements first. Most often I use openssl s_client
or sometimes the tstclnt from NSS (while most of the NSS tools can be found in the SUNWtlsu package for Solaris, tstclnt is not there).

Starting with openssl, check the available options (showing the path in Solaris here):

% /usr/sfw/bin/openssl s_client -h

If all you need is a SSL-enabled ‘telnet replacement’ and are not looking for info about the SSL connection itself simply run the following and (assuming a successful connection) enter the HTTP request as desired.

% openssl s_client -host localhost -port 8080 -quiet

Using the same command above but omitting the -quiet flag you can see information about the connection, such as the server DN and certificate, negotiated cipher suite and so forth. There are additional flags such as -showcerts, -debug and -msg which will produce additional output.

For testing particular cipher suites, check the -cipher option for tuning the list of ciphers openssl will negotiate with the server. For example:

% openssl s_client -host localhost -port 8080  -cipher DES-CBC-SHA

Be sure to check the manpage for s_client for additional details.

As I mentioned, NSS also has a similar tool, called tstclnt. Run the command with no options for a description of all options.

% tstclnt
Usage:  tstclnt -h host [-p port] [-d certdir] [-n nickname] [-23BTfosvx]
                   [-c ciphers] [-w passwd] [-q]
[.. additional output omitted ..]


% tstclnt -h localhost -p 8080

You may have to provide the -o option in order to be able to connect if the server certificate chain can’t be verified (such as for a self-signed cert). Similar to openssl s_client you can specify which specific cipher suites to negotiate.

So that’s it.. either of these tools will allow you to connect to an SSL/TLS-enabled web server and issue requests as well as obtain diagnostics about the SSL handshake itself.

Posted in Sun

ECC Support in JES WS 7.0

Recently I’ve been working on adding ECC (Elliptic Curve Cryptography) support for the upcoming JES Web Server 7.0. It was nice to see some coverage of this at the RSA Conference last week.

Sun had a press release describing the upcoming ECC support. I’ve also seen articles from news.com, infoworld.com, java.ittoolbox.com, governmententerprise.com, hardwarezone.com, vnunet.com, networkworld.com, eetimes.com, zdnet.com and internetnews.com.

You can watch the keynotes at the conference web site and the Sun Labs ECC page has pointers to technical information.

Posted in Sun

Java Servlet Access Control in JES Web Server 6.1

JES Web Server has a built-in Java Servlet engine for running Java web applications. When running Java webapps there are basically two choices for access control – ACL-based (or so-called native) or J2EE/Servlet-based. These choices are described in the product documentation.

I’ll start with a very simple Java webapp hello.war consisting of the only following content:

  ./WEB-INF/sun-web.xml
  ./WEB-INF/web.xml
  ./index.html

I want to limit access to the content (what little there is..) of this webapp. First I will use servlet container-based authentication (as defined by the Java Servlet specification, which is definitely recommended reading if you wish to work the Servlets). The access control requirements are specified in the web.xml file, mine contains:

  <security-constraint>
    <web-resource-collection>
      <web-resource-name>Protected Area</web-resource-name>
      <url-pattern>/*</url-pattern>
      <http-method>DELETE</http-method>
      <http-method>POST</http-method>
      <http-method>GET</http-method>
      <http-method>PUT</http-method>
    </web-resource-collection>
    <auth-constraint>
      <role-name>TestRole</role-name>
    </auth-constraint>
  </security-constraint>

  <login-config>
    <auth-method>BASIC</auth-method>
    <realm-name>file</realm-name>
  </login-config>
  <security-role>
    <role-name>TestRole</role-name>
  </security-role>

The key bit above is that I want the servlet container to use HTTP BASIC authentication as the <auth-method> and I want the client to be authenticated against the ‘file’ realm.

I should also point out that the role name TestRole must be mapped to one or more user(s) and/or group(s) in that realm (in this case, ‘file’). This mapping occurs in sun-web.xml, where I chose to map it to a single user, user1:

<security-role-mapping>
   <role-name>TestRole</role-name>
   <principal-name>user1</principal-name>
</security-role-mapping>

Going back to <realm-name> which has the value ‘file’; this must map to a realm which exists in the server’s configuration file server.xml. There I have an <AUTHREALM> entry with attribute name=’file’ (in fact, this is the default file auth realm entry, I didn’t modify it from the out-of-box setting):

<AUTHREALM name="file" classname="com.iplanet.ias.security.auth.realm.file.FileRealm">
  <PROPERTY name="file" value="/tmp/https-hostname/config/keyfile"/>
  <PROPERTY name="jaas-context" value="fileRealm"/>
</AUTHREALM>

I also created a user called user1 to populate the user list contained in /tmp/https-hostname/config/keyfile, since that is the user name I mapped to the role back in sun-web.xml (your server is probably not installed in /tmp.. I just used that as an example).

With that, I’m done. After deploying hello.war onto this server I can now access it only after succesfully authenticating as user1.

This is a minimal example that shows the basics, there are certainly more possibilities. If you’re interested in more please read the product documentation, in particular the section on securing Java web applications in addition to the Servlet specification previously mentioned. I also recommend reading an article I wrote some time back, Application Server 7 Access Control Guide for more background on this general topic. While that article focuses on a different product, the access control mechanism defined by the Java Servlet specification is standard and applies equally to Web Server 6.1. Moreover, while there are differences in syntax between the configuration files of the two products, the underlying logic is very similar.

Posted in Sun