We had a small issue today that's taken the best part of the day to sort out. The issue was a weird problem where a Java applet worked absolutely fine on one server, but not on the other.
The setup was slightly different in the 2 machines, however, as the issue was to do with the client side, we weren't sure that this would make a difference.
Config1 used the following configuration:
CLIENT -> IIS 6.0(mod_jk) -> JBoss
Config2 used this configuration:
CLIENT -> Apache2(ReverseProxy) -> JBoss
Config1 is Windows 2003 and Config2 is Windows 2008 R2
We wanted to remove IIS as we're more comfortable with Apache, and we think it will run more smoothly (note that this is in a test environment, and we're not stupid enough to just go and do this on live!). The eventual goal is to remove the Windows server completely from the equation and run entirely on Linux (forgetting that one of the many parts to our company uses ASP.NET and isn't compatible with mono).
The symptom of the issue was that the applet would download, however, when it ran, there was a Java error in the console stating that the "Certificate Path" wasn't valid, or couldn't be determined. This was particularly strange as when we loaded other things from the proxy in the browser, there were no complaints about the certificate path.
I should add at this point that my company has taken a path that they only support IE7/8/9 and therefore the majority of testing and debugging is done in one of those browsers. This is important as you will see.
We tried to run everything through Fiddler to see if the certs were coming back differently, but for some reason, this was a non starter as nothing would run at all.
We tried to look at the certificates that were stored in Internet Explorer, but nothing seemed different.
Now, this is where the epiphany came. Internet Explorer uses the Certificate Stores from Windows, and I had a suspicion that the JRE used its own Certificate Stores, this means that the fact that a certificate works in Internet Explorer doesn't mean that it will work in the JRE.
This led to the next epiphany, Firefox also uses its own certificate stores. So I quickly installed Firefox and connected to the site over SSL and BOOM, there was the error in the browser for a standard page. This was the breakthrough we needed, it meant that Java wasn't the issue, it was Apache or the Server in some way.
We did find that if we accessed Config1 using this instance Firefox, then accessed Config2, Config2 started working (presumably because the certificate was installed at that point), so this confused us for a minute, but after finding the offending certificate in Firefox's store, removing it, and then clearing the browsing history, all was well again (well not, it was still broke, but you know what I mean).
The next step was to work out what the problem was. My thought was always that Apache was somehow not sending the certification path, or sending a wrong certificate so I started to pursue this avenue,
There was nothing online that I could find to tell me how to tell Apache2 to send the certification path.
I tried various things to try and create a certificate file that contained all the relevant paths, but just couldn't get it right.
It then occurred to a colleague that when we bought the Wildcard SSL certificate, there was a manual that came with it that specifically talked about installing in Apache.
This detailed an additional download of the CA File that is to be referenced by Apache that will cause the certification path to be sent.
So in short, we should have RTFM, if we had read the FM, then we wouldn't have spent the day looking for this solution to a problem that was already in the FM!
On the flip side, atleast now we know why it has to be done that way!
