Fixing an IBM Connections Social Mail CPU spike problem

The other day we did a test upgrade of our internal IBM Connections 4.5 environment from CR3 to CR4 before doing the real upgrade. After the upgrade the CPU of the WebSphere Application Server node (we are in a single node architecture) would spike to a 100%. After some digging and perusing of log files we narrowed the problem down to IBM Social Mail and that component being loaded. Actually even more specifically to the Discovery Servlet which is used to discover the mail service for a particular user. The issue appeared to be a hung thread as indicated by the below stacktrace. See highlight in bold.

[4/30/14 13:39:51:534 CEST] 00000040 ThreadMonitor W WSVR0605W: Thread "WebContainer : 5" (0000014b) has been
active for 770854 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.getBundle(DefaultClassLoader.java:273)
at org.apache.aries.jndi.Utils.getBundleContext(Utils.java:111)
at org.apache.aries.jndi.Utils.doGetBundleContext(Utils.java:99)
at org.apache.aries.jndi.Utils.access$100(Utils.java:43)
at org.apache.aries.jndi.Utils$1.run(Utils.java:68)
at org.apache.aries.jndi.Utils$1.run(Utils.java:66)
at java.security.AccessController.doPrivileged(AccessController.java:229)
at org.apache.aries.jndi.Utils.getBundleContext(Utils.java:66)
at org.apache.aries.jndi.OSGiInitialContextFactoryBuilder.getInitialContext(OSGiInitialContextFactoryBuilder.java:44)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:232)
at javax.naming.InitialContext.initializeDefaultInitCtx(InitialContext.java:318)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:348)
at javax.naming.InitialContext.internalInit(InitialContext.java:286)
at javax.naming.InitialContext.(InitialContext.java:211)
at javax.naming.directory.InitialDirContext.(InitialDirContext.java:91)
at com.ibm.social.pim.discovery.ldap.domino.DominoLDAPConnector.connect(DominoLDAPConnector.java:68)
at com.ibm.social.pim.discovery.services.domino.LDAPPersonData.findPerson(LDAPPersonData.java:43)
at com.ibm.social.pim.discovery.services.domino.LDAPPersonData.findPerson(LDAPPersonData.java:69)
at com.ibm.social.pim.discovery.services.domino.DominoMailServiceLDAPConnector.connect(DominoMailServiceLDAPConnector.java:69)
at com.ibm.social.pim.discovery.services.domino.DominoMailServiceLDAPConnector.connect(DominoMailServiceLDAPConnector.java:61)
at com.ibm.social.pim.discovery.DiscoveryServiceManager.findUserByEmail(DiscoveryServiceManager.java:163)
at com.ibm.social.pim.discovery.servlet.DiscoveryServlet.doDiscovery(DiscoveryServlet.java:229)
at com.ibm.social.pim.discovery.servlet.DiscoveryServlet.processRequest(DiscoveryServlet.java:198)
at com.ibm.social.pim.discovery.servlet.DiscoveryServlet.doGet(DiscoveryServlet.java:139)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)

We dug around a little without success so I reached out to a friend at IBM and the answer came back. This is an issue that has been seen before and is solved by fixpack 8 of IBM WebSphere Application Server so we upgraded to 8.0.0.8 and sure enough we are back up and running. Apparently fixpack 8 is now supported and actually reading through the detailed system requirements lists that (“IBM Connections 4.5 CR4 and above recommends WAS 8.0.0.8. WAS 8.0.0.6 with required fixes is still supported (see the detailed report for CR3) .”)

Thanks to friends at IBM.

Hiding the Social Mail username and password from socialmail-discovery-config.xml

It’s been bothering me a while that the username and password for our LDAP user was visible in clear text in our socialmail-discovery-config.xml. After going looking for a solution by using very specific searching I found a solution where you can hide the username and password and – stupid as I am – it’s actually right there in the install docs. Stupid is as stupid does. The solution is to remove the authentication data from the socialmail-discovery-config.xml and replace the <DirectoryUser> and <DirectoryPW> tags with a single <DirectoryAuthAlias> tag. This tag should reference a J2C alias configured in the Websphere ISC. Simple and effective.

The above solution is for Domino only – if you’re using Exchange mail you need two additional tags. For complete instructions on the tags refer to the wiki at Enabling the discovery service for IBM Connections Mail.