IBM Connections application development state of the union – part 1

IBM Connections has been on the market for a lot of time now and has always been a real strong player when it comes to application development. I thought it was time to review where we are application development wise over what will probably be a couple of posts. First off is API’s…

IBM Connections is and has always been strong from the point of API’s – there is an API for almost all areas of the product and always has. I think IBM Connections was the first product (maybe still the only one) that was built by IBM Collaboration Solutions using an API first approach or at least with a strong emphasis on API’s.

The API’s are extensive and are pretty straight forward to use. The main caveat about the IBM Connections API’s is that they were designed a looooong time ago and hasn’t been updated since. I went to the documentation to check and the majority of the API’s hasn’t changed one bit since v. 2.5. This means they are still using the once cool Atom publishing protocol using lots of XML and lots of namespaces. Stability is very nice but an XML based API is really appdev unfriendly these days and the use of namespaces makes it even worse and difficult to handle – you either handle the complexity or you simply choose the parse without namespaces. Extracting data is either done using XPath or string parsing none of which is easy or performs well and again the namespaces makes XPath notoriously difficult.

This being said there is one exception to the XML based nature of the API’s. When Activity Streams were added in IBM Connections 4.5 it was brought to market with a JSON based API and is still the only component in the IBM Connections suite using JSON based API’s. Due to its roots in OpenSocial the API was however hard to grasp and I don’t think it got much traction despite my best efforts to make it approachable. My presentation on how to use the Activity Stream remains one of my most popular presentations at conferences.

When this is said and done I think that the most important thing IBM could do for IBM Connections API wise would be to update the API’s. In todays world a JSON based API is really expected and is the defacto way of accessing data. Do away with XML and namespaces and adopt a standards based JSON approach to messages being sent to the server and returned from the server. Of course the legacy Atom based API should be left in but it should really be augmented with a JSON ditto as soon as possible.

Besides the API’s there is also a set of SPI’s (Service Provider Interfaces) for stuff like hooking into the event subsystem, discovering features etc. The event SPI is pretty well designed and is very useful. However it seems that it was never really polished, completed and designed for customer / partner consumption as much of the functionality is reserved for IBM widgets and never documented or supported. Other pieces of the SPI’s can only run within the same JVM as IBM Connections which really makes it unusable from a partner perspective.

The worst thing about the API or SPI is however what is not there…

There is no support for programmatic user registration or user deletion / deactivation. There is no support for feature discovery from outside the JVM. There is no support for getting a complete member lists for a community based on a (LDAP) directory groups. Using the community membership API will return a list of members some of which may be groups. If that’s the case good luck. Your best / only option there would be to use WIM (WebSphere Identity Manager) to resolve the group and then manually combine that result with the direct community members. Of course if you are using IBM Domino as the LDAP server start by figuring out how IBM mangled the group UID before resolving it in WIM. That’s really a can of worms of its own and maybe worth a blog post of its own. There is no support for a super-user that can access data on all users behalf. There is no support for easily reusing the IBM Connections header bar if using IBM Connections on-premises.

I don’t want to finish this post being too pessimistic but there is really room for improvement from the API / integration standpoint. IBM needs to step up to the plate and update the platform and make it current from an API perspective. Oh and while you are at it document the API’s, create up to date examples and actually make the documentation discoverable using Google…

Software Dependency Management and the associated risks

Being a Maven convert and a guy that likes to dabble in programming this topic is very interesting albeit not one I’ve thought much about – and I guess this is true for most. Or let’s put it another way. After you start using Maven, npm, flask or whatever other dependency management tool you use for the job you think of dependency management as a done deal. Not having to download a jar / package makes it easier and thus, for some reason, less worrisome to add a dependency. Until this morning where I read a great post titled Developer Supply Chain Management by Ted Neward. If you’re a programmer and if you use Maven or npm or flask or whatever other automated dependency management tool you really should read this.

And if you use it as part of your product development cycle you should read it. Twice… And then act – part of which is talking to the rest of the team about it.

Thinking about dependency management and how to save dependencies should probably come back front and center and this should be a lesson to us all. If nothing else you should implement a local – dare I say on-premises – caching dependency and/or artifact server so that all dependencies are cached, stored and backed up locally (in a datastore you control). If nothing else enforce that all automated build servers download through the artifact server so that all dependencies that goes into a build is known, cached and kept.

It’s definitely something to think about.

Simple tool to save certificate chain certificates as PEM files

It’s been increasingly frustrating to support our OnTime Group Calendar for Microsoft customers with on-prem Exchange as they usually use a self-signed certificate for TLS resulting in Java throwing a fit. Getting the certificate chain using a browser or OpenSSL is easy enough but for some customers that still prove too difficult. I couldn’t find a tool to automate the export so I wrote a small tool in Java. The tool simply takes the address of the site to contact and saves the certificate chain as individual PEM files ready for import into the Java keystore. Now there is no fingerprint check so use at your own risk. Using the tool is like so:

java Main http://www.ibm.com

The code is available on Github and doubles as an example of how to accept all certificates using a custom TrustManager and HostNameVerifier. I even threw in some Java 8 to make Rene happy 🙁

YMMV…

Actually making Eclipse work for plugin appdev on Windows 10 64 bit

Just yesterday I blogged about how easy it was to get Eclipse configured for IBM Notes 9.0.1 plugin appdev. And it was easy – it just didn’t work for real development. After I imported all the plugins for the OnTime Group Calendar clients nothing would compile. After looking for a while I could see that most errors was from resolving the SWT classes such as Display, Canvas and so on and that made me think of a similar issue I had on Mac. I dove into the target platform definition. In the target platform definition I went on the Environment tab and set the following:

  • Operating system: win32
  • Windowing system: win32
  • Architecture: x86

I also set the Java Runtime Environment to the IBM Notes JVM I defined yesterday. After that the change to the target platform made everything rebuild – now without any errors and now I could launch the products from Eclipse.

Fake names

Needed to generate fake names and emails today for a stub API I’m developing. Found a github gist that did the trick. Very easy. Just had to install the faker gem first:

$ sudo gem install faker

The example generates to CSV but I needed from object instances for C# so changed the code as such:

require 'faker'
require 'securerandom'

File.open("output.txt", "wb") do |file|
  i=0
  until i == 500
    uuid = SecureRandom.uuid
    fake = "new SearchUserResult("" + Faker::Name.name + "", "" +
        Faker::Internet.email + "", UserType.Person, "" + uuid + ""),n"
    file << fake
    i=i+1
  end
end

Configuring Eclipse Neon on Windows 10 64 bit for Notes plugin development

A member of the community reached out to me yesterday to ask whether I recognized a specific error message he was encountering trying to make Eclipse launch Notes 9 correctly for plugin development. I came back with a few suggestions but as I hadn’t tried on Windows 10 yet I really couldn’t offer much help. This morning I tried configuring Notes 9.0.1 for plugin development on Windows 10 and it went smoothly. Here are the steps I took:

  1. Download Eclipse Neon for RCP and RAP development bundle for Windows 64 bit
  2. Unzip bundle and launch Eclipse
  3. Follow the steps described in my Configure Eclipse 4.2 for Notes 9 post
  4. When configuring variables I used the following values:
    • install_id: 1460097942140
    • rcp.base_version: 9.0.1_20131002-1404
    • rcp.home as described in above post

That’s it really…

Reserved characters in WebSphere Application Server passwords… Really!?

Had somewhat of a surprise today when IBM Support informed us that the issue our customer was experiencing could be due to unsupported characters in the password of the user mapped to the connectionsAdmin J2C alias. Say what!? But apparently there are restrictions on the different characters one can use. The password we were using had exclamation point (!) in it which is a no no. The customer is currently on WebSphere Application Server 8.5.5.6 and support suggested we try and upgrade to 8.5.5.7. Funny thing is that the customer has been using that password for years so it must have worked previously.

IBM Connections wiki: Special characters in password

WebSphere Application Server 8.5.5 InfoCenter: Characters that are valid for user IDs and passwords

First Git hook for Atlassian Bitbucket (formerly Atlassian Stash)

For my current project I’ve setup a full CI pipeline to automate the build process of the application (an EAR-file in this case) and deploy it to the test server. The build itself is a Maven build that runs all the tests and builds the EAR file. We are a number of people working on the application – some do frontend work (mainly JavaScript) and I do the backend. The Git repository we use is split into three branches as it concerns this project – one for backend (feature/eventboard_backend), one for frontend (feature/eventboard_frontend) and one that merges the two into the final result for building (feature/eventboard). So I was setting all this up – had the build script ready, the build server ready (Atlassian Bamboo), the deployment script working over SCP/SSH but I needed a nice way to automatically merge the two development branches into the main branch for the build.

The way I solved it was to write a Git post-receive hook on the Git server side (Atlassian Bitbucket). This post-receive hook detects a push to either of the two development branches and when it does merges the two into the main branch and pushes it branch back up. This push is in turn detected by Atlassian Bamboo that then kicks of the build and the deployment. So nice. Even though it took me a couple of hours to configure it has already saved so much time and all builds and deployments are consistant.

Today I extended the build script to monitor another branch so I now both deploy into our “bleeding edge” environment and our test environment.

The post-receive hook is written in bash and is as below. It took me a while to grok but a hook is simply a script that runs as the server OS user when ever something happens. The script is free to run as another user so my script runs as a special Git user so we can distinguish between which users does what. It also means that I could restrict access to feature/eventboard branch so it’s only writable by this build user.

The only caveat about this hook was that we are using Atlassian Bitbucket which apparently only accepts hooks written in Java. There is however a way to add bash-based hooks directly in the file system on the server under /<bitbucket-home>/shared/data/repositories/<repoid> where the repoid can be found in the repository settings on the Bitbucket server if logged in as admin. 

#!/bin/bash

CHECKOUT_NAME=eventboard
MERGE_INTO_BRANCH=feature/eventboard
MONITOR_BRANCH1=feature/eventboard_web
MONITOR_BRANCH2=feature/eventboard_backend
WORKING_DIR=/local/stash-hooks-work

while read oldrev newrev refname
do
        branch=$(git rev-parse --symbolic --abbrev-ref $refname)
        echo "Currently on branch '$branch'"
        if [ "$MONITOR_BRANCH1" == "$branch" ] || [ "$MONITOR_BRANCH2" == "$branch" ]; then
                echo "Detected commit on $MONITOR_BRANCH1 or $MONITOR_BRANCH2 - merging..."
                if [ ! -d "$WORKING_DIR" ]; then
                        mkdir -p $WORKING_DIR
                fi
                cd $WORKING_DIR
                unset GIT_DIR
                if [ ! -d "$CHECKOUT_NAME" ]; then
                        # repo doesn't exit - abort
                        echo "*** Required repo for post-receive hook not configured - exiting..."
                        exit
                else
                        cd $CHECKOUT_NAME
                        git reset --hard
                        git checkout $MERGE_INTO_BRANCH
                        git pull origin $MERGE_INTO_BRANCH
                fi
                git fetch origin $MONITOR_BRANCH1:$MONITOR_BRANCH1
                git fetch origin $MONITOR_BRANCH2:$MONITOR_BRANCH2
                git merge $MONITOR_BRANCH1 $MONITOR_BRANCH2 -m "Merged 
                          '$MONITOR_BRANCH1' and '$MONITOR_BRANCH2' into 
                          '$MERGE_INTO_BRANCH'"
                git push origin $MERGE_INTO_BRANCH
        fi
done

Using Tomcat APR (Apache Native Runtime) on Mac

I had to document some steps using the Apache Portable Runtime (APR) and TLS configuration and for that needed APR on my Mac. I couldn’t really make it work at first but after fiddling a bit I figured it out. There are the steps in bullet form:

Download APR and compile

  • Download APR from Apache (http://apr.apache.org/). I downloaded v. 1.5.2.
  • Compile in Terminal.
    • CFLAGS=’-arch x86_64′ ./configure
    • make
    • make test
    • make install

Install OpenSSL with headers

The OpenSSL on Mac doesn’t come with the header files so you cannot compile the Tomcat native library by default. To fix that use Homebrew to install a new version of OpenSSL first.

  • Install Homebrew per instructions on the website
  • brew install openssl

Compile Tomcat native library

The Tomcat native library is supplied with the Tomcat download. My Tomcat was v. 8.0.17. Steps as below:

  • cd Tomcat8.0.17/bin
  • gunzip tomcat-native.tar.gz
  • tar xf tomcat-native.tar
  • cd tomcat-native-1.1.32-src/jni/native
  • CFLAGS=’-arch x86_64′ ./configure –with-apr=/usr/local/apr –with-ssl=/usr/local/opt/openssl
  • make
  • make install

Configure Tomcat to use APR

This step is basically just to make sure that the Tomcat native library is on the Java Library path. Do as follows:

  • cd Tomcat8.0.17/bin
  • vi setenv.sh
  • Add text: JAVA_OPTS=”-Djava.library.path=/usr/local/apr/lib”

Now when you run Tomcat using catalina.sh you should see a line like below stating what version of the native library was loaded.

15-May-2016 18:14:01.106 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
     Loaded APR based Apache Tomcat Native library 1.1.32 using APR version 1.5.2.

Further reading:

Test agents in Eclipse by extending AgentBase

I continuously get questions on how I do Java agent development and I’m happy to answer them as I hope that some of the answers I provide means that more and more Domino developers see Java as a strong alternative to LotusScript for background processing. Most times the approach I recommend is a mock object approach that I wrote about waaaaaay back in 2006 (did I really write that 10 years ago?!?!?).

If / when you want to read the posts please start from part no. 1. Here’s a link to all 5 parts:

The approach doesn’t handle importing the code back into Domino Designer but it does allow you to “mock” the Notes side of the API and allows you to write, test and debug your agents in Eclipse. Then when they’re completely done you can import it back into Notes without any changes. This is the approach I would choose and that I still use when writing Java agents here 10 years later.

Hope this helps but if it doesn’t please let me know.