kafka-node with Heroku Kafka

The other day I was building a demo for a customer around Heroku and Heroku Kafka. My language of choice these days is node.js so I needed a Kafka library for node.js and I settled on kafka-node. Now I needed to figure how to use the library using the environment variables provided by Heroku to access my Kafka cluster.

After having bound my Kafka add-on to the app I had 5 environment variables added to my app. The variables are well explained in the Heroku DevCenter but how to use them with kafka-node had me spend some time on it. The root cause seemed to be that the name in the certificate presented by the Kafka brokers did not match the name in the certificate provided to me by Heroku. Since the library uses the node.js TLS module under the covers the solution was to implement some of the verification login myself using the checkServerIdentity method. In the method I verify the cryptographic fingerprint of the root issuer certificate as provided by Heroku (KAFKA_TRUSTED_CERT) with that of the issuing certificate of the Kafka broker. I also had to remove the “kafka+ssl://” part from the Kafka broker URL’s. I do that using regular expressions.

Below is the code. YMMW.

const kafka = require("kafka-node");
const x509 = require('x509');
const Client = kafka.KafkaClient;

const kafkaHosts = process.env.KAFKA_URL.replace(/kafka\+ssl:\/\//gi, "");
const kafkaCert = x509.parseCert(process.env.KAFKA_TRUSTED_CERT);

const options = {
  "key": process.env.KAFKA_CLIENT_CERT_KEY,
  "cert": process.env.KAFKA_CLIENT_CERT,
  "ca": [process.env.KAFKA_TRUSTED_CERT],
  "checkServerIdentity": (host, cert) => {
    if (kafkaCert.fingerPrint === cert.issuerCertificate.fingerprint) return undefined;
    return Error('Not authentic')

module.exports = {
    "client": () => {
        return new Client({
            "kafkaHost": kafkaHosts,
            "sslOptions": options
    "topic": `${process.env.KAFKA_PREFIX}safe-habour`

Websockets in an Express node.js app on Heroku

Last night I was having an issue with websockets and TLS in an Express.js node.js app. My websocket was working just fine when developing locally over plain HTTP but when I deployed the app to Heroku I received an error as that app runs over HTTPS but the websocket was still plain HTTP (using ws:// instead of wss://). Hmmm…. I started digging into websockets over TLS and how that would work without any luck. So I asked around but then it dawned on me and I answered my own question… Sometimes finding the answer is all about the question you ask 🙂

So the main realisation is that TLS connections are terminated by the Heroku router and forwarded to a web dyno hence there was no need to listen for TLS based websocket connections in my app. Also remembering how websocket connections are created is important. A websocket connection is a normal HTTP connection which is then upgraded to a websocket connection. So the real solution was to understand how a web dyno in node.js using Express could share it’s port with websockets using a HTTP server and that the same HTTP server would be used for both HTTP transport and websocket connections.

The solution was as follows:

const express = require('express')
const http = require('http')
const WebSocket = require('ws')

const port = process.env.PORT || 8080
const app = express()
const httpServer = http.createServer(app)
const wss = new WebSocket.Server({
    'server': httpServer

So in essence:

  1. Create the Express.js app (but do not set it up to listen in a port)
  2. Create HTTP server in node.js passing in the Express.js app
  3. Create websocket server agin using the HTTP server as the server
  4. Make the HTTP server listen on the port provided through environment variable

Flip Chrome flag to easily inspect TLS certificates (from Chrome 60)

As a developer – or a security conscious user – you may want to inspect TLS certificates from time to time. However inspecting them in Chrome is hard as access to the certificate hierarchy dialog has been tucked away in the Developer Tools. Happily Chrome 60 has added a flag to add an easy to reach option back to the TLS dropdown in Chrome.

Please note that manually editing Chrome browser flags may mess up your browser – don’t say I didn’t warn you…

In the below video I show you how…

Simple tool to save certificate chain certificates as PEM files

It’s been increasingly frustrating to support our OnTime Group Calendar for Microsoft customers with on-prem Exchange as they usually use a self-signed certificate for TLS resulting in Java throwing a fit. Getting the certificate chain using a browser or OpenSSL is easy enough but for some customers that still prove too difficult. I couldn’t find a tool to automate the export so I wrote a small tool in Java. The tool simply takes the address of the site to contact and saves the certificate chain as individual PEM files ready for import into the Java keystore. Now there is no fingerprint check so use at your own risk. Using the tool is like so:

java Main http://www.ibm.com

The code is available on Github and doubles as an example of how to accept all certificates using a custom TrustManager and HostNameVerifier. I even threw in some Java 8 to make Rene happy 😦


IBM announce dates for bringing TLS v. 1.2 to IBM Domino

So in October of 2014 I wrote about the upcoming TLS (transport layer security) enhancements that IBM was planning to bring to IBM Domino as part of the industry wide panic about the POODLE attack which I still consider mainly theoretical. I was a bit critical towards IBM as they chose to patch their seriously lacking SSL v. 1.3 implementation and implement TLS v. 1.0 on top of IBM Domino v. 9.0.x (IBM Domino, POODLE, SHA-1 and why it’s also sad when IBM decides to update the security stack). The reason I was critical was that I thought that you either take security serious and bring the stack to the front of the line (TLS v. 1.2, v. 1.3 in draft) or get out of the game.

Since then I have been pleasantly surprised to hear about the initiatives IBM has going on. At IBM ConnectED 2015 I attended a very nice session by David Kern from IBM and Daniel Nashed (IBM Business Partner) on the TLS and security improvements planned for IBM Domino. Among others was massive cipher suite updates incl. upcoming support for Diffie-Hellman and perfect-forward-secrecy. Cool stuff! Yesterday I was very pleased to see that IBM now has announced the support for TLS v. 1.2 coming in Q1/Q2 of 2015 (the technote is a bit confusing as to when it will be out).

So all appears to be good and IBM is moving in the right direction with this. Very nice.

IBM Domino, POODLE, SHA-1 and why it’s also sad when IBM decides to update the security stack

Over the last few weeks the news hit about the PODDLE attack and the withdrawal of SHA-1 as an acceptable hash algorithm by Google Chrome. This is turn has prompted IBM to update the security stack in IBM Domino for all web protocols incl HTTP, LDAP and SMTP. While this is VERY good news and it will be very welcomed that we do no longer have to resort to fronting IBM Domino by IBM HTTP Server or Apache to get adequate TLS protocol support I find the whole situation a bit sad. In full disclosure I have to say that I get most of my security updates these days from the Security Now! podcast on the TWIT network and the discussion on both POODLE and the SHA-1 debacle as opened my eyes. The sad part about these updates to IBM Domino is that it has taken a theoretical attack on SSL v. 3 (POODLE) and a premature hash algorithm withdrawal by a single browser vendor (SHA-1 and Google) to have IBM update the stack. To be fair Microsoft is also removing SHA-1 support from their security stack in their OS’es but from 2017 giving customers ample time to fix it.

In other words if these attacks hadn’t come out IBM would have left IBM Domino customers with ancient protocols and keystore formats – remember it takes Windows XP to run an iKeyman old enough to edit the .key files used in Domino.

Besides being good marketing and blowing some life into the dying embers of IBM Domino it’s almost a sad move when it’s done so late. And then IBM doesn’t even take it seriously enough to go all the way. Instead they outlines their “plan to deliver SHA-2 support for Domino 9.x” and promises a fix to bring TLS 1.0 to IBM Domino. Version 1.0 – seriously?! TLS is in version 1.2 at present and the draft for v. 1.3 is out. Now I know that implementing TLS for SMTP is much different from doing so for HTTP but security cannot be done half heartedly so if you want to make it a priority do that. Do not stop short and plug a hole by not going all the way. In all honesty I would rather have IBM discontinue SSL/TLS all together on Domino than doing this. I know it’s sad but it’s how I feel about it right now.

For a very nice discussion of the PODDLE attack, and why it’s a theoretical attack, do listen to Security Now! episode 478 from 33:22 minutes in.

Remember to secure your IBM HTTP Server when implementing IBM Connections

In Security Now! episode 396 starting at 12:22 (to 25:25) Steve and Leo were talking about various SSL attacks and how one could verify sites. I decided to check out one of my own stock IBM Connections installs i.e. I verified the stock IBM HTTP Server (IHS) install. That was not a pleasant experience as the default IBM HTTP Server is very insecure in that it accepts SSL v.2 and hence some very weak ciphers. Using SSLLabs.com and their SSL Server Test it is very easy to test a SSL site.

Below is the results from a standard IHS install using a commercial SSL certificate. A grade of F isn’t nice.

After reading a bit on mod_ssl (the SSL module in Apache / IHS) I added the below lines to the mod_ssl section in the httpd.conf file.

## SSLv3 128 bit Ciphers
SSLCipherSpec SSL_RSA_WITH_RC4_128_MD5

## FIPS approved SSLV3 and TLSv1 128 bit AES Cipher

## FIPS approved SSLV3 and TLSv1 256 bit AES Cipher

Now I’m not a SSL wizard by any means so I suggest you do your own research as well but when I restarted the IHS I got a rating of A. BAM!! How’s them apples!?

How secure is the SSL stack for your IBM Connections environment?