32 bit or 64 bit

As part of upgrading the servers here at IntraVision I had to figure out which servers were running 32 bit and which servers were running 64 bit. It turns out there is a very simple way of doing this as there is a server statistic. Simple do a “show stat Server.Version.Architecture” and it will show the currently installed “bit-ness”.

New IBM Notes and Domino Certification Available – get 50% off until 25 June 2013

IBM has announced a nice new entry level certification for Notes and Domino and until 25 June 2013 you get 50% off the certification test so if it fits a suggest you go and get certified. The page I link to at the bottom has the promo code you need for the rebate.

“IBM Collaboration Solutions is pleased to announce a new associate level certification: IBM Certified Associate – Notes and Domino.
This credential requires successful completion of the test LOT-442: IBM Notes and Domino Fundamentals. This test covers IBM Notes and Domino material as it relates to competencies within the following areas:

  • Architecture
  • General Administration
  • Calendaring and Scheduling
  • Replication
  • Mail
  • Clientv
  • Security
  • XPages
  • Non-XPages Design
  • Troubleshooting

Read more: New IBM Notes and Domino Certification Available – IBM Certified Associate – Notes and Domino.

IBM Notes and IBM Domino Social Edition – FINALLY!!!

If you’re part of the Notes/Domino community you are probably excited today!

Unless you’ve been living under a rock you have seen and/or heard the announcements from IBM yesterday about the upcoming IBM Notes and IBM Domino Social Edition. If not do yourself a favor and go watch the recording. And a new major release – no longer is it 8.5.4 but 9.0!

I’m SO pleased that IBM/Lotus/ICS FINALLY took the plunge and stripped the “Lotus” from the product – not that I dislike the Lotus brand but we (the community and the customers) have been in a strange state of flux for so long. From Lotus, to IBM Collaboration Solutions (ICS) to IBM Social Business. We’ve been wondering what the product we love and work with every day would be called or whether it would actually go away and be replaced with some “Websphere thingy”. With the announcement of IBM Notes and IBM Domino Social Edition yesterday IBM finally tore of the Band-Aid that should probably have been torn off back when IBM bought Lotus (or at least shortly thereafter).

With the move to IBM Notes and IBM Domino we can move past all that and look forward knowing that there’s at least a 3 year roadmap and the product isn’t going anywhere. With the best (and only?) public webcast in a long time IBM made it official! IBM has got our back!

And even better – we will also have a PUBLIC beta (see the sign up form). The “public” part is important – no longer is the beta and code drops for the select few of us in the design partner programs but for all. So be sure to take this opportunity and sign up for the beta. Download it. Use it. And most important of all – tell IBM what you think!

Domino rules as an API application server platform

Over the last 1.5 years we at OnTime have been making a serious investment into rewriting our backend for the OnTime Group Calendar product. The most fundamental change has been the change from a traditional Notes/Domino application (if it ever was a such) to be a server/API first approach. This means that there’s now a layer of separation between the business logic on the server and the business login in the clients namely the API. The API is now king!! It means that most, if not all, knowledge about the server operation and data storage has been removed from the clients and server and clients may evolve indepently of one another.

Another key benefit of an “API first” approach is that new exciting UI’s and uses of the OnTime data has started to pop up as it’s now easy to get going. As an application developer you do not have to worry about the OnTime backend or even worry about Domino. As long as you have HTTP access and may use JSON you’re all set.

As an example of where you’ve seen the power of the API approach – maybe without thinking about it – is in Mail and Calendar in Notes 8+. Think about how adding Java views to the “mail and calendar experience” in Notes 8+ has allowed the “mail and calendar experience” to leapfrog the rest of the client. This is because there’s a layer of separation between the data and the views and it allows them to evolve separately and at different paces.

Redesigning OnTime to an “API first approach” hasn’t been a smooth ride all along and sure there has been things we’ve changed along and way and the API backend has been rewritten more than once. The key is however that the interface to the API has only changed once which really wasn’t so much a change as a move to a new, additional, data format for the response messages. This has been key as it allowed the clients to stay constant while the API changed.

The server side API has been written in Java all along which has been a very good choice for us. The request and response messages started out being in plain text in our own proprietary format which worked out well in the beginning but it has become clear that the end user (programmer) still needed a fair amount of knowledge about OnTime to use the API. Over the last 7-9 months we’ve made a transition from first plain text in/plain text out, over plain text in/JSON out to a place where we’re now completely JSON in/JSON out. Taking it slow has been important as not to break any clients meaning our own clients or any customer API programs there might be out there now since we opened up the API for purchase.

One thing we’ve learned however is that API versions in the “real world” needs to coexist and we really couldn’t rip and replace. We’ve now move to a model where the user of the API sends an expected API version number along with the requests to signal what API version he/she expects to be returned and the API will honor that. This means that we’ve managed to move along while still maintaining backwards compatibility. Furthermore we are quite safe that we will be able to futher evolve the API while shielding customers and clients.

The real beauty about the API is the Domino application server though.

We have implemented and evolved on the API from day one using nothing more than a standard Domino server – running the HTTP task if HTTP access to the API is required – using standard Java agents. No fancy OSGi plugins here (yet!) – everything is plain ol’ Domino. Keeping everything on Domino has also allowed us to easily leverage the API in an integration problem we’re doing at a Danish customer at the moment (more on that in a later post). How? Using the web services framework available in Domino of course. Nothing fancy there. Just add a web services endpoint to respond to a SOA server and we were golden – it too scales extremely well.

Builing on Domino hasn’t limited us in any way yet – quite contrary. It has allowed us to add additional application endpoints using the easy of development approach of Domino and it has allowed us to scale well beyond thousands of users (think 15.000+ users) without problems. All this and still allowing us to seamlessly service clients from Domino 7 and upwards. So sweet.

Below is the API stack we’re using as you can see Domino feature prominently at the bottom of the stack.

(click image for a larger version)

So start building you own API’s for your Domino apps. It will be very much needed to get the full potential from Notes 8.5.4 Social Edition and the embedded experiences support in there. It’s all nothing if there is no API to read from. If only there were a Domino oAuth provider… Oh well – I’m sure it will all be there in good time.

Authenticating a web service request

In my current project I needed to place an authenticated web services request from Lotus Domino to an Oracle SOA endpoint. Turned out to be extremely easy using the Lotus Domino web services consumer feature as I just used the setCredentials(String, String) method which then adds the necessary Authorization header to the HTTP call. Below is an example.

MyEchoServiceLocator l = new MyEchoServiceLocator();
MyEcho u = l.getDomino();
u.setCredentials("Domino Admin", "password");
String result = u.echo("HelloWorld");
System.out.println("Result: " + result);

Windows 2008 64-bit can cause a significant CPU increase and I/O degradation when Domino opens many databases

Had a customer report the above issue and have it fixed by IBM so I thought others might benefit from it. The issue has been fixed as IBM SPR# KBRN8AKKA9. The technote is 1449825 and contains a lot of good info. Setting notes.ini “Disable_Random_RW_File_ATTR=1” fixes the issue.

“After Domino opens many NSF files in quick succession during a backup, the Virtual Address Space of the OS system cache may be completely used up (for example, 1TB of data may be used in this OS cache). Successive calls into the OS cache manager to get memory from the OS system cache then results in mapping/unmapping of views from the system cache. These mapping/unmapping operations takes a lot of CPU time and as a result shows as high OS CPU usage. In addition, because the large OS system cache may now reside on the disk (the RAM is not large enough to hold the OS system cache) this results in significant I/O on the system.”

IBM Lotus Domino 8.5.3 server performance and IBM Lotus Notes 8.5.3 client performance

Saw this very nice – although very technincal – whitepaper on developerWorks on how Notes 8.5.3 has been optimized to reduce the number of transactions with the Domino server. A small little thing that I haven’t seen mentioned before but which can have a big impact on your environment and users. Probably shouldn’t be considered light reading.

“IBM Lotus® Domino® 8.5.3 and IBM Lotus Notes 8.5.3 have been optimized to reduce the transactions from the client to the server. These optimizations yield a reduction in server resources, thus lowering the Total Cost of Ownership. This white paper shows the reductions in processor and disk utilization achieved with large numbers of Notes 8.5.3 clients running on a single Domino 8.5.3 server.”

IBM Lotus Domino 8.5.3 server performance: IBM Lotus Notes 8.5.3 performance

DOTS and automatic startup of bundles

Domino OSGi Tasklet Container (or DOTS for short) is an uber-cool OpenNTF project that allows you to write addins for the Domino server in Java. The project used to be called JAVADDIN which kind of gives the purpose away.

At UKLUG I was asked if there was a way to specify which bundles to automatically start at server startup. I didn’t know the answer so I wrote an e-mail to David Taieb who’s the author of the project and he quickly responded that this is indeed possible. Below is Davids answer:

Yes, you can create a file called config.ini in
{data}/domino/workspace directory and add the following line

Restart http task and verify that the plugins specified are
>tell http osgi diag pluginId1
>tell http osgi diag pluginId2
>tell http osgi diag pluginId3

Best Regards

The above is for Domino 8.5.3 – for Domino 8.5.2 the config.ini file is in the workspace.config directory.

There is a caveat though. The automatic startup isn’t performed until an XPage request is made. An extension point to make bundles automatically startup when the server is started will be added for Domino 8.5.4.

RedWiki Residency: Domino Development Best Practices

The RedWiki team is now accepting nominations for a new RedWiki project on Domino Development Best Practices. The (remote) residency will cover a total period of 5 weeks, from August 8, 2011 through September 9, 2011 and requires 6 residents. If you want to give back to the community and share your expert experience this is for you.

As one who has been part of the RedWiki process I can recommend it although it does require sufficient time being set aside for this and some writing experience would be good.

A tale from a customer reaching (and exceeding) the 64 gb limit

As I’ve tweeted I have spent the last couple of days (and the weekend) helping out a customer that exceeded the hard 64 gb database size limit in Lotus Domino. Before discussing how we solved the problem and got the customer back in business I would like you to think about how situations like this could be avoided. And avoiding it is key as once you exceed the size you’re doomed.

First — how and why database platform would EVER allow a database to cross a file size that makes it break. Why doesn’t Domino start to complain at 50gb and make the warnings progressively harder to ignore as the database gets closer to 64gb. Why doesn’t it refuse data once it reaches 60gb? I find it totally unacceptable that a software product allows a database to exceed a size it knows it cannot handle.

Now I know that there are considerations for such a warning and that it could be done in application code (e.g. database script, QueryOpen event) but it really isn’t something an application developer should think about. Also it should be applied to backend logic as well and really doesn’t lend itself to a UI computation. I also know that DDM or similar could warn about it but it still doesn’t change my stance. The 64gb limit is a hard limit and reaching, and exceeding it, shouldn’t depend on me configuring a specific piece of functionality.

Second — having the option of keeping the view index in another location/file than the database would have helped. This has been brought up a number of times including at Lotusphere Ask-The-Developers sessions. One could argue that externalizing the view index from the database would just have postponed the problem but the view index takes up a substantial amount of disk for databases of this size.

Now on to how we saved the data.

The bottom line in this is that the customer was lucky. VERY lucky. The customer uses Cisco IP telephones and keeps a replica of the database in question on a secondary server for phone number lookup using a Java servlet. Due to the way the way the servlet is written only as single, very small, view was built on the secondary server. This is turn meant that the database that had exceeded 64 gb on the primary server was “only” 55 gb on the secondary server. The database on the primary server was toast and gave out very interesting messages if attempting the access or fixup the database:

**** DbMarkCorruptAgain(Both SB copies are corrupt)

So thank God they had the secondary server otherwise the outcome of the story would have been less pleasant because using the secondary server we were able to:

  1. Take the database offline (restrict access using ACL)
  2. Purge all view indexes (using Ytria ViewEZ)
  3. Create a database design only copy to hold archived documents
  4. Delete all views to avoid them accidentally being built
  5. Build a very simple view to prepare for data archiving
  6. Write a LotusScript to archive documents (copy then delete) from the database
  7. Use Ytria ScanEZ to delete deletion stubs from the database (this works for them because the database isn’t replicated to user workstations or laptops)
  8. Do a compact to reclaim unused space
  9. Make the database available on the primary server

Whew! They are now back in business after building views in the database. They were lucky – VERY lucky. If they hadn’t had that secondary replica the data would probably have been lost to much distress. To them and me.

So what are the main take aways from this?

  1. UI check — in the future all databases that I develop will have a database script check on the database size to try and prevent situations like this
  2. DAOS — enable DAOS for databases to keep attachments out of the database and keep the size down
  3. Monitoring — monitor databases either using DDM or other tools to try and prevent sitations like this

And so concludes a story from the field. 4 days later where my hair have turned gray from watching copy/fixup/compact progress indicators the customer is back in and happy once again. Whew!!