Concept Proven: A Complex Production XPages App Running Outside Domino

Oct 7, 2019 1:13 PM

Tags: javaee xpages
  1. Letting Madness Take Hold: XPages Outside Domino
  2. XPages on Android
  3. Concept Proven: A Complex Production XPages App Running Outside Domino

Around the start of the year, I took a little time to see if I could get XPages as it exists today running outside of a Domino server, specifically inside Open Liberty. I met with a good deal of success, finding that I could run stripped-down apps "freely" inside a normal WAR web app, and I could run full-fledged NSF-hosted apps using the LCDEnvironment container-within-a-container technique that normal XPages uses.

Since then, the prospects had been percolating in my mind, including coming back around a few months later to get XPages running on iOS and Android by way of Darwino. At the end of that post, I mused a bit about what a proper setup would look like, which would be essentially taking XPages, custom controls, and supporting resources and putting them into a Maven-structured webapp.

This past week, I decided to put WoW Classic aside for a bit and put some evening-and-weekend time into seeing if I could my get client's huge, 500-XPages-and-CC, plugin-backed, JAX-RS-heavy app working in this setup. And, like before:

Short Answer

Yep!

Long Answer

Building on what I had before, there were some medium- and large-scale hurdles I had to overcome:

  • Hook up the remaining Servlets and listeners at the right times
  • Figure out what to do about transpiling XSP source
  • Add and adapt the remaining required IBM Commons Extensions
  • Tweak the OSGi bridge for more bundle-like behavior and remove Equinox dependencies in our code
  • Make the stack have a Notes runtime, but not think it's too much like Domino
  • Adapt Java EE standards use to ensure it'd work with different implementations

A lot of this work ended up being finding the right little bit to tweak or add - an environment variable here, a META-INF/services file there - but a couple of the topics warrant expansion.

App Legwork

The core of what made this possible was a combination of coincidental and intentional work I had been doing in the main app for a good while. The first big part was that I had started bringing in components of my separate XPages Jakarta EE Support project, in particular bean validation and JAX-RS 2.1. We had previously used an older version of Hibernate Validator, and we were using the ExtLib-provided Wink for a good while to build out our REST services. Moving to RESTEasy's JAX-RS 2.1 implementation not only let us use the newer features it provided, but also let us set aside the Wink-isms we had been using in a couple places in favor of what became standard after Wink went moribund.

I also stripped out as many Equinox-isms as I could that I had made assumptions about over the years, such as removing home-brewed Equinox extension points in favor of portable IBM Commons extensions and reducing our use of Require-Bundle in favor of Import-Package, which pointed out all the areas where we depended on a specific implementation of a standard as opposed to just the cross-server spec.

Notes Runtime

One of the pleasant things I found out in my original experiments was that, while a lot of the newer and more-complex components of the XPages stack end up having some assumptions about the presence of a Notes or Domino environment, there's no dependency on nHTTP specifically. As long as the stack can call into the lotus.domino and NAPI classes, it's happy.

Really, the only thing I had to tiptoe around was making sure that I didn't include the LCDEnvironment stuff and it's associated platform assumptions. At one point, I did investigate whether I could use that and make the running web app just another "module" like an NSF - there's even partial support classes like FileModule for this kind of thing in a "mashupmaker" package - but I ran into too many assumptions about the environment and class hierarchies that way. It's a bit of a shame, since that would have allowed transparently referencing NSF-housed apps in the same server, but that was only a "nice to have" anyway. Plus, it would have had the down side of keeping everything within the LCD wrappers, which do things like report the environment as Servlet 2.5 no matter how they're running (which is, horrifically, an upgrade over the underlying 2.4 on Domino).

XSP Source

My original experiment and my bootstrapping here involved copying the generated Java xsp.* files from Designer into the src/main/java build path of the Maven project, which are then picked up by the CompiledPageDriver used by the XPages runtime. This works, but it's not exactly developer-friendly. To make it in any way practical, I'd need to be able to continue working with the XML source like in Designer.

The trouble here is conceptually similar to the impediment to incremental compilation in the ODP Compiler: due to the way XSP Libraries work, they require not only the presence of the full classpath to know what components are available, but also an active running Java application. They can't just be derived statically - this is why you have to install library plugins into Designer-the-application.

However, I had a distinct advantage here: though I didn't have a running app at editing/compile time, I sure would have a running app while the app is, uh, running. And that is a problem I did solve in the ODP Compiler, thanks to the Bazaar's hooks into the XSP interpreter inside the XPages stack. I realized that I could implement a FacesPageDriver that received requests for pages and compiled them on the fly. The interface is very simple: it contains only one method, which takes a FacesContext and a page name (like "/foo.xsp") and returns a FacesPageDispatcher, which is the obliquely-named interface implemented by the translated xsp.* Java classes.

Really, this could be anything; it's not actually tied to the Java translation and compilation process at all, and could be something like a live translator of XSP into spitting out UIComponent objects on the fly, which is something I considered. Though the use of compiled classes is an implementation detail, I ended up deciding to still piggyback on that, since a dynamic interpreter would have extra legwork to do to make sure the behavior was the same, whereas using the translator would guarantee the same results as if they were compiled the normal way. In practice, the main differences between my code here and the code in the ODP Compiler is that I could use the existing component registry from the running app and that I ended up compiling the classes as Groovy source instead of Java. I did the latter because the Groovy in-memory compiler didn't require the same kind of classpath crawling that the Java compiler does, and which had ended up being a major performance sink in practice. Groovy is almost a strict superset of Java syntax, so it only took a little bit of tweaking to the generated source to make it work - tweaking that could be done simply since the problem domain is so small.

Liberty-Specific Gotcha

Though I'm still smitten with Open Liberty as my Java host of choice, I did pull my hair out a bit over some of the side effects of it. Specifically, I ran into trouble when having the JSP feature enabled (which is on by default), where it would load up com.sun.faces.config.ConfigureListener much earlier in the process than it's supposed to be. It's something to do with an optimization for when you have real JSF enabled as well, but, since IBM didn't rename the core packages, Liberty picked up on the presence of it and kicked it off on server start, instead of during app load when it's expected.

Fortunately, Liberty's architecture is such that, if you don't choose to enable a feature, it's entirely absent at runtime, so there was a clear workaround. It's a bit of a shame, since going from XSP ? JSP is a legitimate potential path, but it's not the end of the world. Additionally, there's nothing Liberty-specific in my approach here. I haven't tried it, but the same app should work in Tomcat (if you bring in some standards implementations) or in another JEE server like Glassfish.

The Results

After wrangling with this stuff enough, though, the results were ideal: the full app, with all its dynamic content, runtime component additions, use of home.xsp/foo/bar path info shenanigans (its own hurdle in the Servlet world), dependency on ODA and half a dozen other XPages Libraries, and its plugin-served resources works just as it does on Domino. And, though some of the runtime setup was a bit hairy, the app itself is pretty svelte and straightforward. All the Java code is where it's expected to be, resources in src/main/webapp show up like you'd want, and it all just acts like it should.

The Future

So now I have an odd conundrum. I prefaced my post in January by saying that I didn't plan to do anything with the experiment, but now I'm looking over a precipice where I could theoretically plop the WAR file into the Domino Open Liberty Runtime, add a redirection rule to point to it, and call it a day. I certainly don't plan to, since there could be any number of weird problems with this and the app still points back to normal classic Domino web resources in some places, but now there's a dark voice whispering into my mind. "No more Designer," it says. "No more Tycho, or writing OSGi plugins just to get a third-party library to work nicely. No more Java policy nonsense. No more Servlet 2, or missing Javadoc, or the server not loading pages because the wrong logging JAR went into jvm/lib/ext. You could be free!" The dark voice makes a good case, I'll give it that. We'll see.

Medium-Term Ways to Improve XPages

Aug 28, 2019 1:18 PM

Tags: xpages
  1. What Makes XPages "Not Modern"?
  2. Medium-Term Ways to Improve XPages

Following up on my jeremiad the other day, I've been thinking about some of the short- to medium-term ways that XPages could be improved. I'm not fully sold on the idea that it should be massively improved in-place due to some of the systemic decisions and the general goal of future-proofing, but some improvements that HCL could make in the near future would make everyday XPages development better and would help people like me in our tasks of bringing in other tools.

The big caveat here is that I'm labeling these as small-scale tasks in theory. Since I don't know about all the workings of the lowest layers, in particular the way the XPages stack interacts with traditional nHTTP, this could easily fall into "this should be easy!" territory for reasons I can't know.

The other caveat is that I'm largely going to leave out bringing in new runtimes as a stated goal: no integrating Open Liberty, no cramming in a Node runtime, or so forth. The most expedient fix for some of this may actually be to do something like that, depending on how the "WebSphere" parts of Domino work to begin with, but that would be an implementation detail for the purposes of this list.

With those out of the way, let's proceed!

Servlet API

This has been a bugbear for a long time, and being able to bump up the Servlet API version would be huge for both its immediate features and for compatibility with newer standards and third-party libraries. I'm not sure what the source of the limitation is - it could be primarily to do with the deprecated web app part of Equinox Domino uses or it could be due to nHTTP-side C code.

Remove or Don't Export Apache Commons Logging From the JSF Libs Plugin

This may seem overly specific at first, but it's been the source of an outsized amount of hair-pulling for me and others.

I'm referring here to the "com.ibm.designer.lib.jsf" plugin that's a central part of the XPages stack. In addition to the obligatory "jsf-api" and "jsf-impl" JARs, it also contains very-old versions Apache Commons BeanUtils, Collections, Digester, and Logging. The trouble is that the Logging packages are made available outside this bundle and then these are automatically on the classpath of every XPages application and plugin. In a void, that's not terrible, but logging packages are a source of tremendous heartache when developing XPages and Java agents, and in particular they can often end up in jvm/lib/ext and cause ClassLoader trouble by mixing with the version in this plugin.

Ideally, it'd be removed outright or taken out of the exported packages and instead included just where it's needed. Next best would be to include newer versions as proper OSGi bundles with package-version information (more on that next).

Add version information to exported packages

What I'm referring to here is the OSGi mechanism for assigning versions not just to a bundle as a whole, but also to individual packages. As an example, here's a trimmed-down and cleaned-up version of the export list from the aforementioned JSF libs plugin:

Export-Package: javax.faces,
 org.apache.commons.logging,
 org.apache.commons.logging.impl

With version information (including a made-up "1.3" version to differentiate XPages JSF), it'd be more like:

Export-Package: javax.faces;version="1.3.0",
 org.apache.commons.logging;version="1.0.3",
 org.apache.commons.logging.impl;version="1.0.3"

Doing this across the board would make the XPages runtime much more OSGi-friendly.

Provide Public or OpenNTF-Permission-Gated p2 and Maven Access (With Javadoc!)

I've been on this soap box for a long while, and it comes up pretty frequently: because the XPages runtime exists solely as a component of Notes and Domino, you have to jump through hoops to depend on it as part of an external project. The "Update Site for Build Management" was a good step, in that it became one place to get a nicely-formatted p2 repository, but that was never updated past 9.0.1 and even then was just a ZIP download, not an HTTP-accessible p2 site like you'd get with things like Eclipse. Additionally, there's never been a Mavenized version of these artifacts, meaning anyone wanting to use XPages JARs without PDE or Tycho has to reinvent the wheel.

I made tools to automate these processes, but it'd be up to HCL to give the legal approval to make the results of that available generally (or to authenticated OpenNTF users who agreed to an EULA).

Shared-Source the Stack

Open-sourcing XPages would likely be a large task just because of the legal side of it, but perhaps just giving read-only access to licensed customers would be an easier step to take. There's so much about the inner workings of XPages that are effectively a black box, and it's very difficult to figure out how to accomplish a lot of advanced tasks. Having source access would make this sort of thing much, much smoother.

Support WABs or Similar

Alongside XPages came the ability to deploy servlets and JEE web apps via OSGi, but these use the deprecated Equinox extension points and an IBM-Expeditor-specific point, respectively. From what I gather, the more-normal way to do this in OSGi is via Web Application Bundles. That wouldn't necessarily change the capabilities of development for Domino dramatically, but it'd make it a smaller step to go from "normal web app" to deploying on Domino, and wouldn't necessarily involve a full revamp of the Expeditor layers.


That's it for this list, at least for now. I may end up thinking of enough for another one of these posts, but I wouldn't want to load HCL up with too much homework right after the summer break.

What Makes XPages "Not Modern"?

Aug 23, 2019 1:53 PM

Tags: xpages
  1. What Makes XPages "Not Modern"?
  2. Medium-Term Ways to Improve XPages

A big part of figuring out how to move past XPages is discussing what makes it no longer modern in the first place. Some of these reasons will be gimmes - out-of-date or outright-missing features - while some will be less about what XPages is strictly capable of and instead more about how XPages development and deployment works. I'm going to leave out some of the ones that are just from recent lack of development attention, such as Bootstrap 4 support remaining in alpha state.

Admittedly, I suspect that this post will come across as an enumeration of complaints, but I don't really mean it that way. XPages served us well for a while, but it's one thing to say "ah, XPages, it's a bit old and creaky" and another to think about just how pervasive that is.

Technical Limitations

The most straightforward reasons are the Java technologies that are included in the XPages stack, but are significantly out of date. This includes big-ticket core standards like Servlet 2.4 (faux 2.5), the forked JSF 1.x, JavaMail 1.3, pre-Unified EL, and JAX-RS 1.1; lesser-known standards like Activation 1.1; and even third-party libraries that show up in the fixed XPages classpath like Apache BeanUtils 1.6, Apache Collections 2.1, Apache Logging 1.0, and Guava 19 (a recent addition).

Some of these can be worked around relatively simply (including a newer JAR in your app/plugin or adding an OSGi bundle), but others are either more pernicious (like Logging and Guava, causing runtime ClassLoader trouble) or are outright showstoppers. I've done a lot of work to bring newer and missing Jakarta EE standards to XPages, but the ancient Servlet version is a hard limit on a lot of things. In addition to its own limitations like the lack of Web Sockets, third-party libraries and implementations often expect Servlet 3 as a bare minimum, and nowadays make use of Servlet 4.

Self-Imposed Hurdles

Beyond the technical limitations that largely come from outdated implementations, XPages is afflicted by what I'll deem "self-imposed hurdles": difficulties that spring from choices that Lotus/IBM/HCL actively made or chose to not make that resulted in things being a bit more difficult than in other environments. These decisions were often even good decisions in the aggregate, but they have tradeoffs.

The first big one is the use of the NSF container. Overall, this is a positive: not only did it create a more unified development experience with classic Notes elements, but the NSF itself brings attributes like external-to-the-webapp ACLs and seamless replication, which are extra addons at best with other frameworks. However, because the NSF is essentially a proprietary binary blob, it means that it can't readily participate with other tooling. You can't crack one open with a ZIP file tool, inside-JVM reflection/classpath tools can stumble over it, and it can't inherently participate in source control or automated builds. As above, some of that can be worked around with immense effort, but even then not nearly as smoothly as elsewhere.

Along the lines of the NSF, due to Designer's lineage and (presumably) the state of technology at the time that XPages was integrated, the normal XPages development process has no participation in the larger unified Java development world that Maven largely ushered in. Even besides not fitting into the build system, this introduces just a bit more friction in normal development. Yes, an XPages app can use many third-party Java libraries by just adding them to the Code/Jars section, but they don't benefit at all from version management or automated fetching of source and Javadoc. It may sound like a small thing, but it makes a world of difference once you're used to it.

That plays in to a general Domino/Designer issue that stretches back to the introduction of Java to the platform: the libraries include no parameter info or Javadoc. Designer has its help sidebar for Notes.jar classes and the Javadoc exists for a subset of the XPages stack as of 9.0.1, but they're not integrated in the IDE, and large swaths of the platform don't have any Javadoc at all. This leads to stumbling blocks where you'll seek out documentation for a class that nominally exists in XPages, but you have to be wary of the "Since" note for every method (and even then you may be thrown for a loop, like UIViewRoot#getViewMap() that's "since" 2.0 but does exist in XPages).

Then there's a big sticking point that's created a weird cargo cult within the community: the default Java policy. While the file itself does indeed come from stock Java (including the adorable "properies" typo), the way it interacts with the SecurityManager and ClassLoader setup in XPages has an outsized negative impact. XPages apps butt heads with this policy constantly, in particular when using reflection - which is something that third-party libraries do often as a matter of course. It's had the effect of making many people shy away from using third-party libraries or code that uses normal Java features like this out of a partially-justified desire to not tweak the settings.

The last one in this category is the use of OSGi generally and Equinox specifically. I can't knock the original choice too hard: OSGi is a logical pick for making a large, pluggable system and IBM institutionally had tremendous experience with it by way of Eclipse and WebSphere. However, while it's a solid choice for system architecture, it's often very awkward for app architecture. And, hypothetically, XPages is designed to avoid this. Like the various Java EE servers that are implemented with OSGi but run plain-old WAR-based apps, a basic XPages app isn't meant to care about the OSGi layer at all. However, a combination of the above security-policy troubles, Designer's limitations as an IDE, explicit blocks on what classes an NSF can access, and the severely-limited EE services provided by the XPages runtime means that OSGi plug-in development came immediately to the fore. And developing plug-ins for XPages is hard. It involves all sorts of ceremony and incantations just to get started, and maintaining even a well-structured plug-in project is much more difficult than a normal Maven or Gradle project. Some of this is OSGi, some of it is Equinox specifically, and some of it is unique to XPages, but it all combines to make it all a daunting process even for people who aren't put off by Java-the-language. So, much like with the security policy, you have a large subset of the XPages community that for either developer or admin reasons don't develop XPages libraries and often don't bring in any outside of that HCL ships in the box.

Development Model

And, finally, I'd like to cover the development model created by XPages even when everything is firing on all cylinders.

The first one is the lack of MVC (or similar) structure. XPages was introduced as bringing MVC to Domino development, in large part thanks to the underlying JSF nature, but this was something of a dirty trick. While it's theoretically possible to make a cleanly-structured XPages app, the framework provides no useful model layer, and Designer is almost actively hostile to this kind of separation. And, because the framework doesn't give any guidance and few of us in the community came in as seasoned web developers, the discussion about XPages and MVC is largely about each person's individual version of "MVC-style" development, and Lord knows I'm as guilty as anyone. This, like the lack of dependency management, is something that you don't fully appreciate until you work with a system with real structure and clear guidance on how to do it.

The second point in this section is one that I actually debated putting on this list, and that's XPages's use of server-side state. Depending on who you talk to, this alone is something that makes XPages dead in the water, an ancient relic of a forgotten era. Personally, I'm not so universally down on the notion: I feel that the worries about scalability are usually based on hypotheticals, and being able to make complicated dynamic apps maintained by the server cuts down on the work of writing and securing REST APIs for absolutely everything. Still, it's pretty clear that the server-state model has fallen out of favor, and that's worth at least considering when determining "not modern".

So What About Workarounds?

A running theme through all of this has been "X is behind the times or weird, but can be overcome with a bunch of work". And this shows up a ton in practice: a lot of XPages developers have gone to Herculean lengths to bring in other technologies or carve out a way to do structured programming, and it sometimes works. Others eschew XPages UI components entirely and build JavaScript apps backed by REST services written with the ExtLib components, JAX-RS, or other mechanisms. Especially since the Java 8 switch, you can do a lot to drag XPages forward.

But that doesn't make it "modern" really. This is all work fighting against the platform, work that doesn't need to be done with other tooling. In other environments, if you want to use the latest version of a spec, you make sure your container is up-to-date and then you just use it. If you want to bring in a dependency, you declare it in your "pom.xml" and it shows up, source and all. And then there are the things that other environments provide for you, like OpenAPI documentation, and concepts that don't even exist in XPages, like fault tolerance. Maybe some of those things could be brought to XPages too, but, again, it'd be a struggle every step of the way. There'd always be some weird thing to do with the Servlet version, or Tycho, or ClassLoaders, or loggers, or some of the many other little asterisks that accompany the platform.

The Work To Move Forward

My plan for future entries in this series is to discuss some of the specific steps that I've been taking with my largest active client project to prepare it for the future. That future is as-yet-undefined - it may stay within the Domino web container, it may not - but the good part is that a lot of the work is in common regardless of where we take it, and I think that it will prove useful to others facing similar situations.

Converting Tycho Projects to maven-bundle-plugin, Initial Phase

Aug 22, 2019 3:27 PM

Tags: maven osgi tycho
  1. Developing an Open/WebSphere Liberty UserRegistry with Tycho
  2. Developing Open Liberty Features, Part 2
  3. Converting Tycho Projects to maven-bundle-plugin, Initial Phase

To date, Tycho has been my tool of choice for developing Domino-targeted Maven projects. However, it's not without protest.. Unlike most Maven plugins, Tycho inserts itself at the very start of the build process and takes over dependency management. Purely in Maven, you can use normal Maven dependencies, but only so long as you're pointing to a dependency that already has OSGi metadata (which, fortunately, most do), and only then to satisfy a Require-Bundle or Import-Package that also has to be present. This gets more annoying, though, when dealing with Eclipse, which removes the notion of Maven dependencies entirely when using Tycho and forces you to jump through hoops to do what you want. And, as a final kicker, Tycho's p2 repository support is completely broken in the latest release version of Maven.

So why do I keep using it, anyway?

Well, it brings a couple major benefits that are of particular importance for Domino:

  • It can use p2 repositories for dependencies. This matters because the XPages runtime plugins are not (yet?) available as normal Maven dependencies. Years back, IBM [provided a "Build Management" update site](https://openntf.org/main.nsf/project.xsp?r=project/IBM Domino Update Site for Build Management), which is helpful, but it's still an Eclipse-style p2 repository, not a Maven repository. Tycho can use p2 repositories natively, though, just as Eclipse does.
  • It constructs a true Equinox environment. This matters both when compiling your project and when running automated tests. The environment created by Tycho is the same Equinox OSGi runtime that Domino uses, and so it supports the same styles of bundle resolution and extensions that you get in Domino. Without this happening during the build, you lose some assurance that things at runtime will match your expectations.
  • It spawns tests in a separate process. This is a little esoteric, but it matters because launching a Notes environment on a non-Windows platform more-or-less requires setting up environment variables for the Notes/Domino directory and others, and these variables are not successfully set when using the normal maven-surefire-plugin runtime. This means that reliably running tests requires setting up the environment ahead of time, which is fiddlier and less automated.
  • It can generate new- and old-style Eclipse Update Sites. To be used in Designer and NSF-based Update Sites, an OSGi project has to be packaged up into a p2 repository along with an old-style "site.xml" file. Tycho can generate these (and can be assisted with "site.xml" when using the newer style), and it can also auto-generate source bundles, features, and repositories.

Alternatives and Workarounds

Some of the "hard" requirements for Tycho can be at least worked around.

Years ago, I wrote a Ruby script that would take a p2 site like IBM's or one generated from a newer version and "Mavenize" it by creating artifact information based on each bundle's OSGi manifest. I since converted it to Java and included it in Darwino's Studio plugins, and yesterday added it to the generate-domino-update-site Maven plugin. Using that lets you declare dependencies on any of the bundles or embedded JARs in a normal Maven project:

1
2
3
4
5
6
7
        <dependency>
            <groupId>com.ibm.xsp</groupId>
            <artifactId>com.ibm.notes.java.api.win32.linux</artifactId>
            <version>[10.0.0,)</version>
            <classifier>Notes</classifier>
            <scope>provided</scope>
        </dependency>

This isn't perfect, since it's neither standardized nor generally available (go vote for the aha idea!), but at least it's reproducible and can be something of a de-facto standard if used enough.

Then there's the matter of generating appropriate OSGi metadata. Outside of the Tycho-using world, the main way that generating this is via a tool called bnd and its related tools. bnd is kind of a parallel world and there's even an alternate tooling set for Eclipse instead of the default PDE. There are a couple ways to use bnd in a Maven build, but the one I'm familiar with to date is the maven-bundle-plugin. I've used this with Darwino to incidentally create OSGi metadata for the otherwise non-OSGi core modules, and I suspect that it gets used heavily this way. It's more powerful than that, though, and is a nice wrapper for bnd under the hood, supporting Declarative Services annotations and all the other OSGi goodies. In my case, I used it to generate the MANIFEST.MF with most of the defaults, but then added in some specifics to play nice in my Domino Equinox target.

I suspect that these bnd-based tools can also be a route to solving my automated-testing woes. For the Open Liberty Runtime project, I don't have to worry about that, since it's so dependent on running in actual Domino that the return-on-investment for setting up JUnit tests wouldn't be worth it. However, I recall seeing some Maven testing plugin that let you spawn an OSGi environment of your choice, and I think that something like that may be able to replace Tycho for me there.

Since p2 repositories/update sites are entirely an Eclipse-ism, most OSGi tooling doesn't care about them. That's where p2-maven-plugin comes in. Not only will it allow you to create p2 repositories, but it lets you define features in the configuration, meaning they don't have to be separate modules like in Tycho. And not only that, but it will also auto-OSGi-ify any Maven dependencies you bring in if they don't already have OSGi bundle information. It also lets you override existing bundle data on the fly if needed, such as if the dependencies and imports conflict with something on Domino.

Eclipse Friendliness

Since I still use Eclipse to develop these projects, I want to be able to make use of the XPages SDK's ability to run Domino's HTTP stack pointed at my active workspace. For that to work, I need to be able to get Eclipse to recognize my projects as functional PDE-compatible bundles even if I'm not using PDE for them. Fortunately, that process isn't difficult: once I set the location for MANIFEST.MF to be in "META-INF" in the project root, maven-bundle-plugin started generating the files there instead of within "target", and Eclipse started working with the projects as OSGi bundles. The only thing left to do then was to gitignore the generated files, since they don't need to be checked into source control anymore.

Future Improvements

The big thing that is still an open problem is dealing with testing. I have some ideas for taking a swing at it, but for now it's the main thing preventing me from doing this for all of my Tycho projects.

Beyond that, I want to look a bit into bnd-maven-plugin. This diverges from maven-bundle-plugin in that it's geared towards using bnd configuration files directly. During the build process, I think the results would be the same, since maven-bundle-plugin can already pass through whatever configuration I want, but it would be a better match for the Eclipse bndtools tooling. Additionally, externalizing the bnd config files would mean they'd be the same if I decided to switch to Gradle, as Open Liberty uses.

Finally, and specific to this Open Liberty project, I may want to consider using bnd to generate Liberty Feature manifests, as it itself does. These features are implemented as OSGi "subsystems" packaged .esa files. Currently, I'm using esa-maven-plugin to generate their specialized manifests, but I've already hit some limitations in the area of cross-feature dependencies. Apparently, bnd takes some wrangling to suit this, but is worth it. I'll consider that one a "stretch goal", though.

For now, I'm pretty pleased with the new setup. The projects still work on Domino, I can run them on there from the workspace, I was able to eliminate the p2 feature projects outright, and now I don't have to worry about packaging up a dependencies site just to have something to point at in Eclipse. Heck, I can even use Visual Studio Code now! It's pretty nice.

Developing Open Liberty Features, Part 2

Aug 18, 2019 9:14 AM

  1. Developing an Open/WebSphere Liberty UserRegistry with Tycho
  2. Developing Open Liberty Features, Part 2
  3. Converting Tycho Projects to maven-bundle-plugin, Initial Phase

In my earlier post, I went over the tack I took when developing a couple extension features for Open Liberty, and specifically the way I came at it with Tycho.

Shortly after I posted that, Alasdair Nottingham, the project lead for Open Liberty, dropped me a line to mention how programmatic service registration isn't preferred, and instead the idiomatic way is to use Declarative Services. I had encountered DS while fumbling my way through to getting these things working, but I had run into some bit of trouble or another, and I ended up settling on what I got working and not revisiting it.

Concepts

This was a perfect opportunity to go back and do things the right way, though, so I set out to do that this morning. In my initial reading up, I ran across a blog post from the always-helpful Vogella Blog that talks about coming at OSGi DS from essentially the same perspective I have: namely, having been used to Equinox and the Eclipse plugin/extension mechanism. As it turns out, when it comes to generic OSGi, Equinox can kind of poison your brain. The whole term "plug-in" instead of "bundle" comes from earlier Eclipse; "features", "update sites", and all of p2 are entirely Equinox-specific; and the "plugin.xml" extension mechanism is of a similar vintage. However, unlike some other vestiges that were tossed aside, "plugin.xml" is still in active use.

At its core, the Declarative Services system is generally similar to that route, in that you write classes to implement a given interface and then declare that your bundle provides that using XML files. The specifics are different - DS is more type-safe and it uses individual XML files in the "OSGI-INF" directory for each service - but the concept is similar. DS also has an annotation-based mechanism for this, which allows you to annotate your service classes directly and not worry about maintaining XML files. It's something of a compiler trick: the XML files still exist, but your tooling of choice (PDE, bnd, etc.) will generate the files based on your annotations. It took a bit for Eclipse to get on board with this, but, as of Neon, you can enable this processing in the preferences.

Implementation

Fortunately for me, my needs are simple enough that making the change was pretty straightforward. The first step was to delete the Activator class outright, as I won't need it for this. The second was to add an optional import for the org.osgi.service.component.annotations package in my Liberty extension bundle. I suspect that this is a bit of a PDE-ism: the annotations aren't even retained at runtime (and the package isn't present in the Liberty server), but this is the only mechanism Eclipse has to add a dependency for a plug-in project.

The annotation for the user registry was as straightforward as can be, needing a single line in this heavily-clipped version of the class:

1
2
3
@Component(service=UserRegistry.class, configurationPid="dominoUserRegistry")
public class DominoUserRegistry implements UserRegistry {
}

With that, Eclipse started generating the associated XML file for me, and the registry showed up at runtime just as it had before.

The TrustAssociationInterceptor was slightly more complicated because it had some extra initialization properties set, in particular the one to mark it as executing before normal SSO. This was a little tricky in two ways: Java annotations don't have any mechanism for specifying a literal Map for properties, and the before-SSO property is a boolean, but I could only write a string. It turned out that the property, uh, property on the annotation has a little mini-DSL where you can mark a property with its type. The result was:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
@Component(
	service=TrustAssociationInterceptor.class,
	configurationPid=DominoTAI.CONFIG_PID,
	property={
		"invokeBeforeSSO:Boolean=true",
		"id=org.openntf.openliberty.wlp.userregistry.DominoTAI"
	}
)
public class DominoTAI implements TrustAssociationInterceptor {
}

Further Features

This is proving to be a pretty fun side project within a side project, and I think I'll take a crack at developing some more features when I have a chance. In particular, I'd like to try developing some API-contribution features so that they can be deployed to the server once and then used by web apps without having to package them (similar in concept to XPages Libraries). This is how Liberty implements its Jakarta EE specifications, and I could see making some extra ones. That's also exactly what CrossWorlds does, and so I imagine I'll crib a bunch of that work.

Upcoming Webinar with Cameron Gregor: How We Are Using XPages

Aug 17, 2019 10:05 AM

Tags: xpages

I ended my XPages post the other day with a request for people who are working on large XPages applications to hit me up on Twitter to tell me about them. Shortly thereafter, the estimable Cameron Gregor did just that. Moreover, he had the suggestion of turning the discussion into an open webinar, so that others can join.

He made a post on his site with the details and a handy time-zone table to account for our respective locations, and the summary is:

  • Sydney Time: 7:00 AM on August 27th, 2019
  • US Eastern: 5:00 PM on August 26th
  • GMT: 10:00 PM on August 26th
  • Zoom Link

I'm pretty curious to take a look myself, and I hope you'll join us in just over a week!

Developing an Open/WebSphere Liberty UserRegistry with Tycho

Aug 16, 2019 3:08 PM

  1. Developing an Open/WebSphere Liberty UserRegistry with Tycho
  2. Developing Open Liberty Features, Part 2
  3. Converting Tycho Projects to maven-bundle-plugin, Initial Phase

In my last post, I put something of a stick in the ground and announced a multi-blog-post project to discuss the process of making an XPages app portable for the future. In true season-cliffhanger fashion, though, I'm not going to start that immediately, but instead have a one-off entry about something almost entirely unrelated.

Specifically, I'm going to talk about developing a custom UserRegistry and TrustAssociationInterceptor for Open Liberty/WebSphere Liberty. IBM provides documentation for this process, and it's alright enough, but I had to learn some specific things coming at it from a Domino perspective.

What These Services Are

Before I get in to the specifics, it's worth discussing what specifically these services are, especially TrustAssociationInterceptor with its ominous-sounding name.

A UserRegistry class is a mechanism to provide a Liberty server with authentication and user info services. Liberty has a couple of these built-in, and the prototypical ones are the basic and LDAP registries. Essentially, these do the job of the Directory and Directory Assistance on Domino.

A TrustAssociationInterceptor class is related. What it does is take an incoming HTTP request and look for any credentials it understands. If present, it tells Liberty that the request can be considered authenticated for a given user name. The classic mechanisms for this are HTTP Basic and form-cookie authentication, but this can also cover mechanisms like OAuth. In Domino, this maps to the built-in authentication mechanisms and, more particularly, to DSAPI filters.

How I Used Them

My desire to implement these developed when I was working on the Domino Open Liberty Runtime. I wanted to allow Liberty to use the containing Domino server as a user registry without having to enable LDAP and, as a stretch goal, I wanted to have some sort of implicit SSO without having to configure LTPA.

So I ended up devising something of an ad-hoc directory API exposed as a servlet on Domino, which Liberty could use to make the needed queries. To pair with that, I wrote a TrustAssociationInterceptor implementation that looks for Domino auth cookies in incoming requests, make a call to a small servlet with that cookie, and grabs the associated username. That provides only one-way SSO, but that's good enough for now.

The Easy Part

The good part was that my assumption that my comfort with Tycho going in would help was generally correct. Since the final output I wanted was a bundle, I was able to just add it to my project structure like any other, and work with it in Eclipse's PDE normally. Tycho and PDE didn't necessarily help much - I still had to track down the Liberty API plugins and make a local update site out of them, but that was old hat by this point.

What Made Development Weird

I went into the project in high spirits: the interfaces required weren't bad, and Liberty uses OSGi internally. I figured that, with my years of OSGi experience, this would be a piece of cake.

And, admittedly, it kind of was. The core concepts are the same: building with Tycho, bundle activators, MANIFEST.MF, and all that. However, Liberty's use of OSGi is, I believe, much more modern than Domino's, and certainly much less focused on Equinox specifically.

For one, though Liberty is indeed OSGi-based, it doesn't use Maven Tycho for its build process. Instead, it uses Gradle and the often-friendlier bnd tooling to handle its OSGi composition. That's not too huge of a difference, and the build process doesn't really affect the final built feature. The full differences are a whole big topic on their own, but the way they shake out for this purpose is essentially a difference in philosophy, and the different build mechanism was something of a herald of the downstream distinctions.

One big way this shows is in service registration. Coming from an Eclipse heritage, Equinox-based apps tend to use "plugin.xml" to register services, Liberty (and most others, I assume) favors programmatic registration of services inside the bundle activator. While this does indeed work on Equinox (including on Domino), this was the first time I'd encountered it, and it took some getting used to.

The other oddity was how you encapsulate your bundle as a feature in Liberty parlance. Liberty uses the term "feature" to refer to individual components that make up the server, and which you can configure in the "server.xml" file. These are declared using files similar to MANIFEST.MF with specialized headers to declare the name of the feature, the bundles that make it up, and any APIs it provides to the server and apps. In my case, I wrote a generic mechanism to deploy these features when a server is established, which writes the manifest files to the server's feature directory. Once they're deployed, they become available to the server as a feature with the "usr" prefix, like "usr:dominoUserRegistry-1.0" for my case.

In The Future

I have some ideas for additional features I'd like to develop - providing implicit APIs for Darwino and Jakarta NoSQL/JNoSQL would be handy, for example. This way went pretty smoothly, but I'll probably develop non-Domino ones using either Gradle or Maven with the maven-bundle-plugin. Either way, it ended up fairly pleasant once I discarded my old assumptions, and it's another good entry in the "pros" column for Liberty.

How Do You Solve a Problem Like XPages? Redux

Aug 13, 2019 5:33 PM

Tags: xpages

The better part of a year ago, I mused about what to do about the XPages problem. For better or for worse, I'm still musing about it to this day, without a good answer. The nice part for me is that it's not actually my job to come up with a real plan to "solve" XPages in a grand scale, but I do have my own set of clients I'm responsible for, and I've been personally invested in it for so long that I'd like to be involved in bringing it to a safe landing.

Moving Away

And I do think that that "landing" almost definitely has to be a path from XPages to something non-XPages. Hypothetically, a path forward would be for HCL to staff up on a new XPages team and improve the platform. Even if they did, though, I don't think it'd be wise for customers to rely on that - not even because I'd doubt the intent, but rather because any single-vendor, closed-source web stack without a large developer community to buffer it is an unreliable foundation.

If it'd be unsafe to rely on a revitalized platform, I think it's certainly unsafe to rely on one that's clearly in maintenance mode. The nature of web development is such that a stationary platform may as well be moving backwards, and not just because it will miss out on Web Workers or other new technology. Sooner or later, Safari or Chrome will remove a capability for security purposes and either Domino or the Dojo version XPages uses will be caught flat-footed. We've already been dealing with different definitions of "define" and various security improvements tripping us up for pretty much the entire life of XPages, and that kind of thing certainly isn't going to go away. Heck, how long to you figure user agents are going to remain reliable? It's unfortunate, but the fact that XPages came out of that "Web 2.0" era means a given page is less likely to function properly in five years than a JavaScript-free page made in 1995 that's still going strong.

Candidates for a Path

So I do think it's important to have a path, but it's not yet defined. A couple candidates spring to mind for me, but each one has one major drawback or another:

Hoist XPages Back to Jakarta EE

By this I mean taking XPages more-or-less as-is and running it on a normal JEE server. This certainly works, and access to the source would let me make it work better, but it kind of kicks the can down the road. XPages itself would still be moribund, and just running it on, say, a Domino-connected Open Liberty runtime wouldn't magically make it modern.

This would, though, provide some breathing room to manipulate an app in a better environment and transform it gradually. A JEE app can use a number of technologies that an XPages app can't currently, and so this would be a way to migrate the code without ever having a hard cutoff.

Bring to XSP Components to JSF 2.x

This is essentially the "upgrade JSF" request that has followed XPages since just after its birth, but with the slightly-lower goal of leaving XPages-the-stack where it is but making a copy of the components and infrastructure that can be brought into a normal JSF runtime like any other set of components. This would possibly be the hairiest of all options, since it wouldn't really be worth it unless things work near-100%, and there are so many little edge cases that it's harrowing to think about. Take the _xsp* methods grafted onto XPages's javax.faces.component.UIComponent alone, or whatever weird ways the XPages Ajax model differs from JSF's.

Still, it'd be doable if desired, and it'd provide a reasonable path to progressively "melt" the XSP components down until they're very-thin wrappers over normal JSF stuff, until they don't really diverge at all. With infinite resources, this would probably be the nicest route.

Try Transforming XSP Markup

By this I mean two similar possibilities: making an XSP-to-Java "compiler" that emits stock JSF components, or one-pass transforming XSP XML into JSF-compatible XML, though I think only the latter would be worth pursuing. While this could potentially rival the complexity of the JSF "update" route, I think that this would allow more room for things to break. For example, if you made an XML transformer, it could target a subset of controls but emit standard comments with TODOs to cover the parts it doesn't handle. That wouldn't be perfect, but you'd end up with real-world-compatible code without having some sort of intermediary translation layer like keeping the old components would essentially be.

This would probably have to involve porting over the SSJS EL extension and thus retaining support for various uncomfortable XPages- and Domino-isms, but them's the breaks, I suppose.

Focus on REST APIs Only

This is the route where we would basically wash our hands of traditional XPages applications (minus bug fixes) and instead target writing only REST services, whether it be in the NSF, in plugins, or in normal JEE apps. This has the advantage of being easiest for HCL, since it more or less works (though the ExtLib's use of Wink holds back the JAX-RS version significantly), but still would mostly be a "rewrite all your applications" route. The only salvaged code would be anything that's already cleanly separated in Java or SSJS, and I suspect that that's not the bulk of it.

Progressing Without a Defined Path

In the mean time, aside from personally becoming acquainted with other technology, I think it behooves all of us with actively-maintained XPages apps to step up the progress on making them portable. I've been doing this heavily for one of my client projects, and I'll have more specifics to say about that in the future. Some parts are straightforward and have remained good advice for a long time: don't use SSJS, do adopt Jakarta EE technologies, and adopt automated builds (including for your non-XPages NSFs).

The specifics get a lot more complicated, unfortunately. Since we've been swimming in the same XPages pond for a decade, even mostly-clean Java code is likely to be infested with XPages-isms, both out of habit and out of necessity. For example, hooking up file uploads to a Java bean requires using an XPages-specific class, which barely transfers to OSGi-based servlets, let alone any other environment. And there are tons of little things, like using XSPContext to get URL parameters. It's going to be a messy process, but I think it will be necessary for any apps that you plan to keep using for more than a year or so.

I'll probably end up turning this into a series, where I'll discuss the various hurdles I've overcome in taking a complicated XPages apps and gradually laying the groundwork for a different UI technology.

And, in the mean time, if you're working with large active XPages applications, hit me up on Twitter and let me know how you've gone about making them. I realized earlier that, while I certainly have detailed knowledge of how people could write XPages applications, I don't have a good bead on how many actual XPages apps of each stripe exist and what the prevailing methods still are.

How the ODP Compiler Works, Part 7

Jul 8, 2019 12:24 PM

Tags: nsfodp
  1. Next Project: ODP Compiler
  2. NSF ODP Tooling 1.0
  3. NSF ODP Tooling Example Project
  4. NSF ODP Tooling 1.2
  5. How the ODP Compiler Works, Part 1
  6. How the ODP Compiler Works, Part 2
  7. How the ODP Compiler Works, Part 3
  8. How the ODP Compiler Works, Part 4
  9. How the ODP Compiler Works, Part 5
  10. How the ODP Compiler Works, Part 6
  11. How the ODP Compiler Works, Part 7

In this probably-final entry in the series, I'd like to muse a bit about possible future improvements and additions to the compiler and the NSF ODP Tooling generally. For the most part, the big-ticket future additions seek to answer one question:

Could this be used to replace Designer?

The quick answer is "yes, it could", but that would take a lot of work. There are a couple things inherent in the task and specific to my implementation that both help and hinder this kind of thing.

Notes Runtime

The biggest stumbling block is the hard requirement on a Notes or Domino runtime initialized for the current process. Being able to use C API calls is required by both my code and some of the underlying XPages bits, and that means initializing the runtime. The good news here is that it doesn't require any specific Notes-based program - it can be run with either the libraries that come with Notes or Domino, and it doesn't require Designer at all. That loosens things up a bit, but still means that one of the supported platforms is obligatory at some step of the process.

Even on a supported platform, though, it's not just as easy as calling an init function - the process's environment needs to be set up specifically to know about the Notes program and data directories, and this varies platform-by-platform. This means that it wouldn't be straightforward to have, for example, an Eclipse plugin that initializes the process, since it would depend on initialized environment variables and loading paths implicitly referenced by lower layers, and over which the programmer doesn't have much control after the fact.

The good news here is that the tooling is already designed to support remote work for compilation and export, both truly remote and with the local Equinox runners. For a true IDE experience, the communication between the IDE and compiler would have to be more complex than the "tell the compiler what to do and hear messages back" simple mechanism it has now, but it'd still be a natural evolution.

OSGi Runtime

The requirements posed by compiling complicated XPages applications presents a similar dependency as above, but on an Equinox environment. Though it's possible to fake the basics of OSGi for known plugins, that wouldn't work for arbitrary third-party libraries.

For integration in Eclipse, this wouldn't necessarily mean any new work - Eclipse is already the premier Equinox product, and so it supports what XPages compilation needs innately. However, Eclipse wouldn't be the only target; any work done here should work with other IDEs like IntelliJ, but also continue to work IDE-free via a Maven or Gradle environment.

So this ends up being another strong argument for retaining the "separate process" model that already exists.

Incremental Compilation

Beyond retaining the runtime requirements, the big thing would be a switch to supporting incremental compilation. Currently, the compiler is designed to do everything in one pass: you point it at an ODP and it spits out a freshly-created NSF. This allows it to build up and tear down its environment cleanly, initializing the OSGi plugins for any XPages libraries at the start and doing similarly for any custom classpath jars to be included in the Java runtime.

What supporting incremental compilation would require to be at all speedy and efficient is having a persistent compilation environment. Instead of everything happening sequentially, the IDE would init the compiler process and then send it requests as files need compilation. This has implications for both local and server-based compilation.

Local complication would need to change less: mostly, it would require picking an IPC mechanism and having the launched Equinox process remain alive until it's no longer needed.

Server-based compilation would be similar in implementation, probably using something like HTTP long polling to be able to run in the Domino HTTP container. The trouble would be that a straightforward implementation of this would mean that the Domino server would pretty much have to be dedicated to a single IDE. There's already a potential conflict scenario with two developers doing compilation at the same time: since the XPages compiler needs to install and uninstall OSGi bundles, they could step on each others' toes if any of them overlap. Keeping the compiler environment resident on the server would mean it would have to be effectively locked out to one connection for long periods of time. Assuming HCL continues the Community Edition licensing model, this will be legal to do, but it's still cumbersome.

This could lead into something I've been mulling over: running a Domino compiler server in Docker. This would loosen a lot of the runtime requirements and mean that the encapsulated Domino server would be both dedicated to the purpose and consistent from the perspective of the compiler. Domino's setup requirements initially made it an awkward fit for Docker, but it looks like things have progressed along nicely.

This would all tie perfectly into the Language Server Protocol, which is an IDE-agnostic way to do basically this: have a little running process that knows about the nitty-gritty of the language, and then tell the IDE only what it needs to auto-complete and other features.

Live NSFs

Currently, the compiler starts with an ODP and emits a clean NSF with each build, and this is absolutely, 100% the correct way that it should work. However, Notes being what it is, it'd be expected that a Designer replacement would be able to work with a live NSF, so you could just crack one open, change a view, and be done. The second part of this process is in there, since the compiler uses the normal Notes APIs to store in an NSF as it is. It's the first part that would have to be new, allowing the tooling to selectively look into an NSF.

The exporter already does this, but, like the compiler, goes in one pass. What would potentially make sense would be to do essentially what Designer does: implement a VFS layer to represent an NSF in an equivalent way to the on-disk project. It's more easily said than done, but would be particularly straightforward for Eclipse, for the same reasons that it was straightforward for IBM to do it for Designer.

The secondary question here would be if it would be better to do continue to use DXL as the sole transport mechanism (so editing a view in a live NSF would export it to DXL and then re-import on save) or to instead try to represent things differently. Though DXL is less efficient, particularly for large notes, I think it'd make sense to stick with it - there would be tremendous work involved in trying to make it smarter, and that would be a breeding ground for bugs that just wouldn't exist with DXL.

IDE Features

Getting an NSF to compile dynamically is one thing, but the other part of this kind of project would be making the experience of working with design elements pleasant. In Designer, we have the benefit of having purpose-built editors for each design element type, but these aren't portable even if licensing allowed: the legacy ones all are just wrappers around C++-based "native" UIs, while the newer-era ones are based on Designer's bizarre internal RPC system.

I've done some work along these lines, initially to add autocomplete for custom controls and known core+ExtLib controls to .xsp files. Since that earlier work, I also added a contributor that tells Eclipse to use the DXL schema file for DXL files. While this doesn't give a proper GUI editor, it does provide enough information for Eclipse to pick up on the allowed elements and properties:

Eclipse DXL schema support

While I don't think it'd be worth trying to fully reproduce the various WYSIWYG editors Designer gives you (particularly the view editor, which is laughably bad for data-centric use), I think it'd be worth adding some editors along the lines of my old Forms 'n' Views project. Having some basic editors with a strong focus on the resulting data structure would be perfect for XPages support use and even mostly useful for legacy use.

Time

The core trouble with getting to all these goals, though, is time. For the main compilation and export work, I could justify spending a good amount of time because it eventually more than paid off in less time fighting with Designer to create consistent builds. For this other stuff, though, it's more dependent on whether my hatred for using Designer is enough to tilt the scales. Sometimes, it almost gets there, but I do also need to be able to pay my mortgage, so that puts a bit of a limit on things. It sure would be nice to leave Designer in the dust for good, though.

How the ODP Compiler Works, Part 6

Jul 7, 2019 12:46 PM

Tags: nsfodp
  1. Next Project: ODP Compiler
  2. NSF ODP Tooling 1.0
  3. NSF ODP Tooling Example Project
  4. NSF ODP Tooling 1.2
  5. How the ODP Compiler Works, Part 1
  6. How the ODP Compiler Works, Part 2
  7. How the ODP Compiler Works, Part 3
  8. How the ODP Compiler Works, Part 4
  9. How the ODP Compiler Works, Part 5
  10. How the ODP Compiler Works, Part 6
  11. How the ODP Compiler Works, Part 7

In this post, I'd like to go over another main component of the NSF ODP Tooling project, the ODP Exporter. The exporter is significantly simpler than the compiler, but still had a surprising number of gotchas of its own.

My goal in writing the exporter was to replace the need to use Designer to create an on-disk project out of an NSF - in one of my projects, in addition to the primary NSF we use, there are also a dozen or so secondary NSFs inheriting from templates and being modified by people not using Git (I know.), and keeping them all in sync is a giant PITA. Previously, I had a dedicated VM just to open the DBs periodically to sync them, but even that took a long time and got error-prone when Designer would miss a change or generally trip over itself.

So I set out to create a compatible replacement, so that I could run a script and update the ODPs en masse.

The Basics

At its core, the exporter does what you might expect: it reads through each design note and sends them through a DXL exporter. For its work, it makes use of the aforementioned design collection and IBM's NAPI. I went with IBM's variant in this case for one class: com.ibm.designer.domino.napi.design.FileAccess. Though this class let me down when it came to importing, it has just enough encapsulated composite-data reader methods to save me a ton of work here, though I had to cheat to access one of them.

For each design note, it determines its type, which contains behavioral information for each type, including whether it should be included in the normal export process at all, where it's placed in the ODP, and the type of export treatment it should get. The main categories of exported note types line up with what the compiler had to know about, with some special knowledge of whether a note is one of many in a folder (e.g. an XPage) or one-per-database (like the icon note).

The Gotchas

Unsurprisingly, things aren't quite as simple as a basic loop. For elements that are just DXL files, it is that easy, but the ones that exist either as just file data (e.g. "plugin.xml") or file data plus metadata require special handling.

Skipped Items

The first thing to note with split data/metadata items isn't complicated, but bears mentioning: the metadata file is generated by exporting the design note as DXL but ignoring data items. Some of these are common among all types, but others (like indexed "$ClassData0", etc. fields) are best matched and excluded with a regex.

LotusScript, Again

LotusScript libraries threw me for an unexpected loop this time. Their storage format is actually not even composite data: the script itself is stored in plain non-summary text items, multiple ones with the same name. However, the NAPI's DXL exporter doesn't actually export the full script test properly, instead only outputting the content of the first item. Additionally, the legacy Java API shows the presence of multiple items with the same name, but also only gives you the value for the first.

So I ended up having to use the raw note format in memory, which DOES include all of the items, and then stitch the script content together onto the filesystem.

Other File Data

The other data types aren't too complicated, but need special cases for each composite data structure, which is where the FileAccess class comes in. Without the convenience methods there, I would have had to write CD iterators to read the data based on the appropriate structures - not terribly difficult, but it's all the better to have the work already done for me. Especially so since FileAccess pleasantly writes directly to a java.io.OutputStream, just like I'd want if I wrote it myself.

Special-Case Files

There are three final special cases that the exporter handles:

  1. The icon note is specially-exported not once, but twice. It's exported using an NAPI-specific special method to create the "database.properties" file, which includes the ACL and and formatted settings alongside the icon note, and then also exported specially after the main loop as "Resources/IconNote". I've always appreciated how much Lotus wildly overloaded the icon note.

    • There's also a distinct "$DBIcon" note that houses the 32-bit icon introduced in R8 (if I recall correctly), but that's just a normal old image resource with a special name and not related to the icon note.
  2. The "META-INF/MANIFEST.MF" file resource became important in 9.0.1 FP10, but Designer's handling of it is a little schizophrenic. FP10+ will usually fill it in with plugin information when it rebuilds an NSF, but nonetheless exports it as a zero-byte file. It's important for it to exist, so I create a blank file if it doesn't exist.

  3. The Eclipse ".project" file is also not present in older NSFs, but is critical for ODPs. If it wasn't exported, I create a generic stub version.

Swiper

When dealing with on-disk projects, Swiper is a mandatory tool, cleaning up the generated XML and (critically) removing extraneous items that change too frequently to be source-control-friendly.

The core of Swiper is an XSLT stylesheet to do the transformation, and I incorporated this wholesale, with a minor modification to retain the ACL that's stripped out by stock Swiper. I then created an OutputStream implementation that passes DXL output through Swiper if configured. As a small note, I think there was a specific reason why I have the Swiper path buffer the DXL into an in-memory ByteArrayOutputStream first instead of just wrapping the file output stream, but I don't remember what that was.

Final Steps

With this post, I think I've covered the big topics I set out to with the two main components of the Tooling. I plan on having at least one final post in the series to cover some potential future additions and enhancements, since I have a lot of ideas in mind for it. Unfortunately, a lot of the most-useful ideas would also be tremendous amounts of work, but the payoff may eventually be worth it.