One of the problems with majority of Open Source projects is the lack of support for previous version. By the time you or your IT department deploys the software stack needed to run your fancy new application, it is a good chance that those components making your software stack have already released multitude of new version, patches or security updates. If the component breaks backward compatibility (deliberately or unintentionally) you are in big problems. You will either need to fix your application or stay with previous version of the component. The first one will cause you to make the quality testing all over again and if by the time you have tested again a new version emerges, you are in the endless loop.

As an example I’ll give the latest Apache Tomcat 5.5 release.
Version 5.5.32 was released on February 1st. Around that time a JDK Double.parseDouble Denial-Of-Service security issue was discovered (CVE-2010-4476) for which we created a workaround and on February 10th the 5.5.53 was released fixing that issue. You may think, wow! that’s cool, open source at its best since a traditional software vendor would need months for that. Not so fast!

Together with CVE-2010-4476 fix, a whole bunch of other code changes came along with version 5.5.33, which you can see listed in changelog. Those changes won’t probably break anything from 5.5.32 but you or your QE team will have to verify that statement for sure.

All those problems are foundations upon which companies like mine are basing their existence. We make sure that version we give you doesn’t break the compatibility and that it has all the relevant security issues fixed. We have our QE team that verifies all that before shipping the updates to our customers. At the end this makes your life and your IT department’s life a lot easier. At least you can blame someone if it’s not working.

So, how that works with JBoss Enterprise Web Server? Basically the same way as with any Red Hat Enterprise Linux component. If you are familiar with RHEL updates and .rpm files (or even Fedora for a more limited time) you know that component versions doesn’t change which is a first premise in ensuring the backward compatibility. Only security features and limited
number of enhancements (mainly performance related) are added to base code, tested and made as an update. Inside Red Hat we have a huge engineering team that implements those code updates and patches and handing over the updated code to equally huge Quality engineering team that verifies the resulting package doesn’t break anything.

However there is one small problem with all that. It’s targeted for Linux operating system. Rpm packaging system is deeply embedded inside Linux distribution so although the source package is multiplatform the end result isn’t. Since JBoss Enterprise Web Server was meant to be used on multiple platforms like Microsoft Windows and Oracle Solaris we needed something that would allow us to leverage exiting update code base while at the same time making is platform neutral. That’s how the new build system was born.

It uses Hudson for managing the produced artifacts and for kicking the builds on a particular native platform. Hudson was used because our existing JBoss lab which was the only one inside Red Hat having platforms other then Linux was using it. Although Hudson is designed primary for building Java applications, it has an option to execute external programs as part of a build process, so we used that feature to simply fire a shell script that actually does a build job.

On the following picture you can see how that looks in practice:

As you can see, quite complex, but at the end it’s a professional stuff. The Hudson build consists of two parts. The Master build prepares the source package from RHEL CVS server using the same logic rpm packaging does internally. It uses the package’s original source code and applies a set of patches. The build system does that in a loop for every component that makes EWS, starting from major components to a particular component dependencies. It also applies a set of local patches that are not part of original rpm .spec file. Those patches are used mainly for enabling the multiplatform builds and they don’t change a package functionality.

When the master job finishes it fires number of sub tasks, or Slave builds, which are executed on a particular native platform. The system itself uses a group of Solaris boxes for example and assigns a slave job to a fist free box. After the builds are done, produced artifacts are gathered by Hudson as a final build. Our QE team uses those builds to verify their integrity and functionality and if everything goes well they are delivered to the customers.

Updates are done in a similar way the full builds are done. Actually we always do a full build which is suboptimal and probably an area for future improvements. A special file is manually maintained that lists the actual changes. According to that list, the final update package is created which is basically a subset of a full build. The update might contain just a single entry (e.g Apache Tomcat’s catalina.jar) or any number of them.

This build system is also used for producing JBoss Enterprise Application Platform native components and connectors, and is currently in testing stage for producing PostgreSQL ODBC drivers as part of JBoss SOA.

I was intrigued by the fact that majority of Microsoft products are linked only to MSVCRT.DLL and that those CRT’s are not the same as one used or produced by Visual Studio 6.0 compiler and CRT sources. There are few articles you can find on the Internet about the topic, but I didn’t found any of them actually helpful, so I decided to dig in and create something usable.

With Visual Studio .NET back in 2002 Microsoft introduced a new CRT DLL and all hell broke loose. Since then each new version of Visual Studio ships with a new CRT DLL making the applications or libraries bound to that particular DLL. Here is the table that shows those version

• MSVCRT70.DLL Visual Studio .NET
• MSVCRT71.DLL Visual Studio 2003
• MSVCRT80.DLL Visual Studio 2005
• MSVCRT90.DLL Visual Studio 2008
• MSVCRT100.DLL Visual Studio 2010

The major problem that arises when using multiple CRT versions is explained in this and this MSDN article.

As you can see, one thing that Microsoft tries to convince you is that MSVCRT.DLL is of no interest to you and that you should use the one that comes with Visual Studio, and even more that you should make sure that each and every component is build using exactly the same version of C compiler. Nice in theory but if it happens that you need to embed already produced binaries compiled by someone else then this is not an option. So if MSVCRT.DLL is of no interest to you how come that lots of Microsoft programs are bound to that DLL instead eating their own dog food? I suppose there is a bit truth in that old latin say “Quod licet Iovi, non licet bovi”

Few things that MSDN article doesn’t mention are inherent cross bounday objects like stdio objects. If you redirect your stdout results can be surprising.

DLL

#include
#include

__declspec(dllexport) void __stdcall dllhi(void)
{
    printf("Hi from DLL\n");
}

Test program

#include
#include

__declspec(dllimport) void __stdcall dllhi(void);

void redirect(const char *fname)
{
    /* Reassign "stdout" to fname */
    freopen(fname, "w", stdout);
}

void sayhi(void)
{
    printf("This will go to the test.out\n");
}

int main(int argc, char *argv[])
{
    redirect("test.out");
    sayhi();
    dllhi();

    return 0;
}

Compile DLL

    cl /MD /DLL dll.c

Compile Test program

    cl /MD tst.c dll.lib

If you compile both DLL and Test  program with the same compiler everything works as expected. However if you compile tst.c with VS2003 and dll.c with VS2008 the application won’t even initialize complaining about Runtime initialization. Other way around works if you compile tst.c with VS2008 and dll.c with VS2003 but the message from DLL won’t be present in the output file. The reason is because each MSVCRTnn.DLL maintains its own copy of those system objects like stdio and stderr streams.

First problem is more easily handled although the error message box doesn’t say much about the reason of failure, but the second is more subtle. Everything seems to be working but you are loosing the data from the DLL that would go to redirected file. This means that each and every CRT stdio function is unsafe when crossing DLL boundaries even if you don’t pass them directly. Also almost all functions that deal with memory allocation are unsafe as well. This means that you cannot free memory that was allocated in a DLL. You cannot use Posix file descriptors as well. For example using an descriptor obtained by calling open() in DLL would result in access violation for any operation on that descriptor performed from the object file linked with different CRT. Just like for stdio and stderr each CRT maintains it’s own list of pseudo file handles. In worse case your data can end up in unexpected location if it happens that you call open() from different DLL’s.

So what happens if you link with system MSVCRT.DLL. Well it doesn’t solve the issues, but it won’t crash your application neither will your DLL (if that’s what you are building) won’t crash the host.

To fight the information loss across different CRT DLL boundaries one solution is to update the data in all CRT DLL’s currently loaded in process. This can be very complex as shown on PostgreSQL example. As you can see they try to load each and every possible MSVCRTnn.DLL and obtain a pointer to putenv() function and calling that function. However this still won’t work for a simple printf() with redirected stdio stream. Acutally I’m not even sure there is a solution for that, beside using the same CRT of course.

The only safe solution is to use the same CRT for your application and your application’s modules. This has some drawbacks of course, and the major one is that you simply must ensure that all dependencies are using the same compiler generation and thus linking to the same CRT DLL. Within Apache Httpd we were using good old Visual Studio 6.0 for producing official binaries, but recently there were some discussion to switch to Visual Studio 2010. One of the major reasons for that is because you just cannot download Visual Studio 6.0 any more, even if you’ve purchased MSDN subscription. It’s not maintained any more as well, and it cannot create 64 bit binaries so it’s logical to move forward to something more advanced.

There is however a solution to use the system MSVCRT.DLL without the need to ship the CRT redistributables which mandate using some sort of installer and Microsoft insist that this should be .msi. Solution is to use the Windows Driver Kit which is freely downloadable from MSDN and can be used for producing a binaries linked with MSVCRT.DLL. Driver kit is meant to be used for producing OS drivers and since this is very sensitive part of your system it must be very stable and error prone. You might ask yourself like I did: OK if drivers which first premise is stability must be compiled to MSVCRT.DLL then why everything else must be compiled with MSVCRTnn.DLL. Weird statement and I really have no clue why Microsoft decided to go that way. At least I didn’t find any technological reason for such a move. Well, I understand that MSVCRT.DLL in Windows XP is different the in Windows 7, but that could at least make sure that higher MSVCRTnn.DLL use data structures from MSVCRT.DLL each of them maintaining its own copy or parallel implementation.

Anyhow, I decided to go with the Windows DDK path. However this created a whole bunch of problems. One of the the first one I faced was that msvcrt.lib from DDK doesn’t export few symbols for functions or data that is in the MSVCRT.DLL like getpid(), _environ and _wenviron. I would like to think that this is just a bug rather then intentional. This can be solved pretty easy by creating a library which will export those symbols, but the drawback is that you must link to that library. At the end this means you must modify you make files to include that thunk library. Create a file named msvcrt_compat.def with the following content:

EXPORTS
    _getpid=msvcrt._getpid
    _environ=msvcrt._environ
    _wenviron=msvcrt._wenviron

Then create a thunk library with those missing symbols using

    lib /NOLOGO /NODEFAULTLIB /DEF:msvcrt_compat.def /MACHINE:X86 /NAME:msvcrt.dll /OUT:msvcrt_compat.lib

All you have to do now is to link your existing application with msvcrt_compat..lib and msvcrt_win2003.obj or msvcrt_winxp.obj from DDK. I have made a toolkit as part of my next pet project that does that automatically for you. It creates a custom Visual Studio compilation using Windows Driver Kit and Windows Software Development Kit. It’s currently hosted on sourceforge and it contains some additional stuff which you are probably not interested in, like optionally installing Cygwin and Perl. Anyhow for those that just need a compiler they will need the following.

• Windows Server 2003 R2 Platform SDK
  Make complete install inside default directory
  C:\Program Files\Microsoft Platform SDK for Windows Server 2003 R2
• Windows Software Development Kit for Windows 7
  Make complete install inside default directory.
  C:\Program Files\Microsoft SDKs\Windows\v7.0\
  It will also install a subset of Visual Studio 2008 (9.0) at
  C:\Program Files\Microsoft Visual Studio 9.0
• Windows Driver Kit version 7.1.0
  This is a DDK version for Windows 7 and Windows Server 2008r2 There are two versions of those at MSDN, so make sure to download version 7.1.0 or later. It installs in: c:\WinDDK\7600.16385.1
• Toolkit for making a custom MSVC distribution
  svn co https://myomake.svn.sourceforge.net/svnroot/myomake/trunk/misc/tools/windows/cmsc
  Make sure you read the README.txt files

If you install all the mentioned packages and checkout the toolkit from MyoMake sourceforge project, open a command prompt and cd to tools directory. Execute cmsc_compile.bat script and it will create a workable compiler distribution for you. Note that you are probably not allowed to redistribute that to someone else according to the Microsoft license terms (don’t say you haven’t be warned). You can call cmsc_makedist.bat which will create a .zip file which you can use for backup purposes or for your dev box. I have created a setup script for command line compilation (setenv.bat) which  when called set’s up the correct compiler in a similar way SetEnv.cmd does with Platform SDK. It also sets EXTRA_LIBS environment variable which you can use to add required thunk at link time as well as BUILD_CPU with selected architecture (i386, amd64 or ia64). After calling setenv.bat /x86 the upper examples can be compiled

    cl /MD /DLL dll.c %EXTRA_LIBS%
    cl /MD tst.c dll.lib %EXTRA_LIBS%

Note that this is not a users manual for that custom compilation script. I’m not even sure it will survive the next DDK version, but who knows. Microsoft might surprise us. Anyhow I’d like to hear your experiences with the subject. When making a comment make sure you read the rules.

In today’s world Java is used mainly to develop server based applications. On the other hand Java doesn’t offer API that can fully benefit from the features most modern operating systems provide.

The reason why I started the Apache Commons Runtime was the lack of features inside two other projects I’m working on: Apache Commons Daemon and Tomcat Native. Those two projects cover two areas in developing server based applications but are not related one to another in any way. Tomcat Native uses APR for creating high performance networking applications, and Apache Commons Daemon transforms the Java applications into daemons (or services for ones coming from Microsoft world) allowing those server based applications to run as such.

Major feature missing is a concept from another project I’m working on: The nice and shiny Apache Httpd that basically drives the nowadays Internet. That feature is graceful restart which allows zero down time for networking applications. In theory it is very simple concept. You create a socket in one process that then creates a process that actually handles the connections. Since one can bind a socket to an port only once, it has to be done only once for any ip:port combination. That socket is passed to the handler process allowing unlimited number of them. When you need a zero down time restart, you just create a new process informing the old one to stop accepting new connections. The old process will handle all pending connections until they are all closed and then shut down itself. However the new process will continue accepting any new connection thus basically although you restarted the process your server didn’t refuse any connection. Something like that is impossible with Java, because there is no way to pass the native object descriptors between distinctive processes. This leads to the service down time during shutdown/restart period which can last from few seconds to couple of minutes. Something like that is obviously not acceptable within the production systems. The standard way of handling restarts is by introducing multiple backend instances and using front end proxy that knows how to handle the session affinity and then restarting the backend nodes in a sequential order. Anyone that tried to setup such system will know how painful the maintenance of such systems can be.

To be able to pass the native object descriptors from one process to another inside JVM we need a native JNI layer that would allow us to duplicate those objects inside child process. Unfortunately duplicating is not enough because you don’t have an option to create Socket object directly from the native descriptor using standard Java runtime classes. Thus the complete Socket class rewrite was needed and we already had that inside Tomcat Native project. But…

The problem with APR

Tomcat Native uses Apache Portable Runtime (APR) that is operating system abstraction layer. When you look at the APR it is basically what rt.jar is for Java developer. It offers consistent API across multiple operating systems. However APR has few minor and one major drawback. The major one comes from design perspective for using it inside Apache Httpd. For each and every operation APR uses it’s own memory pool which allows easier programming by taking care of object life cycle and memory leeks. Sounds familiar? When you try to embed such system inside the environment that has it’s own memory and life cycle management system like Java with its Garbage Collector you are obviously duplicating things. Not only that, APR pools are not thread safe, so everything accessing the pools has to be synchronized. Inside Tomcat Native the APR pools are used in a way they were not designed for. Each and every object creates it’s own pool, thus introducing a huge memory overhead for small objects. Each APR pool when created uses 8K of memory which makes an 90% memory overhead for each socket. One other major problem is object destruction which makes things really hard to sync with Java’s GC which decides to destroy objects on its’s own order. APR requires however to destroy the objects in LIFO order, and that requires some nasty tricks just to prevent the JVM from crashing. Inside Tomcat Native we represent each native object by primitive long value, which allows zero GC for those objects, but on the other hand it doesn’t allow to use those objects as standard Java objects. Writing InputStream implementation is almost impossible because there is no way to inform the object about native’s object life time. At the end we cannot pass the Servlet socket to a third party library for custom processing, which is something anyone that tried to use the PDF library with Tomcat Native is aware of.

For Apache Commons Runtime we are using forked APR. Well, it’s not actually forked, but rather rewritten without using APR pools, and without the things that just sits there doing nothing. Like any abstraction layer APR comes with huge code base that is already done inside JVM like string, xml, date and url processing, to name a few, so we obviously don’t need that code.

Design

Apache Commons Runtime major design requirement was the ease of use. Java developers are simply scared of anything having native in the title, because this sort of things break their vision of the JVM as platform independent system. Thus we have a nice automatic native loader that loads all the needed libraries in the correct dependency order directly from the.jar file. All the user has to do is to call the o.a.c.r.Library.load method and the runtime will dynamically load the native library according to the operating system and the cpu JVM is running on.

Supported Operating systems

Unlike JDK itself, the Apache Commons Runtime is focused on the platforms that are used for server type of applications. Also since it’s a new projects we can focus on future, because applications using it still have to be developed. This means that selection of supported operating system is based on their deployment for server based applications

Initially supported operating systems are:

• Linux
  Kernel 2.6.18 and later. The reason is because we need the kernel to support the AIO
• Solaris
  Version 10 and later.
• Mac OS X
  Version 10.5 ad later
• HP UX
  Version 11 and later on PARISC and Intel Itanium

Adding support for other operating systems is relatively easy because the way how native source code is organized. Each platform has it’s own implementation directory with its own source files for platform specific code, together with common shared code.

This also allows to have strictly platform specific code together with corresponding Java classes. Windows Registry and WMI as well as SE Linux are examples of those platform specific features. Exception will be thrown if one tries to use Windows Registry classes on Linux platform. Although not portable it allows to write the portable applications that can behave differently and use platform specific features depending on the platform the application is running on. For example if the application is running on windows the developer can choose to store some configuration data inside windows registry, but use files or something else on other platforms.

Where to use it and for what

Hard to answer, but its primary target would be high performance server applications. As an example take a look at any networking server that occasionally needs a full restart.

Although modern Java application consists usually of some container that uses class reloading and dynamically reloads almost all classes that make a server application, if you need to reload the container itself, you will need a full restart. Also if your system starts throwing memory exceptions you will need to reconfigure the JVM and again do a full restart. With Apache Commons runtime it is possible to modify the application to be aware of multiple generations and while you start a new application instance with reconfigured JVM the old generation will still continue to serve the pending requests.

Other feature that is consequence of passing the socket descriptors between processes is the possibility to run the connection handler or application itself under a different security context from the one used to create those sockets. On Unixes only the privileged processes can create sockets that are bound to ports lower then 1024. However running a process under such high privileges is a security concern, and Apache Commons Runtime allows to dynamically downgrade the security context under which the application runs. Unlike traditional Apache Commons Daemon where the effective user id is only changed, here you can fully change the user credentials. This means that your custom application will not be able to create any objects that require the privilege not granted to the effective user.

As final you have an option to launch multiple child instances or workers. This increases the application availability, because if one of the instances dies because of JVM issues or faulty code, only the client connections that were bound to that instance will be lost. Today this is usually done by running multiple application instances on the same box and fronting it with Apache Httpd web server with mod_proxy connecting to those instances. This of course introduces additional point of failure (actually three of them) into the system.

Additional benefit of worker concept is that you can fine grade your JVM. Since multiple instances of the JVM can be launched there is no need to use huge memory settings or use 64-bit JVM’s that will allow +3GB limit since multiple worker instances can be launched each serving limited number of connections.

Conclusion

Apache Commons Runtime is currently in development stage as sandbox project meaning it’s currently my own toy. When I finalize the API I’ll apply for a standard commons project. Any comments are more then welcome, but keep in mind that sandbox mean:

Don’t knock over my castle.
If you want to create your own castle, then ask for your own sandbox
(I’ll copy your castle later if it looks better than mine).