10 years of SOAP!

This is a historic week for SOAP .. it was on April 26, 2000 that the SOAP v1.1 specification was published. Then, it was 10 years to the date today that we published IBM SOAP4J, the first ever SOAP implementation!

Wow time flies when you're having fun.

This is a good time for me to remember some of the various milestones that I've been part of in the last 10 years related to SOAP and WS-*, which of course lead me to where I'm now in WSO2. This is a rambling post that I'm writing down to remember some fun things that happened in the last 10 years. If I missed acknowledging anyone that was not intentional! If I misquoted or misrepresented anyone again please accept my apologies.

Matt Duftler, Paco Curbera and I wrote IBM SOAP4J. When Microsoft released SOAP 0.9 back in Nov/Dec 1999, I ended up getting involved with the IBM team that was formulating a "response". That was basically because I was an "XML expert" at the time in IBM as I was one of the representatives on the XSLT Working Group and had done BML (Bean Markup Language, very similar to Spring but several years before Spring) and other various XML things. In other words, I got involved with SOAP pretty randomly :).

Early in 2000 IBM decided to join with Microsoft to help push SOAP. IBM was already working (secretly) with Microsoft on what would eventually become UDDI, so it was a natural thing to do. Before the v1.1 spec came out, I helped refine the drafts and at that time my group in IBM Research started the Java implementation.

(On a side note, before IBM decided to join SOAP, we created an internal alternative .. called SCUM ;-). It (luckily) never saw the light of day!!!)

I was also dead keen to release the source code for it. We worked hard to convince the IBM software group management that IBM would gain a lot by open sourcing the thing .. basically it was a chance to get an implementation out quickly which would allow people to play with it freely without fear. IBM had a process for approving open source contributions (of course). In an incredible chain of events, we managed to start and finish the process (including getting legal & IP clearance) in 3 days .. the code was ready on around April 25th and we took it thru the system and had it ready to go on April 28th! (Matt and I wrote most of the original bits and Paco came on board a bit later .. we had many late nights getting it done on time.)

So on April 28th, exactly 2 days after the spec was released, IBM announced the availability of IBM SOAP4J via alphaWorks, with full source code! I think the code was then under the IBM Public License but I can't remember for sure.

In May 2000, Matt & I attended WWW 9 in Amsterdam. That's where I first met Glen Daniels and also the ever-so-colorful Dave Winer. We also went for an interesting dinner, including a visit to an infamous "coffee shop" as well as stroll down the interesting parts of town! Later Glen was one of the first people to join the Apache SOAP project and became a major contributor very soon. Later of course he would lead Apache Axis, which was the first re-write of the Java SOAP stack.

While the source code was available, the project was still an IBM project - that is, it was not really open source. So then, with a lot of help from Sam Ruby, we ended up donating IBM SOAP4J to form the Apache SOAP project, then under the XML project. That was in June 2000 if I recall correctly.

In the meantime, Matt, Paco & I also came up with an interface description language for SOAP services. The original version was called XIDL and the later version (which was very much like WSDL but even more powerful) was called NASSL (Network Application Service Specification Language). The creation of NASSL had a huge impact on the direction of several Web services specs and the mindset IBM (in particular Don Ferguson, then chief architect of IBM middleware, now CTO of CA) drove into the specs: that the world of SOA was not just about SOAP. In fact Don was our God father .. he was the main guiding hand behind what we were doing.

Microsoft in the meantime published something called SDL (Service Description Language). Then we worked (secretly) with Microsoft to combine SDL with NASSL to form WSDL 1.0 and released it in September 2000. That was my first experience with inter-company techno-political negotiation! Quite fun :). Every time we hit a wall we'd escalate to Don and they'd escalate to Andrew Layman .. and Don & Andrew would have a "parental" meeting and resolve the conflict and the kids would be off playing again. A few months later IBM & Microsoft jointly contributed WSDL 1.1 to the W3C and set it off on the standards path. WSDL 2.0 (of which I was one of the editors and Jonathan Marsh, now VP Bizdev in WSO2, was chair of the working group) would come out much much later .. and unfortunately too late for wide adoption as WSDL 1.1 is with us (forever :-().

In January 2001, the W3C held their infamous Workshop on Web Services. IBM's position paper (written primarily by Don Ferguson) became a roadmap of what we ended up driving towards for the next 5+ years. I was a lowly Research Staff Member in IBM Research at the time, but Don (who was God of IBM Software Group) was an incredible mentor and he gave me opportunities that I can't imagine anyone giving a young kid (yeah I even had some hair). I now try to do that for other people .. probably not as well as Don. Rod Smith, who was then VP of Emerging Technologies in IBM, and an absolute STAR of IBM's executive family, became the spokesperson for IBM's Web services & SOA strategy. Rod is an amazing presenter and can articulate business value in incredible ways. The presentation that IBM gave at that workshop is here. As I made many of the slides (which is evidenced by how boring the slides are) I got my name put on it along with Don & Rod :-). If you read thru the paper and the presentation, you can see much of the WS-* vision laid out there!

In that presentation we had WSEL and WSFL. I was sitting next to Andrew while Rod was presenting and I remember his leaning over me and asking whether these were specs or placeholder names. I think I gave a mixed answer .. and that set us up to working on parallel specs and having to merge them. That's how WSFL and XLANG came separately and got combined into BPEL4WS.

It was during this time that I met Dr. Frank Leymann, Distinguished Engineer in IBM and head of IBM workflow technologies. Frank taught me how to spell workflow and gave Paco and me the opportunity to work with himself, Mark Thomas-Schmidt and others on WSFL. Paco and I got involved because we had in the meantime expanded BML work to add recursive composition and we convinced Frank that in addition to being a flow language, WSFL needed to define a service itself .. that is, it had to be a recursive composition language. WSFL didn't quite get that right but we got it right in BPEL4WS. Working on that spec was an absolute joy and a learning experience for me. Frank, Dieter Roller (now retired from IBM), Satish Thatte (Microsoft) and Jo Klein (Microsoft), Paco and I met a bunch of times to work out issues and merge XLANG and WSFL. Oh yes we needed parently supervision multiple times in that process too :-). Eventually we published the spec and my group of course did an immediate implementation (IBM BPWS4J, first impl of BPEL4WS) and had it ready on alphaWorks.

WSEL eventually got done as WS-Policy and then of course various policy domains (WS-SecurityPolicy etc.) got defined.

On the Apache front, there was incredible progress too. Apache SOAP was really a quick hack type thing. It used DOM, had a relatively constrained type mapping model and really had no support for SOAP headers. Glen lead the thinking about how to re-do it resulting in a large f2f meeting in Washington DC somewhere in December 2001 (I think). After that Glen and a few others started hacking hard on Axis which shipped sometime in 2002 and instantly became the #1 SOAP implementation in Java. Axis had its issues but it was dramatically more powerful than Apache SOAP and quite a bit more performant too.

I had run into Paul Fremantle in 1998 or so when he wrote an IBM Redbook on XML and XSL processing. I had also done BSF (Bean Scripting Framework, which is now part of the JDK as JSR 223 and is an Apache project). Sometime in 98/99 and Matt Duftler and I integrated BSF into the IBM JSP engine to make JSP pages multi-lingual. One of the languages we integrated to BSF was XSLT (via then LotusXSLT which became Apache Xalan) and Paul ran into this while talking about how to do XML and XSLT in JSP pages. He was either incredibly brave or incredibly stupid to even try that stuff at the time (!!!) but we started working together as he kept calling saying "the shit don't work"!

After Paul returned to UK, Paul took BML and made an EJB version of it called EJBML. I think we released the spec and code as part of BML but I can't find it any more. EJBML basically was inversion of control for EJB applications .. without us knowing that's what it was! Concepts from EJBML and BSC (Bean Scripting Components, which we even filed a patent for .. before I realized what patents were and stopped filing them) all went into SCUM (the SOAP competitor which never saw the light of day).

Paul then created Generalized Services Framework (GSF) which eventually got morphed and came out as WSIF - Web Services Invocation Framework, which later became another Apache project. WSIF was a runtime cousin of WSDL - had a pluggable binding model and standard API to interact with services, no matter how the wire interaction looked like. In fact, Paul and I wrote a CACM article in 2002 where we talked about taking COBOL stuff and making it into services thru WSIF. Interestingly, IBM actually shipped support for that in the early Process Server versions (which used WSIF for all invocations).

WSIF later became major input to JSR 109.

I also remember getting a request in early 2002 (approximately) saying "write up a JSR to standardize everything around WSDL and Java" (IBM and Sun were fighting at the time and the way the Java Community Process was set up, he who lead the expert group controlled all). That's what lead to JSR 110 - Java API for WSDL. Unfortunately, we didn't include language binding into that JSR .. and later Sun took control of that and went onto defining JAX-RPC JSR 101, lead by my friend Roberto Chinnici.

I moved back to Sri Lanka in August 2001, while still keeping my IBM Research job in New York. Around that time, I had convinced myself that implementing WS-* by layering it in front of J2EE as IBM was doing was the wrong way to do it. So in December 2002 (2001?) I remember giving a presentation to Rod Smith and David Bolokker suggesting a total rewrite of the entire WebSphere platform around XML and SOAP. Rod always thought WebSphere was too bloated and wanted to redo it but never could get resources to do it. Hey we were in research .. so no one could tell us what not to do :). So we started a project called the "Colombo Project" (because I was living in Colombo) which was to build a service execution platform from scratch.

In the meantime, I had helped start the Lanka Software Foundation in Sri Lanka to help Sri Lanka developers contribute to open source (not use FOSS but manufacture it). As I was working in Web services heavily, I knew where there were opportunities to write code. Axis/C++ was the first project done by LSF - 4 developers (donated by JKCS and Virtusa), 4 Pentium3 machines with 512MB RAM and a lab at Univ of Colombo due to the vision of (late) Prof. V.K. Samaranayake)).

In 2003 (I think) LSF took 4 Univ of Moratuwa interns to do a project. The task I assigned them was to take Apache Axis and rewrite it using a pull architecture (using Alek Slominski's XmlPull library) and make it run 10x faster. The team consisted of Srinath Perera (now Dr. & an architect in WSO2), Dimuthu Leelaratne (now a senior member of WSO2's security team Jeykumaran Chandrasegaram (now in UK) and Vairamuthu Thayapavan (working in Sri Lanka).

They delivered and the resulting effort, AxisMora, was contributed in 2003 to Apache Axis to form the seed for the Axis2 effort.

LSF applied for a grant from the Swedish International Development Agency and got $100k for a 1year effort. That's what funded the original Apache Axis2 team in Sri Lanka. We started the Axis2 effort with a f2f meeting in Colombo where Glen, Paul, Dims and various other people came to help do the initial design of Axis2. Srinath (who had just graduated) lead the effort along with several others (all of who are now doing PhDs in the US!). Axis2 too has been an incredible success with it now being the most popular Web services platform in Java.

So going back to the Colombo project, we got a working system done sometime in 2004. Paul was the software group supporter for that project and was actively involved with us. In September 2004 I took Colombo thru the entire software group hierarchy trying to get it out as a new product direction. Unfortunately it was too threatening to WebSphere (I even made a business plan for IBM!) and so it was to be killed and "knowledge transferred" to various work going on in IBM. In fact that was probably the right business decision for IBM!

That's when I decided to quit from IBM. Paul, myself and another person (who didn't end up joining WSO2 (yet)) had a secret meeting at his mother's place in London on December 21, 2004 to figure out plans to start a company to take the Colombo idea forward. We had expanded vision by then - our initial plan was for 3 products: an app server, an integration server and a process server.

Then of course 5 days after that the massive Boxing Day Tsunami struck Sri Lanka and Asia in general. 40,000 people died in Sri Lanka within a couple of hours. That's how Sahana was born .. now the world's leading disaster management system. I was very involved with that stuff for a few months so my company plans got delayed.

I finally quit IBM (with a lot of sad feelings) on April 15th, 2005. WSO2 was eventually formed in August 2005 and started on a journey which its still on ..

IBM went onto re-do their internal SOAP stuff taking various ideas from Colombo etc.. 6 months after I left however, we were able to convince IBM to kill that and join Apache Axis2. A bunch of us (6 people including Paul and myself IIRC) went to Austin and gave IBM a week of deep deep Axis2, Neethi etc. training to get them started. IBM is of course now a major contributor to Axis2 and ships it in WebSphere and a ton of IBM products.

So going back to SOAP for a second, I've had the luck and privilege to be deeply involved with 3 generations of SOAP implementations (IBM SOAP4J / Apache SOAP, Apache Axis and Apache Axis2). While each iteration has done major improvements, I still don't think we got it right! Axis is pretty much the front line of SOAP implementation architecture (yes I know there are many impls now and their differences) but there's room for a significant new rewrite :-). [I have recently started to supervise an MSc thesis which is going to do some PoC work in Erlang to do just that .. let's see where it goes!]

I would be remiss if I didn't specifically acknowledge the incredible role the Apache Software Foundation (of which I'm a proud member) has had in furthering the SOAP, WS-* and SOA agenda. If not for Apache SOAP there would not have been such rapid adoption of SOAP. Apache Axis made sure that the open source impls would continue to lead. Apache Axis2 made sure that that happened again. A ton of supporting projects (Sandesha, Kandula, Neethi, WSS4J, Rampart etc. etc.) all exist in Apache to give full coverage of WS-*. Now there's also Apache CXF which provides another (independent) implementation. Going beyond SOAP, Apache also hosts Woden, the only impl of J-WSDL and a ton of other projects like Tuscany, Ode, Synapse etc. which totally round out the SOA platform.

In short, if not for Apache's major support, Web services & SOA would not be where it is today. Thank you.

Update: Corrected typo in SOAP 0.9 date.