Ride the Camel!

or whatever ...

Flexible Filebased-Testing in Apache Camel

Introduction

Well it’s been “long time no see”, but finally I got to do some cameleering again ;-).

In my most recent project, I used a Test-Setup for testing a Camel-Route, I came to think, it may be worth sharing:

So, Let’s Start:

The actual work of Camel-Route that is to be tested (from now on called: Working-Route) is actually not important to understand the blog. But you need to know, that the route is servlet that wraps SOAP-Service-Calls. It transforms schemes and the Transformation-Result at the output-endpoint has to be validated.

For the Unittest of the Camel-Route I have some Testsamples organized in Files. The idea is now to read the files with another route and send them to the Files as new Exchanges to the servlet-input-Endpoint the Working-Route exposes. In the Test-Setup, the Working-Route sends it’s (transformed) Exchanges to a instantiated Validation-Route (configured to test for the transformed scheme), filling in for the SOAP-Service behind. The Response that goes through the initiating “file-reading”-Route wiretaps also to another instance of the Validation-Route testing for the backward-transformed Scheme.

File-Route

The File-Route looks like this: The “identifier” in the following code helps to the identify the Scheme to test for in this instance of the route.

    from("file:" + routeFolder + "?noop=true&readLock=rename&filter=#testfileFilter")                   // (*1)
           .autoStartup(false)                                                                          // (*2)
        .routeId("fileread" + identifier)
        .throttle(4)                                                                                    // (*3)
        .convertBodyTo(String.class)
        .wireTap("file://target/output/"+identifier)                                                    // (*4)
        .choice()                                                                                       // (*5)
            .when(simple("${in.body} not contains 'soap:Envelope'"))
                .setBody().simple(TransformationConstants.soapStart + "${in.body}" +             
                                                                      TransformationConstants.soapEnd)
        .end()
        .to(transformationProxyIN)                                                                      // (*6)
        .to(validateendpoint)                                                                           // (*7)
        .setHeader("CamelFileName", simple("${file:name.noext}_TransformationResult.${file:name.ext}")) // (*8)
        .to("file://target/output/"+identifier)                                                         // (*9) 

Explanation:
(1) The Fileroute is defined to read from a “routefolder” but doesn’t change or remove files (like the Camel-File-Consumer would in default)
A filter makes sure only Testfiles are read. If noop is on, the file-component automatically uses a idempotent-repository to ensure the same file is not read more than once.
(
2) Any instance of this Route is not active from the get-go. Each Unit-Test uses its own folder, starts the correspondent Route, so only the folder with the associated Testcases are read
(3) For Performance-Reasons on the Test-Machine, the parallel-processing of files is throttled to 4 in parallel.
(
4) A Requirement for the test is to direct the input & the output of the whole processing to a specific-folder in target. (Mainly to be able to compare them if not rightfully converted). This command-config directs a copy to the folder via the wiretap-EIP.
(5) If the sample-File doesn’t contain a SOAP-Header (Well, some Files do, some don’t, you’ll never know ;-)) the sample gets embedded into a soap-envelope
(
6) Sends the file to the Working-Route
(7) Is feeded with the response of the Working-Route and sent to the Validation-Route
(
8) Changes the CamelFileName-Header, which is used if the file-component is ever used again, which in fact is in 9.
Isn’t the “simple”-language a cute little thing. See how easy it is to change a file based on an old one.
(
9) See *4.s.. Sends the result to the same folder like the original read file.

Validation-Route

The Validation-Route has two interesting aspect - the Route that validates against a Scheme. And the errorhandling for validation-failures. Validation failures get send to the “mock:failure” endpoint. From this MockEndpoint the Unittest is able to evaluate the testresult.

 /**
  *  The "validationroute" configured in this Method, strips the soap-parts off the message and validates it against the xsd
  *  if validation succeeds, the Message-Exchange is sent to MockEndpoint "success".
  *  If Validation fails, a ValidationException is thrown and handled by the configured ErrorHandling
  * @throws Exception
  */
 @Override 
 public void configure() throws Exception {
     Namespaces soapNS = new Namespaces("s", "http://schemas.xmlsoap.org/soap/envelope/");

     onException(ValidationException.class)
         .maximumRedeliveries(0)
         .handled(true)
         .to("mock:failedValidation");

     from(validateendpoint)
         .routeId("validate" + identifier)
         .setBody().xpath("/s:Envelope/s:Body/*[1]", soapNS)
         .to("validator:" + schemaLocation)
         .wireTap("mock:success")
         .process(new ResponseFileProcessor())
         .setBody().simple(TransformationConstants.soapStartTixBank + "${in.body}" + TransformationConstants.soapEnd);
     }
 } 

Unit-Test

The test itself is relatively simple: - Count the Files (so you know how many results to expect - another requirement: If there are new samples put into folder, tests and assertions should work without changing code) - Start the file-reading route. - Assert the results from mock:success and mock:failedValidation - and finally stop the file-reading route

    @Test
    public void testBasicTransformationFunctionalityIN_10() throws Exception {
        int files = countFiles("filereadV10");
        camelContext.startRoute("filereadV10");
        validationAsserts(files, 0);
        camelContext.stopRoute("filereadV10");
    }

It may look a bit tricky and strange not to use java.io.File for this. It was just a try, but it worked, so i kept it, because it was the solution, that made the least work. You can use a Endpoint-Definition (that I already had for the filereading-route) to look how many exchanges may wait there for me. And then I used it to find out how many positive result have to be asserted for a successful test.

See the following piece of code:

    private int countFiles(String routeId) {
        BrowsableEndpoint browsableEndpoint = (BrowsableEndpoint) camelContext.getEndpoint(
                camelContext.getRoute(routeId).getConsumer().getEndpoint().getEndpointUri());
        return browsableEndpoint.getExchanges().size();
    }

The good thing about this implementation is also: it counts only the files the fileroute would also include. So let’s say you have a filefilter in place that takes only the xml-files. The BrowsableEndpoint respects that.

That’s it folks, hope you find it useful.

Easy Handmade SOAP-Webservice-Versioning With Apache Camel

Introduction

Have you ever wondered, how to create a lightweight versioning of your SOAP-Webservices? Or a simple way to create a middle-tier for scheme-conversions between a Webservice-Client and a Webservice-Server using different Schemes?

Then this post may be interesting to you.

A short while ago I was confronted with the exact problem I described as an Introduction. It was to create an integration help for a Business Process Management System. I mulled over that problem somewhat and came up with an example, how you could do this very easily (as i find) with my favorite Integration Framework.

As you may have noticed I am talking about Apache Camel. In this Post I’d like to explain the solution i have found.

Before i get into the details, you could check and try (see readme) that example for yourself: It’s located at github:

BPEL-Camel Example

Here in this post i will only repeat the interesting parts of it.

… Now let’s begin …

Business Case

Lets suppose, we have a Webservice for an specific business-purpose. The business case is invented, so please don’t give too much on the details of it.

I had to find a way to service webservice-clients that for example, know a Version 1-Schema of our Webservice, but need to be serviced by the Version 2-Webservice.

And vice-versa, meaning a V2-Scheme-Call should be able to be serviced by the V1-Webservice.

Configuration

I have two WSDLs in this example that differ only by the imported Xsd-Documents for V1 and V2 Schemas.

As Camel is well integrated with the CXF-Framework to provide Webservices, you only have to create some configuration. In my example i use Spring for my configuration purposes. Here’s a part of the camel-context.xml -Spring config:

For this simple example i use the the Jetty-Component to create a Http-Endpoint as a servlet for both webservices:

<bean id="jetty" class="org.apache.camel.component.jetty.JettyHttpComponent" /> 

<cxf:cxfEndpoint id="insuranceEndpoint-v1"
    address="http://localhost:9000/process1"
    wsdlURL="/wsdl/insuranceV1.wsdl">
    <cxf:properties>
        <entry key="dataFormat" value="MESSAGE" />
        <entry key="publishedEndpointUrl" value="http://localhost:9080/insuranceservice" />
    </cxf:properties>
</cxf:cxfEndpoint>
<cxf:cxfEndpoint id="insuranceEndpoint-v2"
     address="http://localhost:9000/process2"
    wsdlURL="/wsdl/insuranceV2.wsdl">
    <cxf:properties>
        <entry key="dataFormat" value="MESSAGE" />
        <entry key="publishedEndpointUrl" value="http://localhost:9080/insuranceservice" />
    </cxf:properties>
</cxf:cxfEndpoint>

The Endpoint-Address defined in the WSDL is overwritten with a configuration property: publishedEndpointUrl that points to that servlet i wanna use as a entrance to my camel-routes. This configuration takes care for the Endpoint that is printed into the wsdl when you call: http://localhost:9000/process1?wsdl

With this i can create a unified entrance to both webservices.

Camel-Routes: unified entrance for the webservices

But how is this handled in the camel-routes? See yourself:

package com.innoq.bpelcam;

public class SchemaEvalutionRouter extends RouteBuilder {

    @Override
    public void configure() throws Exception {

        Namespaces ns_v1 = new Namespaces("c1", "http://bpel.innoq.com/insurance/v1/types");
        Namespaces ns_v2 = new Namespaces("c2", "http://bpel.innoq.com/insurance/v2/types");

        from("jetty:http://localhost:9080/insuranceservice?minThreads=5")
            .choice()
            .when().xpath("//c1:CarInsuranceFindProcess", ns_v1)
                .setHeader("schemaVersion", constant("1"))
                .setHeader("targetVersion").xpath("//c1:targetVersion", String.class, ns_v1)    
                .to("seda:evaluateV1Scheme")
            .when().xpath("//c2:CarInsuranceFindProcess", ns_v2)
                .setHeader("schemaVersion", constant("2"))
                .setHeader("targetVersion").xpath("//c2:targetVersion", String.class, ns_v2)
                .to("seda:evaluateV2Scheme")
            .otherwise()
                .to("seda:fault");

        ...

So there’s a Jetty-Endpoint which starts the jetty-container implicitly.

If you’d like to use your average WebSphere-Web Container ;-) or a Tomcat you have to configure it a little bit different in the Spring-config, but this should be the topic for another post (or the Camel-documentation).

We have here a WS-Addressing-like (correct me, when i’m wrong) determination which Schema the Webservice-Caller uses with a simple XPath-Expression. The corresponding namespaces have to be defined at the beginning of the configure method.

For later evaluation this information is stored in the Camel-Exchange as Meta-Information (Header). Also the targetedVersion is recognized and stored for later evalutaion as a Header. Depending on the recognized Schema as V1 or V2, we incorporate a evaluation-route, which is build for each Webservice-Schema-Version:

    from("seda:evaluateV1Scheme")
        .choice()
            .when(header("targetVersion").isEqualTo("2"))
                .pipeline("direct:transformV1V2")
                .to("cxf:bean:orderEndpoint-v2")
            .otherwise()
                .to("cxf:bean:orderEndpoint-v1");           

    from("seda:evaluateV2Scheme")
        .choice()
        ...

    from("direct:transformV1V2")
        .bean(new V1TransformerBean(), "transformV1toV2");  

There the targetVersion comes in handy. So you don’t need no more XPath-evaluations to know where to go. So in that matter, for each non-original schema-version (V2,V3 and so on), you need to expand this route with another when()-Block.

The pipeline(“direct:transformV1V2”)-pattern, which enables to incorporate camel-routes (here: “direct:transformV1V2”) for definition is used in this example to show, that you can distribute your schema-conversion on different beans or/and routes as you wish.
The V1TransformerBean takes care in my case for the whole V1 to V2-conversion of the sent document. This bean has grown in a not so easy component, so i’m gonna explain in one of my next posts.

Last trick…

The last trick is that the whole exchange is sent in the end to .to(“cxf:bean:insuranceEndpoint-v2”) . This is a cxf-endpoint. This could be your actual implementation of the versioned Webservice:

 package com.innoq.bpelcam;

 public class WebservicesRouter extends RouteBuilder { 
    @Override
    public void configure() throws Exception {
        DataFormat jaxb10 = new JaxbDataFormat("com.innoq.bpel.insurance.v1.types");
        DataFormat jaxb11 = new JaxbDataFormat("com.innoq.bpel.insurance.v2.types");

        Namespaces ns_v1 = new Namespaces("c1", "http://bpel.innoq.com/insurance/v1/types");
        Namespaces ns_v2 = new Namespaces("c2", "http://bpel.innoq.com/insurance/v2/types");

        from("cxf:bean:insuranceEndpoint-v1")
            .id("handleProcess_v1")
            .transform().xpath("//c1:CarInsuranceFindProcess", ns_v1)
            .unmarshal(jaxb10)
            .process(new OrderContentsProcessor())
            .marshal(jaxb10)
            .to("seda:save")
            .process(new ResponseBuilderProcessor(10));

But you could also easily redirect the whole soap-document to another system (i.e. BPEL). Therefore it only needs to be configured in your camel-spring-configuration as Endpoint pointing to another system.

A side note here is: I always use the Java-Fluent-API, because with a little discipline in formatting, you could have your routes configured in a very transparent and clear matter. I’d prefer that over xml- or even graphical-configuration every time.

That’s it

So that’s it folks. If something is not clear to you, i am happy for feedback/questions ;-)

Cheers, Martin

Camel: Inherit Errorhandling

Hello All,

today i want to write sth new from my ongoing understanding of my work with Apache Camel. This time it’s not that of a big story as last time, but quite useful: Up to this point (actually a little bit in the past now) i thought when building Routes with Apache Camel in a org.apache.camel.RouteBuilder you have to write everything you want to configure in that respective configure()-method and that’s it.

Well that’s not true…

Camel: Application-Monitoring

I’ve been working with Apache Camel for a year now and i am still scraping at the top of all the possibilities you have with Camel.

In this blog i wanna show something, that might be helpful when Camel is already used and someone want to keep the statistics that your routes generate in a database for some sort of long time evaluations. I think it’s quite easy to do so and definitely not limited to handling statistics of camel-routes. If you have absolutely no idea what i could possibly mean with routes and statistics, i recommend some reading: Camel WebSite and for the jmx part itself: Camel JMX

Because this example here is build around camel-routes you’ll see the statistics kept by a ‘org.apache.camel.management.mbean.ManagedPerformanceCounter’
I want to persist those stats in a database using JPA. So i need a @Entity for this:

@Entity(name = "CAMELSTATENTRY")
@Table(name = "MyStats")
public class CamelStatEntry {

    private Long id;
    private String nameIdentifier;
    private Date statisticTimeStamp;
    private Long totalExchanges;
    private Long exchangesCompleted;
    private Long exchangesFailed;
    private Long minProcessingTime;
    private Long maxProcessingTime;
    private Long totalProcessingTime;
    private Long lastProcessingTime;
    private Long meanProcessingTime;
    private Date firstExchangeCompletedTimestamp;
    private Date firstExchangeFailureTimestamp;
    private Date lastExchangeCompletedTimestamp;
    private Date lastExchangeFailureTimestamp;

    @Id
    @Column(name = "ID")
    public Long getId() {
        return id;
    }
    ...

I think it’s not too complicated to see, what those values in the CamelStatEntry-Entity stand for.

Welcome

Welcome to my random thoughts i find worthy of a writedown. As i am about to start this here, blogs actually twittered out of any fashion, but i like them. You can learn a lot by reading the blogs of others. I won’t presume you could necessarily do the same while reading my blog, but i’ll give it my best shot.

Mostly i am into riding the Apache Camel. Up to now i rode mostly the WebSphere Application Server in my professional Java-Life. In order to keep up a steady progress ;-), i am trying to combine both in my most recent project i am glad to be part of.

Next up: Learning some new language. I am targeting the likes of Erlang, Scala or Ruby

Hopefully you’ll enjoy some of my posts