Environment Management in WSO2 ESB using WSO2GREG

In ESB, when environment changes( dev-->qa-->prod) , that affects the proxy service configuration. Proxy service configuration includes sequences,endpoints,local entries etc. For different environments, endpoint configuration might change.(ie: the service url which is given as address uri)

Sample endpoint configuration 

<endpoint xmlns="http://ws.apache.org/ns/synapse" name="TestEndpoint">
   <address uri="http://abc.com">
      <suspendOnFailure>
         <progressionFactor>1.0</progressionFactor>
      </suspendOnFailure>
      <markForSuspension>
         <retriesBeforeSuspension>0</retriesBeforeSuspension>
         <retryDelay>0</retryDelay>
      </markForSuspension>
   </address>
</endpoint>

When we develop  ESB/Synapse artifacts, we might not prefer to modify the configuration for different environments. Users just want to deploy the same artifact in all different environments. To make the deployment easy in all environments, we need a running WSO2 Governance Registry instance and we need to mount the config/governance registries of WSO2ESB to different target paths of WSO2 G-Reg. The paths will indicate the different environments.

Eg:

<mount path="/_system/config" overwrite="true">
    <instanceId>config</instanceId>
    <targetPath>/_system/config/environment/production</targetPath>
</mount>

We need to upload the endpoint configuration for different environments( with the correct endpoint url) directly to the registry in the relevant environment paths.

The following sample configurations are related to  one production ESB, one qa ESB and a common G-Reg instance. All three instances will ponit same external DB. eg: mysql


Configurations 

1.  WSO2 G-Reg

master-datasource.xml

Create a new data source to point an external database. Eg: mysql.

 <datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource used for ESB mount registries</description>
            <jndiConfig>
                <name>jdbc/esbserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/esb</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

Use the created above datasource in the registry.xml

<!-- This defines the default database and its configuration of the registry -->

    <dbConfig name="wso2registry">
        <dataSource>jdbc/esbserver_config</dataSource>
    </dbConfig>

Copy the mysql jdbc driver in repository/components/lib folder and start the server. Make sure to create tables in the mysql database. You can find the relevant sql scripts in the CARBON_HOME\dbscripts folder.

We need to set different port offsets to all carbon servers in carbon.xml. For G-Reg, let’s keep the default value (ie: ‘0’)

2 WSO2ESB-QA

master-datasource.xml
Create a new data source to point the mysql database which is already created.

 <datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource used for ESB mount registries</description>
            <jndiConfig>
                <name>jdbc/esbserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/esb</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

We need to mount the config registry to different target path to make sure QA ESB server talks to different config registry. Governance registry space of all nodes will be mounted to same location, since governance space is common for all nodes.

  <dbConfig name="config">
       <dataSource>jdbc/esbserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>config</id>
        <dbConfig>config</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/config" overwrite="true">
        <instanceId>config</instanceId>
        <targetPath>/_system/config/env/qa</targetPath>
    </mount>

     <dbConfig name="governance">
       <dataSource>jdbc/esbserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>governance</id>
        <dbConfig>governance</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/governance" overwrite="true">
        <instanceId>governance</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount>

Copy the mysql jdbc driver in repository/components/lib folder. Set different port offset in carbon.xml. For ESB_QA server, let’s keep it as ‘1’. Start the server.
Go to https://localhost:9444/carbon/ URL and check the registry browser, you will see the registries are mounted.

3 WSO2ESB-PROD

master-datasource.xml
Create a new data source to point the mysql database which is already created.

 <datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource used for ESB mount registries</description>
            <jndiConfig>
                <name>jdbc/esbserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/esb</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml
We need to mount the config registry to different target path to make sure PROD_ESB server talks to different config registry. Governance registry of all nodes will be mounted to same location, since governance space is common for all nodes.

<dbConfig name="config">
       <dataSource>jdbc/esbserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>config</id>
        <dbConfig>config</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/config" overwrite="true">
        <instanceId>config</instanceId>
        <targetPath>/_system/config/env/prod</targetPath>
    </mount>

     <dbConfig name="governance">
       <dataSource>jdbc/esbserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>governance</id>
        <dbConfig>governance</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/governance" overwrite="true">
        <instanceId>governance</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount>

Copy the mysql jdbc driver in repository/components/lib folder. Set different port offset in carbon.xml. For ESB_PROD server, let’s keep it as ‘2’. Start the server.
Go to https://localhost:9445/carbon/ URL and check the registry browser, you will see the registries are mounted. 

4. Endpoint configuration

We need to directly upload the endpoint configuration to the registry.Go to G-reg management console and upload the endpoint configurations(xml file) for the QA and Production environment under relevant target paths.
Eg: Upload the testEndpoint.xml of the qa environment to /_system/config/env/qa path. Upload the
testEndpoint.xml of the production environment to /_system/config/env/prod path.

Sample endpoint conf for QA-ESB server

<endpoint xmlns="http://ws.apache.org/ns/synapse" name="TestEndpoint">
   <address uri="http://<QA SERVICE ENDPOINT URL>">
      <suspendOnFailure>
         <progressionFactor>1.0</progressionFactor>
      </suspendOnFailure>
      <markForSuspension>
         <retriesBeforeSuspension>0</retriesBeforeSuspension>
         <retryDelay>0</retryDelay>
      </markForSuspension>
   </address>
</endpoint>

Sample endpoint conf for PROD-ESB server

<endpoint xmlns="http://ws.apache.org/ns/synapse" name="TestEndpoint">
   <address uri="http://<PROD SERVICE ENDPOINT URL>">
      <suspendOnFailure>
         <progressionFactor>1.0</progressionFactor>
      </suspendOnFailure>
      <markForSuspension>
         <retriesBeforeSuspension>0</retriesBeforeSuspension>
         <retryDelay>0</retryDelay>
      </markForSuspension>
   </address>
</endpoint>

5. Proxy Configuration

When we create proxy configuration we should not directly provide any of the address url. Rather we need to point the registry key. This registry key will be same for all environments. That means we do not need to modify the proxy configuration when we deploy that to qa or production environment.
Eg:  < endpoint="conf:/testEndpoint">

Sample proxy configuration

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="testEnvProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target endpoint="conf:/testEndpoint"/>
   <description/>
</proxy>

Related post
Environment Management in WSO2 DSS using WSO2GREG

Environment Management in WSO2 DSS using WSO2GREG

1.0 Introduction

In data services, environment change( dev-->qa-->prod) affects the datasources configuration. In the data source configuration, we need to provide database url, username, password.

Sample datasource configuration

 <datasource>
      <name>dss</name>
      <definition type="RDBMS">
           <configuration>
              <driverClassName>com.mysql.jdbc.Driver</driverClassName>
              <url>jdbc:mysql://localhost:3306/dss</url>
              <username>root</username>
              <password encrypted="true">E61Pys3tVBbLagNo+co9hChhwh6J8MZ/+CU4Z8vw6ABv9YR4SZAf9o3OZmUpLWVz8dKhPGrqPYnwIMw3InkJvpHea7xzEljrX9PF3PY/ax+Fe0upBhGIe+jvSDUai5RAb7cxSwhffnhx1JQZroieeIDxb6n1cpuRN8wd/z4zmtk=</password>
         </configuration>
    </definition>
</datasource>

For different environments , we might need to change

• Database url
• Database username
• Database password

2.0 Governance Registry

To manage different environments, we need a running WSO2 Governance Registry instance and we need to mount the config registries of WSO2 DSS to different target paths. The paths will indicate the different environments.

Eg:

<mount path="/_system/config" overwrite="true">
    <instanceId>config</instanceId>
    <targetPath>/_system/config/env/prod</targetPath>
</mount>

All servers will talk to same database.( That is, wso2greg and wso2dss.)

We need to upload the datasource configurations for different environments directly to the registry in the relevant environment paths. Note that, we can not use carbon datasource UI options available at dataservice server. Because WSO2 G-Reg will be the centralized server to manage different environments, so we need to avoid creating datasources from dataservice server.

3.0 Configurations 

3.1 WSO2 G-Reg

master-datasource.xml

Create a new data source to point an external database. Eg: mysql.

 <datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource used DSS mount registries</description>
            <jndiConfig>
                <name>jdbc/dssserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/dss</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

Use the created datasource in the registry.xml

<!-- This defines the default database and its configuration of the registry -->

    <dbConfig name="wso2registry">
        <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>

Copy the mysql jdbc driver in repository/components/lib folder and start the server. Make sure to create tables in the mysql database. You can find the relevant sql scripts in the CARBON_HOME\dbscripts folder.

We need to set different port offsets to all carbon servers in carbon.xml. For G-Reg, let’s keep the default value (ie: ‘0’)

3.2 WSO2DSS-QA

In this sample, let’s take two DSS servers for QA and production environments.

master-datasource.xml
Create a new data source to point the mysql database which is already created.

<datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource for DSS</description>
            <jndiConfig>
                <name>jdbc/dssserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/dss</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

We need to mount the config registry to different target path to make sure QA DSS server talks to different config registry. Governance registry space of all nodes will be mounted to same location, since governance space is common for all nodes.

<dbConfig name="config">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>config</id>
        <dbConfig>config</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/config" overwrite="true">
        <instanceId>config</instanceId>
        <targetPath>/_system/config/env/qa</targetPath>
    </mount>

     <dbConfig name="governance">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>governance</id>
        <dbConfig>governance</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/governance" overwrite="true">
        <instanceId>governance</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount>

Copy the mysql jdbc driver in repository/components/lib folder. Set different port offset in carbon.xml. For DSS_QA server, let’s keep it as ‘1’. Start the server.
Go to https://localhost:9444/carbon/ URL and check the registry browser, you will see the registries are mounted.

3.3 WSO2DSS-PROD

master-datasource.xml
Create a new data source to point the mysql database which is already created.

<datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource for DSS</description>
            <jndiConfig>
                <name>jdbc/dssserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/dss</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

We need to mount the config registry to different target path to make sure PROD_DSS server talks to different config registry. Governance registry of all nodes will be mounted to same location, since governance space is common for all nodes.

<dbConfig name="config">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>config</id>
        <dbConfig>config</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/config" overwrite="true">
        <instanceId>config</instanceId>
        <targetPath>/_system/config/env/prod</targetPath>
    </mount>

     <dbConfig name="governance">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>governance</id>
        <dbConfig>governance</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/governance" overwrite="true">
        <instanceId>governance</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount>

Copy the mysql jdbc driver in repository/components/lib folder. Set different port offset in carbon.xml. For DSS_PROD server, let’s keep it as ‘2’. Start the server.
Go to https://localhost:9445/carbon/ URL and check the registry browser, you will see the registries are mounted.

3.4 Datasource Configuration

When we create carbon datasource from UI, the password will be encrypted and saved to registry. Here since we need to directly upload the data source configuration to the registry, we can just upload the password in plain text. If user needs to encrypt the password, later he can access the datasource via datasource UI and save it. So, it will be encrypted and saved to registry.

Sample data source for QA-DSS server

<datasource>
    <name>testdatasource</name>
    <definition type="RDBMS">
        <configuration>
            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
            <url>jdbc:mysql://localhost:3306/service_qa</url>
            <username>root</username>
            <password encrypted="false">root</password>
        </configuration>
    </definition>
</datasource>

Save the configuration as testdatasource and upload in WSO2 G-Reg server under qa environment. Go to, https://localhost:9443/carbon and browse to /_system/config/env/qa/repository/components/org.wso2.carbon.ndatasource path and upload.

Sample data source for PROD-DSS server

Copy the same configuration and edit the parameters to reflect the production database configurations and save as testdatasource.

Eg:

<datasource>
        <name>testdatasource</name>
        <definition type="RDBMS">
            <configuration>
                <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                <url>jdbc:mysql://localhost:3306/service_prod</url>
                <username>root</username>
                <password encrypted="false">root</password>
           </configuration>
      </definition>
</datasource>

  Go to, https://localhost:9443/carbon and browse to /_system/config/env/prod/repository/components/org.wso2.carbon.ndatasource path and upload.

Now same datasource configuration is available under two paths with the environment specific parameters. But the datasources’ name is same.(ie: testdatasource)

3.5 Dataservice

When we create dataservice, there is no any environment specific parameters. Only thing is, we need to point the datasource there, which contains environment specific parameters.

We already separated and stored the different datasource configurations with same name in different registry paths. So, if we upload same dataservice in both servers(ie: DSS_QA and DSS_PROD) both servers will talk to the relevant data source configurations, which are stored under relevant config registry without any issue.

Sample dataservice

<data name="dss_test">
   <config id="testdatasource">
      <property name="carbon_datasource_name">testdatasource</property>
   </config>
   <query id="select_person" useConfig="testdatasource">
      <sql>call `person-info`(?, ?, ?);</sql>
      <result element="people" rowName="person">
         <element column="PERSON_NAME" name="name" xsdType="string"/>
         <element column="PERSON_NICK" name="nick" xsdType="string"/>
         <element column="PERSON_PHONE" name="phone" xsdType="string"/>
      </result>
      <param name="pName" sqlType="STRING"/>
<param name="pPhone" sqlType="STRING"/>
      <param name="pNick" sqlType="STRING"/>
   </query>
   <operation name="select_person">
      <call-query href="select_person">
         <with-param name="pName" query-param="pName"/>
         <with-param name="pPhone" query-param="pPhone"/>
         <with-param name="pNick" query-param="pNick"/>
      </call-query>
   </operation>
</data>

Related post
Environment Management in WSO2 ESB using WSO2GREG

Processing large text file using smooks in wso2esb

Smooks  library supports large file processing which is used in ESB to process file which are in text/xml format.
Text files can be a CSV file or with a space delimiter or a fixed length file. To process large file, smooks write the output stream , which is a single line record, to a temporary file or a jms queue. After writing all files to the temporary location, we need to process all those records/files again. This is how we can avoid large memory growth/out of heap space issue which occurs when we try to process large file as in- memory process.

Following is a sample configuration to process fixed length large text file in wso2esb.

Proxy configuration

To pick and process the file form a file location , we could use VFS transport.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="cor_file_pull"
       transports="https,http,vfs"
       statistics="disable"
       trace="disable"
       startOnLoad="true"target>
     <inSequence>
  <property name="DISABLE_SMOOKS_RESULT_PAYLOAD"
                   value="true"
                   scope="default"
                   type="STRING"/>
          <smooks config-key="simple-smook">
            <input type="text"/>
           <output type="xml"/>
         </smooks>
       </inSequence>
      <outSequence/>
   </target>
   <parameter name="transport.vfs.Streaming">true&lt;/parameter>
   <parameter name="transport.PollInterval">10&lt;/parameter>
   <parameter name="transport.vfs.ActionAfterProcess">MOVE&lt;/parameter>
   <parameter name="transport.vfs.MoveAfterProcess">C:/Users/TOSH/Desktop/success</parameter>
   <parameter name="transport.vfs.FileURI">C:/Users/TOSH/Desktop/in&lt;/parameter>
   <parameter name="transport.vfs.MoveAfterFailure">C:/Users/TOSH/Desktop/failure</parameter>
   <parameter name="transport.vfs.Locking">disable&lt;/parameter>
   <parameter name="transport.vfs.FileNamePattern">^(cor).+</parameter>
   <parameter name="transport.vfs.ContentType">text/plain</parameter>
   <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   <description/>
</proxy>

Smooks configuration

simple-smook localentry

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:ftl="http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd" xmlns:fl="http://www.milyn.org/xsd/smooks/fixed-length-1.3.xsd" xmlns:file="http://www.milyn.org/xsd/smooks/file-routing-1.1.xsd">
<params>
            <param name="stream.filter.type">SAX </param>
           <param name="default.serialization.on">false </param>
         </params>

<fl:reader fields="RecordId[4],CorpName[60],FileNumber[9]" skipLines="0">
</fl:reader>
      <resource-config selector="record">
         <resource> org.milyn.delivery.DomModelCreator</resource>
     </resource-config>
      <file:outputStream openOnElement="record" resourceName="fileSplitStream">
         <file:fileNamePattern> ${.vars["record"].FileNumber}.xml</file:fileNamePattern>
         <file:destinationDirectoryPattern> C:\Users\TOSH\Desktop\processed</file:destinationDirectoryPattern>
         <file:highWaterMark mark="10000000"> </file:highWaterMark>
      </file:outputStream>
      <ftl:freemarker applyOnElement="record">
         <ftl:template> /repository/resources/smooks/record_as_xml.ftl</ftl:template>
         <ftl:use>
            <ftl:outputTo outputStreamResource="fileSplitStream"> </ftl:outputTo>
         </ftl:use>
      </ftl:freemarker>
   </smooks-resource-list>

Each record will   be written to the specified file location   "C:\Users\TOSH\Desktop\processed" with the name constructed through the parameter "  ${.vars["record"].FileNumber}.xml ".  Here it is a xml file, which will be named with a file number    record in the text file. We have to select a unique parameter to write the processed files else there will be file overwriting issue will be thrown in smooks.
To process large file we have to use 'SAX' filter type. DOM needs more memory and it will lead to out of memory issue.
In this sample, i have used DOM+SAX mixing model to keep the lower footprint of memory.
High  water mark provides the maximum file records need to be written. provide enough number based on the lines in the text file.
The property ;
 <property name="DISABLE_SMOOKS_RESULT_PAYLOAD"   value="true" scope="default"     type="STRING"/>

will be available in ESB 4.9.0. If you use lower version, you might need a patch to be applied to your system.

XML template file
record_as_xml.ftl

<#assign rec = .vars["record"]>
<p:insert_acc_data_file xmlns:p="http://ws.wso2.org/dataservice"><p:RecordId>${rec.RecordId}</p:RecordId>
<p:CorpName>${rec.CorpName}&lt;/p:CorpName>
<p:FileNumber>${rec.FileNumber}&lt;/p:FileNumber></p:insert_acc_data_file>

Based on above template the records will be formatted and will be written to the .

Using another vfs proxy, we can pick the entries written to the  'processed' file and do further processing.

Get WSDL service address with WSDL4J

Just a short code snippet to show how one may access service, port, address elements from a WSDL definition by using WSDL4J.

Rollback Recovery for Non-Deterministic failed events for web services hosted in Axis2

Rollback techniques in message passing systems have been one of the primary areas in message recovering techniques.One significant area where Rollback recovery has played a crucial role is in the area of web services. Web services are now most popular and widely used mechanism in enterprise business, telecommunication, banking world.In many cases, catastrophic failure is a result of network failures,transport errors and web services are down. If we consider underlying transport is stable and server is stable, we could consider the failures in mainly two categories,which are Deterministic failures and Non-deterministic failures. Recovering from Deterministic failures are much more simpler than Non-deterministic events, since services are aware about the errors. But recovering from Non-deterministic events failure is a challenging part, because it is non determined event failure to the service point of view.

                   System downtime  makes loss to an Enterprise and causes other organizational issues. Over the years, there has been a great amount of effort to improve the recovery techniques in messaging systems. With the increase of recovery techniques, IT systems are able to reduce the cost which occurs due to system errors. Failures due to downtime or incorrect results in an end to end system might have considerable monetary penalties. Failures can occur in several levels and can be categorized in different aspects. For example, client side failures, server side failures, transport level failures, processing time errors

                    From the service level, we could categorize the failures based on two factors, which are known failures to that service and unknown failures to that service. Further we could say, that known failures occur when the service sending out messages. Those can be called as Deterministic events for that service. Unknown failures can occur when the service receives requests and that particular service is not in a state to process those requests. Service is not aware, when the request will be arrived. So, if the service is down when client sends request, service won't be able to receive them. Such errors are called Non-deterministic errors to that service.

                   In case of failures, web services can track , recover failures from Deterministic events easily, since those failures are known to web services. Because, in the Deterministic event errors, service is up and running. But when a failure occurs in Non-deterministic events, web services can not track them or unaware about those failures. Such Non-deterministic errors occur when service is down. That is, when service is not up and running, service can not receive requests. So, those requests will be lost. In this case, we make an assumption that, requests are lost because of the service failure.Those failure events are currently omitted under category , such ,”Requests made to the service within the period of service down time”

                For Non-deterministic events, we can apply the Rollback recovery techniques to Rollback the system/service to a consistent state and can replay the failed events. In case of Non-deterministic events, if we can record all Non-deterministic events happen to a web service we can recreate lost state by replaying those events from a known initial condition. The reconstructed consistent state is not necessarily one that has occurred before the failure. It is sufficient that the reconstructed state be one that could have occurred before the failure in a failure-free, correct execution, provided that it be consistent with the interactions that the system had with the outside world.[1]

                       As a solution for Rollback-Recovery for Non-deterministic events failure in web services hosted in Axis2 , Log-based recovery mechanism contains more advantages than Check-point based recovery technique.

Figure:In a failure situation how log based system performs

Assumptions are, 

• All Non-deterministic events can be identified and their determinants logged to stable storage 
• Failures occur after the messages are logged to the stable storage. 
• Underlying transport channel is reliable. 
• No orphan processes.
• Assuming single service domain. 
• There is no inter process connection in the incoming messages. 
• Server is reliable. 

A proof of concept design for adding "Log based Roll back Recovery" for Non-deterministic failed events in Axis2.:

           The determinant of each Non-deterministic event is needed to log to a stable storage before the event is allowed to affect the computation. An axis2 message log handler is kept at the Inflow of the message. When clients sends request to a service, before service receive those  requests, this handler receives them and log them to the a stable storage(database). All these incoming requests are non-deterministic events for services, which are hosted in a server.

             At initial phase, all incoming requests are logged to the database. At that time , system does not aware about which will be the success message and which will be the failed message. To differentiate both message another Axis2 handler is placed at the outflow of the Message and identified the Non-deterministic failed events .

         When the service becomes to Active state from InActive state, the Observer Process has to monitor it and needs to check the database for any non-deterministic failure messages for that particular service and replay them. If failed requests are found in the database, those are replayed to the service. To replay the failed Non-deterministic events to service, a Serviceclient is used. The service client will construct message payload back from the saved message and service, operation details are set to the message. Using Service client the failed Non-deterministic message can be replayed back to the service successfully.

Reference
[1] E.N. Elnozahy, Lorenzo Alvisi , Yi-Min Wng , David B. Johnson “A Survey of Rollback-
Recovery Protocols in Message-Passing Systems” ACM Computing Surveys (CSUR) ,Vol.
34, Issue 3,pp. 375-408 , sept 2002

Send back a custom error response in Axis2Handler

In some cases, user wants to send back a custom response to client and may want to abort the message flow for some error scenarios. This can be done in a handler, which can be kept at in/out flow.

Eg:
 In a messageprocess handler, if an exception occurs and if you want to endup the message flow and send back a custom message to client, use "AxisEngine.sendFault(messageContext)".
If we try to set up fault message in the MessageContext as soap body, then we need to set quite number of parameters to send back that message to client.eg: "to" address...

public class MessageProcessorHandler extends AbstractHandler {
    public InvocationResponse invoke(MessageContext msgContext) throws AxisFault {
        try{
                .................
           }

       catch (MessageFaultException e) { 
           String faultPayload = getFaultPayload(); //get a fault message
           MessageContext faultContext = getFaultMessagecontext(msgContext,faultPayload);
           AxisEngine.sendFault(faultContext);
       return   InvocationResponse.ABORT;
     }
............
 return InvocationResponse.CONTINUE;
}

Adding simple Axis2 Listener/Observer to monitor services

To monitor  hosted service statuses, register a listener in the axis2.xml.  The listener should implement  the "AxisObserver". Currently four statuses can be monitored ;

• Services
• Start
• Stop
• Deploy
• Undeploy
• Modules
• Engage
• Disengage
• Deploy
• Undeploy

Eg:

package org.test.axisobserver;
import java.util.ArrayList;

import org.apache.axiom.om.OMElement;

import org.apache.axis2.AxisFault;

import org.apache.axis2.description.AxisModule;

import org.apache.axis2.description.AxisService;

import org.apache.axis2.description.AxisServiceGroup;

import org.apache.axis2.description.Parameter;

import org.apache.axis2.engine.AxisConfiguration;

import org.apache.axis2.engine.AxisEvent;

import org.apache.axis2.engine.AxisObserver;

/**

 * Axis observer to get notification of service activate/deactiavte/deploy/undeploy operations

 *

 */

public class ServiceMonitorObserver implements AxisObserver {

    public void init(AxisConfiguration arg0) {

}

     public void serviceUpdate(AxisEvent event, AxisService axisService) {

//if service is activated 

if(AxisEvent.SERVICE_START == event.getEventType()){

//do your stuff

}

      //if service is deployed

       if(AxisEvent.SERVICE_DEPLOY == event.getEventType()){

//do your stuff

}

}

public void addParameter(Parameter arg0) throws AxisFault {

}

public void deserializeParameters(OMElement arg0) throws AxisFault {

}

public Parameter getParameter(String arg0) {

return null;

}

public ArrayList getParameters() {

return null;

}

public boolean isParameterLocked(String arg0) {

return false;

}

public void removeParameter(Parameter arg0) throws AxisFault {

}

public void moduleUpdate(AxisEvent arg0, AxisModule arg1) {

}

public void serviceGroupUpdate(AxisEvent arg0, AxisServiceGroup arg1) {

}

}

The above listener can be registered in the axis2 configuration (axis2.xml) as follows;

 <listener class="org.test.axisobserver.ServiceMonitorObserver"/>

Content-Type Header and Axis2

I see some issues when set the "Content-Type" Header  in synapse. Aixs2 doesn't obey the "Content-Type" header, when the message conatins  different messageType.

Before sending a soap message to an endpoint in synapse, if we set the "Content-Type" as text/plain, axis2 sends out  the message with text/xml content-type. But if we set messageType property as text/plain, then content-type header of the outgoing message is set to "text/plain".  This is bit confusing as in the documentation, i find that, "messageType" property is used to select correct MessageFormatter.
There are three properties ,that are related to message-type,content-type which can be set in aixs2/transport scopes.

As in the documentations,
<property name="messageType" value="text/plain" scope="axis2" type="STRING"/>
can be used to select right Messageformatter.

<property name="ContentType" value="text/plain" scope="axis2" type="STRING"/>
can be used to select right MessageBuilders

To set the Content-Type transport header,we can use following property.

 <property name="Content-Type" value="text/plain" scope="transport" type="STRING"/>

But the issue is,

When i do following setting in synapse, aixs2 sends out the message with text/xml content-type.

<inSequence>
      <property name="Content-Type" value="text/plain" scope="transport" type="STRING"/>
         <send>
            <endpoint>
                   <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
            </endpoint>
       </send>

After setting the "messageType" property only i see the correct content-type is set.

HTTP Basic Auth and WS-Security username/password authentication

Basic-auth and ws-security username/password authentication both are different and independent.

Basic auth is used in HTTP where user name and password will be encoded and passed  with the request as a HTTP header.
Eg:  HTTP header block will have " Authorization: Basic YWRtaW46YWRtaW4="  header element.

Username and password will be  encoded using base64 and which is used in  authorization header.

Eg: base64(username:password) --> base64(admin:admin)

Most of the webservice clients have option to provide basic auth header. In SOAPUI, at "Authentication" tab, we can provide username and password. If we switch to "Raw" format of the request, all the HTTP headers are visible and we can see the Basic Auth header is set.

When we use Basic Auth, the username and password setting is on the HTTP headers. Not in the SOAP message. SOAP message goes with HTTP body.

We can secure webservices using ws-security username/password authentication mechanism that is a simple mechanism to secure services. It enforces user to provide UsernameToken security header in the SOAP requests.

Sample request:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://services.samples">
   <soapenv:Header>
      <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <wsu:Timestamp xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" wsu:Id="Timestamp-10">
         <wsu:Created>t2013-09-05T16:44:03.872Z</wsu:Created>

         <wsu:Expires>2013-09-05T16:49:03.872Z</wsu:Expires>
     </wsu:Timestamp>
     <wsse:UsernameToken xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" wsu:Id="UsernameToken-9">
       <wsse:Username>admin</wsse:Username>
       <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText"&gtadmin</wsse:Password>
       </wsse:UsernameToken>
     </wsse:Security>
   </soapenv:Header>
   <soapenv:Body>
      <ser:getSimpleQuote>  
         <ser:symbol>IBM</ser:symbol>
      </ser:getSimpleQuote>
    </soapenv:Body>
</soapenv:Envelope>

If we check above sample request, the ws-security header is set as part of SOAP message.

WSO2  service hosting servers provide ws-username token security option. If we secure a service using user name token option, (that is, ws-security username/password authentication) we should pass ws-security headers as mentioned above.

But there is POXSecurity handler is available in the platform which converts HTTP-Basic Auth header to ws-security usernametoken header. So, if we enable UserName token security option for a service , we can send HTTP_basic auth header to execute the service.

Or else, we have to send request, with ws-security headers as mentioned in above sample request.

Extracting HTTP level information in Axis2

From the Aixs2MessageContext, we can get the HttpServletRequest object, which contains HTTP request header information,session details, cookies etc..

Sample

public class MessageStoreHandler extends AbstractHandler {

@Override
public InvocationResponse invoke(MessageContext msgcontext) throws AxisFault {
             extractHTTPInfo(msgcontext);
             return InvocationResponse.CONTINUE;
       }

       private void  extractHTTPInfo(MessageContext msgContext) {

HttpServletRequest obj = (HttpServletRequest)msgContext .
                                          getProperty("transport.http.servletRequest");
if (obj != null) {
System.out.println("Method :"+ obj.getMethod());
System.out.println("Content-type :" +obj.getContentType());
System.out.println("Content-length :"+obj.getContentLength());
System.out.println("Remote addtress"+obj.getSession().getLastAccessedTime());
}
}
}

Writing a simple Axis2 handler

If we want to listen all the incoming/outgoing messages coming to/from the service, we can write an axis2 handler which can be placed in a phase, which is executed in order as defined in the axis2 configuration. The phase can contain more than one handler.
Message flow is categorized  as;

• InFlow
• OutFlow
• InFaultFlow
• OutFaultFlow

As name implies, the message flows indicate the request,response and their fault execution paths. In each flow there are phases defined and we can keep our handler in the right flow and right phase.

Sample code:

package org.test.sample;

import org.apache.axis2.handlers.AbstractHandler;

import org.apache.axis2.description.AxisService;

import org.apache.axis2.context.MessageContext;

public class MessageHandler extends AbstractHandler {

private static final Log log = LogFactory.getLog(MessageHandler.class);

@Override

public InvocationResponse invoke(MessageContext msgcontext) throws AxisFault {

if (log.isDebugEnabled()) {

log.debug("Message Store handler is executing");

}

AxisService axService = msgcontext.getAxisService();

//TODO

return InvocationResponse.CONTINUE;

}

The message execution can be continued, aborted or suspended.

Once we created the jar out of this, we need to keep the jar in the classpath(ie:lib folder) and has to register the handler in the axis2 configuration.
We can register the handler programmatically as a module or  we can edit the axis2.xml file based on the requirement.

Editing axis2.xml

Add the following configuration in the suitable flow/phase.

< phaseOrder type="InFlow">
< phase name="Transport">
< handler name="MessageHandler" class="org.test.sample.MessageHandler">
......
< /phaseOrder>

Programmatically adding the module;

axis2Service.engageModule(axisCfg.getModule(testModule));