Monday, December 22, 2014

axiom.soap.SOAPProcessingException: Transport level information does not match with SOAP Message namespace URI

This is a common exception [1]we see when invoke services hosted in axis2 based servers.
It occurs due to the namespace mismatch in the request. Check the service'e version and the request's version.
eg: SOAP 1.2 message with content type text/xml, we get this error. Correct the namespace in the request.

[1]

ERROR - RelayUtils Error while building Passthrough stream
org.apache.axiom.soap.SOAPProcessingException: Transport level information does not match with SOAP Message namespace URI
at org.apache.axis2.builder.BuilderUtil.validateSOAPVersion(BuilderUtil.java:745)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:58)
at org.apache.synapse.transport.passthru.util.DeferredMessageBuilder.getDocument(DeferredMessageBuilder.java:118)
at org.apache.synapse.transport.passthru.util.RelayUtils.builldMessage(RelayUtils.java:107)
at org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:82)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:68)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131)
at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:166)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:411)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:183)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2014-12-22 00:53:36,500] ERROR - SequenceMediator Error while building message
org.apache.axis2.AxisFault: Error while building Passthrough stream
at org.apache.synapse.transport.passthru.util.RelayUtils.handleException(RelayUtils.java:236)
at org.apache.synapse.transport.passthru.util.RelayUtils.builldMessage(RelayUtils.java:111)
at org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:82)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:68)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131)
at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:166)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:411)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:183)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: org.apache.axiom.soap.SOAPProcessingException: Transport level information does not match with SOAP Message namespace URI
at org.apache.axis2.builder.BuilderUtil.validateSOAPVersion(BuilderUtil.java:745)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:58)
at org.apache.synapse.transport.passthru.util.DeferredMessageBuilder.getDocument(DeferredMessageBuilder.java:118)
at org.apache.synapse.transport.passthru.util.RelayUtils.builldMessage(RelayUtils.java:107)

Saturday, November 29, 2014

Synapse Handler

Synapse handler is similar to axis2 handler , that can be used in RESTAPIs to handle incoming request and the responses.

Sample handler ;


package test;

import org.apache.synapse.MessageContext;
import org.apache.synapse.rest.AbstractHandler;

public class TestSynapseHandler extends AbstractHandler {
@Override
public boolean handleRequest(MessageContext messageContext) {
String idKey = "idKey";
String idValue = "abc";
messageContext.setProperty(idKey, idValue);
return true;
}

@Override
public boolean handleResponse(MessageContext messageContext) {
// TODO Response handling
return true;
}

}

Handler has to be registered in the RESTAPI configuration.

Sample configuration;

<api xmlns="http://ws.apache.org/ns/synapse" name="calAPI" context="/cal">
   <resource methods="GET">
      <nSequence>
         <send>
            <endpoint>
               <address uri="http://localhost:9763/services/calservice"/>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <send>
            <endpoint>
               <default/>
            </endpoint>
         </send>
      </outSequence>
   </resource>
<handlers>
        < handler class="test.TestSynapseHandler"/>
   </handlers>
</api>












Tuesday, November 11, 2014

Debugging tests in maven project

If we use 'maven' as build tool for java projects, to run test cases we might use Surefire plugin. Plugin can be added in pom.xml like


To debug testcases; run
# mvn -Dmaven.surefire.debug test

Which will make the debugger to run port 5005

Tuesday, October 21, 2014

Server did not recognize the value of HTTP Header SOAPAction



We face this issue[1] if we send different version of soap message to a soap service endpoint. Check the service endpoint is soap 1.1 or 1.2 and send the right message.


[1]System.Web.Services.Protocols.SoapException: Server did not recognize the value of HTTP Header SOAPAction: urn:process.
at System.Web.Services.Protocols.Soap11ServerProtocolHelper.RouteRequest()
at System.Web.Services.Protocols.SoapServerProtocol.Initialize()

Sunday, August 31, 2014

Invoking SOAP service via an API in WSO2 APIManager

WSO2 APIManager provides  comprehensive features to support API management capabilities. Users can  publish APIs for REST and SOAP services. To test REST service, REST testing tools are available in API store. But for SOAP services there is no inbuilt tool to test the API. Users can use  any SOAP client to test the API. A simple easy to use tool is SOAPUI.
To test SOAP service api, user needs to set a http auth header to invoke api successfully.
There is no any difference  between REST service api and SOAP service api when creating,  publishing or subscribing to API. After generating an accesstoken, user can  use REST client to invoke API by setting "Authorization" header. 

Eg: Authorization: Bearer ozVF2bhIBnd6rgziVJNr1Vu0J54a

Same header has to be set as HTTP header in soap-ui too. Note that, this is not ws-security header.


Saturday, August 2, 2014

Configuring WSO2 APIManager with Apache HTTP server for reverse proxy

Applies to : AM -  1.8.0

APIManager which has two web apps deployed by default which are api_publisher and api_store. When user needs to route the requests through a proxy server for publisher and store, he can do that by editing  site.json file which is available in AM_HOME/repository/deployment/server/jaggeryapps/store(/publisher)/site/conf folder.

Apache HTTP server
  • Download and install HTTP server
  • Check the default installation works fine for HTTP.
    • Try like "http://localhost" you will see the default index page.
  • To set SSL settings, edit the default http_ssl config file with the following settings. User has to create a new ssl cert/key for ssl communication. 
<VirtualHost _default_:443>
SSLEngine on
SSLCertificateFile /Users/ratha/WSO2/apachehttp/server.crt
SSLCertificateKeyFile /Users/ratha/WSO2/apachehttp/server.key

ProxyPreserveHost On
SSLProxyEngine on

# Proxy path which user wants to map with actual backend
ProxyPass /publicstore https://localhost:9443/store/
ProxyPassReverse /publicstore https://localhost:9443/store/
ProxyPassReverseCookiePath /store /publicstore

ProxyPass /publicpublisher https://localhost:9443/publisher/
ProxyPassReverse /publicpublisher  https://localhost:9443/publisher/
ProxyPassReverseCookiePath /publisher /publicpublisher

</VirtualHost>


  • After editing the ssl configuration restart the http server. Try accessing "https://localhost", if settings are fine, user should be able to see the default index page.
site. json configuration

Edit the context and request-url parameters.
context- The url context
request_url- The original request url which hits the proxy server.

"context" : "/publicstore ",
"request_url":"https://localhost/publicstore ",

Now access the store page(/publisher ) using 'https://localhost/publicstore' and  the request  will be routed internally to 'https://localhost/store'

Monday, June 16, 2014

Environment Management in WSO2 ESB using WSO2GREG


In ESB, when environment changes( dev-->qa-->prod) , that affects the proxy service configuration. Proxy service configuration includes sequences,endpoints,local entries etc. For different environments, endpoint configuration might change.(ie: the service url which is given as address uri)

Sample endpoint configuration 

<endpoint xmlns="http://ws.apache.org/ns/synapse" name="TestEndpoint">
   <address uri="http://abc.com">
      <suspendOnFailure>
         <progressionFactor>1.0</progressionFactor>
      </suspendOnFailure>
      <markForSuspension>
         <retriesBeforeSuspension>0</retriesBeforeSuspension>
         <retryDelay>0</retryDelay>
      </markForSuspension>
   </address>
</endpoint>

When we develop  ESB/Synapse artifacts, we might not prefer to modify the configuration for different environments. Users just want to deploy the same artifact in all different environments. To make the deployment easy in all environments, we need a running WSO2 Governance Registry instance and we need to mount the config/governance registries of WSO2ESB to different target paths of WSO2 G-Reg. The paths will indicate the different environments.

Eg:
<mount path="/_system/config" overwrite="true">
    <instanceId>config</instanceId>
    <targetPath>/_system/config/environment/production</targetPath>
</mount>

We need to upload the endpoint configuration for different environments( with the correct endpoint url) directly to the registry in the relevant environment paths.

The following sample configurations are related to  one production ESB, one qa ESB and a common G-Reg instance. All three instances will ponit same external DB. eg: mysql


Configurations 

1.  WSO2 G-Reg


master-datasource.xml

Create a new data source to point an external database. Eg: mysql.

 <datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource used for ESB mount registries</description>
            <jndiConfig>
                <name>jdbc/esbserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/esb</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

Use the created above datasource in the registry.xml

<!-- This defines the default database and its configuration of the registry -->

    <dbConfig name="wso2registry">
        <dataSource>jdbc/esbserver_config</dataSource>
    </dbConfig>

Copy the mysql jdbc driver in repository/components/lib folder and start the server. Make sure to create tables in the mysql database. You can find the relevant sql scripts in the CARBON_HOME\dbscripts folder.

We need to set different port offsets to all carbon servers in carbon.xml. For G-Reg, let’s keep the default value (ie: ‘0’)

2 WSO2ESB-QA

master-datasource.xml
Create a new data source to point the mysql database which is already created.


 <datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource used for ESB mount registries</description>
            <jndiConfig>
                <name>jdbc/esbserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/esb</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

We need to mount the config registry to different target path to make sure QA ESB server talks to different config registry. Governance registry space of all nodes will be mounted to same location, since governance space is common for all nodes.

  <dbConfig name="config">
       <dataSource>jdbc/esbserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>config</id>
        <dbConfig>config</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/config" overwrite="true">
        <instanceId>config</instanceId>
        <targetPath>/_system/config/env/qa</targetPath>
    </mount>

     <dbConfig name="governance">
       <dataSource>jdbc/esbserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>governance</id>
        <dbConfig>governance</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/governance" overwrite="true">
        <instanceId>governance</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount>


Copy the mysql jdbc driver in repository/components/lib folder. Set different port offset in carbon.xml. For ESB_QA server, let’s keep it as ‘1’. Start the server.
Go to https://localhost:9444/carbon/ URL and check the registry browser, you will see the registries are mounted.

3 WSO2ESB-PROD

master-datasource.xml
Create a new data source to point the mysql database which is already created.

 <datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource used for ESB mount registries</description>
            <jndiConfig>
                <name>jdbc/esbserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/esb</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>


registry.xml
We need to mount the config registry to different target path to make sure PROD_
ESB server talks to different config registry. Governance registry of all nodes will be mounted to same location, since governance space is common for all nodes.


<dbConfig name="config">
       <dataSource>jdbc/esbserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>config</id>
        <dbConfig>config</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/config" overwrite="true">
        <instanceId>config</instanceId>
        <targetPath>/_system/config/env/prod</targetPath>
    </mount>

     <dbConfig name="governance">
       <dataSource>jdbc/esbserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>governance</id>
        <dbConfig>governance</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/governance" overwrite="true">
        <instanceId>governance</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount>


Copy the mysql jdbc driver in repository/components/lib folder. Set different port offset in carbon.xml. For ESB_PROD server, let’s keep it as ‘2’. Start the server.
Go to https://localhost:9445/carbon/ URL and check the registry browser, you will see the registries are mounted. 

4. Endpoint configuration

We need to directly upload the endpoint configuration to the registry.Go to G-reg management console and upload the endpoint configurations(xml file) for the QA and Production environment under relevant target paths.
Eg: Upload the testEndpoint.xml of the qa environment to /_system/config/env/qa path. Upload the
testEndpoint.xml of the production environment to /_system/config/env/prod path.

Sample endpoint conf for QA-ESB server


<endpoint xmlns="http://ws.apache.org/ns/synapse" name="TestEndpoint">
   <address uri="http://<QA SERVICE ENDPOINT URL>">
      <suspendOnFailure>
         <progressionFactor>1.0</progressionFactor>
      </suspendOnFailure>
      <markForSuspension>
         <retriesBeforeSuspension>0</retriesBeforeSuspension>
         <retryDelay>0</retryDelay>
      </markForSuspension>
   </address>
</endpoint>


Sample endpoint conf for PROD-ESB server


<endpoint xmlns="http://ws.apache.org/ns/synapse" name="TestEndpoint">
   <address uri="http://<PROD SERVICE ENDPOINT URL>">
      <suspendOnFailure>
         <progressionFactor>1.0</progressionFactor>
      </suspendOnFailure>
      <markForSuspension>
         <retriesBeforeSuspension>0</retriesBeforeSuspension>
         <retryDelay>0</retryDelay>
      </markForSuspension>
   </address>
</endpoint>


5. Proxy Configuration

When we create proxy configuration we should not directly provide any of the address url. Rather we need to point the registry key. This registry key will be same for all environments. That means we do not need to modify the proxy configuration when we deploy that to qa or production environment.
Eg:  < endpoint="conf:/testEndpoint">

Sample proxy configuration

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="testEnvProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target endpoint="conf:/testEndpoint"/>
   <description/>
</proxy>

Related post
Environment Management in WSO2 DSS using WSO2GREG

Monday, May 26, 2014

Environment Management in WSO2 DSS using WSO2GREG

1.0 Introduction

In data services, environment change( dev-->qa-->prod) affects the datasources configuration. In the data source configuration, we need to provide database url, username, password.

Sample datasource configuration

 <datasource>
      <name>dss</name>
      <definition type="RDBMS">
           <configuration>
              <driverClassName>com.mysql.jdbc.Driver</driverClassName>
              <url>jdbc:mysql://localhost:3306/dss</url>
              <username>root</username>
              <password encrypted="true">E61Pys3tVBbLagNo+co9hChhwh6J8MZ/+CU4Z8vw6ABv9YR4SZAf9o3OZmUpLWVz8dKhPGrqPYnwIMw3InkJvpHea7xzEljrX9PF3PY/ax+Fe0upBhGIe+jvSDUai5RAb7cxSwhffnhx1JQZroieeIDxb6n1cpuRN8wd/z4zmtk=</password>
         </configuration>
    </definition>
</datasource>

For different environments , we might need to change
  • Database url
  • Database username
  • Database password
2.0 Governance Registry

To manage different environments, we need a running WSO2 Governance Registry instance and we need to mount the config registries of WSO2 DSS to different target paths. The paths will indicate the different environments.

Eg:
<mount path="/_system/config" overwrite="true">
    <instanceId>config</instanceId>
    <targetPath>/_system/config/env/prod</targetPath>
</mount>

All servers will talk to same database.( That is, wso2greg and wso2dss.)

We need to upload the datasource configurations for different environments directly to the registry in the relevant environment paths. Note that, we can not use carbon datasource UI options available at dataservice server. Because WSO2 G-Reg will be the centralized server to manage different environments, so we need to avoid creating datasources from dataservice server.

3.0 Configurations 

3.1 WSO2 G-Reg

master-datasource.xml

Create a new data source to point an external database. Eg: mysql.




 <datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource used DSS mount registries</description>
            <jndiConfig>
                <name>jdbc/dssserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/dss</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

Use the created datasource in the registry.xml

<!-- This defines the default database and its configuration of the registry -->

    <dbConfig name="wso2registry">
        <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>


Copy the mysql jdbc driver in repository/components/lib folder and start the server. Make sure to create tables in the mysql database. You can find the relevant sql scripts in the CARBON_HOME\dbscripts folder.

We need to set different port offsets to all carbon servers in carbon.xml. For G-Reg, let’s keep the default value (ie: ‘0’)

3.2 WSO2DSS-QA

In this sample, let’s take two DSS servers for QA and production environments.

master-datasource.xml
Create a new data source to point the mysql database which is already created.

<datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource for DSS</description>
            <jndiConfig>
                <name>jdbc/dssserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/dss</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>


registry.xml

We need to mount the config registry to different target path to make sure QA DSS server talks to different config registry. Governance registry space of all nodes will be mounted to same location, since governance space is common for all nodes.

<dbConfig name="config">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>config</id>
        <dbConfig>config</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/config" overwrite="true">
        <instanceId>config</instanceId>
        <targetPath>/_system/config/env/qa</targetPath>
    </mount>

     <dbConfig name="governance">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>governance</id>
        <dbConfig>governance</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/governance" overwrite="true">
        <instanceId>governance</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount>

Copy the mysql jdbc driver in repository/components/lib folder. Set different port offset in carbon.xml. For DSS_QA server, let’s keep it as ‘1’. Start the server.
Go to https://localhost:9444/carbon/ URL and check the registry browser, you will see the registries are mounted.

3.3 WSO2DSS-PROD

master-datasource.xml
Create a new data source to point the mysql database which is already created.

<datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource for DSS</description>
            <jndiConfig>
                <name>jdbc/dssserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/dss</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

We need to mount the config registry to different target path to make sure PROD_DSS server talks to different config registry. Governance registry of all nodes will be mounted to same location, since governance space is common for all nodes.

<dbConfig name="config">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>config</id>
        <dbConfig>config</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/config" overwrite="true">
        <instanceId>config</instanceId>
        <targetPath>/_system/config/env/prod</targetPath>
    </mount>

     <dbConfig name="governance">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>governance</id>
        <dbConfig>governance</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/governance" overwrite="true">
        <instanceId>governance</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount>

Copy the mysql jdbc driver in repository/components/lib folder. Set different port offset in carbon.xml. For DSS_PROD server, let’s keep it as ‘2’. Start the server.
Go to https://localhost:9445/carbon/ URL and check the registry browser, you will see the registries are mounted.

3.4 Datasource Configuration

When we create carbon datasource from UI, the password will be encrypted and saved to registry. Here since we need to directly upload the data source configuration to the registry, we can just upload the password in plain text. If user needs to encrypt the password, later he can access the datasource via datasource UI and save it. So, it will be encrypted and saved to registry.

Sample data source for QA-DSS server

<datasource>
    <name>testdatasource</name>
    <definition type="RDBMS">
        <configuration>
            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
            <url>jdbc:mysql://localhost:3306/service_qa</url>
            <username>root</username>
            <password encrypted="false">root</password>
        </configuration>
    </definition>
</datasource>

Save the configuration as testdatasource and upload in WSO2 G-Reg server under qa environment. Go to, https://localhost:9443/carbon and browse to /_system/config/env/qa/repository/components/org.wso2.carbon.ndatasource path and upload.

Sample data source for PROD-DSS server

Copy the same configuration and edit the parameters to reflect the production database configurations and save as testdatasource.


Eg:


<datasource>
        <name>testdatasource</name>
        <definition type="RDBMS">
            <configuration>
                <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                <url>jdbc:mysql://localhost:3306/service_prod</url>
                <username>root</username>
                <password encrypted="false">root</password>
           </configuration>
      </definition>
</datasource>


  Go to, https://localhost:9443/carbon and browse to /_system/config/env/prod/repository/components/org.wso2.carbon.ndatasource path and upload.

Now same datasource configuration is available under two paths with the environment specific parameters. But the datasources’ name is same.(ie: testdatasource)

3.5 Dataservice

When we create dataservice, there is no any environment specific parameters. Only thing is, we need to point the datasource there, which contains environment specific parameters.

We already separated and stored the different datasource configurations with same name in different registry paths. So, if we upload same dataservice in both servers(ie: DSS_QA and DSS_PROD) both servers will talk to the relevant data source configurations, which are stored under relevant config registry without any issue.

Sample dataservice

<data name="dss_test">
   <config id="testdatasource">
      <property name="carbon_datasource_name">testdatasource</property>
   </config>
   <query id="select_person" useConfig="testdatasource">
      <sql>call `person-info`(?, ?, ?);</sql>
      <result element="people" rowName="person">
         <element column="PERSON_NAME" name="name" xsdType="string"/>
         <element column="PERSON_NICK" name="nick" xsdType="string"/>
         <element column="PERSON_PHONE" name="phone" xsdType="string"/>
      </result>
      <param name="pName" sqlType="STRING"/>
<param name="pPhone" sqlType="STRING"/>
      <param name="pNick" sqlType="STRING"/>
   </query>
   <operation name="select_person">
      <call-query href="select_person">
         <with-param name="pName" query-param="pName"/>
         <with-param name="pPhone" query-param="pPhone"/>
         <with-param name="pNick" query-param="pNick"/>
      </call-query>
   </operation>
</data>

Related post
Environment Management in WSO2 ESB using WSO2GREG

Monday, May 19, 2014

Processing large text file using smooks in wso2esb

Smooks  library supports large file processing which is used in ESB to process file which are in text/xml format.
Text files can be a CSV file or with a space delimiter or a fixed length file. To process large file, smooks write the output stream , which is a single line record, to a temporary file or a jms queue. After writing all files to the temporary location, we need to process all those records/files again. This is how we can avoid large memory growth/out of heap space issue which occurs when we try to process large file as in- memory process.

Following is a sample configuration to process fixed length large text file in wso2esb.

Proxy configuration

To pick and process the file form a file location , we could use VFS transport.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="cor_file_pull"
       transports="https,http,vfs"
       statistics="disable"
       trace="disable"
       startOnLoad="true"target>
     <inSequence>
  <property name="DISABLE_SMOOKS_RESULT_PAYLOAD"
                   value="true"
                   scope="default"
                   type="STRING"/>
          <smooks config-key="simple-smook">
            <input type="text"/>
           <output type="xml"/>
         </smooks>
       </inSequence>
      <outSequence/>
   </target>
   <parameter name="transport.vfs.Streaming">true&lt;/parameter>
   <parameter name="transport.PollInterval">10&lt;/parameter>
   <parameter name="transport.vfs.ActionAfterProcess">MOVE&lt;/parameter>
   <parameter name="transport.vfs.MoveAfterProcess">C:/Users/TOSH/Desktop/success</parameter>
   <parameter name="transport.vfs.FileURI">C:/Users/TOSH/Desktop/in&lt;/parameter>
   <parameter name="transport.vfs.MoveAfterFailure">C:/Users/TOSH/Desktop/failure</parameter>
   <parameter name="transport.vfs.Locking">disable&lt;/parameter>
   <parameter name="transport.vfs.FileNamePattern">^(cor).+</parameter>
   <parameter name="transport.vfs.ContentType">text/plain</parameter>
   <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   <description/>
</proxy>

Smooks configuration

simple-smook localentry

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:ftl="http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd" xmlns:fl="http://www.milyn.org/xsd/smooks/fixed-length-1.3.xsd" xmlns:file="http://www.milyn.org/xsd/smooks/file-routing-1.1.xsd">
<params>
            <param name="stream.filter.type">SAX </param>
           <param name="default.serialization.on">false </param>
         </params>

<fl:reader fields="RecordId[4],CorpName[60],FileNumber[9]" skipLines="0">
</fl:reader>
      <resource-config selector="record">
         <resource> org.milyn.delivery.DomModelCreator</resource>
     </resource-config>
      <file:outputStream openOnElement="record" resourceName="fileSplitStream">
         <file:fileNamePattern> ${.vars["record"].FileNumber}.xml</file:fileNamePattern>
         <file:destinationDirectoryPattern> C:\Users\TOSH\Desktop\processed</file:destinationDirectoryPattern>
         <file:highWaterMark mark="10000000"> </file:highWaterMark>
      </file:outputStream>
      <ftl:freemarker applyOnElement="record">
         <ftl:template> /repository/resources/smooks/record_as_xml.ftl</ftl:template>
         <ftl:use>
            <ftl:outputTo outputStreamResource="fileSplitStream"> </ftl:outputTo>
         </ftl:use>
      </ftl:freemarker>
   </smooks-resource-list>

Each record will   be written to the specified file location   "C:\Users\TOSH\Desktop\processed" with the name constructed through the parameter "  ${.vars["record"].FileNumber}.xml ".  Here it is a xml file, which will be named with a file number    record in the text file. We have to select a unique parameter to write the processed files else there will be file overwriting issue will be thrown in smooks.
To process large file we have to use 'SAX' filter type. DOM needs more memory and it will lead to out of memory issue.
In this sample, i have used DOM+SAX mixing model to keep the lower footprint of memory.
High  water mark provides the maximum file records need to be written. provide enough number based on the lines in the text file.
The property ;
 <property name="DISABLE_SMOOKS_RESULT_PAYLOAD"   value="true" scope="default"     type="STRING"/>

will be available in ESB 4.9.0. If you use lower version, you might need a patch to be applied to your system.

XML template file
record_as_xml.ftl
<#assign rec = .vars["record"]>
<p:insert_acc_data_file xmlns:p="http://ws.wso2.org/dataservice"><p:RecordId>${rec.RecordId}</p:RecordId>
<p:CorpName>${rec.CorpName}&lt;/p:CorpName>
<p:FileNumber>${rec.FileNumber}&lt;/p:FileNumber></p:insert_acc_data_file>

Based on above template the records will be formatted and will be written to the .

Using another vfs proxy, we can pick the entries written to the  'processed' file and do further processing.

Sunday, May 4, 2014

Executing FaultSequence for SOAPFaults in WSO2ESB

In WSO2ESB/Synapse, whenever mediation errors occur, user can execute faultsequence to find error details.
The fault sequence can be defined at a sequence level or at proxy service level. Mediation level issues and endpoint errors can be captured at faultsequence. But SOAPFault can not be captured at fault sequence, since ESB/Synapse will treat it as the response from the endpoint.
But from user's pont of view , those are service endpoint failures which need to be handled properly.
To handle the soapfaults, now we can use a property which forces to execute fault sequence whenever soapfault occurs.
We need to set  following property before sending the message to endpoint.;

<property name="FORCE_ERROR_ON_SOAP_FAULT" value="true"/>

Sample conf;

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="testProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
  <target faultSequence="myfaultSeq">
      <inSequence>
         <property name="FORCE_ERROR_ON_SOAP_FAULT" value="true"/>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
      <endpoint>
         <address uri="http://localhost:9764/services/echo/"/>
     </endpoint>
   </target>
   <description/>
</proxy>
           

In the above sample , whenever we get soapfault, the 'myfaultSeq' will be executed.

Related Post: Sending mails for errors

Tuesday, April 15, 2014

Securevault support for credientials used in mediation configuration

Secure vault tool which  is available in carbon servers, helps to encrypt passwords which are configured in plain text in the configuration files. This tool can be directly used for the passwords which are configured in carbon.xml,  axis2.xml, datasources.xml etc..We can not use secure vault tool directly in the mediation configuration , where we might provide passwords to make secured connection  for any basic auth protected endpoints.
In the older versions of ESB/APIM, we provide basic auth header by setting the "Authorization" transport header,where password was in plain text.

Eg: 
&lt;property name="Authorization" expression="fn:concat('Basic ', base64Encode('<Username>:<Password>'))" scope="transport"/>

In the newer versions (AM 1.6.0 / ESB 4.8.0 onwards), user can store the encrypted password in registry and  the 'alias' can be used to refer the stored passwords.

Eg:

<property name="secPassword" expression="wso2:vault-lookup('securedDS.endpoint.password')"/>
  • securedDS.endpoint.password : - Password alias, which is stored in config registry under /repository/components/secure-vault
  • wso2:vault-lookup :- Is a custom xpath implementation for synapse to do lookups. To use this custom xpath, user needs to add following synapse property in the synapse.properties file.
synapse.xpath.func.extensions=org.wso2.carbon.mediation.security.vault.xpath.SecureVaultLookupXPathFunctionProvider 

In APIManager;
  •  If user needs to encrypt the password in the mediation configuration, he has to enable the <EnableSecureVault> property in the api-manager.xml, and needs to run the cipher tool before running the product. 
  • Cipher tool is available in the  /bin folder.
    • # ciphertool.bat/sh -Dconfigure
  • When user enables securevault option and  publishes an API via the publisher UI, created API configuration will have the securevault enabled password protection. User can check the alias configured in the API configuration and can map that with the configuration registry entry. Alias is created with the string merge of  publisherName+apiName+version;
 In ESB, user can use management console to create aliases.


Monday, March 24, 2014

Axiom.soap.SOAPProcessingException: First Element must contain the local name, Envelope.


I faced this issue [1] and struggled a bit to find the root cause of this issue in synapse. The error was not clear enough to find out what was the issue.
The fix was, we need to set  the 'SOAPAction' property before sending out the message.

<property name="SOAPAction" value="urn:select_address " scope="transport"/>

Axis2 dispatches the message based on ;
  • Request URI based
  • Request URI Operation based
  • SOAP Action based
  • SOAP MessageBody based etc..
If system could not find any of the option to dispatch the request to correct endpoint, axis2 will throw an error states "operation not found/ EPR not found error"  .
But here, the error stack is not descriptive  to find the actual missing configuration.

[1] ERROR - RelayUtils Error while building Passthrough stream
org.apache.axiom.soap.SOAPProcessingException: First Element must contain the local name, Envelope , but found faultstring
        at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.constructNode(StAXSOAPModelBuilder.java:305)
        at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.createOMElement(StAXSOAPModelBuilder.java:252)
        at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.createNextOMElement(StAXSOAPModelBuilder.java:234)
        at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:249)
        at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.getSOAPEnvelope(StAXSOAPModelBuilder.java:204)
        at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.(StAXSOAPModelBuilder.java:154)
        at org.apache.axiom.om.impl.AbstractOMMetaFactory.createStAXSOAPModelBuilder(AbstractOMMetaFactory.java:73)
        at org.apache.axiom.om.impl.AbstractOMMetaFactory.createSOAPModelBuilder(AbstractOMMetaFactory.java:79)
        at org.apache.axiom.om.OMXMLBuilderFactory.createSOAPModelBuilder(OMXMLBuilderFactory.java:196)
        at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:55)
        at org.apache.synapse.transport.passthru.util.DeferredMessageBuilder.getDocument(DeferredMessageBuilder.java:118)
        at org.apache.synapse.transport.passthru.util.RelayUtils.builldMessage(RelayUtils.java:107)
        at org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:82)
        at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:68)

Saturday, February 15, 2014

Sending mails using mail transport in wso2esb

Scenario:

If any error occurs in a message mediation flow, Administrator has to send mail to manager.

Axis configuration

To send mails, we use mail transport. Enable the 'MailTransportSender' to send emails in specific scenarios.

@axis2.xml;

<transportSender name="mailto" class="org.apache.axis2.transport.mail.MailTransportSender">
        <parameter name="mail.smtp.host">smtp.gmail.com</parameter>
       <parameter name="mail.smtp.port">587</parameter>
       <parameter name="mail.smtp.starttls.enable">true</parameter>
       <parameter name="mail.smtp.auth">true</parameter>
       <parameter name="mail.smtp.user">admin@abc.com</parameter>
        <parameter name="mail.smtp.password">XXXXX</parameter>
       <parameter name="mail.smtp.from">Admin</parameter>
  </transportSender>

Proxy configuration


 <proxy xmlns="http://ws.apache.org/ns/synapse"
       name="FileProxy"
       transports="https,http,vfs"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
    <target faultSequence="sendErrMailSequence">

       <inSequence>
.................
</inSEquence>
</proxy>



To get  a mail content in HTML format, we need to add HTML message formatter in axis2.xml
<messageFormatter contentType="text/html" class="org.apache.axis2.transport.http.ApplicationXMLFormatter"/>


SendErrMailSequence  configuration


<sequence xmlns="http://ws.apache.org/ns/synapse" name="sendErrMailSequence">
   <property name="messageType" value="text/html" scope="axis2"/>
   <property name="ContentType" value="text/html" scope="axis2"/>
  <property name="Subject" value=" Error occured" scope="transport"/>
   <property name="OUT_ONLY" value="true"/>
 <payloadFactory media-type="xml">
            <format>
               <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
                  <soapenv:Body>
                     <ErrorMessage xmlns=""> Please check endpoint errors</ErrorMessage>
                  </soapenv:Body>
               </soapenv:Envelope>
           </format>
         </payloadFactory>
   <send>
      <endpoint name="MailEpr">
         <address uri="mailto:manager@abc.com"/>
      </endpoint>
   </send>

</sequence>

Friday, February 14, 2014

XML to Text transformation using smooks mediator

Smooks mediator can be used to apply transformations. Inputs can be provided as text/xml and smooks transformation resource has to be passed as an external resource.
Here is the sample smooks mediator configuration and smooks configuration which can be used to read an xml element from the input and parse it to a preferred output xml format.

Smooks meditaor config;

<smooks config-key="smooks">
               <input type="xml" expression="//csv/arg0"/>
               <output type="text"/>
            </smooks>


Sample request;

<soapenv:Envelope xmlns:soapenv=http://schemas.xmlsoap.org/soap/envelope/ >
 <soapenv:Body>
  <csv>
   <arg>1000,20130926,UAE</arg0>
  </csv>
 </soapenv:Body>
</soapenv:Envelope>

Smooks configuration;

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.0.xsd">
   <resource-config selector="org.xml.sax.driver">
    <resource>org.milyn.csv.CSVParser</resource>
    <param name="fields" type="string-list">RecordID,FileNumber,CorporationName</param>
  </resource-config>
</smooks-resource-list>

Output;
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ax:text xmlns:ax="http://ws.apache.org/commons/ns/payload">
<csv-set>
<csv-record number="1">
<RecordID>1000</RecordID>
<FileNumber>00238374</FileNumber>
<CorporationName>UAE</CorporationName>
</csv-record>
</csv-set>
</ax:text>
</soapenv:Body>
</soapenv:Envelope>

Thursday, February 13, 2014

Creating OMElement from a list of Array Elements

Creating an  OMElement  from an Array of elements , requires a custom xml root element, which will wrap all arguments.
Here is a sample code snippet to do that;

String[] csvArray = csvList.toArray(new String[csvList.size()]);
OMElement csvOMElement =  BeanUtil.getOMElement(qname, csvArray, null, false, null);

eg:
qname = new QName("csv"); // here user can  provide  prefered root element for his OMelement.

Monday, February 3, 2014

SAML 2.0 service provider meta file for WSO2 IS

SAML2.0 identity providers and service providers has to provide a SAML 2.0 metadata file representing entities. Metadata documents provided by a service provider must include an <md:SPSSODescriptor> element containing all necessary elements and identity provider must include an < md:IDPSSODescriptor> element which containing all necessary elements.

WSO2 Identity server can be configured as  a single sign on system, where it can act as identity provider and service provider. There is no option to generate meta files in WSO2 IS. User has to manually write IDP/SP meta files and need to import with other third party systems.
Here is a sample Service-provider meta file for WSO2 IS, which can be used with third party identity providers.



Tuesday, January 28, 2014

Design Time Governance Using WSO2 G-Reg



Governance of services is critically important in the SOA world to know about running services and their service contracts, such as whether service SLAs are violated or not. Governance tools provide support for policy creation, access control etc. for service governance, which are used within a business enterprise. Governance further enforces the created policy rules to be followed by users in design and run time of the services. 
This article explains how users can manage design time governance using WSO2 G-Reg.