Configuring WSO2 APIManager with Apache HTTP server for reverse proxy

Applies to : AM -  1.8.0

APIManager which has two web apps deployed by default which are api_publisher and api_store. When user needs to route the requests through a proxy server for publisher and store, he can do that by editing  site.json file which is available in AM_HOME/repository/deployment/server/jaggeryapps/store(/publisher)/site/conf folder.

Apache HTTP server

• Download and install HTTP server
• Check the default installation works fine for HTTP.
• Try like "http://localhost" you will see the default index page.
• To set SSL settings, edit the default http_ssl config file with the following settings. User has to create a new ssl cert/key for ssl communication. 

<VirtualHost _default_:443>

SSLEngine on

SSLCertificateFile /Users/ratha/WSO2/apachehttp/server.crt

SSLCertificateKeyFile /Users/ratha/WSO2/apachehttp/server.key

ProxyPreserveHost On

SSLProxyEngine on

# Proxy path which user wants to map with actual backend

ProxyPass /publicstore https://localhost:9443/store/

ProxyPassReverse /publicstore https://localhost:9443/store/

ProxyPassReverseCookiePath /store /publicstore

ProxyPass /publicpublisher https://localhost:9443/publisher/

ProxyPassReverse /publicpublisher  https://localhost:9443/publisher/

ProxyPassReverseCookiePath /publisher /publicpublisher

</VirtualHost>

• After editing the ssl configuration restart the http server. Try accessing "https://localhost", if settings are fine, user should be able to see the default index page.

site. json configuration

Edit the context and request-url parameters.

context- The url context

request_url- The original request url which hits the proxy server.

"context" : "/publicstore ",
"request_url":"https://localhost/publicstore ",

Now access the store page(/publisher ) using 'https://localhost/publicstore' and  the request  will be routed internally to 'https://localhost/store'

Processing large text file using smooks in wso2esb

Smooks  library supports large file processing which is used in ESB to process file which are in text/xml format.
Text files can be a CSV file or with a space delimiter or a fixed length file. To process large file, smooks write the output stream , which is a single line record, to a temporary file or a jms queue. After writing all files to the temporary location, we need to process all those records/files again. This is how we can avoid large memory growth/out of heap space issue which occurs when we try to process large file as in- memory process.

Following is a sample configuration to process fixed length large text file in wso2esb.

Proxy configuration

To pick and process the file form a file location , we could use VFS transport.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="cor_file_pull"
       transports="https,http,vfs"
       statistics="disable"
       trace="disable"
       startOnLoad="true"target>
     <inSequence>
  <property name="DISABLE_SMOOKS_RESULT_PAYLOAD"
                   value="true"
                   scope="default"
                   type="STRING"/>
          <smooks config-key="simple-smook">
            <input type="text"/>
           <output type="xml"/>
         </smooks>
       </inSequence>
      <outSequence/>
   </target>
   <parameter name="transport.vfs.Streaming">true&lt;/parameter>
   <parameter name="transport.PollInterval">10&lt;/parameter>
   <parameter name="transport.vfs.ActionAfterProcess">MOVE&lt;/parameter>
   <parameter name="transport.vfs.MoveAfterProcess">C:/Users/TOSH/Desktop/success</parameter>
   <parameter name="transport.vfs.FileURI">C:/Users/TOSH/Desktop/in&lt;/parameter>
   <parameter name="transport.vfs.MoveAfterFailure">C:/Users/TOSH/Desktop/failure</parameter>
   <parameter name="transport.vfs.Locking">disable&lt;/parameter>
   <parameter name="transport.vfs.FileNamePattern">^(cor).+</parameter>
   <parameter name="transport.vfs.ContentType">text/plain</parameter>
   <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   <description/>
</proxy>

Smooks configuration

simple-smook localentry

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:ftl="http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd" xmlns:fl="http://www.milyn.org/xsd/smooks/fixed-length-1.3.xsd" xmlns:file="http://www.milyn.org/xsd/smooks/file-routing-1.1.xsd">
<params>
            <param name="stream.filter.type">SAX </param>
           <param name="default.serialization.on">false </param>
         </params>

<fl:reader fields="RecordId[4],CorpName[60],FileNumber[9]" skipLines="0">
</fl:reader>
      <resource-config selector="record">
         <resource> org.milyn.delivery.DomModelCreator</resource>
     </resource-config>
      <file:outputStream openOnElement="record" resourceName="fileSplitStream">
         <file:fileNamePattern> ${.vars["record"].FileNumber}.xml</file:fileNamePattern>
         <file:destinationDirectoryPattern> C:\Users\TOSH\Desktop\processed</file:destinationDirectoryPattern>
         <file:highWaterMark mark="10000000"> </file:highWaterMark>
      </file:outputStream>
      <ftl:freemarker applyOnElement="record">
         <ftl:template> /repository/resources/smooks/record_as_xml.ftl</ftl:template>
         <ftl:use>
            <ftl:outputTo outputStreamResource="fileSplitStream"> </ftl:outputTo>
         </ftl:use>
      </ftl:freemarker>
   </smooks-resource-list>

Each record will   be written to the specified file location   "C:\Users\TOSH\Desktop\processed" with the name constructed through the parameter "  ${.vars["record"].FileNumber}.xml ".  Here it is a xml file, which will be named with a file number    record in the text file. We have to select a unique parameter to write the processed files else there will be file overwriting issue will be thrown in smooks.
To process large file we have to use 'SAX' filter type. DOM needs more memory and it will lead to out of memory issue.
In this sample, i have used DOM+SAX mixing model to keep the lower footprint of memory.
High  water mark provides the maximum file records need to be written. provide enough number based on the lines in the text file.
The property ;
 <property name="DISABLE_SMOOKS_RESULT_PAYLOAD"   value="true" scope="default"     type="STRING"/>

will be available in ESB 4.9.0. If you use lower version, you might need a patch to be applied to your system.

XML template file
record_as_xml.ftl

<#assign rec = .vars["record"]>
<p:insert_acc_data_file xmlns:p="http://ws.wso2.org/dataservice"><p:RecordId>${rec.RecordId}</p:RecordId>
<p:CorpName>${rec.CorpName}&lt;/p:CorpName>
<p:FileNumber>${rec.FileNumber}&lt;/p:FileNumber></p:insert_acc_data_file>

Based on above template the records will be formatted and will be written to the .

Using another vfs proxy, we can pick the entries written to the  'processed' file and do further processing.

Executing FaultSequence for SOAPFaults in WSO2ESB

In WSO2ESB/Synapse, whenever mediation errors occur, user can execute faultsequence to find error details.
The fault sequence can be defined at a sequence level or at proxy service level. Mediation level issues and endpoint errors can be captured at faultsequence. But SOAPFault can not be captured at fault sequence, since ESB/Synapse will treat it as the response from the endpoint.
But from user's pont of view , those are service endpoint failures which need to be handled properly.
To handle the soapfaults, now we can use a property which forces to execute fault sequence whenever soapfault occurs.
We need to set  following property before sending the message to endpoint.;

<property name="FORCE_ERROR_ON_SOAP_FAULT" value="true"/>

Sample conf;

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="testProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
  <target faultSequence="myfaultSeq">
      <inSequence>
         <property name="FORCE_ERROR_ON_SOAP_FAULT" value="true"/>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
      <endpoint>
         <address uri="http://localhost:9764/services/echo/"/>
     </endpoint>
   </target>
   <description/>
</proxy>
           

In the above sample , whenever we get soapfault, the 'myfaultSeq' will be executed.

Related Post: Sending mails for errors

Axiom.soap.SOAPProcessingException: First Element must contain the local name, Envelope.

I faced this issue [1] and struggled a bit to find the root cause of this issue in synapse. The error was not clear enough to find out what was the issue.
The fix was, we need to set  the 'SOAPAction' property before sending out the message.

<property name="SOAPAction" value="urn:select_address " scope="transport"/>

Axis2 dispatches the message based on ;

• Request URI based
• Request URI Operation based
• SOAP Action based
• SOAP MessageBody based etc..

If system could not find any of the option to dispatch the request to correct endpoint, axis2 will throw an error states "operation not found/ EPR not found error"  .
But here, the error stack is not descriptive  to find the actual missing configuration.

[1] ERROR - RelayUtils Error while building Passthrough stream
org.apache.axiom.soap.SOAPProcessingException: First Element must contain the local name, Envelope , but found faultstring
        at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.constructNode(StAXSOAPModelBuilder.java:305)
        at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.createOMElement(StAXSOAPModelBuilder.java:252)
        at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.createNextOMElement(StAXSOAPModelBuilder.java:234)
        at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:249)
        at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.getSOAPEnvelope(StAXSOAPModelBuilder.java:204)
        at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.(StAXSOAPModelBuilder.java:154)
        at org.apache.axiom.om.impl.AbstractOMMetaFactory.createStAXSOAPModelBuilder(AbstractOMMetaFactory.java:73)
        at org.apache.axiom.om.impl.AbstractOMMetaFactory.createSOAPModelBuilder(AbstractOMMetaFactory.java:79)
        at org.apache.axiom.om.OMXMLBuilderFactory.createSOAPModelBuilder(OMXMLBuilderFactory.java:196)
        at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:55)
        at org.apache.synapse.transport.passthru.util.DeferredMessageBuilder.getDocument(DeferredMessageBuilder.java:118)
        at org.apache.synapse.transport.passthru.util.RelayUtils.builldMessage(RelayUtils.java:107)
        at org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:82)
        at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:68)

Defining JMS proxy to listen to a particular queue in wso2esb

There are  use cases, where user needs to define number of jms queues for different purposes and proxies will listen to those queues and do further processing with messages.

There is a predefined service level parameter available in axis2   to make 'jms' proxies to listen particular queues.
Check the following simple sample(for qpid), where 'ErrorProxy" listens a  queue named as "errorqueue".

<proxy xmlns="http://ws.apache.org/ns/synapse" name="ErrorProxy" transports="https,http,jms" statistics="disable" trace="disable" startOnLoad="true">
   <target inSequence="ErrorInSequence" endpoint="ErrorQueueEndpoint" />
   <parameter name="transport.jms.ContentType">
      <rules>
         <jmsProperty>contentType</jmsProperty>
         <default>text/xml</default>
      </rules>
   </parameter>
   <parameter name="transport.jms.ConnectionFactory">myQueueConnectionFactory</parameter>
   <parameter name="transport.jms.DestinationType">queue</parameter>
   <parameter name="transport.jms.Destination">errorqueue</parameter>
</proxy> 

You have to enable jms transport in your axis2.xml.

eg:

<transportReceiver name="jms" class="org.apache.axis2.transport.jms.JMSListener">
  <parameter name="myQueueConnectionFactory" locked="false">
   <parameter name="java.naming.factory.initial" locked="false"> org.apache.qpid.jndi.PropertiesFileInitialContextFactory</parameter>
   <parameter name="java.naming.provider.url" locked="false">repository/conf /jndi.properties</parameter>
   <parameter name="transport.jms.ConnectionFactoryJNDIName" locked="false"> QueueConnectionFactory</parameter>
   <parameter name="transport.jms.ConnectionFactoryType" locked="false">queue< /parameter>
  </parameter>

Need to add following entry at jndi.properties file
  queue.errorqueue = example.errorqueue


ErrorInSequence Configuration
--------------------------------------

<sequence xmlns="http://ws.apache.org/ns/synapse" name="ErrorInSequence">
   <property name="FORCE_SC_ACCEPTED" value="true" scope="axis2" />
   <property name="OUT_ONLY" value="true" scope="default"/>
     <log level="custom">
      <property name="Picked message" value="************" />
   </log>
   <description></description>
</sequence> 

This sequence receives the error message,which is stored at error queue

Invoking a RESTful service via WSO2ESB

In the ESB, when we define a proxy, it is not a must to provide the service wsdl, unless if you want to hide/add a new elements in your service contract.
Lets check a  very simple basic proxy to process a HTTP GET request

• Get wso2esb binary distribution.
• Deploy the simplestockquote service and start the sample axis2server.
• Start ESB.

Create the following proxy;

<proxy name="RestProxy" transports="https http" startOnLoad="true" trace="disable">         <target>               <endpoint>               <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>             </endpoint>                          <outSequence>                 <send/>             </outSequence>         </target>     </proxy>

Execute the following GETrequest url in your browser;
http://localhost:8280/services/RestProxy/getSimpleQuote?symbol=IBM

Now you will see the response in your browser. If you place the tcpmon between client and ESB, you can check your http 'get' request,

GET /services/RestProxy/getSimpleQuote?symbol=IBM HTTP/1.1

Host: 127.0.0.1:5555

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.24) Gecko/20111103 Firefox/3.6.24

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 115

Connection: keep-alive

Cookie: region1_configure_menu=; region3_registry_menu=visible; region4_monitor_menu=;

In the above sample client sends the 'symbol=IBM' as the parameter for the operation "getSimpleQuote"..Say in the mediation logic user may need to change the parameter as 'SUN'.(ie:symbol=SUN)..For that he/she can use "REST_URL_POSTFIX" property ..Which basically adds the suffix for the REST service endpoint.
eg:
   <property name="REST_URL_POSTFIX" value="/getSimpleQuote?symbol=SUN" scope="axis2"/>

So, the insequence would be;

    <inSequence>

                <property name="REST_URL_POSTFIX" value="/getSimpleQuote?symbol=SUN" scope="axis2"/>

                <send>

                    <endpoint>

                        <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>

                    </endpoint>

                </send>

    </inSequence>

User could add your own mediation logic with whatever the mediators within the sequence..

Related post;
REST support in WSO2ESB - Introduction
REST Support in WSO2ESB - Handling JSON messages