Environment Management in WSO2 DSS using WSO2GREG

1.0 Introduction

In data services, environment change( dev-->qa-->prod) affects the datasources configuration. In the data source configuration, we need to provide database url, username, password.

Sample datasource configuration

 <datasource>
      <name>dss</name>
      <definition type="RDBMS">
           <configuration>
              <driverClassName>com.mysql.jdbc.Driver</driverClassName>
              <url>jdbc:mysql://localhost:3306/dss</url>
              <username>root</username>
              <password encrypted="true">E61Pys3tVBbLagNo+co9hChhwh6J8MZ/+CU4Z8vw6ABv9YR4SZAf9o3OZmUpLWVz8dKhPGrqPYnwIMw3InkJvpHea7xzEljrX9PF3PY/ax+Fe0upBhGIe+jvSDUai5RAb7cxSwhffnhx1JQZroieeIDxb6n1cpuRN8wd/z4zmtk=</password>
         </configuration>
    </definition>
</datasource>

For different environments , we might need to change

• Database url
• Database username
• Database password

2.0 Governance Registry

To manage different environments, we need a running WSO2 Governance Registry instance and we need to mount the config registries of WSO2 DSS to different target paths. The paths will indicate the different environments.

Eg:

<mount path="/_system/config" overwrite="true">
    <instanceId>config</instanceId>
    <targetPath>/_system/config/env/prod</targetPath>
</mount>

All servers will talk to same database.( That is, wso2greg and wso2dss.)

We need to upload the datasource configurations for different environments directly to the registry in the relevant environment paths. Note that, we can not use carbon datasource UI options available at dataservice server. Because WSO2 G-Reg will be the centralized server to manage different environments, so we need to avoid creating datasources from dataservice server.

3.0 Configurations 

3.1 WSO2 G-Reg

master-datasource.xml

Create a new data source to point an external database. Eg: mysql.

 <datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource used DSS mount registries</description>
            <jndiConfig>
                <name>jdbc/dssserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/dss</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

Use the created datasource in the registry.xml

<!-- This defines the default database and its configuration of the registry -->

    <dbConfig name="wso2registry">
        <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>

Copy the mysql jdbc driver in repository/components/lib folder and start the server. Make sure to create tables in the mysql database. You can find the relevant sql scripts in the CARBON_HOME\dbscripts folder.

We need to set different port offsets to all carbon servers in carbon.xml. For G-Reg, let’s keep the default value (ie: ‘0’)

3.2 WSO2DSS-QA

In this sample, let’s take two DSS servers for QA and production environments.

master-datasource.xml
Create a new data source to point the mysql database which is already created.

<datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource for DSS</description>
            <jndiConfig>
                <name>jdbc/dssserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/dss</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

We need to mount the config registry to different target path to make sure QA DSS server talks to different config registry. Governance registry space of all nodes will be mounted to same location, since governance space is common for all nodes.

<dbConfig name="config">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>config</id>
        <dbConfig>config</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/config" overwrite="true">
        <instanceId>config</instanceId>
        <targetPath>/_system/config/env/qa</targetPath>
    </mount>

     <dbConfig name="governance">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>governance</id>
        <dbConfig>governance</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/governance" overwrite="true">
        <instanceId>governance</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount>

Copy the mysql jdbc driver in repository/components/lib folder. Set different port offset in carbon.xml. For DSS_QA server, let’s keep it as ‘1’. Start the server.
Go to https://localhost:9444/carbon/ URL and check the registry browser, you will see the registries are mounted.

3.3 WSO2DSS-PROD

master-datasource.xml
Create a new data source to point the mysql database which is already created.

<datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource for DSS</description>
            <jndiConfig>
                <name>jdbc/dssserver_config</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/dss</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

registry.xml

We need to mount the config registry to different target path to make sure PROD_DSS server talks to different config registry. Governance registry of all nodes will be mounted to same location, since governance space is common for all nodes.

<dbConfig name="config">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>config</id>
        <dbConfig>config</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/config" overwrite="true">
        <instanceId>config</instanceId>
        <targetPath>/_system/config/env/prod</targetPath>
    </mount>

     <dbConfig name="governance">
       <dataSource>jdbc/dssserver_config</dataSource>
    </dbConfig>
      <remoteInstance url="https://localhost:9443/registry">
        <id>governance</id>
        <dbConfig>governance</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
        <enableCache>true</enableCache>
    </remoteInstance>
     <mount path="/_system/governance" overwrite="true">
        <instanceId>governance</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount>

Copy the mysql jdbc driver in repository/components/lib folder. Set different port offset in carbon.xml. For DSS_PROD server, let’s keep it as ‘2’. Start the server.
Go to https://localhost:9445/carbon/ URL and check the registry browser, you will see the registries are mounted.

3.4 Datasource Configuration

When we create carbon datasource from UI, the password will be encrypted and saved to registry. Here since we need to directly upload the data source configuration to the registry, we can just upload the password in plain text. If user needs to encrypt the password, later he can access the datasource via datasource UI and save it. So, it will be encrypted and saved to registry.

Sample data source for QA-DSS server

<datasource>
    <name>testdatasource</name>
    <definition type="RDBMS">
        <configuration>
            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
            <url>jdbc:mysql://localhost:3306/service_qa</url>
            <username>root</username>
            <password encrypted="false">root</password>
        </configuration>
    </definition>
</datasource>

Save the configuration as testdatasource and upload in WSO2 G-Reg server under qa environment. Go to, https://localhost:9443/carbon and browse to /_system/config/env/qa/repository/components/org.wso2.carbon.ndatasource path and upload.

Sample data source for PROD-DSS server

Copy the same configuration and edit the parameters to reflect the production database configurations and save as testdatasource.

Eg:

<datasource>
        <name>testdatasource</name>
        <definition type="RDBMS">
            <configuration>
                <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                <url>jdbc:mysql://localhost:3306/service_prod</url>
                <username>root</username>
                <password encrypted="false">root</password>
           </configuration>
      </definition>
</datasource>

  Go to, https://localhost:9443/carbon and browse to /_system/config/env/prod/repository/components/org.wso2.carbon.ndatasource path and upload.

Now same datasource configuration is available under two paths with the environment specific parameters. But the datasources’ name is same.(ie: testdatasource)

3.5 Dataservice

When we create dataservice, there is no any environment specific parameters. Only thing is, we need to point the datasource there, which contains environment specific parameters.

We already separated and stored the different datasource configurations with same name in different registry paths. So, if we upload same dataservice in both servers(ie: DSS_QA and DSS_PROD) both servers will talk to the relevant data source configurations, which are stored under relevant config registry without any issue.

Sample dataservice

<data name="dss_test">
   <config id="testdatasource">
      <property name="carbon_datasource_name">testdatasource</property>
   </config>
   <query id="select_person" useConfig="testdatasource">
      <sql>call `person-info`(?, ?, ?);</sql>
      <result element="people" rowName="person">
         <element column="PERSON_NAME" name="name" xsdType="string"/>
         <element column="PERSON_NICK" name="nick" xsdType="string"/>
         <element column="PERSON_PHONE" name="phone" xsdType="string"/>
      </result>
      <param name="pName" sqlType="STRING"/>
<param name="pPhone" sqlType="STRING"/>
      <param name="pNick" sqlType="STRING"/>
   </query>
   <operation name="select_person">
      <call-query href="select_person">
         <with-param name="pName" query-param="pName"/>
         <with-param name="pPhone" query-param="pPhone"/>
         <with-param name="pNick" query-param="pNick"/>
      </call-query>
   </operation>
</data>

Related post
Environment Management in WSO2 ESB using WSO2GREG

Processing large text file using smooks in wso2esb

Smooks  library supports large file processing which is used in ESB to process file which are in text/xml format.
Text files can be a CSV file or with a space delimiter or a fixed length file. To process large file, smooks write the output stream , which is a single line record, to a temporary file or a jms queue. After writing all files to the temporary location, we need to process all those records/files again. This is how we can avoid large memory growth/out of heap space issue which occurs when we try to process large file as in- memory process.

Following is a sample configuration to process fixed length large text file in wso2esb.

Proxy configuration

To pick and process the file form a file location , we could use VFS transport.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="cor_file_pull"
       transports="https,http,vfs"
       statistics="disable"
       trace="disable"
       startOnLoad="true"target>
     <inSequence>
  <property name="DISABLE_SMOOKS_RESULT_PAYLOAD"
                   value="true"
                   scope="default"
                   type="STRING"/>
          <smooks config-key="simple-smook">
            <input type="text"/>
           <output type="xml"/>
         </smooks>
       </inSequence>
      <outSequence/>
   </target>
   <parameter name="transport.vfs.Streaming">true&lt;/parameter>
   <parameter name="transport.PollInterval">10&lt;/parameter>
   <parameter name="transport.vfs.ActionAfterProcess">MOVE&lt;/parameter>
   <parameter name="transport.vfs.MoveAfterProcess">C:/Users/TOSH/Desktop/success</parameter>
   <parameter name="transport.vfs.FileURI">C:/Users/TOSH/Desktop/in&lt;/parameter>
   <parameter name="transport.vfs.MoveAfterFailure">C:/Users/TOSH/Desktop/failure</parameter>
   <parameter name="transport.vfs.Locking">disable&lt;/parameter>
   <parameter name="transport.vfs.FileNamePattern">^(cor).+</parameter>
   <parameter name="transport.vfs.ContentType">text/plain</parameter>
   <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   <description/>
</proxy>

Smooks configuration

simple-smook localentry

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:ftl="http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd" xmlns:fl="http://www.milyn.org/xsd/smooks/fixed-length-1.3.xsd" xmlns:file="http://www.milyn.org/xsd/smooks/file-routing-1.1.xsd">
<params>
            <param name="stream.filter.type">SAX </param>
           <param name="default.serialization.on">false </param>
         </params>

<fl:reader fields="RecordId[4],CorpName[60],FileNumber[9]" skipLines="0">
</fl:reader>
      <resource-config selector="record">
         <resource> org.milyn.delivery.DomModelCreator</resource>
     </resource-config>
      <file:outputStream openOnElement="record" resourceName="fileSplitStream">
         <file:fileNamePattern> ${.vars["record"].FileNumber}.xml</file:fileNamePattern>
         <file:destinationDirectoryPattern> C:\Users\TOSH\Desktop\processed</file:destinationDirectoryPattern>
         <file:highWaterMark mark="10000000"> </file:highWaterMark>
      </file:outputStream>
      <ftl:freemarker applyOnElement="record">
         <ftl:template> /repository/resources/smooks/record_as_xml.ftl</ftl:template>
         <ftl:use>
            <ftl:outputTo outputStreamResource="fileSplitStream"> </ftl:outputTo>
         </ftl:use>
      </ftl:freemarker>
   </smooks-resource-list>

Each record will   be written to the specified file location   "C:\Users\TOSH\Desktop\processed" with the name constructed through the parameter "  ${.vars["record"].FileNumber}.xml ".  Here it is a xml file, which will be named with a file number    record in the text file. We have to select a unique parameter to write the processed files else there will be file overwriting issue will be thrown in smooks.
To process large file we have to use 'SAX' filter type. DOM needs more memory and it will lead to out of memory issue.
In this sample, i have used DOM+SAX mixing model to keep the lower footprint of memory.
High  water mark provides the maximum file records need to be written. provide enough number based on the lines in the text file.
The property ;
 <property name="DISABLE_SMOOKS_RESULT_PAYLOAD"   value="true" scope="default"     type="STRING"/>

will be available in ESB 4.9.0. If you use lower version, you might need a patch to be applied to your system.

XML template file
record_as_xml.ftl

<#assign rec = .vars["record"]>
<p:insert_acc_data_file xmlns:p="http://ws.wso2.org/dataservice"><p:RecordId>${rec.RecordId}</p:RecordId>
<p:CorpName>${rec.CorpName}&lt;/p:CorpName>
<p:FileNumber>${rec.FileNumber}&lt;/p:FileNumber></p:insert_acc_data_file>

Based on above template the records will be formatted and will be written to the .

Using another vfs proxy, we can pick the entries written to the  'processed' file and do further processing.

Executing FaultSequence for SOAPFaults in WSO2ESB

In WSO2ESB/Synapse, whenever mediation errors occur, user can execute faultsequence to find error details.
The fault sequence can be defined at a sequence level or at proxy service level. Mediation level issues and endpoint errors can be captured at faultsequence. But SOAPFault can not be captured at fault sequence, since ESB/Synapse will treat it as the response from the endpoint.
But from user's pont of view , those are service endpoint failures which need to be handled properly.
To handle the soapfaults, now we can use a property which forces to execute fault sequence whenever soapfault occurs.
We need to set  following property before sending the message to endpoint.;

<property name="FORCE_ERROR_ON_SOAP_FAULT" value="true"/>

Sample conf;

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="testProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
  <target faultSequence="myfaultSeq">
      <inSequence>
         <property name="FORCE_ERROR_ON_SOAP_FAULT" value="true"/>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
      <endpoint>
         <address uri="http://localhost:9764/services/echo/"/>
     </endpoint>
   </target>
   <description/>
</proxy>
           

In the above sample , whenever we get soapfault, the 'myfaultSeq' will be executed.

Related Post: Sending mails for errors