Step by Step Guide for the Data Migration of WSO2 IS Analytics from DAS to SP

Analytics profile of WSO2 IS 5.6.0 & below [IS 5.x.x] are based on WSO2 DAS. Analytics profile of WSO2 IS 5.7.0 is based on WSO2 Stream Processor. When upgrading from WSO2 IS 5.x.x to 5.7.0, you are moving from an instance of WSO2 DAS to WSO2 Stream Processor. IS Analytics table migration is an indirect process where we need to convert the Analytics tables into RDBMS tables by running spark scripts using the CarbonJDBC provider packed with DAS. The “migISAnalytics.sh/migISAnalytics.bat” scripts create the IS Analytics tables by executing a Java client and the “migISAnalyticsSpark” spark script migrate the data. You can read more about the architectural background behind the data migration in my blog post “Story Behind the Data Migration of WSO2 Analytics Solutions from DAS to SP”

How to build?

• Clone or Download the IS Analytics Data Migration Github Repo from the following link => Roly-Data-Migration-IS-Analytics

git clone https://github.com/RolandHewage/Roly-Data-Migration-IS-Analytics.git

• Go to Roly-Data-Migration-IS-Analytics/is-analytics/migration/migration-IS5.x.x-5.7.0/resources/ folder.

• Go to $project home(mig_is_analytics) folder and execute following command from the project home folder to generate the final Jar file.

$mvn clean install

• A new single jar named “migIS.one-jar.jar” is created in the $project/target folder together with its dependency jars.

How to migrate?

• Copy the built jar file “migIS.one-jar.jar” and “migISAnalytics.sh or migISAnalytics.bat” file into a same directory location.
• Execute the “migISAnalytics.sh or migISAnalytics.bat” file.

$sudo sh migISAnalytics.sh

• Enter the correct database details in the command prompt.

• The tables related to IS Analytics are created in the relevant database specified by the user.
• Next add the datasource configuration to the master-datasources.xml file in {IS_ANALYTICS_HOME}/repository/conf/datasources folder of the old DAS product.

Eg:-

<datasource>
            <name>IS_ANALYTICS_DB</name>
            <description>The datasource used for is-analytics persisted data</description>
            <jndiConfig>
                <name>jdbc/IS_ANALYTICS_DB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/IS_ANALYTICS_DB?autoReconnect=true&amp;relaxAutoCommit=true&amp;useSSL=false</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                    <defaultAutoCommit>false</defaultAutoCommit>
                </configuration>
            </definition>
</datasource>

• Add the JDBC driver to the {IS_ANALYTICS_HOME}/repository/components/lib folder
• Next run the Analytic profile of the old version of WSO2 IS which is based on WSO2 DAS using the following command.

$./wso2server.sh

• Log in to the management console by clicking the URL in the terminal.

• Go to [Main Home -> Dashboard -> Analytics Dashboard] and login

• Click the “View” mode of the dashboard

• You can view the IS Analytics Dashboard of the old DAS based product as follows

• Go to [Main Home -> Manage -> Batch Analytics -> Scripts] and add the New Analytics Script (“migISAnalyticsSpark” spark script).

• Then execute in the background (Important to execute in background. If not it would throw a thread death error) the “migISAnalyticsSpark” spark script to migrate the data related to the analytics profile. The migrated data would be stored in the RDBMS database specified by the user.

• Next add the datasource configuration to the deployment.yaml file in {IS_ANALYTICS_HOME}/conf/{dashboard &worker} folder of the new SP product.
• Run the Analytic profile of the new version of WSO2 IS which is based on WSO2 SP to generate the rest of the IS Analytics tables using the following command.

$./worker.sh

• Run the Analytic dashboard of the new version of WSO2 IS to view the migrated statistics using the following command.

$./dashboard.sh

• You can view the IS Analytics Dashboard of the new SP based product as follows

new SP based dashboard

old DAS based dashboard

We can see that identical statistics are shown in the old DAS based dashboard and new SP based dashboard after the successful data migration of IS Analytics from DAS to SP.

Analytics profile of WSO2 IS 5.6.0 & below [IS 5.x.x] are based on WSO2 DAS. Analytics profile of WSO2 IS 5.7.0 is based on WSO2 Stream Processor. When upgrading from WSO2 IS 5.x.x to 5.7.0, you are moving from an instance of WSO2 DAS to WSO2 Stream Processor. IS Analytics table migration is an indirect process where we need to convert the Analytics tables into RDBMS tables by running spark scripts using the CarbonJDBC provider packed with DAS. The “migISAnalytics.sh/migISAnalytics.bat” scripts create the IS Analytics tables by executing a Java client and the “migISAnalyticsSpark” spark script migrate the data. You can read more about the architectural background behind the data migration in my blog post “Story Behind the Data Migration of WSO2 Analytics Solutions from DAS to SP”

How to build?

• Clone or Download the IS Analytics Data Migration Github Repo from the following link => Roly-Data-Migration-IS-Analytics

git clone https://github.com/RolandHewage/Roly-Data-Migration-IS-Analytics.git

• Go to Roly-Data-Migration-IS-Analytics/is-analytics/migration/migration-IS5.x.x-5.7.0/resources/ folder.

• Go to $project home(mig_is_analytics) folder and execute following command from the project home folder to generate the final Jar file.

$mvn clean install

• A new single jar named “migIS.one-jar.jar” is created in the $project/target folder together with its dependency jars.

How to migrate?

• Copy the built jar file “migIS.one-jar.jar” and “migISAnalytics.sh or migISAnalytics.bat” file into a same directory location.
• Execute the “migISAnalytics.sh or migISAnalytics.bat” file.

$sudo sh migISAnalytics.sh

• Enter the correct database details in the command prompt.

• The tables related to IS Analytics are created in the relevant database specified by the user.
• Next add the datasource configuration to the master-datasources.xml file in {IS_ANALYTICS_HOME}/repository/conf/datasources folder of the old DAS product.

Eg:-

<datasource>
            <name>IS_ANALYTICS_DB</name>
            <description>The datasource used for is-analytics persisted data</description>
            <jndiConfig>
                <name>jdbc/IS_ANALYTICS_DB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/IS_ANALYTICS_DB?autoReconnect=true&amp;relaxAutoCommit=true&amp;useSSL=false</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                    <defaultAutoCommit>false</defaultAutoCommit>
                </configuration>
            </definition>
</datasource>

• Add the JDBC driver to the {IS_ANALYTICS_HOME}/repository/components/lib folder
• Next run the Analytic profile of the old version of WSO2 IS which is based on WSO2 DAS using the following command.

$./wso2server.sh

• Log in to the management console by clicking the URL in the terminal.

• Go to [Main Home -> Dashboard -> Analytics Dashboard] and login

• Click the “View” mode of the dashboard

• You can view the IS Analytics Dashboard of the old DAS based product as follows

• Go to [Main Home -> Manage -> Batch Analytics -> Scripts] and add the New Analytics Script (“migISAnalyticsSpark” spark script).

• Then execute in the background (Important to execute in background. If not it would throw a thread death error) the “migISAnalyticsSpark” spark script to migrate the data related to the analytics profile. The migrated data would be stored in the RDBMS database specified by the user.

• Next add the datasource configuration to the deployment.yaml file in {IS_ANALYTICS_HOME}/conf/{dashboard &worker} folder of the new SP product.
• Run the Analytic profile of the new version of WSO2 IS which is based on WSO2 SP to generate the rest of the IS Analytics tables using the following command.

$./worker.sh

• Run the Analytic dashboard of the new version of WSO2 IS to view the migrated statistics using the following command.

$./dashboard.sh

• You can view the IS Analytics Dashboard of the new SP based product as follows

new SP based dashboard

old DAS based dashboard

We can see that identical statistics are shown in the old DAS based dashboard and new SP based dashboard after the successful data migration of IS Analytics from DAS to SP.