Data Migration of WSO2 IS Analytics from DAS to SP

What is Identity Server?

WSO2 Identity Server (WSO2 IS) is a comprehensive identity and access management (IAM) solution. It caters to identity management requirements across many platforms such as enterprise applications, services, and APIs. The Identity Server enables enterprise architects and developers to reduce identity provisioning time, guarantee secure online interactions, and deliver a reduced single sign-on environment. The WSO2 Identity Server decreases the identity management and entitlement management administration burden by including the role-based access control (RBAC) convention, fine-grained policy-based access control, and Single-Sign-On (SSO) bridging.

WSO2 Identity Server

IS Analytics

WSO2 Identity Server Analytics allows you to view and analyze statistics related to authentication requests that go through WSO2 Identity Server. You can configure WSO2 Identity Server to publish authentication related events to WSO2 Identity Server Analytics so that you view and analyze the following:

  • Login analytics: Includes statistics related to login attempts made via WSO2 Identity Server.
  • Session analytics: Includes statistics related to specific sessions that get created for different applications accessed via WSO2 Identity Server.
IS Analytics

The Analytics profile of WSO2 Identity Server (WSO2 IS) 5.7.0 & above is an instance of the WSO2 Stream Processor (WSO2 SP) runtime. This Analytics profile is, by default, configured to monitor statistics related to authentication requests that go through WSO2 Identity Server.

The new SP based Analytics profile consists of two components: Worker and Dashboard. The worker is the server that processes the data streams that are sent from the IS server and publishes the statistics to a database. The dashboard reads the statistics published by the worker and displays the statistics on the dashboard. The worker and dashboard are connected through the database.

The old DAS based IS Analytics consists of the components: Data Agents, Event Recievers, Analytics REST API, Data store, Siddhi Event Processors, Analytics Spark, Data Indexing, Event Publishers, Analytics Dashboard, Event Sinks which are accessible through the DAS Management Console. WSO2 Identity Server (WSO2 IS) 5.6.0 & below integrates with the WSO2 Data Analytics Server (WSO2 DAS) to monitor statistics of login attempts & sessions that get created for different applications accessed via WSO2 Identity Server.

Data migration of IS Analytics from DAS to SP is migrating persisted analytics data in the Analytics Record Store of the old DAS product to the data store specified by the user & used by the Worker & Dashboard in the new SP product for IS Analytics. We can’t use the same RDBMS data store of the old DAS product as it is in the new SP product because data is stored as blobs in the old DAS product which is encrypted & not human readable. Database level indexing cannot be used for blobs to search by fields.

I had to write a spark script to migrate the IS Analytics data in the Analytics Record Store from CarbonAnalytics to a relational database specified by the user through CarbonJDBC. But database level indexing wasn’t possible through CarbonJDBC provider. So I had to write a JAVA client to create the IS Analytics tables related with the new SP product prior to any data migration. I had to write a migration client to be given to the user. A shell script migration client which calls the JAVA client internally was written for Linux users & a batch script migration client was written for Windows users. Then the data migration client was the Spark Script. So prior to writing any code I had to explore the database schemas of the IS Analytics data stores of the old DAS product & the new SP product & do a relevant mapping of the tables & relevant fields & find common solutions for any issue faced during the data mapping and data migration.

Steps followed for the Data Migration of IS Analytics from DAS to SP

  1. Explore the database schemas of the IS Analytics data stores of the old DAS product & the new SP product.
  2. Data mapping of the IS Analytics tables & relevant fields & find common solutions for any issue faced during the data mapping.
  3. Implement the JAVA Client to create the IS Analytics tables related with the new SP product prior to any data migration.
  4. Implement the Shell Script migration client for Linux users which calls the JAVA Client internally.
  5. Implement the Batch Script migration client for Windows users which calls the JAVA Client internally.
  6. Implement the data migration client which is a Spark Script to migrate the IS Analytics Data in the Analytics Record Store from CarbonAnalytics to a Relational database specified by the user through CarbonJDBC.

A data mapping was made by exploring the underline database schemas of the IS Analytics data stores of the old DAS product & the new SP product. The analytics data of the old DAS based product was explored using the Data Explorer of DAS management console.

The java client creates the IS Analytics tables of the new SP based product in the relevant database specified by the user. The java client was written to create the tables in MySQL, PostgreSQL, ORACLE, Microsoft SQL Server according to the requirement of the user. The logic was handled in the shell script / batch script & the table creation was handled in the java client. Separate functions were written to handle table creation for each RDBMS database.

For MySQL the part of the code segment is as follows.

Function that creates IS Analytics tables in MySQL

For PostgreSQL the part of the code segment is as follows.

Function that creates IS Analytics tables in PostgreSQL

For ORACLE the part of the code segment is as follows.

Function that creates IS Analytics tables in ORACLE

For MSSQL the part of the code segment is as follows.

Function that creates IS Analytics tables in MSSQL

In all the functions query execution was handled through a list. The following figure shows the sample code segment written for MSSQL. Similar method was used for all the other functions MySQL, ORACLE, PostgreSQL etc…

Query Execution handled through a list

The JDBC Driver & JDBC URL setting logic was handled in the following code segment.

JDBC Driver & JDBC URL setting logic

The dynamic loading of the JDBC driver within the java client itself was handled through the following code segment.

Dynamic loading of the JDBC driver within the java client

The database connection establishment & calling of functions to create tables in the relevant RDBMS database were handled in the following code segment.

Database connection establishment

The code segment which shows all the exception & error handling logic while creating the tables is given below.

Exception & error handling logic

A separate class was used to dynamically load the jar within runtime.

Class that dynamically load the jar within runtime

A POM file was used to add the relevant dependencies for the MAVEN project. Log4j plugin was used to log all the errors. The relevant dependency in the POM.xml file is given below.

POM.xml file

The maven-jar-plugin was used to make the jar executable

maven-jar-plugin

One-jar plugin was used to pack all the dependency jars inside one single jar.

One-jar plugin

The following figures show the shell script written to get the relevant RDBMS database specification from the user migrating to the new SP based product & the relevant java client is invoked through the script by passing the database type, host, port, name, user, password & the absolute path of the database driver location as arguments to the jar file.

Shell Script to call Java Client in IS Analytics

The corresponding Batch script for windows users is given below.

Batch Script to call Java Client in IS Analytics

A sample code segment in the spark script which does the data migration for processed data (an aggregation table) is given below.

Data Migration for Processed Data in Spark Script

A sample code segment in the spark script which does the data migration for event data is given below.

Data Migration for Event Data in Spark Script

--

--

--

(Born to Code) | Software Engineer (Ecosystem Engineering) at WSO2 | Bachelor of Computer Science (Special) Degree Graduate at University of Ruhuna, Sri Lanka

Love podcasts or audiobooks? Learn on the go with our new app.

Identity Graph and Identity Resolution in SQL

gqlgen shortcut for script

Hosting your own CTF

Ruby Iterations with each, map, and select

Start moving you operations to the cloud — what you need to know

A Lesson In Effective Innovation From Amazon

Developers, Start Paying For Stuff, Will You? — Done

We can extrapolate username and a password into any block-chain address, here is how.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Roland Hewage

Roland Hewage

(Born to Code) | Software Engineer (Ecosystem Engineering) at WSO2 | Bachelor of Computer Science (Special) Degree Graduate at University of Ruhuna, Sri Lanka

More from Medium

4 days — decent understanding of Big-Query

My PMDojo Learning Journey

Planting Seeds of Success for Data-Enabled Organizations

What is indoor mapping and how to benefit from it?