Historical data warehouse and BI-platform implementation (Telecom)

The Customer

Telecommunication company, which provides mobile telecommunications, internet connection, and cloud computing services for individuals and legal entities.

The problem

As part of the project, the technical specialists were faced with the task of solving performance issues of the existing historical data warehouse, which with the increased inflow of information, ceased to satisfy the Customer. At the beginning of the project, the information in the warehouse was collected from a single data source, while the collection of daily information itself and its processing usually took more than 16 hours.

In addition, the Customer’s company used manual collection and aggregation of data from three different sources to generate reports used for making managerial decisions. The work on the preparation of weekly reports was carried out by the efforts of 3 analysts during 3 workdays. As a part of the project, Invento Labs specialists had to design and implement a BI platform that would automatically receive operational data in a short span of time.

Another problem that the company representatives wanted to solve was the desire to reduce the cost of ownership and maintenance of the existing solution, which normally cost them $ 100,000 per year.

The solution

An audit of the existing solution showed that attempts to optimize it wouldn’t give the desired effect and wouldn’t solve the ownership cost problem.

That’s why the first thing Invento Labs specialists took a special interest in was the choice of the tools needed for the completion of the project.

Invento Labs architects proposed to use the MPP GreenPlum distributed database as a historical data warehouse. The proposed software removed a whole range of problems and provided some additional benefits:

  1. MPP GreenPlum is an open-source and free software product, therefore, its ownership does not imply the cost of purchasing a license.
  2. MPP GreenPlum architecture implies the distribution of data between several servers, which increases the speed of data writing and data reading. As a result, the historical data warehouse would no longer be a bottleneck when collecting data from different data sources.
  3. An additional advantage of using MPP GreenPlum is its ability to duplicate data on different servers, which provides protection against data loss in case of server failure where the historical data storage is located.

After the deployment of MPP GreenPlum the team of Invento Labs began the adaptation of the existing data collection, transformation, and aggregation (ELT / ETL) processes.

To provide visualization of analytical reporting, the team chose the Tableau product, which at the start of the project was already enlisted under the “Leaders” category in the 4th Gartner quadrant. As it was later mentioned in the report to the Customer’s CEO, the proposed product made it possible to visualize more than 50 different indicators using only 4 dashboards. The installed instance of Tableau-server was connected to the Customer’s Microsoft Active Directory authorization service, which allowed managing access rights to various reports in a flexible manner.

The use of Tableau did not end with the report to the CEO. After the training, led by BI experts of Invento Labs, the Customer’s analysts continued the creation of the necessary reports for various company departments.

Technologies
The result

As a result of the implementation of the proposed solution for the source data warehouse and BI platform, the Customer was able to:

  • Increase the performance of the data collection and processing system. The time spent on this operation has decreased from 16 hours to less than 2 hours;
  • Reduce labor costs to ensure data safety;
  • Increase the number of primary data sources from 1 to 3;
  • Improve the quality of data provided for analysis in 9% percent of cases, in comparison with the previously available solution;
  • Receive automatically generated reports in a few hours after the end of the reporting period, which previously took about three working days and three analysts to generate;
  • Reduce the cost of ownership of a complex solution with additional functionality by more than 30%.