Our customer is a British global FMCG organization active in over 100 countries.
- The customer needed to integrate a very large volume of data coming a variety of sources containing multi-structured information such as organisational data, web logs, social media, multimedia (images, audio, video), spatial and GPS data and, finally, devices and sensors information.
- In addition the organisation needed to add data virtualization and cloud integration capabilities on top of the traditional data warehouse
- Architecture needed to be scalable and to empower agile delivery Lastly, the querying requirements called for a smooth search ability leveraging the use of a data catalogue.
In response to our customer’s request, we have formulated a modern data warehouse solution featuring a multi-platform architecture in a hybrid environment.
In order to grow the existing data warehouse and extend it into a modern variant, we worked with the customer to enable them to define and formulate new data modelling strategies for the customer, while creating a new partitioning. We also touched on items such as in-memory structures, MPP and clustered columnstore indexes.
Through this customer modern data warehouse solution, we extended the capabilities by adding complementary storage and analytics tools, on a hybrid infrastructure complete with data virtualization functionalities.
Data marts were put in place to fuel the in-memory model, while a master data capability communicated with the data warehouse.
Furthermore, the solution is set up as scalable with a view of adding data lakes as a history area for the data warehouse in a future development phase. The objective of the data lake architecture is to enable an archival process based on a predefined data retention policy as well as to allow for data retrieval based on federated query. The data lake store capability could further integrate several areas for raw data as a staging environment, as well as an active archive operating in parallel and fueling the near-to-real-time analytics and reporting tools.
In terms of technology, the solution was deployed using the Microsoft stack where our consultants and developers are certified. Other technologies such as Hadoop, NoSQL can also be accommodated and are recommended based on best practice and best fit for the customer need.
- Significant value can be derived from a modern data warehouse solution as it enables data experimentation. This is an essential component of a cross-functional data-driven organizational culture that relies heavily on the intersection of the Business and IT functions within a large company such as our customer’s.
- The hybrid architecture solution was capable of integrating data from cloud based systems such as social media and CRM, as well as on-premises ones such as systems used for sales or inventory.
- Operational reporting as well as historical analysis is now complemented with advanced analytics, self service reports and models.
