Andreas Bartsch, Head of Service Delivery at PBT Group

Companies today want to migrate from their on-premise environments to the cloud and inject more agility into their operations. But even though many of the cloud technologies and concepts have been with us for a while, it is the organisations’ realisation that data will be the critical success factor in their ability to compete and be sustainable, which prompted the need to obtain better understanding and insight.

In the traditional data warehouse and business intelligence environments, we have become familiar with the ‘Big Five’ solution providers – Microsoft, IBM, Oracle, SAP, and SAS. We have also become accustomed to the relevant databases, as well as ETL (extract, transform, load) tools like Microsoft SSIS, IBM DataStage, Oracle Data Integrator, SAP Data Services, Informatica, amongst others. The typical data platform consisted of staging layers, an Enterprise Operational Data Store (EODS), an Enterprise Data Warehouse (EDW), and the Visualisation Layer.

Different ‘viewpoints’

However, for every component available in the cloud, each cloud service provider (CSP) has its own terminologies, solutions, and ways of configuring services. The emergence of the Enterprise Data Lake (EDL) resulting from the need to derive value from unstructured data (Big Data) and in enabling the data scientist. The COVID-19 pandemic has emphasised how data has become a predominant feature in the way business is done. In fact, good data is a non-negotiable ingredient to most aspects of successful Artificial Intelligence.

Already, organisations are embarking on major strategic initiatives to establish data-driven cultures. Regulatory compliance and security remain high profile focus areas and necessitate appropriate data governance. The Internet of Things (IoT), edge computing, and technologies like 5G furthermore enable access to more data, and data in near real-time. Cloud services provide the opportunity to leverage these technologies and the vast volumes of data but require data specialist expertise to do so.

An agnostic perspective

Amidst the mentioned CSPs and associated technologies, we have also noticed emerging roles like Cloud Administrator, Cloud Engineer, and Cloud Architect, to name a few. As with the technologies, these are typically branded as per the relevant CSP. Whilst these roles are necessary for organisations who migrate to the cloud, be aware that they do not replace the likes of a data engineer or data architect. Instead, these are now evolving to become cloud data engineers and cloud data architects.

Within this, remaining technology agnostic is not only crucial for success, but more relevant than ever to deliver an unbiased view. There is no doubt that the cloud has introduced many new technologies and most definitely a new ‘way of work’, which must be adapted to. As data specialists, it is our responsibility to remain informed, knowledgeable, and technology savvy – we must continuously cross-skill and learn.

Strong foundation

It also means that the basic principles of data management still apply and are possibly more important than ever. Data architecture, data modelling, data engineering, and data governance will remain critical competencies, irrespective of the CPS or associated technologies used.

Dare one say, every cloud has a silver lining with the cloud (and technology)-agnostic data specialist remaining relevant, unbiased, and important. So, even though some of the ways that data is managed might differ between cloud systems, the principles of good data practice will always apply. Having a truly agnostic data specialist is therefore fundamental to any cloud data migration journey.