Data science and data engineering trends

AI solutions and other data-related projects depend on having access to data. Data engineers build data lakes and ensure that the right data is available at the right moment.

Data modeling will create clarity when consolidating business domain specific databases, while data governance and other data management skills ensure the integrity and security of data.

Today, our largest clients are already in the process of developing their data warehouses. Smaller players will follow suit just a couple of steps behind.

Data mesh is an emerging paradigm shift in the data architectures and platforms are viewed in the context of business. Forerunners are already in the process of data mesh experimentation and implementation which is first and foremost a cultural shift on how data will be governed in the future.

Image and speech recognition and other solutions to generic problems are already available as easily integrated APIs, and their significance will keep growing. The number of ready machine learning (ML) services will also continue to grow. As a result, developers will be able to implement advanced features without a deep knowledge of data science.

Even in projects that require custom ML solutions, AutoML will partly automate the model optimization and therefore enhance the impact of the data scientists’ work.

Companies want to capitalize on their textual and speech data, and this applies in languages other than English as well.

Cloud platforms already provide NLP tools for a number of languages and plan to release more in the future to cater to additional markets. Even so, plenty of specialized problems remain to be solved that are not universal enough to convince the big players to implement purpose-built tools for relatively small languages.

The need for both simpler rule-based models and more advanced pre-trained deep learning models remains to be solved by smaller boutique players.

MLOps, or the management of the lifecycle of machine learning (ML) projects, will become increasingly important when we develop long-term solutions and not just proofs-of-concept (PoCs).

MLOps means developing systematic approach to the monitoring, scalability and evaluation of data pipelines and ML models. Model training should be reproducible and deployment should be as automated as possible.

The ability to rapidly build end-to-end solutions allows for an improved focus on providing genuine business value.

As many aspects of our daily lives are becoming automated by machine learning systems, it is important to mitigate their inherited risks and potential harm. Data scientists need to be acutely aware of potential sources of bias and have the necessary tools to evaluate the impact of the systems they are building.

Data privacy is important to ensure the end-users’ trust towards the application. Technical solutions, such as federated learning, can ensure that private data is not shared more widely than necessary – but data scientists will also need to be aware of privacy best practices.