Cloud is the most important way for companies to run their workloads, and at the same time, cloud-enabled companies are rapidly adopting more sophisticated cloud technologies. AWS is still the big fish and Google has a huge reach, but Azure is also gaining traction fast. As organizations begin harnessing AI at scale, data engineering and MLOps will be necessary to ensure a proper foundation and tooling for the task.
Cloud adoption and prevalence
Cloud is no longer a distant thing of the future or a secondary option for businesses. Cloud adoption is a must, and it is increasingly becoming the primary – and even only – option for companies to run their workloads.
AWS has remained the market leader year after year. However, in the past few years Azure has made huge leaps, and it’s now becoming quite prominent among our client base.
Cloud-based SaaS products such as Snowflake (Cloud Data Warehouse) are also becoming very prevalent and challenging the incumbent market.
Hybrid cloud and multi-cloud are big topics in enterprise contexts today. In a highly competitive market, enterprise clients are often using (or exploring) combinations of one or two cloud providers and private, on-premises cloud environments. Kubernetes is a key technology to master especially in this space, and Google Anthos and Microsoft are something to keep an eye on as well.
More than ever, companies need guidance in enterprise and solution architecture, strategy as well as data and cloud best practices – for example infrastructure as code (IaC).
Serverless, SRE and low-code
Organizations are becoming increasingly cloud-native. This will inevitably be followed by a huge adoption of serverless architectures and managed cloud services.
There is a huge push towards serverless and low-code or no-code solutions (e.g. MS PowerApps, AWS Honeycode) because of their lower operational overhead and faster time to value especially in internal applications.
Topics such as SRE (Site Reliability Engineering) and chaos engineering are also gaining traction. After an MVP or first release, it’s not enough to simply keep the lights on, and SRE strives for continuous operational improvement, scalability and reliability. Chaos engineering practices, on the other hand, ensure systems can handle and tolerate failure and regularly put assumptions to the test.
Data engineering and MLOps
Data and AI have become massively hyped buzzwords. Data-driven organizations that actively run AI experiments know that a huge part of that work is data engineering. Without a solid technical foundation, there will be neither data nor AI.
With data collection growing at an exponential rate, it is absolutely vital to have the skills necessary to understand how to acquire, store and prepare Big Data solutions. Likewise, machine learning operations (MLOps) is also a trending topic – and for a simple reason: data scientists need proper tooling, interfaces and processes to drive experiments.
Also noteworthy is the emerging popularity and importance of AutoML and readily available pre-trained AI/ML APIs, which will further drive cloud adoption in the data science field.