Data architects need to navigate several challenges when deploying a cloud data architecture, including technical issues like data gravity, political issues such as existing investments, and process issues like security and incomplete data migration.
Data gravity
Data gravity, is by far the most common challenge organizations face as they continue to solve for hybrid or multi-cloud environments. Data gravity issues can arise when applications span multiple environments and when ingesting, transforming and analyzing massive data sets from disparate sources.
These challenges can be overcome by separating data storage, utilizing event-driven architectures, analyzing data at the edge and scaling public cloud compute for batch processes and large-scale analysis.
On-premises investments
Data architects may need to contend with pushback from management and its desire to maximize corporate investments in on-premises processes. The infrastructure necessary to support a sizable on-premises data architecture requires significant capital investment.
One way to mitigate this hurdle is to identify bursty use cases that are impractical for the existing on-premises architecture.
Data privacy and security
Data security is one of the foremost challenges, and this can be addressed by controlling access to the data whether in transit or at rest, ingesting data from a reliable external source and validating data at the point of entry.
A cloud data architecture can take advantage of hardened services to improve security, but that requires upfront work to automate data handoff processes. Consider data governance to address privacy and security issues by classifying data, creating controls and policies, and managing data lifecycles through catalogs and tracking.
Regulatory and compliance obligations
To comply with data quality and protection policies and regulations, on-premises hosting of data may sometimes be favored over the cloud. To counter this resistance, ensure that stakeholders are made aware of successful use case studies and liability protections as a result of cloud data architectures.
Incomplete cloud migration
Companies can get cold feet in the middle of their migration to the cloud, and accidentally use a combination of cloud and on-premises systems. Cloud computing works best when a company totally migrates its data and applications to the cloud.