Part 1: Advanced Healthcare Data Analytics Platforms for Cloud (Strategies and Decisions)

Overview

The series ​“Advanced Healthcare Data Analytics for Cloud” has been created to share insights gained during our journey, aiming to assist others in navigating similar endeavors. The authors chose to publish several whitepapers due to the multitude of lessons learned throughout the course of this journey. Subsequent papers in this series will delve into specific areas such as Financial Operations (FinOps), Enterprise Architecture, Advanced Analytics Application(s), Privacy, and other targeted topics.

Over the past couple of years, a consortium of Maritime based companies collaborated to implement a cost-effective platform for advanced health analytics, emphasizing cost optimization, security, privacy, next generation tools and data residency. The platform provided an environment on which government, researchers, and educators gain access and build insights into healthcare. The platform was created to obtain from following a monolith architecture design to maximize flexibility in implementation.

Healthcare Technical Services and Capabilities

Getting Started

The inaugural paper will concentrate on fundamental strategies and decisions essential for establishing the foundational framework upon which to construct an analytics platform in the cloud. 

“How was the process of choosing a cloud provider undertaken, given the highly competitive market where each provider presents comparable services and capabilities?”

The challenge was selecting a cloud vendor capable of providing services and capabilities to accelerate our program for implementing our platform specific needs. An evaluation matrix was used to identify a provider that would expedite the platform’s delivery. While major cloud vendors offered similar infrastructure services, Microsoft Azure emerged as a leader in the AI/ML and BI space. Its standout position was attributed to existing integrated offerings and strategic roadmap. This decision underscores the significance of not only considering general infrastructure services but also focusing on a vendor’s strengths in specific areas crucial to an organization’s objectives.

Gartner has recognized Microsoft as a Magic Quadrant Leader in analytics and business intelligence platforms for the sixteenth consecutive year. — Forbes

“What considerations factored into your decision-making process regarding FinOps for cost optimization?”

FinOps merits its own dedicated whitepaper but as precursor to that paper here are some preliminary recommendations. The guidance emphasizes the importance of establishing the appropriate enterprise architecture, defining budgets, and implementing robust monitoring and reporting mechanisms. Using Azure, the platform provided budgeting tools, dashboards/​alerts, and a recommendation engine to effectively manage, control, and reduce costs.

Additional guidance underscores the importance of avoiding a direct lift-and-shift of legacy on-premises jobs to the cloud. It cannot be overstated that provisioning cloud compute resources to run continuously ²⁴⁄₇ is a significantly expensive decision. Through monitoring and capitalizing on the economy of scale, consumption-based costs are initially more economical until surpassed by a prepurchase dedicated infrastructure. Only through cloud native designs and architectures will effective operational costs be maintained – this has been proven again and again.

More to come on this topic!

The most common motivator for repatriation I’ve been seeing is cost. In the survey, more than 43% of IT leaders found that moving applications and data from on-premises to the cloud was more expensive than expected. – 2024 Infoworld​.com

“How was the strategy devised to oversee the management of all health data?”

Early in the decision-making process, it was determined to separate compute from storage, enabling the platform to leverage decoupled services for enhanced flexibility and cost efficiency. However, the extent of scalability in utilizing storage for both traditional warehouses and advanced analytics (AI/ML) processes was not initially evident. Amid the journey focusing on Lakehouse architecture, the choice settled on Azure Gen2 storage to support our Delta Lake implementation. Leveraging serverless SQL within Azure Synapse, we successfully constructed semantic views atop our data to satisfy warehouse needs. With data stored as delta tables, seamless integration with a diverse array of big data tools, including Spark, was possible. Our approach to data management centered on aligning with client preferences regarding how they wished to access the data. Fortuitously, our decision to embrace Delta Lake proved to be well-founded, given that Fabric/​One Lake has selected Delta as their foundational format.

More to come on this topic!

With Delta Lake it has everything needed to reduce costs, maintain data correctness, and keep customers happy – and the switch has made the data team’s job significantly easier. – 2023 Delta​.io

“In the context of Healthcare, where privacy is paramount, how was the implementation of privacy by design approached?”

An essential decision involved early assignment of a senior privacy resource during the development of services. Like conducting a threat risk assessment for security, a privacy risks framework was used to identify privacy issues. Working with the senior privacy resource, corresponding mitigation and control measures were implemented. Using Microsoft Purview as a starting point, a successful privacy framework was established that integrates privacy seamlessly into our design, data pipelines, and business processes.

Privacy is more than just technology it is a critical component to inform design! 

More to come on this topic!

Literature confirms that while consumers understand the benefits of health data sharing for research purposes, issues of trust, transparency, and privacy remain central to acceptance of health data sharing policies and programs in the general community. – 2020, BMC Research

“When considering the enterprise solution in the cloud, what were some of the best-practice decisions made and primary challenges encountered?”

In establishing the foundational Azure architecture to underpin the health analytics platform, we made some early decisions to address some key anticipated challenges, namely, 1) the need to create a flexible architecture that will evolve to meet the changing needs of the analytics platform, and 2) the need to create a policy-driven architecture to promote compliance with legislative, regulatory and security requirements.

To address the need for a flexible architecture, we adopted a modular and decoupled design approach, utilizing a combination of Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Serverless offerings to cater to varying analytics workloads. Further, we partitioned the platform architecture into functional segments through thoughtful use of Azure Management Groups and Subscriptions. This modular and de-coupled design allowed us to make local architectural changes, that did not significantly impact the broader architecture and design. We took full advantage of this decision part through the project when we changed our fundamental approach to data storage with little impact to the broader platform architecture and services. 

More to come on this topic!

In support of a policy-driven architecture, we implemented ​“compliance as code” and continuous compliance monitoring through Azure Policy’s features. We combined national, international, and custom policy definitions to address specific business, regulatory and security requirements, providing real-time visibility into the platform’s policy compliance status. Through integration with Azure Defender and Monitor, we were also able to establish automated reporting and remediation tasks to promptly address non-compliance issues. This comprehensive policy approach ensured our Azure enterprise solution remained both flexible and compliant, capable of adapting to evolving business needs while maintaining a robust governance framework.

More to come on this topic!

Ensuring compliance is a key security consideration in the cloud. Non-compliance with an enterprise’s business rules could result in non-compliance with industry and government regulations, leading to heavy penalties and fines. It also could make cloud resources more vulnerable to attacks. — The Register

Authors

Trevor Cook is a founding Principal with Atlantic Canadian-based Mara Consulting, and a Senior Enterprise, Cloud and Security Architect with over 30 years of experience in the IT industry, focused on large public sector and healthcare solutions.

Brian May serves as the Principal Architect, BI Architect and Data Scientist for Mariner, a Maritime company situated in Atlantic Canada. Leveraging his 30 years of expertise, he has successfully developed and used several big data analytical systems using industry architecture patterns for on-premises and cloud-based deployments.

In memory of Neil Foshay, who offered vision, strategy, guidance, and unwavering support, ensuring the successful implementation of the health analytics platform.