HPC — High Performance Cluster

Solution architecture: HPC cluster deployed in the cloud
High performance computing (HPC) applications can scale to thousands of compute cores, extend on-premises big compute, or run as a 100% cloud native solution. This HPC solution including the head node, compute nodes, and storage nodes, runs in AWS or Azure with no hardware infrastructure to maintain.

This solution is built on the Azure managed services or AWS Console: Virtual Machine Scale Sets, Virtual Network and Storage. These services run in a high-availability environment, patched and supported, allowing you to focus on your solution instead of the environment they run in.

HPC head node(jumpbox)

• The HPC head node(jumpbox) runs on-premises or within Azure as a virtual machine.
• Virtual networking available with all instances is used to achieve high bandwidth and microsecond latencies between compute nodes.
  Storage nodes can also be run within virtual machines.

Virtual Machine Scale Sets

• Availability sets ensure that the application is available and resilient to updates and hardware faults.
• Virtual machines communicating via Virtual network(NSG — network security group) are placed within the same availability set.

Virtual Network

• Virtual Network provides IP connectivity between the head node, compute nodes, and storage nodes.

Storage

• Storage blobs(or shared disks storage) store the disks backing the virtual machines and provides long-term storage of unstructured data and executable files used by the HPC application.
• can be network-sharing services (NFS), storage account manager services, or shared virtual disks in the VMs between the containers.

CI/CD Proccess

• The CI/CD(Continuous Integration/Continuous Deployment) has a constructs one of the most important parts of the application life cycle.
• We build important rules during the design, design and development of the apps that are implemented through cicd pipelines.
• For each Client there are two different pipelines, develop and master.
• Pipeline develop builds all the rules necessary to move up test and local development automation environments or in test VMs in the cloud.
• Master pipeline covers rules that involve approves of other team members in the demands created through build, cluster configurations, and all-in-one VMs, and business rules for various deployments in production environments.

Monitoring & Instrumentation

• With performance management quickly identify if there are any problems, how many users are impacted, and perform a risk analysis to find and fix the problem.
• We get effective performance monitoring, alerts, and easy-to-use dashboards to help ensure applications are available and performing as expected.
• Seamless integration with your DevOps pipeline using Team Services, Git, AWS and our webhooks.
• Detect and diagnose application performance exceptions and issues.
• Detect Azure Cloud Services behaviors, anomalies, failure counts, and performance changes. Receive notifications and alerts with rich diagnostic information to enable you to respond to problems at the speed desired by your customers.
• With distributed instrumentation in the cloud (AWS, Azure, Elasticsearch, Logstash, Kibana, App Insights) we detect cloud service behaviors, anomalies, failure counts, and performance changes. Receive notifications and alerts with rich diagnostic information to enable you to respond to problems at the speed desired by your customers. Integrated with SLACK 😃