AWS: A parallel cluster creation example

AWS
1

A process of parallel cluster creation from its user guide [1] will be shown in this topic. The creation will be done by using AWS CLI.

The process includes following steps:

  • Python virtual environment setup and install AWS ParallelCluster module​
  • Install NVM​
  • AWS configure for AWS CLI operation​
  • Configure and create a cluster
  • Connect to the cluster head node and use​ the cluster
  • Delete the cluster​

Python virtual environment setup and install AWS ParallelCluster module​

$ python3 -m pip install --upgrade pip​
$ python3 -m pip install --user --upgrade virtualenv​
$ python3 -m virtualenv -p $(which python3) ~/apc-ve​
$ source ~/apc-ve/bin/activate​
(apc-ve)~$ python3 -m pip install --upgrade “aws-parallelcluster”

Install NVM​

$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash
$ chmod ug+x ~/.nvm/nvm.sh
$ source ~/.nvm/nvm.sh nvm install --lts

AWS configure for AWS CLI operation​

  • Find the AWS Access key id and Secrete Access key from IAM>Users>username>Access Keys. If you don’t have it already, create it and save for AWS CLI configuration​
  • Run the configuration: $aws configure and type the user access key information
    Screenshot from 2024-06-25 22-28-11
    Screenshot from 2024-06-25 22-28-111074×190 19.2 KB
    Screenshot from 2024-06-25 22-28-11100%75%50%

Screenshot from 2024-06-25 22-37-05Screenshot from 2024-06-25 22-37-05100%75%50%

Configure and create a cluster

  • Run the cluster configuration script and answer to choose the cluster options:​

$ pcluster configure --config mycluster.yaml​

  • After the intearcive configureation session following mycluster.yaml file will be created.​

Screenshot from 2024-06-25 23-14-38

  • Create the cluster instance with the configuration file​

$ pcluster create-cluster --cluster-name my-test-cluster --cluster-configuration mycluster.yaml ​

  • Check the cluster creation status with the following command. It takes some time the creation get completed.​

$ pcluster describe-cluster --cluster-name my-test-cluster

Connect to the cluster head node and use​ the cluster

  • Use following command to ssh to the head node of the cluster.​

$ pcluster ssh --cluster-name my-test-cluster -i ~/aws-key.pem ​

  • A use example with hello.c mpi code:​

Screenshot from 2024-06-25 23-27-57
Screenshot from 2024-06-25 23-27-57900×485 57.2 KB

Delete the cluster​

  • Use the following command to delete the cluster​
  • $ pcluster delete-cluster --cluster-name my-test-cluster​
  • Again check the deleting process using the “describe-cluster” command shown previously.

Reference
[1] AWS parallelcluster userguide: What is AWS ParallelCluster - AWS ParallelCluster