Amazon EC2 cloud setup

Initial Set Up

Following the instructions with Amazon documentation: Getting Started Guide

You have your account setup, and credit card submitted.

Going to use the Java toolset, lookup where java is installed on hgwdev, set environment:

JAVA_HOME=/usr/java/jre1.6.0_11

From your Amazon account management screen, save the cert-*.pem and pk-*.pem files and save in your home directory ~/.ec2/ directory, chmod 700 ~/.ec2/ for security, and set the files at chmod 600 to be read-only for yourself.

Set environment:

EC2_PRIVATE_KEY=$HOME/.ec2/pk-*.pem
EC2_CERT=$HOME/.ec2/cert-*.pem

Record your Amazon account number for later reference, it looks like: 1234-5678-9012

Pick up Java toolset from: Java Toolset

Unpack in some directory, and set environment:

EC2_HOME=/some/directory/ec2-api-tools-1.3-34128

Add $EC2_HOME/bin to your PATH

And to protect Java from the huge memory on hgwdev, set environment:

EC2_JVM_ARGS=-Xmx256M

Verify regions available with the toolset command:

$ ec2-describe-regions
REGION  eu-west-1       eu-west-1.ec2.amazonaws.com
REGION  us-east-1       us-east-1.ec2.amazonaws.com

Set environment:

EC2_URL=https://us-east-1.ec2.amazonaws.com

Create SSH keys:

$ cd $HOME/.ssh
$ ec2-add-keypair gsg-keypair > id_rsa.gsg-keypair
$ chmod 600 id_rsa-gsg-keypair

Sample AMI Images

Initial sample images are listed with:

$ ec2-describe-images -o self -o amazon > describe-images.txt

And, we want to use this one for a test first:

$ grep getting-started.manifest describe-images.txt 
IMAGE   ami-3c47a355    ec2-public-images/getting-started.manifest.xml ... etc...

Note the identification ami-3c47a355, used in this command:

$ ec2-run-instances ami-3c47a355 -k gsg-keypair
RESERVATION     r-cee26ea7      891631566226    default
INSTANCE        i-32a7c65b      ami-3c47a355                    pending gsg-keypair     0               m1.small        2009-04-02T22:47:17+0000        us-east-1c      aki-a71cf9ce    ari-a51cf9cc

Means it is running, Note the instance identification: i-32a7c65b

Use in this command to display the status of the instance:

$ ec2-describe-instances i-32a7c65b
RESERVATION     r-cee26ea7      891631566226    default
INSTANCE        i-32a7c65b      ami-3c47a355    ec2-72-44-35-57.compute-1.amazonaws.com domU-12-31-39-03-46-52.compute-1.internal       running gsg-keypair    0m1.small        2009-04-02T22:47:17+0000        us-east-1c      aki-a71cf9ce   ari-a51cf9cc

Note the name of the machine in that output: ec2-72-44-35-57.compute-1.amazonaws.com

Open SSH and HTTP ports:

$ ec2-authorize default -p 22
GROUP           default
PERMISSION              default ALLOWS  tcp     22      22      FROM    CIDR   0.0.0.0/0
$ ec2-authorize default -p 80
GROUP           default
PERMISSION              default ALLOWS  tcp     80      80      FROM    CIDR   0.0.0.0/0

You can now see the Apache WEB server at: http://ec2-72-44-35-57.compute-1.amazonaws.com/

To see SSH keys:

$ ec2-get-console-output i-32a7c65b
ec2: 2048 92:81:4f:19:e6:5b:23:e6:78:b7:e2:5d:be:ea:70:1d /etc/ssh/ssh_host_key.pub
ec2: 2048 20:9d:15:98:f6:71:fc:8d:4f:82:b6:e7:3f:76:e0:cc /etc/ssh/ssh_host_rsa_key.pub
ec2: 1024 4f:18:83:90:7b:11:be:c3:b4:c5:7a:a3:11:9d:26:76 /etc/ssh/ssh_host_dsa_key.pub

Login via ssh to this system:

$ ssh -i $HOME/.ssh/id_rsa.gsg-keypair root@ec2-72-44-35-57.compute-1.amazonaws.com

Verify the displayed RSH key is one of those from the console-output command:

RSA key fingerprint is 20:9d:15:98:f6:71:fc:8d:4f:82:b6:e7:3f:76:e0:cc.
Are you sure you want to continue connecting (yes/no)? yes

         __|  __|_  )  Fedora 8
         _|  (     /    32-bit
        ___|\___|___|

 Welcome to an EC2 Public Image
                       :-)

    Getting Started

 --[ see /etc/ec2/release-notes ]--

[root@domU-12-31-39-03-46-52 ~]# 

There is your login prompt. You are now a Linux system administrator.

Mounting snapshots

Creating a snapshot based on the Ensembl image, identification found at: Public Datasets

$ ec2-create-volume --snapshot snap-c78360ae -z us-east-1b
VOLUME  vol-399d7d50    550     snap-c78360ae   us-east-1b      creating       2009-04-03T17:19:02+0000

Attach that volume to a running image, as device /dev/sdh:

$ ec2-attach-volume vol-399d7d50 -i i-4bc1a022 -d /dev/sdh
ATTACHMENT      vol-399d7d50    i-4bc1a022      /dev/sdh        attaching      2009-04-03T17:25:04+0000

Taking a look at those:

$ ec2-describe-volumes
VOLUME  vol-399d7d50    550     snap-c78360ae   us-east-1b      in-use  2009-04-03T17:19:02+0000
ATTACHMENT      vol-399d7d50    i-4bc1a022      /dev/sdh        attached       2009-04-03T17:25:04+0000

Examining instances:

$ ec2-describe-instances | grep running
INSTANCE        i-4bc1a022      ami-3c47a355    ec2-75-101-203-203.compute-1.amazonaws.com      ip-10-251-126-22.ec2.internal   running gsg-keypair     0      m1.small 2009-04-03T17:03:17+0000        us-east-1b      aki-a71cf9ce    ari-a51cf9cc

Logged into the instance, mounting that volume:

# mkdir /mnt/ensembl
# mount /dev/sdh /mnt/ensembl
# cd /mnt/ensembl
# df -h .
/dev/sdh              546G  535G   11G  99% /mnt/ensembl

After it was mounted:

$ ec2-describe-volumes
VOLUME  vol-b49d7ddd    550     snap-c78360ae   us-east-1c      in-use  2009-04-03T17:57:54+0000

ATTACHMENT vol-b49d7ddd i-bac4a5d3 /dev/sdh attached 2009-04-03T17:59:17+0000

Creating a snapshot

Create a 1000 Gb volume:

$ ec2-create-volume -s 1000 -z us-east-1c
VOLUME  vol-5e9e7e37    1000            us-east-1c      creating        2009-04-03T18:09:45+0000

Attach that volume to a running image, as device /dev/sdi:

$ ec2-attach-volume vol-5e9e7e37 -i i-bac4a5d3 -d /dev/sdi

Make a new filesystem on this volume, logged into the instance:

# yes | mkfs -t ext3 /dev/sdi
This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

That took about 15 minutes, 8000 inode tables was the longest bit of the process. Create a mount point and mount it:

# mkdir /mnt/ucsc
# mount /dev/sdi /mnt/ucsc

Load your data into that filesystem. It is going into the S3 storage system as that takes place. When it is complete, create a snapshot:

$ ec2-create-snapshot vol-5e9e7e3
SNAPSHOT        snap-a8a352c1   vol-5e9e7e37    pending 2009-04-03T23:48:55+0000

Turning Off an Instance

You need to make sure you turn these instances off, they consume accounting charges if they are left on:

$  ec2-terminate-instances i-32a7c65b
INSTANCE        i-32a7c65b      running shutting-down