Amazon Web Services

I created this article in 2011, it was copied across to the wordpress wiki in the summer of 2013, and then amended in November 2013. I returned to it in Feb 2015, to document my adventures with wordpress. This article talks about Launching, Amazon Storage, console access, Fedora, Ubuntu, Hadoop, Essential Services, ddclient, zabbix, java, VNC, Links, and Futures in that order, with WordPress added in 2015. Some of the older sections are now both out of date and wrong.

Originally I needed to get an AMI up and running to host a previous version of this wiki. I remain most proficient using Ubuntu and it is significantly easier to install planet venus and redland rdf using Ubuntu’s package manager which were also required. I also experimented with Hadoop and VNC. In November 2013, I needed a WordPress host, but eventually went with 1and1. In 2015, my employer needed a wordpress host and so I returned to AWS.

Launching an Instance

Amazon have a good page, Getting Started Guide. This focuses on Launching an Instance. Launching your 1st instance allows you to obtain an ssh key and declare some firewall rules. Make sure that ssh is enabled and that you store the key. The user and instance name are available from the aws console.

Saving to persistent storage

Amazon have a documentation page, Where to go from here? which says,

One of the first tasks you’ll want to do after completing this guide is to create one or more custom AMIs. You can modify Amazon AMIs or public AMIs as needed (a process called re-bundling). For general information about AMIs, go to AMIs and to Preparing and Creating AMIs in the Amazon Elastic Compute Cloud User Guide. See also Bundling an AMI.You can choose between Amazon S3 or Amazon Elastic Block Store as the root device for your AMI (for a brief description of Amazon EBS, see Amazon Elastic Block Store later in this section). We recommend using instances backed by Amazon EBS, because they launch faster and use persistent storage. For more information, go to AMIs Backed by Amazon EBS in the Amazon Elastic Compute Cloud User Guide.

Console Access

I use putty or bash ssh. The oldest comment on this page documents what I originally wrote.

Experimenting with Hadoop

Single Node Cluster

Importing a VM

I plan to upload a single node hadoop cluster to AWS. I plan to locate it on EBS so it can survive a system halt. I originally planned to create it on a Virtual Box image, and upload the image. concluded that the process of creating an image from scratch when the distro authors have AMIs available is too high. One would need a good reason.  20th Sept 2011

Two Node Cluster

Michael Noll has written a tutorial for multi-node clusters, so I plan to make one of these.

The first thing is that the two nodes need to be able to talk to each other, this requires enabling in the firewall rules. It is not possible to change the ruleset name for an instance once created. It is possible to change the rules and so I have created a private AMI with a single node hadoop server configured with firewalls that permit ping, http and ssh. I later enhanced these rules to include ports 54310-54311 and other ports mentioned on the Cloudera Hadoop Ports Quick Reference page. If I want to monitor my AMIs with Zabbix, I will need to add the Zabbix ports to the firewall rules.

Any way I have distributed the master public key to slave node. I now need to consider how dynamic to make the name discovery. I could use dyndns. Probably best to just get it done.

Important Amazon Support

Essential Services

  1. SSH – don’t forget to allow it in the firewall rules
  2. DNS – I use DynDNS, whose home page is at http://www.dyndns.com/, who have a page called DNS Update Clients. I usedddclient on the late lamented Qube. It’s download page and a how to page are on the site, and some aspects of my Cobalt Installation might be on the Dell or even elsewhere on this site.
  3. apache – need to test that dyndns is working
  4. git – planet venus now uses git, see also my planet planet page
  5. scp/ftp – I need a file transfer program to up load stuff, it seems I have one

Installing ddclient

I use this to maintain the server’s dns name. My notes have been transferred to my ddclient page on this site. They include the magic runes needed to obtain the public IP address, which are documented on both my ddclient page and on the internet at Amazon EC2 – What You May Not Have Known a blog article at codesta.com. Ubuntu users installing from the tarball will need to install the PERL SSL libraries. The ddclient page was updated in November 2013 when I created a new AMI. The ubuntu install package has superb post install script. I looked for the runes again, and found these articles at stackoverflow.com.

Zabbix

I downloaded the source and created the zabbix agents having installed gcc.

  1. Create the User
  2. Create the Config directory, i.e. /etc/zabbix
  3. Copy the conf file to the directory
  4. Set the configuration options in the conf file to hold pid & log file in /var/tmp
  5. Reserve the ports by editing /etc/services
  6. Get my initialisation script from the Zabbix forums
  7. Ensure that all local configurations required are are accuratly stated in the script e.g. /usr/local/[create s]bin, /etc/zabbix and /var/tmp

I have not installed this in the /etc/init.d directories. I need to open the zabbix ports, and I need to sort my server out. (Its a bit crook at the moment!) See above, I’ll need a new instance, with new network firewall rules. The Zabbix ports, 10050 & 10051 need to be enabled.

Java

I need Java 5 for Snipsnap. This needs to be downloaded from the Oracle site.

I used wget to download the file, I renamed it, and moved it to /usr/lib/jvm and then ran the downloaded file. It seems that U10 doesn’t come with the Java 5 repositories enabled, and I couldn’t work out how to fix it. Java 6 requires one to enable the partner repositories insources.list.

Running a remote desktop using VNC

I started my build from http://cloud.ubuntu.com/ami/, for various reasons I( used a 32 bit image, based on Natty hosted in the EU. This is a server.

sudo tasksel install ubuntu-desktop
sudo apt-get install update
sudo apt-get install upgrade

I then referred to

which took me to tightvncserver, this is a wrapper for x11vnc and so I lookedup

I set the firewalls to 590[01] and downloaded http://www.uvxstartupnc.com/ to my desktop to act as my local client. I started a server, see below, I have enscripted this and needed to customise the xstartup file.

xstartup_sh

vncctl_sh

This works quite nicely, with just one problem. In my VNC sessions, the ‘D’ key is bound to ‘Hide All Windows’, this thread, tightvnc Keyboard Oddity, documents the solution, using ‘System -> Prefs -> Keyboard Short Cuts’. I ran into some dead ends while looking for a command line solution, and so to ensure that GNOME was fully upto date I ran,

sudo apt-get install gnome

tightvnc needs to initialise the user file system; I do not run it as root, but it does have a password function. I used ideas documented on

to start and stop the server. I have written a vncctl script and again inspired by the ideas on ubuntu’s vnc page I wrote an xstartup script.

Links

I originally used Second Brain to manage my AWS links, but they gave up and so I now have a bunch of links related to AWS. Here they are,

One day

  • Need to sort Rightscale vs Landscape, one day.
  • Need to understand the billable activity of the AMI, its more expensive than I expect
  • Get Zabbix working, fix ddclient’s syslog interface and write zabbix monitors for that and mingle
  • Get Snipsnap working.

Worpdress

How to install wordpress on ubuntu, prequisites, and configuration. It may be that wordpress can’t be installed on micro servers on EC2.

Related Posts

  • System V init scripts I really like Ubuntu's system V initialisation scripts and features. This is now based […]
  • Ubuntu 13.10 I am installing this on the Mac under Virtual Box. This snip is about Ubuntu. The […]
  • File Transfer I need to revise file transfer over TCP/IP. This article consists of links on […]
  • Centos 6.x I downloaded this from http://centos.mirroring.pulsant.co.uk/6.4/isos/x86_64/ The […]
  • Disqus This is page contains complex html i.e. javascript, the page must only be edited as […]

8 thoughts on “Amazon Web Services

  1. I am amending the page, I’ll remove the following content.

    I have used AWS Management Console and filtered on EBS storage and on the string 10.10. I have chosen to use ami-f46f5a80, which is an Ubuntu 10.10 Server i386. I created a new key pair. 12th Jan 2011. This is the old way to select a source image.

    Since I am using PUTTY, I need to convert the key to PUTTY format, using PUTTYGEN.

    The AWS instances disconnect from the network after periods of non-use. There is a flag in Putty,

    PuTTY Configuration > Categories > Connection > Seconds between keepalives

    Using a .ssh/config file and setting the following flags

    ServerAliveInterval 60
    ServerAliveCountMax 5

    works for bash ssh via cygwin on windows.

  2. I documented how I built EC2 instances for Fedora and Ubuntu. The library is now much tighter and the technique no longer works. This has been removed.

Leave a Reply

Your email address will not be published. Required fields are marked *