I created this article in 2011, it was copied across to the wordpress wiki in the summer of 2013, and then amended in November 2013. I returned to it in Feb 2015, to document my adventures with wordpress. This article talks about Launching, Amazon Storage, console access, Fedora, Ubuntu, Hadoop, Essential Services, ddclient, zabbix, java, VNC, Links, and Futures in that order, with WordPress added in 2015. Some of the older sections are now both out of date and wrong.
Originally I needed to get an AMI up and running to host a previous version of this wiki. I remain most proficient using Ubuntu and it is significantly easier to install planet venus and redland rdf using Ubuntu’s package manager which were also required. I also experimented with Hadoop and VNC. In November 2013, I needed a WordPress host, but eventually went with 1and1. In 2015, my employer needed a wordpress host and so I returned to AWS.
Launching an Instance
Amazon have a good page, Getting Started Guide. This focuses on Launching an Instance. Launching your 1st instance allows you to obtain an ssh key and declare some firewall rules. Make sure that ssh is enabled and that you store the key. The user and instance name are available from the aws console.
Saving to persistent storage
Amazon have a documentation page, Where to go from here? which says,
One of the first tasks you’ll want to do after completing this guide is to create one or more custom AMIs. You can modify Amazon AMIs or public AMIs as needed (a process called re-bundling). For general information about AMIs, go to AMIs and to Preparing and Creating AMIs in the Amazon Elastic Compute Cloud User Guide. See also Bundling an AMI.You can choose between Amazon S3 or Amazon Elastic Block Store as the root device for your AMI (for a brief description of Amazon EBS, see Amazon Elastic Block Store later in this section). We recommend using instances backed by Amazon EBS, because they launch faster and use persistent storage. For more information, go to AMIs Backed by Amazon EBS in the Amazon Elastic Compute Cloud User Guide.
Experimenting with Hadoop
Single Node Cluster
Importing a VM
I plan to upload a single node hadoop cluster to AWS. I plan to locate it on EBS so it can survive a system halt. I originally planned to create it on a Virtual Box image, and upload the image. I concluded that the process of creating an image from scratch when the distro authors have AMIs available is too high. One would need a good reason. 20th Sept 2011
Two Node Cluster
Michael Noll has written a tutorial for multi-node clusters, so I plan to make one of these.
The first thing is that the two nodes need to be able to talk to each other, this requires enabling in the firewall rules. It is not possible to change the ruleset name for an instance once created. It is possible to change the rules and so I have created a private AMI with a single node hadoop server configured with firewalls that permit ping, http and ssh. I later enhanced these rules to include ports 54310-54311 and other ports mentioned on the Cloudera Hadoop Ports Quick Reference page. If I want to monitor my AMIs with Zabbix, I will need to add the Zabbix ports to the firewall rules.
Any way I have distributed the master public key to slave node. I now need to consider how dynamic to make the name discovery. I could use dyndns. Probably best to just get it done.
Important Amazon Support
- SSH – don’t forget to allow it in the firewall rules
- DNS – I use DynDNS, whose home page is at http://www.dyndns.com/, who have a page called DNS Update Clients. I usedddclient on the late lamented Qube. It’s download page and a how to page are on the site, and some aspects of my Cobalt Installation might be on the Dell or even elsewhere on this site.
- apache – need to test that dyndns is working
- git – planet venus now uses git, see also my planet planet page
- scp/ftp – I need a file transfer program to up load stuff, it seems I have one
I use this to maintain the server’s dns name. My notes have been transferred to my ddclient page on this site. They include the magic runes needed to obtain the public IP address, which are documented on both my ddclient page and on the internet at Amazon EC2 – What You May Not Have Known a blog article at codesta.com. Ubuntu users installing from the tarball will need to install the PERL SSL libraries. The ddclient page was updated in November 2013 when I created a new AMI. The ubuntu install package has superb post install script. I looked for the runes again, and found these articles at stackoverflow.com.
I downloaded the source and created the zabbix agents having installed gcc.
- Create the User
- Create the Config directory, i.e. /etc/zabbix
- Copy the conf file to the directory
- Set the configuration options in the conf file to hold pid & log file in /var/tmp
- Reserve the ports by editing /etc/services
- Get my initialisation script from the Zabbix forums
- Ensure that all local configurations required are are accuratly stated in the script e.g. /usr/local/[create s]bin, /etc/zabbix and /var/tmp
I have not installed this in the /etc/init.d directories. I need to open the zabbix ports, and I need to sort my server out. (Its a bit crook at the moment!) See above, I’ll need a new instance, with new network firewall rules. The Zabbix ports, 10050 & 10051 need to be enabled.
I need Java 5 for Snipsnap. This needs to be downloaded from the Oracle site.
- Oracle’s Java 5 download page.
I used wget to download the file, I renamed it, and moved it to /usr/lib/jvm and then ran the downloaded file. It seems that U10 doesn’t come with the Java 5 repositories enabled, and I couldn’t work out how to fix it. Java 6 requires one to enable the partner repositories insources.list.
Running a remote desktop using VNC
I started my build from http://cloud.ubuntu.com/ami/, for various reasons I( used a 32 bit image, based on Natty hosted in the EU. This is a server.
sudo tasksel install ubuntu-desktop sudo apt-get install update sudo apt-get install upgrade
I then referred to
which took me to tightvncserver, this is a wrapper for x11vnc and so I lookedup
I set the firewalls to 590 and downloaded http://www.uvxstartupnc.com/ to my desktop to act as my local client. I started a server, see below, I have enscripted this and needed to customise the xstartup file.
This works quite nicely, with just one problem. In my VNC sessions, the ‘D’ key is bound to ‘Hide All Windows’, this thread, tightvnc Keyboard Oddity, documents the solution, using ‘System -> Prefs -> Keyboard Short Cuts’. I ran into some dead ends while looking for a command line solution, and so to ensure that GNOME was fully upto date I ran,
sudo apt-get install gnome
tightvnc needs to initialise the user file system; I do not run it as root, but it does have a password function. I used ideas documented on
to start and stop the server. I have written a vncctl script and again inspired by the ideas on ubuntu’s vnc page I wrote an xstartup script.
I originally used Second Brain to manage my AWS links, but they gave up and so I now have a bunch of links related to AWS. Here they are,
- Need to sort Rightscale vs Landscape, one day.
- Need to understand the billable activity of the AMI, its more expensive than I expect
- Get Zabbix working, fix ddclient’s syslog interface and write zabbix monitors for that and mingle
Get Snipsnap working.