Travis Hegner | Ramblings of an I.T. Guy tag:www.travishegner.com,2009-05-01://2 2010-05-05T20:21:22Z A conglomeration of thoughts, projects, tutorials, and whatever else I feel like writing. Movable Type 4.25 For Sale: 2004 Suzuki GSX-R600 tag:www.travishegner.com,2010://2.26 2010-04-12T00:46:13Z 2010-05-05T20:21:22Z I am selling my motorcycle because I don't really ride it much anymore, and the day the hood flew off of the mustang I decided it was time for a new (to me) car. If you are viewing this from outside of driving distance, then be aware that I will not be making any arrangements to ship/move or otherwise relocate the motorcycle. That will be completely up to the buyer. The Specs: 2004 Suzuki GSX-R 600 7600 Miles New back tire (less than 1k miles) $5500 obo $5000 obo The Mods: Yoshimura exhaust Passenger Seat Rear fender Eliminator kit... Travis Hegner http://www.travishegner.com/about-me.html I am selling my motorcycle because I don't really ride it much anymore, and the day the hood flew off of the mustang I decided it was time for a new (to me) car. If you are viewing this from outside of driving distance, then be aware that I will not be making any arrangements to ship/move or otherwise relocate the motorcycle. That will be completely up to the buyer.

The Specs:
2004 Suzuki GSX-R 600
7600 Miles
New back tire (less than 1k miles)
$5500 obo
$5000 obo

The Mods:
Yoshimura exhaust
Passenger Seat
Rear fender Eliminator kit
LED Tail-light, with integrated blinkers
Flush-mount LED front blinkers
Woodcraft CFM Frame Sliders

The Problems:
The bike was laid down before I owned it. Most damage was cosmetic, except for the shift lever, foot peg, and clutch handle. All of those parts were replace except for the stator cover and the left side plastic. I've been riding the bike for 5 years as is and haven't had any issues with it. The frame is not bent, and the bike still rides like new. All parts were replaced/installed after I purchased the bike, and I have never laid it down. The bike does need a new battery. Since it has been sitting the battery has gone kaput.

The Included stuff:
A woodcraft CFM rear jack stand.
Original stock exhaust pipe, still in excellent condition.
Woodcraft CFM Aluminum Starter Cover, never installed.

I'll probably be getting a new battery for it this week, so the buyer doesn't have to worry about it.

The Pics:




















]]> Hbase 0.20: Low Memory Performance Tuning tag:www.travishegner.com,2009://2.23 2009-11-23T15:35:33Z 2010-02-25T21:39:51Z Hey Everybody! As promised, though a little later than I'd originally hoped, here is a quick blurb on what I've done to make my fully distributed Hbase instance run with a little bit more stability. hbase.hregion.max.filesize &#09134217728 Default: 268435456 This setting is the maximum file size of a region in any given Hbase table. Setting this to a lower value will inevitably increase the number of regions served, but it seems to help prevent time-outs when dealing with desktop grade hardware. It's a little easier for the system to move around and work with slightly smaller files. hbase.hstore.compactionThreshold &#092... Travis Hegner http://www.travishegner.com/about-me.html Hey Everybody!

As promised, though a little later than I'd originally hoped, here is a quick blurb on what I've done to make my fully distributed Hbase instance run with a little bit more stability. 


	hbase.hregion.max.filesize
	134217728
	Default: 268435456

This setting is the maximum file size of a region in any given Hbase table. Setting this to a lower value will inevitably increase the number of regions served, but it seems to help prevent time-outs when dealing with desktop grade hardware. It's a little easier for the system to move around and work with slightly smaller files. 


	hbase.hstore.compactionThreshold
	2
	Default: 3

This setting controls how often a mem store file will be compacted into a full region file. The mem store file is created when a mem store is flushed to disk. Setting this to a lower value, increases the number of compactions you have, but it decreases the amount of time it takes to compact. During my testing, my clients were timing out during compactions, so this setting helped to prevent that. 


	hbase.hregion.memstore.flush.size
	33554432
	Default: 67108864

This is the mem store mentioned earlier. Once the store reaches the configured size, it is flushed to disk. Setting this value lower gives you your biggest bang for saving memory, as it forces data to disk sooner, and uses less memory overall.

Please experiment with these settings on your own accord. Your results may differ, but for me, these settings made the difference that allowed my very large import to complete without a single failure.

]]> Ubuntu 9.10 Karmic Koala Coming Soon! tag:www.travishegner.com,2009://2.22 2009-10-02T13:57:05Z 2009-10-02T15:28:43Z I am pleased to report that the 9.10 release of Ubuntu is available for Beta testing. I have had the Alpha running on a test machine for several weeks now with daily updates applied. My First impressions of the release are pleasing, but I haven't done too much with it yet. The first thing you'll notice when starting your new 9.10 machine is that the new kernel now seems to have some major video improvements. The screen resolution is automatically detected on my test machine, and the native resolution is applied even before the splash screen. If you use... Travis Hegner http://www.travishegner.com/about-me.html
I am pleased to report that the 9.10 release of Ubuntu is available for Beta testing. I have had the Alpha running on a test machine for several weeks now with daily updates applied. My First impressions of the release are pleasing, but I haven't done too much with it yet. The first thing you'll notice when starting your new 9.10 machine is that the new kernel now seems to have some major video improvements. The screen resolution is automatically detected on my test machine, and the native resolution is applied even before the splash screen. If you use the virtual terminals (ctrl-alt-f1 through f6), then this is a awsome benefit, as the virtual terminals now run in your monitors native resolution, and switching between them is lightning fast.

The next thing you'll notice is the new splash screen in 9.10. This new piece of work is nothing short of awsome looking. The default user selection/logon screen is a bit boring though. First, I don't like the "user picker" style login screens. For an attempting intruder, half the battle of gaining access is done for you, you don't have to guess the username. It's also not quite as fancey as the new splash screen we just drooled over. I very much prefered the 9.04 default login screen, perhaps they'll change it before the final release. A quick glance at System->Administration->Login Screen tells me that they've taken away the ability to change the default login screen. This is dissapointing for the reasons I stated above, and for those that like to completely customize their system.

The third observation you'll have is the default color scheme. The ubuntu team has finally let go of the orange title bars in gnome which have served so well for the last... well... since I started using ubuntu anyway. The have moved to a much darker brown, which is fine with me, I prefer most dark color schemes anyway. This is a very customizeable option anyway, and if you don't like it, it's very easy to change under System->Preferences->Appearance. For those that really want to spruce up ubuntu, or any gnome based distribution, have a look at http://www.gnome-look.org/.

For those of us that use ubuntu in a corporate environment which runs Microsoft Exchange 2007, I have some good news, and some bad news. The long awaited, and long sought after, evolution-mapi plugin is installable, and you can create and authenticate to your exchange mailbox with the native mapi library. And the bad news? It (still) doesn't seem production ready! I have my account configured and was able to use it for a few minutes the first time, but when loading certain messages, evolution crashes (completely disappears from the screen) and you lose anything not saved. Worse yet, when I open evolution now, it crashes (disappears) within 5 to 10 seconds, rendering the app useless until I delete my ~/.evolution folder to delete my configuration. It's so close, yet so far away!

One thing I don't understand is why someone hasn't tackled this evolution to exchange problem using EAS or Exchange Active-Sync. Exchange has a web service that runs, which is designed for mobile applications, and any windows based mobile phone can link seemlessly and download any and all emails, tasks, and calander apppointments. Even Google's android, and the new Palm Pre can utilize this service to accomplish the same thing opensource users have been asking for years. Exchange mailbox access, from a non microsoft client. I realize that the actual API is probably licensed, and costs money to use, but I know there are some brilliant people out their who can design their own API, open source of course, that sends requests and responses to the EAS service the same way the microsoft licensed API would. It's not much different than mimicing MAPI the way the new mapi libraries do, you'll just be mimicing EAS instead. Oh well, maybe one day it will work.

That is about as far as I've taken my 9.10 box for a test drive. Only around the block. When the 9.10 final is released, I will be installing it on my production workstation, and reporting on all the goodies that I've been waiting to be fixed... Like nvidia support for xrandr, so I can get rid of the deprecated xinerama.

Until Next Time!
]]> Adding Hbase 0.20 to Our Hadoop Cluster tag:www.travishegner.com,2009://2.21 2009-08-13T03:17:11Z 2009-11-23T16:27:53Z OK, OK, I know what you're thinking. "Where has this guy been with some more of this hadoop goodness that we just can't get enough of?" Well, I am (finally) back, and you'll be happy to learn what I brought with me: A whole mess more Hadoop and Hbase knowledge that I can't wait to show you!Nevertheless, the time has finally come. The brilliant developers over at Hbase have made a release candidate for version 0.20 available for download. I believe that the changes have slowed enough to document the installation procedures a bit. The link is for RC1, but... Travis Hegner http://www.travishegner.com/about-me.html
Nevertheless, the time has finally come. The brilliant developers over at Hbase have made a release candidate for version 0.20 available for download. I believe that the changes have slowed enough to document the installation procedures a bit. The link is for RC1, but be on the lookout for RC2, which should be available very soon. I will document the procedures for both downloading the pre-built package, and installing from svn.

First, I am going to recommend you follow my hadoop tutorial, because the power of Hbase only really begins to shine when you have it installed fully distributed on a cluster.

Please make a note that in order to get Hbase running with some real stability, I was forced to upgrade my cluster a bit. First, I moved my master onto a virtual machine, so that it was off of the "workhorse" nodes, and on a more redundant set of "hardware". Effectively, this gave me another slave node to do some work, so now I'm up to a 7 node cluster! I also upgraded the ram on each node to 1GB. Still low, but much more stable since doing so.

Log into your master node, and navigate to the /hadoop directory:

cd /hadoop
Make sure you have subversion and ant installed:

sudo apt-get install subversion ant
Check out the code from the apache subversion repository:

svn co http://svn.apache.org/repos/asf/hadoop/hbase/branches/0.20 hbase-svn
This will always be the latest and greatest code for the 0.20 branch. (The trunk is now working towards 0.21). If you would like a "release" that is not under development, then have a look at http://svn.apache.org/repos/asf/hadoop/hbase/tags/ and you can pick from previously released "snapshots" of the code. Just adjust the "svn co" command accordingly. If you have downloaded a pre-built package, then simply extract the contents to the /hadoop/hbase-<version>/ directory. Assuming all has gone well, we are ready to configure our hbase install. Navigate to the hbase/conf directory that you are currently using:

cd /hadoop/hbase-svn/conf
Add the following line to your hbase-env.sh file:

export JAVA_HOME=/usr/lib/jvm/java-6-sun
In the same file change this line:

export HBASE_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
to look like:

export HBASE_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true"
Now, modify your "regionservers" file to list all of the machines you want to host regions. Think of an Hbase region as a small chunk of the data in your database. The more regionservers you have, the more data you can reliably serve. In my cluster, the regionservers are the same nodes as all of my datanodes, and all of my tasktrackers. So, essentially, the "regionservers" file should be identical to your "slaves" file from the hadoop tutorial.

Next, modify the hbase-site.xml file. The settings in this file over-write those in hbase-default.xml, so if you want to see a list of available settings to configure, then study that file, but only make changes to your hbase-site.xml. Add the following settings to hbase-site.xml:

<property>
    <name>hbase.rootdir</name>
    <value>hdfs://$master$/hbase</value>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>$slave1$,$slave2$,$slave3$</value>
</property>
<property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/hadoop/zookeeper/data</value>
</property>
Please remember to replace $master$ and $slaveX$ with your master and slave host names respectively. You may have read that Hbase 0.20 now requires zookeeper, but fear not, the above configuration directives allow hbase to completely manage zookeper on it's own, you never have to mess with it. Now, it is typically recommended to always run zookeeper on dedicated zookeeper only servers. If you are running a small cluster, then this is hardly efficient, because you want as many nodes "working" as possible. While I can't give you recommendations of the maximum cluster size you can have before requiring dedicated zk nodes, I can tell you that my 6 slave nodes run datanode, tasktracker, regionserver, and zookeeper without too much of a problem. I would imagine that if you have over 10 nodes in your cluster, then you shouldn't have a problem dedicating a few for zookeeper. They also recommend (maybe even require) that zookeeper runs on an odd number of machines. I don't completely understand how zookeeper works, but basically as long as you still have more than half of your "quorum" in tact, then your cluster won't fail. In essence, if your zk quorum has 7 nodes, you can lose 3 nodes without any adverse affects, a 35 node quorum could theoretically lose 17 nodes, and still operate. I think basically zookeeper is used to keep track of the locations of regions, so your quorum will notify any clients, and fellow regionservers where to find the data they are looking for. If zk becomes overloaded, then your regionservers can time out and crash, and potentially lose data if they haven't flushed to disk yet. So make sure you have enough horsepower for your application. In my cluster, the hbase.zookeeper.quorum directive is simply a comma separated list of all of my slave nodes, including my master. If you have an odd number of slaves (even number counting your master), then just leave the master out of the list. If you have more than ten slaves, then consider dedicating 3 of them to zookeeper if you have problems with regionservers timing out. The logs will tell you if that is the case.

Now, if you are building from svn, then run the following command from your hbase-svn directory:

ant
This command is like "make", but for java. It will compile the java code into .class binary files, .jar files, and even copy our config to hbase-svn/build. This new build directory is basically the same as a pre-built downloaded package, so I always link to it from /hadoop/hbase:

cd /hadoop && ln -s hbase-svn/build hbase
If you are not building from svn, then simply link /hadoop/hbase to your hbase-<version> directory:

cd /hadoop && ln -s hbase-<version> hbase
Simply remember to replace hbase-<version> with your actual directory name. Please also remember that only hbase-0.20.X will run on our hadoop cluster.

Next we have copy our hbase install to all of the nodes that will be running a regionserver. I wrote a handy script to do this for me, that way as I build, and rebuild hbase, I can simply run my 'deploy-hbase.sh' script to spread it around my cluster:

#!/bin/bash

rsync -r --progress --delete hbase-svn slave2:/hadoop/
rsync -r --progress --delete hbase-svn slave3:/hadoop/
rsync -r --progress --delete hbase-svn slave4:/hadoop/
rsync -r --progress --delete hbase-svn slave5:/hadoop/
rsync -r --progress --delete hbase-svn slave6:/hadoop/
rsync -r --progress --delete hbase-svn slave7:/hadoop/
Add or remove lines to correspond to your slaves, and put the proper names/directories in as necessary. After hbase has been copied to each of the slaves, we must log into each slave for some final steps. Every node in a cluster has to have the code-base installed in the same location so we have to link each of our nodes /hadoop/hbase directory to the same location that we linked to on the master. For each of our zk nodes, we must also create a data directory, and id file:

mkdir -p /hadoop/zookeeper/data && echo 'X' > /hadoop/zookeeper/data/myid
It is imperative that you replace the 'X' with '0', on the first node in your quorum, '1' on the second, '2' on the third and so on. This file allows the node to identify itself in the zk quorum.

Once all that per node work is done, you can finally start your hbase instance. From the master /hadoop/hbase directory run:

bin/start-hbase.sh
If all goes well, you can navigate your browser to http://<master>:60010/ to see some stats about the running hbase and zk instances.

Keep checking back, as there is plenty more to come:
* Tuning Hadoop/Hbase for low memory clusters (like mine, truly commodity hardware!)
* A full map/reduce example job utilizing hbase as a source and destination for data, written natively in java (a first for me).
* A document search algorithm utilizing your map/reduce output to find exactly what we are looking for, with ranking included.

Thanks for reading, come back soon!
]]> Hadoop 0.20 on Ubuntu Server 9.04 Jaunty tag:www.travishegner.com,2009://2.20 2009-06-29T16:08:31Z 2009-11-23T16:29:08Z In the last few weeks, I have become completely fascinated by clustered computing, and solving very large complex problems with a somewhat large number of cheap computers. While researching more efficient ways to store and search through my employers internal resume database, I stumbled onto this software called Hbase, which is a column-oriented database, modeled after Google BigTable. To increase speed and reliability, Hbase runs best on a distributed file system provided by the Hadoop Core project (HDFS).Documentation is plentiful for older versions of Hadoop, installed on older versions of Ubuntu, but there were significant changes in both Hadoop 0.20... Travis Hegner http://www.travishegner.com/about-me.html Hbase, which is a column-oriented database, modeled after Google BigTable. To increase speed and reliability, Hbase runs best on a distributed file system provided by the Hadoop Core project (HDFS).

Documentation is plentiful for older versions of Hadoop, installed on older versions of Ubuntu, but there were significant changes in both Hadoop 0.20 and Ubuntu Server 9.04 that created a few stumbling points for my test installations. That being said, this tutorial will be on how to install a hadoop 0.20 cluster on any number of Ubuntu Server Jaunty 9.04 machines.

In light of the current economic times, we have quite a few extra desktops sitting around here at work, so I took the liberty of installing ubuntu server on a few (6) of them to build a test cluster. I relied heavily on the Hadoop online documentation as well as the tutorials by Michael G. Noll on single-node clusters and multi-node clusters. The only problem I have with my cluster at this point is not having enough data to process with it!

Before you begin, make sure you have a dedicated switch for you cluster, an ubuntu server 9.04 install cd, and all the machines ready for an OS that you want to put into your cluster. I highly recommend running static IP's on all the machines in your cluster, as well as an internal authoritative DNS server. You could even install BIND on the "Namenode" for a dedicated solution. If not, you'll have to manually keep every hosts file up to date on every node. I've found that Hadoop can be very finicky about DNS name resolution, so you have to be careful about what's in your hosts file, as well as your nsswitch.conf file. You definately don't want your machine to grab a dhcp address at any point in time during the install, as it will write your hosts file and potentially your resolv.conf file differently than you need it. The minimum requirements for you cluster machines is really dependant on the problem you want to solve, or the amount of data you want to analyze. Each node in my cluster has about 512 MB of RAM, 30-80 GB of hard disk space, and a 3.0 GHz HT P4 processor. I am also only running on a 10/100 Mb switch. If you want to do something serious, then you may want to look at some better hardware, and definately a 1Gb switch, but if your just fooling around, your dusty 686's, and a simple 10/100 Mb switch should do the trick. Your nodes will need internet access, so be sure that is available on your cluster network as well.

First, you'll want to install ubuntu server on each of your nodes. I usually just set the user during the install to hadoop, to avoid having to create this user later. Make sure after the install you run a dist-upgrade,

sudo apt-get update
sudo apt-get dist-upgrade
and then install sun-java6-jdk.

sudo apt-get install sun-java6-jdk
Next, you'll have to select a single machine to be your "NameNode" and your "JobTracker". This is basically the Master node in the cluster, and will do a little bit more work than the rest of the nodes, so choose your most powerful machine. If you are building a large cluster for a production system, you'll want your NameNode and JobTracker to be on separate, dedicated boxes, and You'll also want to make sure you have a "SecondaryNameNode" for redundancy. Since my cluster is small, and only for experimentation, I only have a single master node and all of the master processes combined on to it. It even runs the slave processes as well to help store the data, and spread the work load around.

The next step is to setup ssh key authentication from the master to each slave node. It is imperitive that the following commands be executed as the hadoop user on the master node.

ssh-keygen -t rsa -P ""
ssh-copy-id hadoop@<master-node>
ssh-copy-id hadoop@<slave-node>
... (repeat for each slave) ...
This will create a key for your hadoop user on your master node, and give that user password-less access to itself and each of the slaves. While copying the keys it will ask to store the fingerprint, answer yes each time, so that it doesn't interfere with the master's commands later. Hadoop uses ssh to start and stop the slave processes. Repeat these steps on any other "master" machine if you have opted to have a secondary name node and/or separate job tracker machines. You can get a bit more advanced and find the same key on the first master, and distribute it to all the other masters, instead of generating a new key for each master, but you'll still have to connect to each slave and each other master at least once manually to store the fingerprint.

Download hadoop core. I typically create a directory on the root called hadoop,

sudo mkdir /hadoop
and expand the tar.gz we just downloaded into it. Then, change the permissions on the directory to be owned by our hadoop user and group:

sudo chown -R hadoop:hadoop /hadoop
I usually then sym-link "core" to the "hadoop-0.20.0" directory, so it's a little easier to reference.

ln -s /hadoop/hadoop-0.20.0 /hadoop/core
Remember that all of your nodes will have to have the same version of hadoop installed, and in the same location on each node, so you'll be repeating these steps for every node in the cluster. Make sure the following directories exist in the /hadoop directory as well: hdfs/name, and hdfs/data.

mkdir /hadoop/hdfs/name && mkdir /hadoop/hdfs/data
My intention is to use the /hadoop directory as a top level holding place when I start experimenting with Hbase and/or any other hadoop based projects. The code base for any project can be stored in that directory to keep things organized.

A short cut is available using rsync to get this set up on all of our slave nodes. First, log in to each slave, create the /hadoop directory, modify permissions (command above), then run

rsync -r --progress hadoop@<master-node>:/hadoop/* /hadoop/
ln -s /hadoop/hadoop-0.20-0/ /hadoop/core
This should copy our entire set-up to the node without having to go through unzipping and all the subsequent steps from that. You can do this after configuring hadoop so that you don't have to configure each slave independently, but only do this if you don't plan to have any custom settings per node. Please also do not do this if you've already attempted to start hadoop.

So with our install in place on all of our nodes, it's time to begin configuring hadoop. On the master node, go to /hadoop/core/conf, and open the core-site.xml file for editing.

cd /hadoop/core/conf && pico core-site.xml
Between the <configuration> and </configuration> tags, add the following:

<property>
  <name>fs.default.name</name>
  <value>hdfs://$master-node$/</value>
</property>
Next, open the hdfs-site.xml, and add the following properties:

<property>
  <name>dfs.name.dir</name>
  <value>/hadoop/hdfs/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/hadoop/hdfs/data</value>
</property>
Then, open mapred-site.xml and add the following:

<property>
  <name>mapred.job.tracker</name>
  <value>$master-node$:54311</value>
</property>
Please remember to replace $master-node$ in each of the above files with the actual dns name of your master node, and leave out the $'s, that is just so you recognize that it must be your custom name.

Next, you have to add a couple of settings to the hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-6-sun
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
If you've followed some of the older tutorials for hadoop on ubuntu, namely those by Michael Noll, then you've realized that you can't (yet) disable IPv6 in 9.04. The newer kernels recognize some optional paramters to disable it, but 9.04's current kernel does not recognize those options. The second line we add to the hadoop-env.sh forces hadoop (through a java option) to use our IPv4 address, instead of the IPv6 one. For some reason, hadoop will only listen on IPv6 when it's available. The NameNode process ties into this as well because it will grab it's listening address from the hosts file based on the machines name. You must verify that your hosts file has your master node's dns name listed next to the intended network address, and not 127.0.0.1, or some other address.

It may not be necessary, but for good measure I even modified my /etc/nsswitch.conf so that the "hosts:" entry only checks "files" then "dns". The other entries (mdns) cause hostnames to be re-written to <host-name>.local, and during some troubleshooting, I worried that the re-written host names may affect hadoop.

The above configurations should all also be made on every slave, with the exception of the dfs.name.dir property. This property is only used by the namenode, so it does not need to be configured on any of the slaves, although I think a slave will safely ignore it.

Now, we need to modify the "slaves" file. This file is a list of all the machines in the cluster who should be running the "DataNode" and/or the "TaskTracker" processes. Please use the actual dns name of the master server and not "localhost" as is in there by default. Since we used the dns name earlier in the process, it will hang when it tries to ssh to "localhost" because we never accepted and saved the fingerprint for that host. Localhost will work if you save the fingerprint, but I prefer to keep things somewhat consistent. If you have separated your NameNode from your JobTracker, then you'll need to have the same list on each master machine.

In our test cluster, we have no use for the SecondaryNameNode, so I removed all entries from the "masters" file. If you will be needing redundancy, then you'll definitely want to look into the SecondaryNameNode Process, and how to set it up. Any machine in the cluster that should run the SecondaryNameNode, should be listed in the masters file.

Assuming you have everything configured properly, then from the master server, in the /hadoop/core directory, you can run:

bin/hadoop namenode -format
bin/start-dfs.sh
bin/start-mapred.sh
The first command should only be run once to initialize hdfs, and should never be run while the cluster is running. The second two commands will start the NameNode/DataNodes and the JobTracker/TaskTrackers respectively for every machine in the cluster. Once the cluster has had a few minutes to register itself to the master, you can point your browser to http://<master-node>:50070/ for the current cluster health, and lists of slave nodes. You can also point to http://<master-node>:50030/ for lists of jobs and their current status.

The start-hdfs.sh, stop-hdfs.sh, start-mapred.sh, and stop-mapred.sh scripts can all be used to quickly stop and start the entire cluster, just always be certain that the slaves file is up to date. If you have separated your JobTracker from your NameNode, then you must also execute those commands separately. The cluster-wide start/stop commands should only be executed from the master of that service. To manage individual slave nodes independently of the cluster, you can use bin/hadoop-daemon.sh. This script allows you to dynamically add and remove individual machines from the cluster without affecting the rest of the cluster.

Usage: hadoop-daemon.sh [--config <conf-dir>] [--hosts hostlistfile] (start|stop) <hadoop-command> <args...>
After completing my cluster, I decided to do some benchmarking to see what we can actually use this thing for. I stole about 1.6 GB worth of raw log files from our web server, and copied them into hdfs:

bin/hadoop dfs -copyFromLocal temp/* logs
Watching the cluster health page, you can see how it balances the storage load equally among all of the nodes. Once the copy was complete, I ran the provided wordcount job as an example:

bin/hadoop jar hadoop-0.20.0-examples.jar logs logs-out
This breaks the logs up into individual words, and then counts the number of occurrences of each word and writes it to an output file. To see how the sheer number of nodes affects the processing time of this job, I re-ran this job on my cluster with from only one node, all the way up to six. My results are as follows:

1 node:      16m59.997s
2 nodes:     8m34.724s
3 nodes:     5m49.434s
4 nodes:     4m33.927s
5 nodes:     3m29.834s
6 nodes:     2m57.472s

As you can see, there is a sort of nuclear decay with adding nodes verses the amount of time it takes to process. If you have researched the beowulf project at all, they talk a lot about parrallel computing theory, and explain that only so much work can be done in parrallel, and that as you get to an infinite number of nodes, the parrallel portion of the job would be done nearly instantaneously, but there always will be certain steps that can not be skipped, and must be processed serially. Not to mention the added overhead that happens as you add nodes to the cluster.

Now imagine that I had to process 16 GB of apache logs... One node would have taken roughly 170 minutes, while 6 nodes would have taken roughly 30 minutes. When you start to look at extremely large quantities of data, you can quickly see the benifit to having several machines working on a single problem.

If you are interested, you should check out the "powered by" section of the hadoop website, to see how this type of computing is being used in the real world.

Until Next Time...
]]> Authoritative DNS tag:www.travishegner.com,2009://2.19 2009-06-10T00:51:55Z 2010-01-22T18:33:21Z As much as I love exploring and learning the newer protocols and technologies like SIP, and Enterprise Virtualization, there is something to be said for those standards and protocols that the internet is built from on and designed around. It is quite refreshing to work with a protocol that is not under constant revision and modification, since the software that runs it is rock solid, stable, and just runs until you tell it not to.If you haven't caught on, or are new to my site, I prefer running everything I can on ubuntu server edition. I've found it to be... Travis Hegner http://www.travishegner.com/about-me.html
If you haven't caught on, or are new to my site, I prefer running everything I can on ubuntu server edition. I've found it to be stable, fast, reliable, lightweight, and still powerful enough to handle anything from large database back-end systems, to lightweight web hosting systems, and even to, you guessed it, authoritative DNS servers. I won't suggest an ubuntu server version here, since it is nearly the same procedure on every version I have used (I've been using ubuntu server since 2006.)

First, you need a server to run it on. If you have a virtual infrastructure already handy, DNS is ideal as a virtualized system because it uses very, very little system resources, unless you are hosting DNS for some EXTREMELY busy domains. If you don't have a virtual infrastructure, or any handy hardware, you can comfortably share your DNS services on your web or email server. Just bear in mind that this server will have to be publicly available, so choose a machine that is safe for that purpose.

After several difficult and dissapointing conversations with first level technical support from various ISPs, I decided a few years ago that I would manage my own authoritative DNS solutions. I was pretty much fed up when the last ISP I had hosting my DNS told me that there was no such thing as a TXT Record.

So once you have a server ready to host your DNS, you'll want to install BIND:

sudo apt-get install bind9
Once bind is installed, you can immediatly begin to configure it. My standard procedure is to create the directory where I store all of my "zone files".

sudo mkdir /etc/bind/zones
A zone file is a file which describes your domain, it's hosts, and how other users of the internet should use the information you provide. Next we edit the file /etc/bind/named.conf.options. In here, you can specify some of the many, many options of bind. I typically will set up 'allow-recursion', and 'version' directives here.  If you are setting up an internal DNS server, or would like bind to do recursive queries for your internal clients, then you would want to include your internal address range:

allow-recursion { 192.168.0.0/16 };
If this server will be relying on itself for DNS lookups, then you'll want to make sure that the localhost can recurse as well:

allow-recursion { localhost; 192.168.0.0/16; };
There are some other tricks you can find at:
http://www.zytrax.com/books/dns/ch7/queries.html
http://www.zytrax.com/books/dns/ch7/address_match_list.html
Just be careful and put some thought into who you will, and will not allow to do recursive queries.

The version directive is recommended by http://www.cyberciti.biz/faq/hide-bind9-dns-sever-version/ to be set to something generic like "BIND":

version "BIND";
Otherwise it is possible for an attacker to obtain your version number, and exploit some yet undiscovered vulnerabilities in your version. Some of the newer versions of BIND instituted a new set of logging directives which I typically set, otherwise your syslog will be littered with tons of error messages from servers in the world that don't follow the DNS RFC standards properly:

logging {
  category lame-servers { null; };
  category edns-disabled { null; };
};
Next we have to modify the file /etc/bind/named.conf.local, which holds all of our yet to be configured "zones". Zones are usually domains, but can be sub-domains, or even in-addr.arpa domains for doing reverse DNS. Once inside the file, add a zone like so:

zone "myexampledomain.com" {
  type master;
  file "/etc/bind/zones/db.myexampledomain.com";
};
This tells BIND that it is authoritative for the domain "myexampledomain.com", and that the zone file can be found at "/etc/bind/zones/db.myexampledomain.com". The zone file name and location is my personal preference, you can store it wherever, and name it whatever you'd like. Once you configure a zone file, you can restart the service (/etc/init.d/bind9 restart) and begin testing your newly created DNS server. I won't go into detail about zone files, but a simple google search for "bind zone file" will get you going. Here is an example zone file:

$TTL 86400
myexampledomain.com.  IN  SOA  ns1.myexampledomain.com. hostmaster.myexampledomain.com. (
                                2007100301     ; Serial
                                3600           ; Refresh 3 hours
                                3600           ; Retry 1 hour
                                604800         ; Expire 1 week
                                86400          ); Negative TTL: Minimum 5 minutes

@            IN      NS              ns1.myexampledomain.com.
@            IN      NS              ns2.myexampledomain.com.
@            IN      A               192.168.0.1 ;web site ip
ns1          IN      A               192.168.0.2 ;bind server ip
www            IN      CNAME           myexampledomain.com.
With the zone file in place, restart your BIND server and we can begin testing. Type nslookup and hit enter. Here you can issue dns queries against your default server or specify a server:

server localhost
Now query for your domain and you should get the following response:

> myexampledomain.com
Server:        localhost
Address:    127.0.0.1#53

Name:    myexampledomain.com
Address: 192.168.0.1
If you see something like this, then your configuration is successful. If not, check your logs, and your configuration files, and try again.

]]> PostgreSQL 8.3 Warm Stand-by Replication tag:www.travishegner.com,2009://2.18 2009-06-05T18:02:43Z 2009-06-12T18:58:32Z In an effort to make things a bit more redundant at my work place, I have implemented PostgreSQL 8.3 warm stand-by replication, as outlined in the pgsql 8.3 manual.The manual leaves the possibilities for implementation pretty wide open, which is good for making things scalable, and applying the process in many different ways, but it is bad if you are looking for more detail as to how to set it up. In this tutorial, I will outline my exact methods for implementing warm stand-by for PostgreSQL, running on Ubuntu Server 9.04.The basic jist of postgresql's warm stand-by system, is based... Travis Hegner http://www.travishegner.com/about-me.html PostgreSQL 8.3 warm stand-by replication, as outlined in the pgsql 8.3 manual.The manual leaves the possibilities for implementation pretty wide open, which is good for making things scalable, and applying the process in many different ways, but it is bad if you are looking for more detail as to how to set it up. In this tutorial, I will outline my exact methods for implementing warm stand-by for PostgreSQL, running on Ubuntu Server 9.04.

The basic jist of postgresql's warm stand-by system, is based on the "continuous archiving" ability of pgsql. It utilizes this feature to do "log shipping" of the write ahead log (WAL) files, where the logs are then retrieved by a stand-by server in "continuous recovery" mode. The stand-by server remains in continuous recovery until a given condition has been met, at which point the server comes up into normal operation.

First, you must start with two "as close to the same as possible" servers, as recommended by the postgresql manual. In my case, I have a virtual pgsql server in our primary location, and another virtual pgsql server in our disaster recovery location. Both servers run ubuntu server 9.04 jaunty 32-bit, with postgresql installed from the repositories. PostgreSQL also requires that the two servers be the same architecture (32 or 64 bit).

Once both servers are running, you must create a location to store the archived WAL files. It is highly recommended to have the archive storage located remotely from the primary server, that way, in the event of a primary server failure, you have the latest and greatest WALs for your stand-by server to import. The archive must be readable, and writeable from both the primary and secondary servers. For my installation, I created a NFS share on the stand-by server itself, and then the stand-by server imports the WAL files locally. Just be sure that the archive on the stand-by server is owned by "postgres:postgres", and that the world permissions are "0" because all data written to your database will be visible here as it is written. Remember to take precautions, security is your number one concern at all times.

With the remote archive mounted locally, we must configure pgsql for continuous archiving. Edit /etc/postgresql/8.3/main/postgresql.conf and set the "archive_mode" directive to "on". For my purposes, I set the "archive_command" to "test -f /psql_archive/mounted && test ! -f /psql_archive/%f && rsync -a %p /psql_archive/%f". I created an empty file called "mounted" on the NFS share, so that I could verify that the share was actually mounted, and prevent writing WAL files to the local place-holder archive directory. So our archive_command basically first checks to see if the share is mounted, If it is, then it checks to see if the current WAL already exists, if it doesn't, then it writes the file with rsync. The -a parameter of rsync is important because it first copies the data to a place-holder file name, and then changes the name of the destination file only after all the data was successfully copied. This prevents the stand-by server from trying to load a partial log file.

The archive_timeout directive will require some careful thought on your part. Postgresql will only archive a WAL when it reaches 16MB worth of writes, or when the data becomes older than the "archive_timeout" in seconds. Unfortunately, the WAL file is ALWAYS 16MB, whether we have written that much data or not. So setting this value too low, will exponentially increase the amount of bandwidth required to transmit your WALs. If the stand-by server is on the LAN, this is less of an issue, but over the WAN, you'll need to really be careful how often the the archive times out. Please also remember that if you have 16MB of data written to the database before your time-out, it will ship the log at that point in time, so if you are replicating over the WAN, plan your bandwidth capacity according to the requirements of your database's write activity. For me, the archive_timeout is only every 10 minutes, which is safe because we don't have rapidly changing data, and losing 10 minutes worth of data in the event of a failure would not be catastrophic. This only averages out to around  220kbps of continuous bandwidth utilization.

Your main concern with WAN replication during idle database times, is that you do not want the next WAL file attempting to transmit before the previous one has finished. This will cause a chain reaction which will lead to your secondary database falling further and further behind, without an opportunity to catch up. If you happen to run into this during peak periods of activity, it shouldn't hurt, so long as it's able to catch up within a time frame that is acceptable to you.

After setting these parameters, and restarting the postgresql service (/etc/init.d/postgresql-8.3 restart) you can watch the archive directory, and you should start seeing WAL files being transmitted. We have to be careful, because if we run in this state too long, we will run our archive out of disk space.

Now we can begin configuring our stand-by server for continous recovery mode. It is my preference to create a new file in /etc/postgresql/8.3/main/recovery.conf and symbolic link to it, but you can optionally create the file in the final destination directory which I'll mention later.

The continuous recovery portion would be quite a bit more complicated if it weren't for "pg_standby". Luckily, pg_standby is installed and compiled by default in ubuntu server 9.04, but you must create a symbolic link to pg_wrapper from /usr/bin.

sudo ln -s /usr/share/postgresql-common/pg_wrapper /usr/bin/pg_standby
Once that is complete, just open your recovery.conf file and add the line (please disregard the line wrap around, this should be a single line in the file):

restore_command = 'pg_standby -d -l -r 3 -s 60 -t /psql_archive/role.master /psql_archive %f %p %r 2>>/var/log/postgresql/pg_standby.log'
This line basically tells postgresql to continue recovery forever until the file /psql_archive/role.master exists. In order to make my fail-over in the event of a disaster a bit more seamless, I have written a custom cron job to create this file automatically, based on an internal DNS change. To put it as simply as possible, I have a generic CNAME "postgres" in my internal DNS, which all of my postgresql clients point to for service. Under normal circumstances, the CNAME points to "pg-primary". In the event of a fail-over, I will change the CNAME so that it points to "pg-secondary". Adding the following line to your /etc/crontab file on your stand-by machine will automatically create your "trigger file" after making your DNS change:

* * * * * postgres test -n "`nslookup postgres | grep pg-secondary`" && touch /psql_archive/role.master >/dev/null 2>&1
I still wouldn't go as far as calling this a "hot standby" just yet, but it does simplify the process of failing over.

So to actually get this thing up and running, we must first take a base backup of the data directory of our primary server. It is neither necessary, nor recommended to shut the server down for the base backup. From a SQL prompt, issue the query:

select pg_start_backup('base_backup');
After that, create a tar archive of the entire data directory. On ubuntu server 9.04, the postgresql data directory is located at /var/lib/postgresql/8.3/main. To do this, issue the command:

tar -czf base_backup.tar.gz /var/lib/postgresql/8.3/main
Once the tar file is made, connect back to the database and issue the query:

select pg_stop_backup();
Copy the resulting file to your stand-by server. Once there, shut down postgresql on the stand-by server, and extract the contents of the tar.gz archive, overwriting the existing data directory. Navigate to the data directory:

cd /var/lib/postgresql/8.3/main
Then create a symbolic link to /etc/postgresql/8.3/main/recovery.conf, if you chose to make it there, or create your recovery.conf file here.

ln -s /etc/postgresql/8.3/main/recovery.conf
Assuming everything is set up right, you should be able to start the postgresql service (/etc/init.d/postgresql-8.3 start). Monitoring the archive directory, you should see the WAL log files start to be consumed and removed every minute or so after one appears.

Unfortunately, there is no way to verify that the data is actually being replicated and recovered into the stand-by server, because during "recovery mode" the database is not operational. To test that everything is working properly, you have to manually create the trigger file:

touch /psql_archive/role.master
This should trigger the recovery process to complete, and the server should come on-line automatically. To verify, check the pgsql data directory (/usr/lib/postgresql/8.3/main) and see if the recovery.conf file has been re-named to recovery.complete. If so, then connect to the secondary database and check for the existance of some known changes since the base backup. If all looks well, then you must repeat the steps above starting from creating the original base backup, in order to get the stand-by server into recovery mode again. Simply re-naming the recovery.complete back to recovery.conf WILL NOT WORK because once the server has been started, it creates a unique 'instance' or 'timeline' and it will not start a recovery process unless the 'instance' of the WAL files match that of the recovering server. Hence the necesity of re-doing the base backup.

I also recommend testing the automatic trigger file by changing the DNS with a NON PRODUCTION set of DNS names, to make sure that it works properly. What works for me, may not work for you.

There are a number of options for recovery to status quo after a complete fail-over scenario. You could reconfigure each of the two servers so that the secondary becomes the primary. You could use the pg_dump and pg_restore tools to migrate your data back to the original primary, and follow the steps in this article from the base backup again to get replication going.

Another method may be to simply do a shutdown the original secondary server, then make a tar of it's data directory, extract it over top of the original primary server data directory (once repaired of course), then follow the steps to start up the secondary server in recovery mode again. I have not tested that method of recovery, but theoretically it should work as both instances, or timelines should be the same. Please try this last method at your own risk.

Disclamer: Please note that this set up may not work for any/every environment, and I can not and will not be held responsible if the use of this information results in data loss or damage. It is your responsibility to make backups and use precautions while working in a production environment with production data. The information here is provided as-is, and without warranty. That being said, I will be willing to help you out if you run in to a snag, or my directions aren't quite clear enough.

Further Reading, related articles, and sources:
http://www.postgresql.org/docs/8.3/static/backup-file.html
http://www.postgresql.org/docs/8.3/static/continuous-archiving.html
http://www.postgresql.org/docs/8.3/static/warm-standby.html
http://www.postgresql.org/docs/8.3/static/pgstandby.html
]]> Google AdSense tag:www.travishegner.com,2009://2.17 2009-05-20T17:21:43Z 2009-05-20T18:25:39Z So, if you haven't already noticed, I have clogged up my blog site with a bunch of those nifty google ads. The main reason for doing so is to make some extra pocket money, if anyone ever visits me anyway, and because I need a new car.Since it was so easy to install, I decided to write a little blurb about how to do it in movable type 4. First and Foremost, you have to sign up for google adsense at http://www.google.com/adsense. After that you have to wait to be approved. It only took around 8-12 hours for me, but... Travis Hegner http://www.travishegner.com/about-me.html I need a new car.

Since it was so easy to install, I decided to write a little blurb about how to do it in movable type 4. First and Foremost, you have to sign up for google adsense at http://www.google.com/adsense. After that you have to wait to be approved. It only took around 8-12 hours for me, but they say it could be up to two days. Once approved, you can log in and begin creating ads, customizing the styles and colors, and creating what google calls "channels" to keep track of which ads get clicked most often. Once you have followed the steps to creating your ads, they will give you the code to paste onto your site, just like google analytics. I designed my ads to go into the sidebars, so in movable type 4, I created a couple of new widget templates. In each template I added the code <div class="widget"></div> and pasted the google provided code between the <div> and </div> tags. I also had to add a couple of css overrides in the style.css template to get things to layout a bit more smoothly, but other than that it was a snap.

So as you rummage through all of my mediocre content, please feel free to click on an ad or two, and support the "Retire the grass-stained bumble-bee fund".
]]> Ubuntu Server 9.04 Jaunty: open-vm modules fail to build tag:www.travishegner.com,2009://2.15 2009-05-13T18:15:59Z 2009-05-20T18:02:01Z I have decided to start upgrading from Ubuntu Server 8.10 Intrepid to Ubuntu Server 9.04 on all of my server machines. In 8.10, I was able to get the open-vm-tools to build and install, but they suffered from a bug which caused the vmware-guestd service to crash soon after starting. This was mildly acceptable and my servers have been running that way for a while, because I wanted the fix to be easily installed from the ubuntu repositories.Well after waiting 6 months, with no fix in the repo's, Canonical released Ubuntu Server 9.04. I upgraded a test server "sudo do-release-upgrade"... Travis Hegner http://www.travishegner.com/about-me.html I have decided to start upgrading from Ubuntu Server 8.10 Intrepid to Ubuntu Server 9.04 on all of my server machines. In 8.10, I was able to get the open-vm-tools to build and install, but they suffered from a bug which caused the vmware-guestd service to crash soon after starting. This was mildly acceptable and my servers have been running that way for a while, because I wanted the fix to be easily installed from the ubuntu repositories.

Well after waiting 6 months, with no fix in the repo's, Canonical released Ubuntu Server 9.04. I upgraded a test server "sudo do-release-upgrade" and rebooted, only to find that the new vmware-guestd installs and runs, but the new open-vm modules fail to build properly with module-assistant (sigh). So I decided that I don't want my VM's to run without modules, because they are very important for the way ESXi handles memory allocation and management, but I still want to upgrade my 8.10 servers because having vmware-guestd not crash is a huge plus. vmware-guestd handles tasks like allowing the ESXi host to shutdown the linux guest safely, and synchronizing the VM clock with the host clock.

So after some searching, I came across this open-vm patch which allows the modules to build successfully. If you are not comfortable with patching it yourself, read on, and I'll explain how I did it, and provide a pre-patched source archive for you to replace on your machine, in order to allow module-assistant to build the modules successfully.

Many people (myself included) are a bit weary when it comes to compiling and installing linux modules. Probably mostly due to lack of research, because if it's like anything else in linux, it is very straght-forward and easy to understand. ModuleAssistant is an awsome tool for debian (and inherently debian-based) linux distros which nearly automates the process of compiling and installing modules. Assuming the code is correct and m-a is installed, we could build and install the open-vm modules and tools in one fell swoop by typing:

sudo m-a a-i open-vm

But since we don't get that lucky, we have to do a little work to get the modules installed and compiled properly. First, take all the necessary precautions and backup your data, or take a snapshot. Then, run the command from above to prepare the machine for the new modules. Go through the prompts accepting the changes, and eventually you will get an error stating that the build has failed. At this point, choose "stop" and you should be back at the command prompt. Navigate to '/usr/src' and list the contents.

cd /usr/src && ls

You should see a source archive titled 'open-vm.tar.bz2', rename this archive to something else, and replace it with my pre-patched open-vm.tar.bz2. Also rename the 'modules' directory.

sudo mv open-vm.tar.bz2 open-vm.tar.bz2.orig
sudo mv modules modules.orig
sudo wget http://www.travishegner.com/stuff/open-vm.tar.bz2

Once the file has been replaced, re-run the auto-install command from above and the build should complete successfully. The modules and tools should also install. Reboot for good measure to make sure everything comes up like it's supposed to.

You should now be running both the open-vm modules, and vmware-guestd without any issues. AWSOME! No more errant clocks, and even the "Shutdown Guest" and "Restart Guest" scripts work like they are supposed to.

]]> Google Analytics tag:www.travishegner.com,2009://2.14 2009-05-12T14:19:05Z 2009-05-20T17:34:03Z I learned about Google Analytics from my friend/co-worker Kyle Dye while we both worked at Trillium Staffing. It is an awesome tool that google provides for free to anyone who wants to use it. It provides a TON of information and statistics about who visits your web site, and how often, and what keywords they searched for or where they were linked from. And that is just scratching the surface. It is a very good system for keeping stats on your site and trying to improve your traffic. It's only a few short lines of pre-generated javascript code which I... Travis Hegner http://www.travishegner.com/about-me.html Google Analytics from my friend/co-worker Kyle Dye while we both worked at Trillium Staffing. It is an awesome tool that google provides for free to anyone who wants to use it. It provides a TON of information and statistics about who visits your web site, and how often, and what keywords they searched for or where they were linked from. And that is just scratching the surface. It is a very good system for keeping stats on your site and trying to improve your traffic. It's only a few short lines of pre-generated javascript code which I will describe how to install into Moveable Type 4.25 below.

First you have to register your URL on the Google Analytics site, which is very easy if you already have an account with google. They use a single sign-on system so if you already have gmail, or a google profile, or anything with them, you can just log in. If you don't already have one they will guide you through creating one. After you register your URL, they will give you a piece of javascript code, which they explain should be put just before the </body> tag in your HTML code. In Moveable Type 4.25, you simply have to go to "Design" --> "Templates" and click on the "Banner Footer" template to edit it. Paste the code at the very bottom of the template, save, and publish your blog, and within a day or two you will start seeing some statistics. You have to be patient, as they only update the statistics once per day, as it would take way too many resources to provide this kind of information real time, even for google.

If you view the source of your blog, near the bottom you should see the pasted javascript code. It won't be exactly before the </body> tag as there are a couple of </div> tags between them, but that won't hurt anything. For a live example, just view the source of this page and you should see my google analytics code, right after my RefTagger code.

Thanks and Enjoy!
]]> RefTagger tag:www.travishegner.com,2009://2.13 2009-05-12T14:00:01Z 2009-05-20T17:34:33Z I have just installed "RefTagger" from http://www.logos.com/reftagger it's a free blog/site parsing javascript code set that automatically links any bible verses to their on-line bible http://bible.logos.com. It also includes the verse you reference in a tool tip which looks like a really nice feature.So in the future, you will see any references to bible verses automatically linked with the verse in a tool tip. So if I mention my one of my favorite verses in a blog, for instance, 1 Corinthians 13:4-8 it will magically parse it and create a tool tip. (Hover your mouse cursor on the verse now.)If... Travis Hegner http://www.travishegner.com/about-me.html http://www.logos.com/reftagger it's a free blog/site parsing javascript code set that automatically links any bible verses to their on-line bible http://bible.logos.com. It also includes the verse you reference in a tool tip which looks like a really nice feature.

So in the future, you will see any references to bible verses automatically linked with the verse in a tool tip. So if I mention my one of my favorite verses in a blog, for instance, 1 Corinthians 13:4-8 it will magically parse it and create a tool tip. (Hover your mouse cursor on the verse now.)

If you haven't noticed, I use Moveable Type 4.25 for my blog, and it was very easy to install reftagger. First, go to http://www.logos.com/reftagger#code and generate your javascript code. Then under "Design" --> "Templates" edit the "Banner Footer" template, and paste the pre-generated code at the end of the template somewhere. I put mine right above my Google Analytics code.
]]> In Honor of Mothers Day tag:www.travishegner.com,2009://2.12 2009-05-11T12:44:15Z 2009-05-12T14:18:51Z Happy Mothers Day to all the Moms out there!Remember, "Treat her right"... Travis Hegner http://www.travishegner.com/about-me.html


Remember, "Treat her right"
]]> Congratulations! tag:www.travishegner.com,2009://2.11 2009-05-08T15:47:33Z 2009-05-08T15:52:24Z I'd like to take a second to congratulate a good friend on a new coaching position at Maranatha Baptist Bible College.Congrats Coach Andy Peterson, I wish you very good luck in your new career!... Travis Hegner http://www.travishegner.com/about-me.html Maranatha Baptist Bible College.

Congrats Coach Andy Peterson, I wish you very good luck in your new career!
]]> Just My Luck tag:www.travishegner.com,2009://2.9 2009-05-06T12:07:54Z 2009-05-08T00:00:13Z Why is it, that on the exact same stretch of road where I got pulled over yesterday, someone can blow by me doing like 80 mph and never have any problems. Where's the county sheriff then?Is it just my luck or what? Maybe it's because the grass-stained bumble-bee is a bit too noticeable? Maybe I should get a new car...... Travis Hegner http://www.travishegner.com/about-me.html pulled over yesterday, someone can blow by me doing like 80 mph and never have any problems. Where's the county sheriff then?

Is it just my luck or what? Maybe it's because the grass-stained bumble-bee is a bit too noticeable? Maybe I should get a new car...
]]> Prevention of Bankruptcy... tag:www.travishegner.com,2009://2.8 2009-05-05T22:50:30Z 2009-05-06T12:35:13Z In the past 9 or so months, I have not-so-happily, and not-so-voluntarily, donated great sums of money to various governments throughout the state of Michigan. I almost believe that my speeding habits have single-handedly prevented the bankruptcy of any of these entities.Today marks three tickets in 9 months. That is quite a record. Believe or not, non of them were on the motorcycle. The worst part of all, is that I had to go through the misery of disappointing my wife with something so stupid and preventable for the third time.If you ever do read this Babe, I'm truly sorry,... Travis Hegner http://www.travishegner.com/about-me.html
Today marks three tickets in 9 months. That is quite a record. Believe or not, non of them were on the motorcycle. The worst part of all, is that I had to go through the misery of disappointing my wife with something so stupid and preventable for the third time.

If you ever do read this Babe, I'm truly sorry, and I hope you can forgive me... Again...

]]>