Monthly Archives: June 2013

Software Development, Technology

Amazon AWS – spinning up a micro for Gearman Client

Leave a comment

Walkthough for spinning up a micro EC2 for use as a GearMan worker node.

This is a preliminary effort, and the steps may very well change over the next few weeks as these are tested out. The documentation I’m presenting here is based on my previous work [HERE] Install Gearman + Gearman-PHP on AWS ec2.

Getting Started

This assumes you already have an AWS account setup. If you don’t you need to go do that now. You can get started [HERE].

Once logged in, go to your dashboard. From here you’ll be selecting the EC2 area.
Screen Shot 2013-06-17 at 10.26.23 AM

Under resources you’ll find a button “Launch Instance”. This is the button you want to click. This is where the fun begins.
Screen Shot 2013-06-17 at 10.27.57 AM

In this case I’m going to use the ‘Classic Wizard’. It’s really not that magical but Wizard is such a super-awesome-cool-name (ala Windoze ’95) you’ll see it used here. Anyhow, I’m going classic:
Screen Shot 2013-06-17 at 10.30.04 AM

I want to keep things EVERY simple here so I’m going use the default Amazon AWS distribution/AMI.
Screen Shot 2013-06-17 at 10.31.52 AM

For this exercise, I’m using an ‘On Demand’ Micro instance:
Screen Shot 2013-06-17 at 10.34.01 AM

No Advanced features should be required, so I’ve left everything on this page set to it’s default settings:
Screen Shot 2013-06-17 at 10.36.20 AM

Next the storage requirements are defined. Since this is a worker that should be able to be spun up on need, and shouldn’t require much in the way of local storage, I’m going to opt out of defining and EBS volume and rely upon Ephemeral storage.
Screen Shot 2013-06-17 at 10.40.43 AM

At this point, I don’t see the need to define any keys for EC2 management, so I’m leaving this area blank:
Screen Shot 2013-06-17 at 10.41.30 AM

Next, select the .ssh key file (it’s stored in a .pem file) that you want to use to access this system. If you don’t have a set of these keys setup already, you’ll want to define them. I’m using one specific to this node class already defined.. you’ll need to handle this step as your policy/needs dictate. It’s not that complex, but word to you, DOWNLOAD THAT KEY, once you create it, there is no known way (according to all places I’ve checked) to download it again. BE WARNED.
Screen Shot 2013-06-17 at 10.47.16 AM

Next step will be do select a security profile for your node. I’ve found that the default one is sufficient for these purposes. You may want to further restrict the number of ports open, as the default opens a few extra things you might not want. This is another area where you’re own needs an policy will need to be carefully considered. Otherwise, start with default and iterate to the optimal configuration.
Screen Shot 2013-06-17 at 10.52.02 AM

Check your settings on this review page, and if everything is to your preference, then you can spin it up!!
Screen Shot 2013-06-17 at 10.55.38 AM

One you click launch, it will take a little while for the instance to go live.
Screen Shot 2013-06-17 at 10.57.20 AM

Once launched, you’ll be able to see your instance in the dashboard! Sorry bout all the greyd out information, but the instance I’m talking about is slightly highlighted in blue.
Screen Shot 2013-06-17 at 11.00.22 AM

NOW, YOU CAN START TO INSTALL YOUR GEARMAN COMPONENTS

Motorcycles, travel

Nor Cal Ducati – Monterey CA meetup

1 Comment

Last night we had the 2nd meeting up of the Nor Cal Ducati Club this evening in at Cibo, in Monterey CA.. It was a nice evening, warm, sunny and the food was quite enjoyable, but the company made it all worth the ride. And a bit of a challenging ride it was!

I met up with Dimitri on his cool Monster900 (a deal may be in the works soon!), at the Vista Point on Highway 1, for the ride to Monterey.

View Larger Map

We waited a little while to try and catch a couple of riders coming down from the North Bay, but by 5:30 PM we decided it was time to head south. We didn’t make it far before we were caught in a massive traffic jam on Hwy 1. It was a virtual parking lot, we were going nowhere.

Fortunately we both knew of an alternate route and turned around on Hwy1. By the time we reached Castroville, the traffic heading north was also backed up for miles. A check of the local news sources didn’t reveal the cause of all this chaos, but I heard that at least one person was life-flighted from and accident scene on the highway.

We arrived in Monterey a little later than planned but it was no matter, there were two parking spots for us in front of Cibo, so we pulled in, popped off the helmets and met up with the southern contingent of the Nor Cal Ducati Club. It was nice little array of fine Italian machines on display:
IMG_3008

IMG_3009

IMG_3011

Thanks to The Bear for setting this all up. Looking forward to the next Ducati Bike Night.

Uncategorized

Memories of VME

Leave a comment

vme.seattle.aug.2005.05
While enjoying a meeting up of the Nor Cal Ducati Club this evening in at Cibo, in Monterey CA., the subject of the Seattle VME (Vintage Motorcycle Enthusiasts) came up, specifically the wonderful and wacky “Isle of Vashon” event. The coolest aspect of the bikes you’ll see in these videos, is that they are ALL ridden to and around the island.

What others have posted about the TT

Sample videos shot by others:
VME Isle of Vashon 2011

VME Isle of Vashon 2008

Here are some articles about the event itself:

VME Isle of Vashon
The Northwest Classic Old Bike Event
http://www.soundrider.com

What, you missed it again? VME’s Isle of Vashon? Nooooo? Why? Didn’t know when the date was? Where was it listed?[…]

Motorcycle club appreciates the Island’s support
http://www.vashonbeachcomber.com
The Vintage Motorcycle Enthusiasts (VME) would like to thank the Vashon-Maury Island community for supporting our 29th annual old bike rally, known as the Isle of Vashon TT.

Thanks to the event’s con[…]

Now this is really cool, and geeky.. but a big post about VME on a Gamers forum!!

Vintage Motorcyle Rally (Pic intensive)

Where is this, “Vashon Island” place?

Vashon island is located in lower ventricle of Puget Sound:
Screen Shot 2013-06-16 at 10.46.29 PM

Situated above Tacoma, south-west of Seattle and south-east of Bremerton, Vashon Island is one of the few Puget Sound island that is serviced only by ferry. No conventional roads exist to get to this island, unlike the rest within Puget Sound that are connected via some sort of bridge or another.
Screen Shot 2013-06-16 at 10.58.30 PM

VME events, and the Isle of Vashon in particular, you can find not only a nice variety of vintage bikes, but a number of strange and often amusing customs.

I certainly wouldn’t mind attending another one of these unusual events some time in the near future!

Motorcycles

Tale of two Italians

1 Comment

Awaking this morning, I decided to put together this little rundown of my two liter-class Italians. For a pair of bikes with remarkably similar style, their characters are vastly different.

Here is the 411:


2008 Ducati 1098 Superbike
2007 MV Agusta F4 1000R
bike_banner
Dimensions
Wheel Base:
 56.3 in. Wheel Base:
 55.4 inches
Seat Height:
 32.2 in. Seat Height:
 31.9 inches
Fuel Capacity:
 4.1 gal. Fuel Capacity:
 5.5 Gal
Dry Weight:
 381 lbs. Dry Weight:
 423.3 pounds
Suspension / Wheels
Front Tire:
 120/70 ZR17 Front Tire:
 120/70 ZR17
Rear Tire:
 190/55 ZR17 Rear Tire:
 190/55 ZR17
Front Forks:
 45mm Showa Inverted, adjustable preload, adjustable compression & rebound damping Front Forks:
 50mm Marzocchi Inverted, adjustable preload, adjustable compression & rebound damping
Engine and transmission
Displacement:
 1099 cc Displacement:
 998 cc
Bore and Stroke:
 104 x 64.7mm Bore and Stroke:
 76 mm x 55 mm
Compression:
 12.5:1 Compression:
 13.1:1
Fuel System:
 Marelli EFI w/ elliptical throttle bodies Fuel System:
 Fuel Injected
Horsepower (bhp):
 160 bhp Horsepower (bhp):
 174 hp
Torque (ft./lb.):
 90.4 Torque (ft./lb.):
 81.9
Transition:
 6 speed Transition:
 6 speed cassette
Fleet Status
Acquired:
 Feb. 2008 (new) Acquired:
 May 2013 (used)
Mileage:
 14,581 Mileage:
 11,203
Duty:
 Occasional track days, special riding excursions, photography prop Duty:
 Special riding excursions, photography prop
Modifications:
 * CRC forged levers
* Woodcraft billet clutch cover
* Ducati Performance spider pressure plate
* AVVI Billet Rear Sets
* Custom painted brake, clutch and slave reservoir caps
* Ducati Performance exhaust and 1098R ECU
* H11 HID low-beam lamp
* Reflector delete 		Modifications:
 none
Riding Impressions
  • Wicked fast.
  • Scalpel sharp handling.
  • Loud and Proud
  • Incredible performance
  • Scalpel sharp handling.
  • Head turning design (motorcycle art)
  • Closest thing to an F1 engine on the street
Software Development, Technology

Install Gearman + Gearman-PHP on AWS ec2

1 Comment

The fun and games continue!! As with every Gearman implementation I’ve done, there are trick for each environment. Here are the Cliff Notes (originally sourced from [Planet MySQL page] with my own twist) for getting Gearman setup on an Amazon Web Services (AWS) EC2 (Elastic Computing 2) node running the default AWS distribution. As always, your experience may vary.

Install required libraries

First, get all the required libraries installed using yum:

[ec2-user@]$ sudo yum install -y gcc
[ec2-user@]$ sudo yum install -y gcc-c++
[ec2-user@]$ sudo yum install -y gperf
[ec2-user@]$ sudo yum install -y boost
[ec2-user@]$ sudo yum install -y boost-devel
[ec2-user@]$ sudo yum install -y memcached
[ec2-user@]$ sudo yum install -y libuuid
[ec2-user@]$ sudo yum install -y libuuid-devel
[ec2-user@]$ sudo yum install -y libevent-devel
[ec2-user@]$ sudo yum install -y php-devel
[ec2-user@]$ sudo yum install -y php-xml

Compile Gearmand from Source

Very straight forward config and build.

[ec2-user@]$ cd gearmand-1.1.9
[ec2-user@]$ sudo ./configure --with-boost=/usr/include --prefix=/usr
[ec2-user@]$ sudo make
[ec2-user@]$ sudo make install

Compile Gearman PHP Library from Source

Fairly simple build, but you must first phpize.

[ec2-user@]$ cd gearman-1.1.1
[ec2-user@]$ sudo phpize
[ec2-user@]$ sudo ./configure --prefix=/usr
[ec2-user@]$ sudo make
[ec2-user@]$ sudo make install

Run ldconfig to Reload Dynmaic Library Cache

If you don’t run ldconfig, you’re going to get errors when you edit the php.ini file (last step).


bad:
PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/gearman.so' - libgearman.so.8: cannot open shared object file: No such file or directory in Unknown on line 0
[ec2-user@]$ sudo ldconfig

Edit the PHP ini file

This is the last step.

finding location of your php.ini file
[ec2-user@]$ php -i | grep php.ini
Configuration File (php.ini) Path => /etc
Loaded Configuration File => /etc/php.ini

Edit your config file, adding these lines:

[ec2-user@]$ sudo vi /etc/php.ini
[Gearman]
; Add Gearman shared object to config
extension="gearman.so"

Now your install is complete!!

Software Development, Technology

Installing HomeBrew for OSX

1 Comment
Installing Homebrew

This process appeared to be fairly trivial. Run the following command. The script will tell you want it wants to do. I stuck with the default responses and let it do it’s worst. You will probably want to read the HomeBrew page [HERE] first, before just trusting me.


david$ ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

Next step was to run the Brew Doctor and address any major issues raised there, of which I had a few. The most pressing of which was this:

Warning: Xcode is installed to a directory with a space in the name.
This will cause some formulae to fail to build.

I deleted XCode 4.3.3 (my latest installed version) and re-installed which gave me version 4.6.2. Next I had to install the ‘Command Line Tools’ (no longer embeded in XCode). That can be found in the XCode Preferences -> Downloads.

I had a few other issues, stray libraries from botched builds past. I worked through my issued and then tried to installing components.

Since Boost seems to be very commonly referenced, I went ahead and installed it with Brew as it’s first task:

Installing Boost using Brew

Then went about installing boost with HomeBrew, since it did not seem to get all the components I needed when installed via MacPorts/Fink.


david$ brew install boost

This appears to have installed boost-1.53.0

* DONE *

Now, you may Brew!

Software Development, Technology

Installing Gearman PHP components for OSX- (less fun that it should be)

2 Comments

gearmanlogo

NOTES: Installing Geaman’s PHP components on OSX is a frustrating and rather complex task. Know this going in. The only way I was finally able to do this was by reading this page here, from one of PHP’s own engineers, got me pretty close but it was still not a full solution. [HERE]. YOU HAVE BEEN WARNED!!

FIRST THINGS FIRST — Get the right versions!!

Do not use gearman.1.1.7!
As of this writing (7-JUN-2013) the current version of gearmand (gearmand-1.1.7) has a bug that prevents it from properly building on OSX. I waste probably 2 days before in a deep corner of the mind I thought.. “If 1.1.7 has a known bug in OSX, and they have not fixed it yet.. let’s try 1.1.6!, and that worked!! The main Gearman download page has multiple versions so just avoid 1.1.7.

Get the lateset PHP source, (gearman-1.1.2) NOT the one linked off the Gearman page!
Yes.. this wasted even more time. The Gearman page didn’t have a quick easy link to the full set of available versions, and the linked version from the page was very much out of date. There is another build bug regarding PHP. One of the engineers decided to get fancy and change the privacy of the objects members somewhere in 1.0.x tree. This BREAKS build on OSX. They did as recently as this year release a FIX for this which is version gearman-1.1.2. All of them can be had on this page [HERE].

Getting Library Dependancies Worked Out

After fighting with source code, screaming at the screen, and even getting completely frustrated with what I was able to (or not able to) install with MacPorts.. I decided to install HomeBrew and give that a run. It’s not a big deal but I moved that to it’s own page located [HERE].

libevent must be built/installed

You’re going to need libevent, and installing it straight up from brew (nor Mac Ports) did the job for me. Check out my previous pages on installing libevent located [HERE] for details on that exciting exercise.

You Must Install Gearmand (sever) regardless

Regardless of how you plan to user Gearman with PHP, you must have the GearmanD server compiled to create the required libraries. There are no two ways about it, just resign yourself to that and keep moving forward!

First, obtain the Gearmand source code for compile from [HERE]. I dropped min in /usr/local

Unball the file and cd into source code directory, configure and build with the following commands:

david$ cd gearmand-1.1.12/
david$ ./configure -disable-shared -prefix=/usr/local
david$ make && make install

A couple of adjunct notes

If you are having problems location the libevent.. or basically seeing this error:

checking test for a working libevent… no
configure: error: Unable to find libevent

Try setting these two environment variables, to tell the configurator exactly where to locate these libraries, if you’ve managed to build libevent from source:

[gearmand]$ export CPPFLAGS=’-I/usr/local/include’
[gearmand]$ export LDFLAGS=’-L/usr/local/lib’

Software Development, Technology

Gearman — Starting the Java-Gearman-Service process

1 Comment

gearmanlogo
These notes apply to testing on a MAC OSX portable, and may or may not apply to your implementation. They are provided as an adjunct to my main Getting Gearman Going post elsewhere in this blog.

The project page for Java-Gearman-Service is located [HERE] on Google Code.

The full set of instructions for staring up Gearman’s Java-Gearman-Service were not clearly linked to the main Gearman project page, so I’m including the link [HERE] to save you the few Google dorkings I did to find it.

Starting up the Java service should be as simple as this:


java -jar java-gearman-service-0.6.6.jar

HOWEVER, I received this instead.. an error:


david$ java -jar java-gearman-service-0.6.6.jar
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/gearman/impl/Main : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Checking my version shows that I am on 1.6 not 1.7 as I had thought;


david$ java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06-451-11M4406)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01-451, mixed mode)

To get the proper version, I navigated to the Oracle page located [HERE], agreed to their terms (do I really have a functional choice… not if I want/need to use Java…) and pushed forward.

If you are running the install for 1.7, you should see a dialog like this:
Screen Shot 2013-06-03 at 2.12.16 PM

That has at least resolved this part of the issue, and will attempt to restart the server.


david$ java -version
java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b12)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)

Now I’m going to restart, specific a custom port and get it kicked off in the background:


david$ java -jar java-gearman-service-0.6.6.jar -p6315 &

At this point the process is running on my Job Service box and next step will be to craft some code to see how it all works.

Software Development, Technology

Deploying Java Gearman Job Server

Leave a comment

gearmanlogo
OK, after more than a year it’s time to get down to really building out a Gearman system.

I’ve recently taken on a project that I believe to be the perfect fit for the pipelined distributed work manager of Gearman. I’m of course, not at liberty to discuss the various details of this project, but I can provide some high-level description for the purposes of justification.

The Project – distributed harvesting

The objective of the project is to provide a distributed method of web page scraping and parsing. This project requires that the scraping and profiling occur for 2,000,000+ websites in under 18 hours. No small feat for certain. The good news is that I’ve built a systems in the past (circa 2006) that did just this using MySQL as the task manager. It worked, but it had it’s issues, and almost every single one of them can be mitigated by using Gearman. The rest will be mitigated with the application of NoSQL solutions for site list management.

What is Gearman

Here is the synopsis from the Gearman.org main page.

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates.

The Job Server — implementing in Amazon’s ES2 environment

For the project I’m working on, I’ve opted for the Java implementation of the Job Server. This implementation’s main page is located [HERE].

Information about the Java Job Server:

Java Gearman Service is an easy-to-use distributed network application framework implementing the gearman protocol used to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages.

04-23-2012 java-gearman-service v0.6 has been released. [DOWNLOAD]

  • The service now uses the slf4j logging facade, allowing the user to have better control over logging
  • Persistent background jobs are now supported though an application hook
  • The API has been updated to be more user friendly, and it makes it easier to create divide-and-conquer/mapreduce applications (breaks the code of previous versions)
  • A .properties file now may be used to set property values and fine-tune the application.
Requirements to deploy the Java job server:
  • Java SE 7
  • slf4j 1.6.4+

For my implementation, I’ve extracted the zip into a vendor directory form where I’ll plan to launch the .jar. Development is occurring on an Apple OSX portable, then deployed to the AWS EC2 cluster for production. It’s expected that some library pathing and configuration will be required to make this all work.

Starting up the Java Gearman Service

It took a little time to locate the instructions for Starting up the Java implementation of the Gearman Service. It is located [HERE].

Instructions on how I started the GearMan server on my OSX development machine are located [HERE].

Once started, you should be able to communicate with it with your Client and Worker code!!

Next steps:

Installing Gearman PHP components

Build a GearMan Client Demonstrator

Build a GearMan Worker Demonstrator