All posts by David

Software Development, Technology

Installing Gearman PHP components for OSX- (less fun that it should be)

2 Comments

gearmanlogo

NOTES: Installing Geaman’s PHP components on OSX is a frustrating and rather complex task. Know this going in. The only way I was finally able to do this was by reading this page here, from one of PHP’s own engineers, got me pretty close but it was still not a full solution. [HERE]. YOU HAVE BEEN WARNED!!

FIRST THINGS FIRST — Get the right versions!!

Do not use gearman.1.1.7!
As of this writing (7-JUN-2013) the current version of gearmand (gearmand-1.1.7) has a bug that prevents it from properly building on OSX. I waste probably 2 days before in a deep corner of the mind I thought.. “If 1.1.7 has a known bug in OSX, and they have not fixed it yet.. let’s try 1.1.6!, and that worked!! The main Gearman download page has multiple versions so just avoid 1.1.7.

Get the lateset PHP source, (gearman-1.1.2) NOT the one linked off the Gearman page!
Yes.. this wasted even more time. The Gearman page didn’t have a quick easy link to the full set of available versions, and the linked version from the page was very much out of date. There is another build bug regarding PHP. One of the engineers decided to get fancy and change the privacy of the objects members somewhere in 1.0.x tree. This BREAKS build on OSX. They did as recently as this year release a FIX for this which is version gearman-1.1.2. All of them can be had on this page [HERE].

Getting Library Dependancies Worked Out

After fighting with source code, screaming at the screen, and even getting completely frustrated with what I was able to (or not able to) install with MacPorts.. I decided to install HomeBrew and give that a run. It’s not a big deal but I moved that to it’s own page located [HERE].

libevent must be built/installed

You’re going to need libevent, and installing it straight up from brew (nor Mac Ports) did the job for me. Check out my previous pages on installing libevent located [HERE] for details on that exciting exercise.

You Must Install Gearmand (sever) regardless

Regardless of how you plan to user Gearman with PHP, you must have the GearmanD server compiled to create the required libraries. There are no two ways about it, just resign yourself to that and keep moving forward!

First, obtain the Gearmand source code for compile from [HERE]. I dropped min in /usr/local

Unball the file and cd into source code directory, configure and build with the following commands:

david$ cd gearmand-1.1.12/
david$ ./configure -disable-shared -prefix=/usr/local
david$ make && make install

A couple of adjunct notes

If you are having problems location the libevent.. or basically seeing this error:

checking test for a working libevent… no
configure: error: Unable to find libevent

Try setting these two environment variables, to tell the configurator exactly where to locate these libraries, if you’ve managed to build libevent from source:

[gearmand]$ export CPPFLAGS=’-I/usr/local/include’
[gearmand]$ export LDFLAGS=’-L/usr/local/lib’

Software Development, Technology

Gearman — Starting the Java-Gearman-Service process

1 Comment

gearmanlogo
These notes apply to testing on a MAC OSX portable, and may or may not apply to your implementation. They are provided as an adjunct to my main Getting Gearman Going post elsewhere in this blog.

The project page for Java-Gearman-Service is located [HERE] on Google Code.

The full set of instructions for staring up Gearman’s Java-Gearman-Service were not clearly linked to the main Gearman project page, so I’m including the link [HERE] to save you the few Google dorkings I did to find it.

Starting up the Java service should be as simple as this:


java -jar java-gearman-service-0.6.6.jar

HOWEVER, I received this instead.. an error:


david$ java -jar java-gearman-service-0.6.6.jar
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/gearman/impl/Main : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Checking my version shows that I am on 1.6 not 1.7 as I had thought;


david$ java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06-451-11M4406)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01-451, mixed mode)

To get the proper version, I navigated to the Oracle page located [HERE], agreed to their terms (do I really have a functional choice… not if I want/need to use Java…) and pushed forward.

If you are running the install for 1.7, you should see a dialog like this:
Screen Shot 2013-06-03 at 2.12.16 PM

That has at least resolved this part of the issue, and will attempt to restart the server.


david$ java -version
java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b12)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)

Now I’m going to restart, specific a custom port and get it kicked off in the background:


david$ java -jar java-gearman-service-0.6.6.jar -p6315 &

At this point the process is running on my Job Service box and next step will be to craft some code to see how it all works.

Software Development, Technology

Deploying Java Gearman Job Server

Leave a comment

gearmanlogo
OK, after more than a year it’s time to get down to really building out a Gearman system.

I’ve recently taken on a project that I believe to be the perfect fit for the pipelined distributed work manager of Gearman. I’m of course, not at liberty to discuss the various details of this project, but I can provide some high-level description for the purposes of justification.

The Project – distributed harvesting

The objective of the project is to provide a distributed method of web page scraping and parsing. This project requires that the scraping and profiling occur for 2,000,000+ websites in under 18 hours. No small feat for certain. The good news is that I’ve built a systems in the past (circa 2006) that did just this using MySQL as the task manager. It worked, but it had it’s issues, and almost every single one of them can be mitigated by using Gearman. The rest will be mitigated with the application of NoSQL solutions for site list management.

What is Gearman

Here is the synopsis from the Gearman.org main page.

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates.

The Job Server — implementing in Amazon’s ES2 environment

For the project I’m working on, I’ve opted for the Java implementation of the Job Server. This implementation’s main page is located [HERE].

Information about the Java Job Server:

Java Gearman Service is an easy-to-use distributed network application framework implementing the gearman protocol used to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages.

04-23-2012 java-gearman-service v0.6 has been released. [DOWNLOAD]

  • The service now uses the slf4j logging facade, allowing the user to have better control over logging
  • Persistent background jobs are now supported though an application hook
  • The API has been updated to be more user friendly, and it makes it easier to create divide-and-conquer/mapreduce applications (breaks the code of previous versions)
  • A .properties file now may be used to set property values and fine-tune the application.
Requirements to deploy the Java job server:
  • Java SE 7
  • slf4j 1.6.4+

For my implementation, I’ve extracted the zip into a vendor directory form where I’ll plan to launch the .jar. Development is occurring on an Apple OSX portable, then deployed to the AWS EC2 cluster for production. It’s expected that some library pathing and configuration will be required to make this all work.

Starting up the Java Gearman Service

It took a little time to locate the instructions for Starting up the Java implementation of the Gearman Service. It is located [HERE].

Instructions on how I started the GearMan server on my OSX development machine are located [HERE].

Once started, you should be able to communicate with it with your Client and Worker code!!

Next steps:

Installing Gearman PHP components

Build a GearMan Client Demonstrator

Build a GearMan Worker Demonstrator

Software Development, Technology

macports.conf points to a non-existing directory.

1 Comment

QUICK TIP:

If you receive the following error while attempting to install MacPorts, there is a quick and easy fix:

Warning: Your developer_dir setting in macports.conf points to a non-existing directory. Since this is known to cause problems, please correct the setting or comment it and let macports auto-discover the correct path.

The following commands will let you locate the current path, then set this into the macport.conf file, making installation of your MacPorts package more robust.

First, determine the actual path on your machine. Your path may vary. The path shown below is for my OSX 10.7 box.


xcode-select --print-path
/Applications/Xcode 4.3.3.app/Contents/Developer

Next, edit the macports.conf file, replacing the current entry with the one you just found above:


sudo vi /opt/local/etc/macports/macports.conf

[...]

# Directory containing Xcode Tools
#developer_dir /Developer
developer_dir /Applications/Xcode 4.3.3.app/Contents/Developer

[...]

Once you have done that, when you run the installation command, you’ll no longer see that annoying warning, and your MacPorts should be installed, ready for you to enjoy!


sudo port -c install coreutils +with_default_names

Santa Cruz, Technology, YouTube

Dash Cam Test Run – Santa Cruz to SJC, CA

1 Comment

Sunday afternoon, I mounted my new $25 dash cam (purchased 2 of them from Amazon, $25 each to my door, I figured why not!) inside the windshield of my 2002 BMW ///M3 Convertible, and drove from Santa Cruz to San Jose (SJC airport) California.

Highway 17, has earned this dubious distinction as the most dangerous highway in California. With it’s steep decent from it’s 1800′ Santa Cruz Mountains summit to the Santa Clara Valley Floor below, it’s often this site of injury and fatality crashes.

Audio in the clip is horrible, and unedited. Wind noise due to the top on car being down, plus the fact I was wasting no time on “The Hill“.

While growing up in “The Valley”, heading over the hill to Santa Cruz always seemed like such a journey. Families would pack up food and drink to make it over the mountain. In reality, it’s a fairly short dive (about 20 miles) but back in the 70’s when I grew up, summer time temps well into the 90’s, combined with the fairly steep ascent to the summit (at 1800′) meant there was a pretty good chance you’d be taking a break in one of the turnouts while the car’s radiator cooled off enough to continue the drive. Using A/C on the hill was basically an impossibility, unless you enjoyed being temporarily stranded on the side of the highway while possibly awaiting a tow hook.

As I grew older, cars were built better, and I was able to afford these better cars, the drive to the beach became less and less an practice of gambling upon one’s luck, and simply being able to afford a 1/2 tank of gas on a high school or college student’s budget.

photography, Santa Cruz, Sports, travel

Steamer Lane Surfing Pics – Santa Cruz, CA

Leave a comment

Between storms, the waves really picked up at Steamers Lane, Santa Cruz, CA.


View Larger Map

Steamer’s Lane has a rich surfing history. It is just off a point on the side of cliffs in the West Cliff residential area near downtown Santa Cruz, providing a good opportunity to view the surfing. The Santa Cruz Surfing Museum is in the lighthouse there. “Steamer Lane” is the preferred form of the name used by the locals. It was at Steamer Jack O’Neil invented the modern surfing wetsuit, “Just because he wanted to surf longer”.

I went out to capture some of the Saturday action:

photography, Santa Cruz, travel

Pacific Storm slams Santa Cruz, CA

Leave a comment

It was an ugly, ugly morning in Santa Cruz.

Following almost a week of non-stop unsettled weather, Mother Nature decided to give us a little reminder of how insignificant and inconsequential we are in the grand scheme of things.

Obviously, this is nothing like the horrid disaster of Hurricane Sandy, but and angry sea, is, an angry see.

Here are some photos takes along West Cliff Drive in Santa Cruz on the 2nd of December. The sunlight was a horrible blue from the deep clouds and reflection out of the ocean. I did the best I could do on the color correction:

Santa Cruz Storm Waves - 2-Dec-2012
Santa Cruz Storm Waves – 2-Dec-2012
Santa Cruz Storm Waves - 2-Dec-2012
Santa Cruz Storm Waves – 2-Dec-2012
Santa Cruz Storm Waves - 2-Dec-2012
Santa Cruz Storm Waves – 2-Dec-2012
Santa Cruz Storm Waves - 2-Dec-2012
Santa Cruz Storm Waves – 2-Dec-2012
Santa Cruz Storm Waves - 2-Dec-2012
Santa Cruz Storm Waves – 2-Dec-2012
Santa Cruz Storm Waves - 2-Dec-2012
Santa Cruz Storm Waves – 2-Dec-2012
photography, Sports

Some photos from the O’Neil Coldwater Classic

Leave a comment

Granted, it’s been a couple of weeks since the pros packed up and headed off to Pipeline for the last event on their years tour. I’m not always the quickest to get photos uploaded, with clients to service, mouths to feed, cars and bikes to work on and in there somewhere I try to have a little fun for myself.

So this morning I’m pulling up some pics from a few weeks ago that I’ve wanted to post up. One of these days I’ll get my watermarking done, but for now.. I’ll just have to trust that nobody finds these compelling enough to steal (here’s hoping).

Health, travel

Santa Cruz – a Day in the Life

12 Comments
8:00 AM
  • Woke up to a sunny morning
  • Spoke to my son about he latest developments in his Minecraft adventures
8:30 AM
  • Researched local prices on BMW M3 convertibles. Not ready to pull the trigger, YET.
  • Got pricing and installation instructions for trailer hitch on the ML320
  • Started research on attending next year’s Isle of Man TT

  • Hugged everyone in the house
9:15 AM
  • Hopped on mountain bike and rode Wilder Ranch
  • Noticed my tires were a little on the flat side and decided to not go full agro
  • Checked out the coastal loop trail for the first time

  • Watched seals play in the surf, and dolphins swim by for about 15 minutes
  • Marveled at the beauty and splendor that is California
11:30 AM
  • Walked down to the pool, slipped into the hot tub
  • Sunned poolside and mused about the fact I feel like I’m vacation every day here
  • Watched Monarch butterflies fly by the pool, expressed need to walk down to beach and see if they have officially arrived
  • Reaffirmed the strong desire to buy a two seat convertible, even though it’s totally impractical with 2 kids
1:30 PM
  • Returned home, relaxed, agreed that we need to go to Costco
  • Looked for clothes to wear, considered shoes for about 3 seconds, decided that would be lame
1:45 PM
  • Costco run to get veggies for juicing
  • Checked out the Food Bin for organic foods to cook for dinner (success!)
3:15 PM
  • Hiked down to the beach and checked out the incoming Monarch Butterfly migration, in the eucalyptus grove
  • Checked out the many butterflies resting in the trees, and flying about the grotto
5:00 PM
  • Dinner preparations underway
  • Opened a Rothchilds Cabernet for dinner
6:15 PM
  • Organic pasta (whole wheat) dinner with a sausage, mushroom and onion red sauce
  • Discussed the wonders of the day
7:00 PM
9:15 PM
  • Retired the crew to their quarters, myself included
  • Turned on iTunes on the Apple iMAC, streaming ‘On the Edge Radio‘ from Seattle, set the sleep timer to 90 minutes
9:18 PM