While looking at metrics an monitoring the processing of a CBMHDHS*, I noticed that the active job queue would become quiescent for minutes at a time. Most markedly with the one specific tasklist. My gut told me it was an issue with the way PHP was handling my simple list (700,000 items give or take). So some performance testing was required. What I found was astonishing. PHP’s array_shift is horribly inefficient.
Here is an example to demonstrate this.
When shifting 2500 items (one at a time) off the array (list), it can take 30 or more seconds. Keep in mind this is all in memory! This test is with only 108,000 items:
2013-08-16 18:21:06 # STARTING ARRAY_SHIFT TEST: Q:108581
2013-08-16 18:21:43 # DONE removed 2500 Q:106081
It took 37 seconds, to be exact. To me, that seemed like a long time. Doing a little research I found that when you rewind an array to it’s beginning (using PHP), one of it’s side effects is that it returns the element at that pointer. Reset looks like a scary operation. It does not ‘reset’ the array in the sense of flushing it out, it just resets the pointer to the top. At any rate. the code looked like this:
$element = reset($array);
So, in theory one could use this to get the first element off an array automatically, without removing it. OK, but shift_array does this ANY removes the element. So this is not really the same action. However… (and you know there is a point there), there is another PHP function with a useful side effect. The ‘key() function‘ that returns the key value of at the current pointer (which in this case is the same location we returned the element above). It looks like this:
$key = key($array);
Now, with those two lines of code I have the element (value) and the key at the top of the array. So this is 2 lines of code where array_shift() is just one.. but we’re not even done yet.. we have to perform the most important part (for this exercise) and remove the element. So.. that’s a 3rd line of code like this, right?
unset($array[$key]);
OK.. so taking a line of code right out of the processor, there is a direct comparison. array_shift on the left, reset, key, unset on the right.
| Array Shift method |
Array Reset, Key and Unset method |
$key = array_shift($this->QUEUE)) {
$payload = $this->DATA[$key];
|
$appkey = reset($this->QUEUE);
$payload = $this->DATA[$appkey];
$key = key($this->QUEUE);
unset($this->QUEUE[$key]);
|
What I didn’t expect to find is how dramatically FASTER the more complex (looking) code is. In The numbers don’t lie.. see for yourself.
Array Shift method.
Total time for 2500 items removed is 37 seconds:
2013-08-16 18:21:06 # STARTING ARRAY_SHIFT TEST: Q:108581
2013-08-16 18:21:43 # DONE removed 2500 Q:106081
Array Reset, Key + Unset method.
Total time for 2500 items removed is <1 second:
2013-08-16 18:21:46 # STARTING ARRAY_RESET TEST: Q:108581
2013-08-16 18:21:46 # DONE removed 2500 Q:106081
Those numbers include getting the payload data out of the 2nd array.
Mind blowing in my opinion. It’s not even a contest. Writing your own is at least 15 times faster than PHP’s native operation, that for all intents, accomplishes the same objective.
The point of all this?
If you are seeing an inexplicable slowdown somewhere in PHP.. and you can narrow it down to 1-2 operations… it’s probably worth your while to try to do it another way, even if that way looks more complex!
I could not believe this when I tried it. When I implement this in the processor, I should be able to remove hours of useless waiting to ‘shift’ items off the list. At some point I’d hope Zend would fix this. Keep in mind I ran this test on PHP 5.5.3 (the latest stable release I could get my mitts on, at the time).
*Cloud Based Massivly Horizontal Distributed Harvesting System
