Tag Archives: Technology

STRATUX – A WIFI CONNECTED GROUND STATION (2020)

It’s 2020, we’re deep in the throws of the COVID-19 PanicDemic, so I decided to build a refreshed receiver. Once completed, this fixed-station Stratux receiver will double as a FlightRadar24 feed node.

The initial effort was quite successful, so, I’m taking recommended ‘Social Distancing’ mandates to re-document the setup process I used to accomplish the following:

Imaging Stratux

The first step was performing a default STRATUX install to a micro SD card. I selected a 16GB card for this project, but Stratux will run on anything down to 4 GB, based on what I observed with the default file system (default size is under 2GB).

I won’t repeat the installation instruction here, since they might have changed since this article was written. The instructions I used, and link to the latest Pi Image are located here: [ STRATUX.me ]

Preliminary Work

Once the card was imaged, I plugged in an HDMI cable, keyboard, mouse and Ethernet cable to start work.

Stratux Pi preliminary setup

Once powered up and the boot sequence completes, your are challenged with a simple login prompt. The initial login is: pi and the password is raspberry.

Once you login, you’ll see this welcome screen (sorry for the crappy photo.. if I can figure out how to PiP the HDMI output to my Mac to snap a good screen shot, I’ll re-do this!

Stratux first login

You will WANT to change that default password first!

Changing Default Password

Switch user to root and change the pi user’s password. You do not NEED to be root to change your own password, but we’ll need to be root from here on out.. and why not just do it that way?
pi@raspberrypi: sudo su -

root@raspberrypi: passwd pi
Enter new UNIX password: enter your password here
Retype new UNIX password: re-enter your password here

Next, will be the modifications to disable the adhoc network and enable local WiFi connectivity.

Disable adhoc ‘stratux’ WiFi

Now that you are logged into the Stratux via ssh, assume root user and then cd to the networking directory again:
pi@raspberrypi:~ $ sudo su -

root@raspberrypi:~# cd /etc/network

root@raspberrypi:~# vi interfaces

Now, this time the interfaces file will be heavily modified to configure the eth0 and wlan0, along with some rational routing, using the ‘metric’ setting to prioritize route assignments.

At the top of the file a new line will be added:
auto lo
auto wlan0

iface lo inet loopback

Then replacing the wlan0 entries with the following block of text. NOTE: The line `wireless-power off` will disable the power management for WiFi. If this is not done, I have found that the chip will shut down after a couple of hours and the device will become unreachable via wifi, until it’s rebooted. This discovery took no short amount of time to discovery and remedy.

iface lo inet loopback

iface lo inet loopback

iface wlan0 inet static
   metric 0
   wireless-power off
   hostname Stratux-eth0
   wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
   address 10.100.0.210
   netmask 255.255.255.0
   gateway 10.100.0.1
   network 10.100.0.0
   broadcast  10.100.0.255
   dns-nameservers 10.100.0.1 8.8.8.8 

Configure the wlan0 WiFi

NOTE!: As of 19-APR-2020 and Raspberry Pi 3b; 5.0GHz Wifi IS NOT SUPPORTED.

Now.. the really fun part.. connecting the Pi to your WiFi network. You will need to find the SSID you want to connect to and the password for that network before starting.

The networking configuration that you setup in the previous step contains this setting: ‘wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf‘. The next step is to create/configure that file.

The file is fairly simple, and with the SSID and Password you already obtained for your network (you did that already.. right?). You’ll simple fill that information into the blanks named _SSID_ and _NETWORK_PASS_.

Open the file and edit:
root@raspberrypi:~# vi /etc/wpa_supplicant/wpa_supplicant.conf

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

Add this block below the original lines, using your settings. _SSID_ for the ID of the 2.4GHz network you are connecting too, and _NETWORK_PASS_ will be the plain-text password for that network. NOTE: For more security you can configure this block to use an encrypted password. That is beyond the scope of this basic tutorial; there are many pages that detail that process if you want to implement it.

network={
        ssid="_SSID_"
        psk="_NETWORK_PASS_"
}

Save this file, restart networking, signal a daemon reload, and then, reboot of the device. Once the networking is back up, check to make sure you are able to contact the machine via it’s WiFi connection.

root@raspberrypi:~# systemctl networking restart
root@raspberrypi:~# systemctl daemon-reload
root@raspberrypi:~# reboot

After 2-3 Min. the Pi should have completed restarting and you should be able to verify a successful WiFi login.
IngeniiGroup:STRATUX$ ssh pi@192.168.1.210

Increasing Root Filesystem Space

By default, the size of the disk partitions on the image is VERY small; less than 2GB. Most of that space is used by by the base Stratux install, leaving about 400MB of space to save logs (and your replay database if you turn it on). You can see this with the ‘df’ command:

root@raspberrypi:~# df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       1.8G  1.4G  324M  81% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           463M     0  463M   0% /dev/shm
tmpfs           463M   12M  451M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           463M     0  463M   0% /sys/fs/cgroup
/dev/mmcblk0p1   60M   20M   41M  34% /boot

This just isn’t enough space, especially for the ‘/var/log’ directory where a lot of transient/logging data is written. My solution is to create a disk partition and mount it to `/var/log`.

Locate the disk device

Instructions on the web are not exactly correct, some suggest /dev/sda as the main device, however my testing shows it’s actually this named ‘/dev/mmcblk0’.
root@raspberrypi:~# fdisk -l | grep Disk
[...]
Disk /dev/mmcblk0: 14.5 GiB, 15523119104 bytes, 30318592 sectors

… with the following partitions:Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 131071 122880 60M c W95 FAT32 (LBA)
/dev/mmcblk0p2 131072 3887103 3756032 1.8G 83 Linux

Running fdisk

With the physical partition located.. start fdisk:
fdisk -u /dev/mmcblk0

Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

First, order of business is to increase the size of the main partition, to give it a big more room than just 1.8GB. I like to bump it up to around 4GB to leave room for installing more system updates and tools. To do this you will need to know the starting and ending blocks of the partition. That is available with the ‘print’ command:
Command (m for help): p

Disk /dev/mmcblk0: 14.5 GiB, 15523119104 bytes, 30318592 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe6a544c8

Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 131071 122880 60M c W95 FAT32 (LBA)
/dev/mmcblk0p2 131072 3887103 3756032 1.8G 83 Linux

Now delete the partition. Yes.. feels VERY dangerous.. and it is.. but as long as the starting block is maintained, and the end block number is increased.. this will end up being a safe operation. Verify that you targeted the correct partition by using ‘p’ again:
Command (m for help): d
Partition number (1,2, default 2): 2

Partition 2 has been deleted.

Command (m for help): p
Disk /dev/mmcblk0: 14.5 GiB, 15523119104 bytes, 30318592 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe6a544c8

Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 131071 122880 60M c W95 FAT32 (LBA)

If this is correct, now recreate it with same number (2), start and type but with a bigger end (taking care not to overlap with other partitions). Try to align things on a megabyte boundary that is for end, make it a multiple of 2048 minus 1. Change the type if needed with t (for partitions holding an extX or btrfs filesystem, the default of 83 is fine). Then `w` to write and `q` to quit.
Command (m for help): n
Partition type
p primary (1 primary, 0 extended, 3 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2): 2
First sector (2048-30318591, default 2048): 131072
Last sector, +sectors or +size{K,M,G,T,P} (131072-30318591, default 30318591): 8451072

Created a new partition 2 of type ‘Linux’ and of size 4 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

root@raspberrypi:~#

The partition table will have been modified but the kernel will not be able to take that into account as some partitions are mounted.

However, if in-use partitions were only enlarged, you should be able to force the kernel to take the new layout with:
root@raspberrypi:~# partx /dev/mmcblk0
NR START END SECTORS SIZE NAME UUID
1 8192 131071 122880 60M e6a544c8-01
2 131072 8451072 8320001 4G e6a544c8-02

If the command works the next step is to expand the filesystem. In my case I needed to reboot before the kernel picked up the new partition size, despite running partx to fill up this new space.
root@raspberrypi:~# init 6

Following the restart, execute `resize2fs` and run an on-line expansion of the filesystem, and finally verify it again with ‘df -h’
root@raspberrypi:~# resize2fs /dev/mmcblk0p2
resize2fs 1.42.12 (29-Aug-2014)
Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mmcblk0p2 is now 1040000 (4k) blocks long.

root@raspberrypi:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 3.9G 1.4G 2.4G 36% /
devtmpfs 459M 0 459M 0% /dev
tmpfs 463M 0 463M 0% /dev/shm
tmpfs 463M 6.2M 457M 2% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 463M 0 463M 0% /sys/fs/cgroup
/dev/mmcblk0p1 60M 20M 41M 34% /boot

The next step is to add a 3rd partition which will then be mounted to `/var/log`

Creating a dedicated filesystem for logging / database

I ended up creating 3 primary partitions, the largest of which will be mounted to `/var/log`.
fdisk /dev/mmcblk0

Command (m for help): n
Partition type
p primary (2 primary, 0 extended, 2 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (3,4, default 3): 3
First sector (2048-30318591, default 2048): 8451073
Last sector, +sectors or +size{K,M,G,T,P} (8451073-30318591, default 30318591): 30318591

Created a new partition 3 of type ‘Linux’ and of size 10.4 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

root@raspberrypi:~# partprobe

Checked to make sure the device was crated by checking the `/dev` directory:
root@raspberrypi:~# ls -l /dev/mmcblk0*
brw-rw---- 1 root disk 179, 0 May 15 15:11 /dev/mmcblk0
brw-rw---- 1 root disk 179, 1 May 15 15:11 /dev/mmcblk0p1
brw-rw---- 1 root disk 179, 2 May 15 15:11 /dev/mmcblk0p2
brw-rw---- 1 root disk 179, 3 May 15 15:11 /dev/mmcblk0p3

Next, put a filesystem on this new partition. Using df to determine the type of filesystem currently in use; I recommend that you stick with it for this most basic of operations:
root@raspberrypi:~# df -T

Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/root ext4 4063680 1392604 2512268 36% /
devtmpfs devtmpfs 469688 0 469688 0% /dev
tmpfs tmpfs 474004 0 474004 0% /dev/shm
tmpfs tmpfs 474004 6340 467664 2% /run
tmpfs tmpfs 5120 4 5116 1% /run/lock
tmpfs tmpfs 474004 0 474004 0% /sys/fs/cgroup
/dev/mmcblk0p1 vfat 61384 20416 40968 34% /boot

Run mkfs to initialize the filesystem.
/sbin/mkfs -t ext4 /dev/mmcblk0p3

Creating filesystem with 2733439 4k blocks and 684096 inodes
Filesystem UUID: 94f004af-7008-4dbe-8805-3eb2d739436b
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information:... this might go on for a bit..

Once completed.. mount this where the logs and databases live. To do this the first thing that needs to happen is to check your current fstab:
cat /etc/fstab
proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults 0 2
/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for that

My first order of business was to copy the current `/var/log` to a new location, create a new clean mount point for `/var/log` and then mount the new filesystem, and then verify it’s mounted using `df -h`
root@raspberrypi:~# mv /var/log /var/log2
root@raspberrypi:~# mkdir /var/log
root@raspberrypi:~# mount -t ext4 /dev/mmcblk0p3 /var/log
root@raspberrypi:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 3.9G 1.4G 2.4G 36% /
devtmpfs 459M 0 459M 0% /dev
tmpfs 463M 0 463M 0% /dev/shm
tmpfs 463M 6.2M 457M 2% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 463M 0 463M 0% /sys/fs/cgroup
/dev/mmcblk0p1 60M 20M 41M 34% /boot
/dev/mmcblk0p3 11G 27M 9.6G 1% /var/log

Edit the fstab file to create a mount point for the new partition where the logs used to be written (added the orange line), and ran mount to verify that it will automount on a restart.
root@raspberrypi:~# vi /etc/fstab

proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults 0 2
/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
/dev/mmcblk0p3 /var/log ext4 defaults,noatime 0 0
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for that

root@raspberrypi:~# mount -a

Restart and verify

Restart the little box and verify that the mount was preserved.
init 6

Log back in, and run df to check the filesystem health. It should now has the the main filesystem has some breathing room again:
pi@raspberrypi:~ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 3.9G 1.4G 2.4G 36% /
devtmpfs 459M 0 459M 0% /dev
tmpfs 463M 0 463M 0% /dev/shm
tmpfs 463M 6.2M 457M 2% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 463M 0 463M 0% /sys/fs/cgroup
/dev/mmcblk0p1 60M 20M 41M 34% /boot
/dev/mmcblk0p3 11G 27M 9.6G 1% /var/log

Setting Time with NTPD

If you do not have a GPS receiver attached to your Stratux, then it might not be able to determine proper system time. When this is the case, enabling ntpd will be your solution. Using national time sync services, it will keep your system clock correct. If you have your GPS plugged in, it’s getting really good timing signals already.. or the location function would not work at all! For the non GPS users:

Install ndptdate:
root@raspberrypi:~# apt-get install ntpdate
Reading package lists... Done
Building dependency tree
Reading state information... Done
[...]
Do you want to continue? [Y/n] Y
Get:1 http://mirrordirector.raspbian.org/raspbian/ jessie/main liblockfile-bin armhf 1.09-6 [18.2 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ jessie/main liblockfile1 armhf 1.09-6 [14.7 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ jessie/main ntpdate armhf 1:4.2.6.p5+dfsg-7+deb8u2 [69.0 kB]
[...]
Setting up ntpdate (1:4.2.6.p5+dfsg-7+deb8u2) ...
Processing triggers for libc-bin (2.19-18+deb8u3) ...

It should now start at bootup and resolve any timing issues you might have.

Update your Pi with the latest updates and security patches

Some might remember “Black Friday” when a worm created some serious disruption in the tech world.. including at hospitals in the UK. And it happened primarily because people are not applying their security patches! Although the risk of your Pi being botified and ransomed my not be high.. you should be updating it regardless! So, let’s do that now.
root@raspberrypi:~# apt-get install ntpdate
root@raspberrypi:~# apt-get dist-upgrade

Finish up with this command to clean up some of the used disk space. Since you’ve already bumped the numbers on your partitions in the previous steps.. this is not nearly as necessary, but why leave unused stuff lying around? Your mother taught you to clean up afteryourself, right?
root@raspberrypi:~# sudo apt-get clean

AND THAT DOES IT FOR THIS EPISODE!

STRATUX – Filesystem Full; Managing disk space redux

Checking in with my Starux project this morning, I found it unresponsive. A physical check shows a flashing red light on the Pi… something has gone haywire, and I couldn’t SSH into the little thing, so a really hard cold restart was in order.

Following the restart I quickly shut Stratux back down to start performing diagnostics:


pi@raspberrypi:~ $ sudo su -

root@raspberrypi:~# service stratux stop

Diagnostics 101

Check Filesystem Health

The last time the little Stratux suddenly had problems it was a filesystem space issue ( previous article ). It turns out the boot drive space was OK but the new Logging partition was again, completely consumed:

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        1815440 1391284    331164  81% /
devtmpfs          469688       0    469688   0% /dev
tmpfs             474004       0    474004   0% /dev/shm
tmpfs             474004    6340    467664   2% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             474004       0    474004   0% /sys/fs/cgroup
/dev/mmcblk0p1     61384   20400     40984  34% /boot
/dev/mmcblk0p4   8125880 8109496         0 100% /var/log

Locate the Culprit

There are a number of ways to locate large file on a *NIX system. My favorite tool is find. First thing I want to do is locate any file that is larger than 1 Gigabyte, and sure enough it located a massive sqlite database file. The same one that ate up all the space on the boot drive. So.. this is going to require some more extrodinary measures to maintain 100% 24×7 operational status.


root@raspberrypi:~# cd /var/log
root@raspberrypi:/var/log# find . -size +1G -exec ls -l {} \;
-rw-r--r-- 1 root root 7463211008 Apr 29 21:48 ./stratux.sqlite
Start Solving

First order of business is to move aside the massive database, but try to preserve the data for examination. Since the filesystem is full, I can’t zip this thing in place, so first some space needs to cleared on the device. The things I’m least interested in go first.. like the zipped syslogs and any other ‘archived’ file (those with a .# suffix).

-rw-r----- 1 root adm      728756 Apr 25 06:25 syslog.5.gz
-rw-r----- 1 root adm     2552845 Apr 26 06:25 syslog.4.gz
-rw-r----- 1 root adm     2447263 Apr 27 06:25 syslog.3.gz
-rw-r----- 1 root adm     2498089 Apr 28 06:25 syslog.2.gz
-rw-r----- 1 root adm    33735478 Apr 29 06:25 syslog.1
-rw-r----- 1 root adm       54307 Apr 30 05:19 debug.1
-rw-r----- 1 root adm      380928 Apr 30 05:19 kern.log.1
-rw-r----- 1 root adm   332365824 Apr 30 06:18 daemon.log.1
-rw-r----- 1 root adm       48723 Apr 30 06:25 auth.log.1
-rw-r----- 1 root adm           0 Apr 30 06:25 syslog.1.gz
-rw-r----- 1 root adm     1032192 Apr 30 06:25 messages.1

root@raspberrypi:/var/log# rm -f *.gz *\.[0-9]

But.. that’s not going to doe the complete trick, especially if after deleting files df still shows 100% utilization. You need to figure out what is holding which deleted file(s).

Normallyh, the best way to do that on *NIX is with lsof. Much to my chagrin, it was not available on the OS… so I had to go grab it. Thankfully I had the main filesystem on a different partition that still had enough space to install more tools! Once lsof was installed, grep through the list of open file handles and find those marked for delete.. and those are the processes that have handles pointing to those files. However, this didn’t help me with Rasperian Jessie. I know that a reboot will recover the space so.. that was the next step.. a brutal warm-boot.


root@raspberrypi:/var/log# lsof
-su: lsof: command not found

root@raspberrypi:/# apt-get install lsof
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libperl4-corelibs-perl
The following NEW packages will be installed:
libperl4-corelibs-perl lsof
...

root@raspberrypi:/var/log# lsof | grep deleted
root@raspberrypi:/var/log#

root@raspberrypi:/var/log# init 6

Archiving Massive SQLite Database

The first thing was to move aside the current database, and then restart stratux to verify it can created a new empty database for it’s purposes.. then shut it right back down again.


root@raspberrypi:/var/log# mv stratux.sqlite stratux.sqlite.1
root@raspberrypi:/var/log# service stratux start
root@raspberrypi:/var/log# ls -l
total 7291480
[...]
-rw-r--r-- 1 root root 4096 Apr 30 14:27 stratux.sqlite
-rw-r--r-- 1 root root 7466160128 Apr 30 14:24 stratux.sqlite.1
[...]
root@raspberrypi:/var/log# service stratux stop

A new empty database file has been created [ 4096 Apr 30 14:27 stratux.sqlite ]. This tells me that moving aside the current database file on a periodic basis, compressing and then archiving it should be sufficient to maintain operational status.

Logging Insanity

Before restarting Stratux, I zerod out these log files. Running for first a few moments these files were already reading up a lot of space. Tailing one of them I see that Stratux, with my current settings is logging A LOT of data to these log files. I feel this was my first error.. enabling too much logging. My settings look like this:

With those settings enabled, there are a lot of GPS and other events that I don’t really have a use for, being dumped into the Statux log.


-rw-r--r-- 1 root root 1361778 Apr 30 14:39 stratux.log

Turning OFF ‘Verbose Message Log’ made that insanity stop.

Replay logging is what is writing to the SQLite database. So the question is. how much of that data do I want to keep, and how much will I lose if I turn off the replay logs. I think that will be research for another day… right now the goal is to recover disk space by compressing the massive database file that was moved aside, and get Stratux stabilized again. Once compressed, check filesystem and file size!


root@raspberrypi:/var/log# gzip stratux.sqlite.1

root@raspberrypi:/var/log# df

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        1815440 1392384    330064  81% /
devtmpfs          469688       0    469688   0% /dev
tmpfs             474004       0    474004   0% /dev/shm
tmpfs             474004   12272    461732   3% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             474004       0    474004   0% /sys/fs/cgroup
/dev/mmcblk0p4   8125880  681792   7008276   9% /var/log
/dev/mmcblk0p1     61384   20400     40984  34% /boot

root@raspberrypi:/var/log# ls -lktr --color --block-size=M stratux.sqlite*

-rw-r--r-- 1 root root 643M Apr 30 14:24 stratux.sqlite.1.gz
-rw-r--r-- 1 root root   8M Apr 30 15:21 stratux.sqlite
-rw-r--r-- 1 root root   1M Apr 30 15:22 stratux.sqlite-shm
-rw-r--r-- 1 root root   5M Apr 30 15:22 stratux.sqlite-wal

With the file compressed, it could be copied elsewhere for analysis.

Is Google looking at a rough 2015?

Screen Shot 2014-12-18 at 10.09.15 AMInteresting read about possibly looming troubles for Google. I will say that in the past I used Google to look for products, but most of the items I found that way were from shaky looking distributors, or links to Amazon, where I found they had a very competitive price.

Perception is reality, my personal perception is that Amazon is a trustworthy enough for me to buy from them. Over the last few months I’ve simply quit Googling for products and checked Amazon first, and only using Google if I felt that Amazon didn’t offer the product or the price was more than I wanted to pay.


Google’s stocks have taken a dive recently. It was a rocky 2014 but the last month has seen a nose dive in stock trading value:
Screen Shot 2014-12-18 at 10.12.44 AM

That’s not all. As the Mercury News (headquartered in Silicon Valley) reported last month, FireFox has dropped Google as it’s default search engine:
http://www.mercurynews.com/business/ci_26971412/firefox-drops-google-yahoo-default-search-engine

Here is a link to an opinion piece on LinkedIn that discusses this further:

https://www.linkedin.com/pulse/googles-very-rough-transition-nicholas

Installing Gearman PHP components for OSX

Locating the latest PHP Components

The Gearman.org page has links to the PHP code on the Downloads page, however the link is very old. The latest code is located at: http://pecl.php.net/package/gearman.

As of 23-OCT-2014, the current stable version is gearman-1.1.2.

I like to drop these files in my /opt directory, and work on them there and unball the package.

mv ~/Downloads/gearman-1.1.2.tgz /opt/.
tar xvzf gearman-1.0.2.tgz
cd gearman-1.0.2

Configuring for Build

The following commands prepared the PHP package to build on OSX Yosemite (10.10).

phpize
Configuring for:
PHP Api Version: 20121113
Zend Module Api No: 20121212
Zend Extension Api No: 220121212

./configure
checking for grep that handles long lines and -e… /usr/bin/grep
checking for egrep… /usr/bin/grep -E
checking for a sed that does not truncate output… /usr/bin/sed
[…]
appending configuration tag “CXX” to libtool
configure: creating ./config.status
config.status: creating config.h

Building the Library

Next step is to run the compile and install the built objects:

make
/bin/sh /opt/gearman-1.1.2/libtool –mode=compile cc -I. -I/opt/gearman-1.1.2 -DPHP_ATOM_INC -I/opt/gearman-1.1.2/include -I/opt/gearman-1.1.2/main -I/opt/gearman-1.1.2 -I/usr/include/php -I/usr/include/php/main -I/usr/include/php/TSRM -I/usr/include/php/Zend -I/usr/include/php/ext -I/usr/include/php/ext/date/lib -I/usr/local/include -I/usr/local/include -DHAVE_CONFIG_H -g -O2 -Wall -c /opt/gearman-1.1.2/php_gearman.c -o php_gearman.lo
mkdir .libs
[…]
Build complete.
Don’t forget to run ‘make test’.

make install
Installing shared extensions: /usr/lib/php/extensions/no-debug-non-zts-20121212/

Telling PHP about gearman

You will need to identify your relevant php.ini file, and edit it, letting PHP know where the library file are located.

Typically under OSX, this file does not exist, and it must be created.

Edit the file:

vi /etc/php.ini

Either way, make sure these two lines are in the file:

Add these lines:

include_path=.:/mnt/crawler
extension=gearman.so

DONE

At this point you should be able to reference Gearman library in your PHP code.

These lines of code, should not throw an error:

$client = new GearmanClient(); // instance
$worker = new GearmanWorker(); // instance

Netflix tries to make nice with customers

As some of you might have read in my Tech Feeds today, Netflix has backed off form the Quickster project.

Here an e-mail I just received from them. Nice Mea Culpa. “Uh.. er.. sorry.. we’re sorry.. prices still going to be same but no new DVD only website.”

Dear David,

It is clear that for many of our members two websites would make things more difficult, so we are going to keep Netflix as one place to go for streaming and DVDs.

This means no change: one website, one account, one password…in other words, no Qwikster.

While the July price change was necessary, we are now done with price changes.

We’re constantly improving our streaming selection. We’ve recently added hundreds of movies from Paramount, Sony, Universal, Fox, Warner Bros., Lionsgate, MGM and Miramax. Plus, in the last couple of weeks alone, we’ve added over 3,500 TV episodes from ABC, NBC, FOX, CBS, USA, E!, Nickelodeon, Disney Channel, ABC Family, Discovery Channel, TLC, SyFy, A&E, History, and PBS.

We value you as a member, and we are committed to making Netflix the best place to get your movies & TV shows.

Respectfully,

The Netflix Team

Some links from my Tech needs today:

Netflix Kills Qwikster

Netflix drops unpopular Qwikster DVD plan

Some history on the controversy, this one about 3 weeks ago:

Netflix Creates Qwikster For DVD Only Business

Setting up a custom ‘Weather Reader’ (Part 3) – Using Data


<– Part 2

It would be short work to grab the current observations from the local station’s file, then grab the full projected forecast from the Zone’s XML formatted data file (example for Bremerton). I checked the full forecast XML, and unfortunately it does not include the local current conditions. In the grand scheme, this is not a big deal since I’ll need to grab three forecast files already. One more file for current observations is not that much more of a burden.

This section contains the Maximum Forecast Temperatures, covering outlook:

<temperature type="minimum" units="Fahrenheit" time-layout="k-p24h-n7-1">
<name>Daily Minimum Temperature</name>
<value>40</value>
<value>45</value>
<value>43</value>
<value>44</value>
<value>43</value>
<value>43</value>
<value>42</value>
</temperature>

This section contains the Minimum Forecast Temperatures, covering outlook:

<temperature type="maximum" units="Fahrenheit" time-layout="k-p24h-n7-2">
<name>Daily Maximum Temperature</name>
<value>59</value>
<value>61</value>
<value>60</value>
<value>61</value>
<value>61</value>
<value>60</value>
<value>60</value>
</temperature>

Also in the file is the projected percentage of precipitation:

<probability-of-precipitation type="12 hour" units="percent" time-layout="k-p12h-n14-1">
<name>12 Hourly Probability of Precipitation</name>
<value xsi:nil="true"/>
<value xsi:nil="true"/>
<value xsi:nil="true"/>
<value xsi:nil="true"/>
<value xsi:nil="true"/>
<value>30</value>
<value>30</value>
<value xsi:nil="true"/>
<value xsi:nil="true"/>
<value xsi:nil="true"/>
<value xsi:nil="true"/>
<value xsi:nil="true"/>
<value xsi:nil="true"/>
<value xsi:nil="true"/>
</probability-of-precipitation>

The XML also is kind enough to include the image names.   Based on this, the actual visually rendered page is a style sheet applied to the XML data, providing a human readable web page <i>(something I need to get more familiar with)</i>.

<conditions-icon type="forecast-NWS" time-layout="k-p12h-n14-1">
<name>Conditions Icon</name>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nbkn.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/sct.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nsct.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/bkn.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nbkn.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/shra30.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nshra30.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/shra.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nshra.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/shra.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nshra.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/shra.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nshra.jpg</icon-link>
<icon-link>http://www.nws.noaa.gov/weather/images/fcicons/shra.jpg</icon-link>
</conditions-icon>

Design and coding of the file downloads, processing and construction of the status file is the next step.  One thought I have, is to download and store locally, all the images.  This will keep my page off of their webserver’s request log.  External includes of images can be a burden on a remote server, if you start to abuse it.  No sense in being a bad netizen.  The question that I need to answer is, <i>CAN I</i> get a list of all the potential images (and there are <i>many</i>) and automate the download to system I plan to run this little product upon.

Automation

Automation of the XML downloading and parsing, will be handled by PERL (at least in this first tool, it may be converted to JAVA on a second iteration, once I’ve performed usability testing of this first iteration).

Additional binary libraries are required first:

apt-get install libxml2-dev

So, here is the skinny. First, have to move into the location where CPAN unravels the packages:

cd /root/.cpan/build

Next, move into the directory for XML::LibXML and manually build with these steps:

cd XML-LibXML-1.70-OucP9U
perl Makefile.PL
make
make install

Next build the XML::Atom package:

cd XML-Atom-0.37-pcVvAG/
perl Makefile.PL
make
make install

Next, a few PERL modules must first be installed. NOTE: the XML::SAX modules are suggested, to provide a big boost in XML parsing performance. You might be surprised at how much processing power the typical XML parsing packages require. XML::SAX and XML::SAX::Expat provide a better code for XML parsing.

cpan LWP::UserAgent
cpan HTML::TokeParse
cpan XML::Feed
cpan XML::SAX
cpan XML::SAX::Expat
cpan Data::Dumper
cpan XML::Feed
cpan XML::Atom

NOTE:
As hard as I tried, CPAN would not install XML::LibXML nor XML::Atom. So, I had to get down into the muck and manually build the packages. The trick was getting the libxml2 package talked about earlier.

One critical required step, if you are going to write and use custom PERL libraries, you need to add the following line to a shell initialization file. I’ve been using bash, for lack of any reason to use other shells. Thus, I added this line to my ~/.bash_profile

export PERL5LIB=$HOME/plib

Once that is done, you’ll need to source the updated config file, or logout/login to gain the benefits of the change.

Step 4 – coming soon

Setting up a custom ‘Weather Reader’ (Part 2)

<-- PART 1

Locating Suitable Data Sources

The process of finding the desirable data sources, can be a challenge. I already know which of the 5 day forecast sources I want to use.

Bremerton, WA

Tacoma, WA

Seattle, WA

Maybe I’ll add this one too, just to so I’m reminded of what I’m missing in CA:

Campbell, CA

Next is a page that contains the CURRENT and HIGH / LOW temps. In the past I have located some text based data files on the NOAA.GOV website. I’ll have to dig up my notes one this. From time to time NOAA moves those files and changes the format, so those notes might not end up being that helpful. It is somewhere to start.

This page here, has a pile of nice detailed information (see compressed screen shot) but does not contain everything I’m looking for:

Recent WX from Bremerton National Airport.

XML DATA!. XML is something I know more about, than I really care too, but it is a portable, easily parsable and text data format. I’ve not yet investigated the contents here, but it does look promising!

Washington State WX XML Feeds

TO BE CONTINUED TODAY

Part 3 –>

Setting up a custom ‘Weather Reader’ (Part 1) – Product Plan

What is a “Weather Reader”? Well, it’s like a “News Reader”, but it’s purpose is to check the web for local weather conditions and display them on a computer screen. For this project I’ve revived an old Averatec laptop, installed Ubuntu 9.10 on it and started to write some PERL.

Beat up old Averatec AMD64 laptop, only 1/2 the keyboard works and the battery is shot.

Weather Data

Where to get the weather data? Personally, I like to go to the source. The public source at NOAA.gov. This is a government agency, paid for by my tax dollars (and yours too if you’re a US tax payer.. of which only 53% of working people in the US are, but that’s another story. So, free for the taking as far as I can tell, and am concerned. Which is good, because the real heavy work is up to me. Finding a URL is one thing, using it for something productive is another thing.

To see what I’m looking at as a source without clicking a bunch of links, here is a screen shot of my indented HTML victim:

Seattle Weather from NOAA website.

Now, the only part that I’m interested in is this one:

I’m sure some of you are asking, “WHAT’S THE POINT!?!?!?”. Well, I have several, not all atop my head. As an avid motorcyclist, weather is almost as import to me as mariners and aviators. Weather might look good for the moment, but around here anything can happen. It’s good to know what the forecast is. And not just in one location. Since I transit two area micro-climates commuting to work, knowing that weather will be at the end points of my journey are are a minimum requirement. Plus, I’m lazy. Not so lazy I won’t write a program to do this, but lazy in the larger sense, of I just want to hit a button and see all the data I want, not have to wait for the Weather Chanel to decided it’s ready to dispense it’s forecast, or hunt around on various local news sites. Nor, frankly, have to click on TWO hyperlinks to see what weather will be like HERE and THERE. Nope, I’m just too lazy for that, so I’ll spend 8-32 hours figuring out a way to save myself 2-3 minutes a day (I didn’t say I’m always smart, but always lazy). Remember; “Laziness is the father of invention!”. Digressing….

The Plan

The plan is to use PERL, run every 30 minutes from 5 AM-> 1 AM, and grab the weather page shown above. Parse out only that which I want, and then re-construct my own page on my own server. It’s a basically a mashup page of what I want to see.

Checking the source code of the page, I see that, ugh, NOAA did not supply and useful DIV markers that I can use. So, this will be a rel brute-force process to detect, select, extract and re-construct the page of my dreams. Yes, I can be easily entertained.

<table width="100%" border="0" align="center" cellpadding="0" cellspacing="0">
<tr valign ="top" align="center">
<td width="11%"><b>Tonight<br></b><br><img src="/images/wtf/nshra50.jpg" width="55" height="58" alt="Scattered Showers Chance for Measurable Precipitation 50%" title="Scattered Showers Chance for Measurable Precipitation 50%" ><br>Scattered<br>Showers<br>Lo <font color="#0033CC">45 &deg;F</font></td><td width="11%"><b>Tuesday<br></b><br><img src="/images/wtf/shra50.jpg" width="55" height="58" alt="Scattered Showers Chance for Measurable Precipitation 50%" title="Scattered Showers Chance for Measurable Precipitation 50%" ><br>Scattered<br>Showers<br>Hi <font color="#FF0000">54 &deg;F</font></td><td width="11%"><b>Tuesday<br>Night</b><br><img src="/images/wtf/nshra30.jpg" width="55" height="58" alt="Scattered Showers Chance for Measurable Precipitation 30%" title="Scattered Showers Chance for Measurable Precipitation 30%" ><br>Scattered<br>Showers<br>Lo <font color="#0033CC">44 &deg;F</font></td><td width="11%"><b>Wednesday<br></b><br><img src="/images/wtf/bkn.jpg" width="55" height="58" alt="Partly Sunny" title="Partly Sunny" ><br>Partly<br>Sunny<br>Hi <font color="#FF0000">57 &deg;F</font></td><td width="11%"><b>Wednesday<br>Night</b><br><img src="/images/wtf/hi_nshwrs30.jpg" width="55" height="58" alt="Chance Showers Chance for Measurable Precipitation 30%" title="Chance Showers Chance for Measurable Precipitation 30%" ><br>Chance<br>Showers<br>Lo <font color="#0033CC">46 &deg;F</font></td><td width="11%"><b>Thursday<br></b><br><img src="/images/wtf/shra30.jpg" width="55" height="58" alt="Chance Showers Chance for Measurable Precipitation 30%" title="Chance Showers Chance for Measurable Precipitation 30%" ><br>Chance<br>Showers<br>Hi <font color="#FF0000">59 &deg;F</font></td><td width="11%"><b>Thursday<br>Night</b><br><img src="/images/wtf/nshra30.jpg" width="55" height="58" alt="Chance Showers Chance for Measurable Precipitation 30%" title="Chance Showers Chance for Measurable Precipitation 30%" ><br>Chance<br>Showers<br>Lo <font color="#0033CC">46 &deg;F</font></td><td width="11%"><b>Friday<br></b><br><img src="/images/wtf/shra.jpg" width="55" height="58" alt="Chance Showers" title="Chance Showers" ><br>Chance<br>Showers<br>Hi <font color="#FF0000">59 &deg;F</font></td><td width="11%"><b>Friday<br>Night</b><br><img src="/images/wtf/nshra.jpg" width="55" height="58" alt="Chance Showers" title="Chance Showers" ><br>Chance<br>Showers<br>Lo <font color="#0033CC">46 &deg;F</font></td></tr>
</table>

Snagging that out of the documents for 2-3 locations will get me what I want. It should look something like this, but the example here does not have any stylesheets applied, it’s a direct rip and mash.

Sample mashup, this was done manually. It at least proved the concept.

There are a couple of problems with that page though, it’s going to take up too much real estate on the final page. Here, it’s almost too large to view on my laptop’s main screen. So, some embedded CSS is in order. This hear overloads the td type, making any text found within anytd rendered at 80% of the normal size. This also overloads creates a style named headish that I is applied to the section titles:

<style>
.headish { 
  font-weight:bold; 
  background-color:#CCC; 
  font-size: 1.2em;}
td { font-size: 0.8em; }
</style>

The result of that very small change, gives me a smaller visual footprint, and formatting at little more pleasing to my sensibilities:

A little more hammering around in the stylesheet, and adding a CURRENT and High/Low temps forecast for the day. A little closer to the desired mockup. I still need to locate a suitable news feed to process:

A little closer to the desired mashup page.

PART 2 –>

Help for Geeks – My Development Bookmarks

I’ve thought about this many times. What info do I find useful, and would other geeks find it useful too?   Perhaps.    In that spirit here is my current list of Development related bookmarks.   Slightly organized.  You might find some nuggets of info in here that relates to a project you are working on.  Or it might spark and idea to build something new, or re-design a process that is not as optimal is you’d like it to be.

I hope some of these help, as they have helped me over the years.  One note though, I removed all my links to PHP development.  I just can’t stand using it any more, it’s just too easy to PWN.


Apache

libapreq2-2.08: libapreq2: Apache2::Request
Apache2::RequestRec – Perl API for Apache request record accessors – search.cpan.org
Apache2::AuthCookie – Perl Authentication and Authorization via cookies – search.cpan.org
Adventures in the Land of Apache and mod_perl
perl. Computational Chemistry List, perl, mod_perl, modperl2, cgi, Apache, cgi, forms
Combining Apache and Perl
Useful OpenSSL Commands
Creating Self-Signed Certs on Apache 2.2 | CB1, INC.
CB1, INC. is a Minneapolis based software and consulting company specializing in custom development and systems integration.
NSS and SSL Error Codes
Smart HTTP and HTTPS RewriteRule Redirects
Smarter SSL HTTPS to HTTP Redirections in .htaccess using RewriteRule to set an environment variable
Apache Week. Using User Authentication
The O’Reilly Network has teamed with Red Hat Apache Week, the leading commercial Apache site to offer comprehensive Apache information and resources. Apache Week offers news, feature articles, reviews, resources, and documentation.
htaccess rewrite tips using RewriteRule and RewriteCond for .htaccess mod_rewrite
mod_rewrite tips and tricks for .htaccess files using RewriteBase, RewriteCond, RewriteEngine, RewriteLock, RewriteLog, RewriteLogLevel, RewriteMap, RewriteOptions, and RewriteRule
mod_perl: mod_perl 2.0 Server Configuration
mod_perl documentation: This chapter provides an in-depth mod_perl 2.0 configuration details.
Dr. Dobb’s | A mod_perl 2 Primer | December 1, 2004
Though it’s technically not quite ready for prime time, it’s high time we all got a taste of the next version of mod_perl.
Apache2::Status 4.00
apache 2: “private key not found”

HTML / CSS

CSS2 Reference
Free HTML XHTML CSS JavaScript DHTML XML DOM XSL XSLT RSS AJAX ASP ADO PHP SQL tutorials, references, examples for web building.
CSS Tutorial – Border
Place customized CSS borders around your HTML elements with the CSS Border attribute.
How can I make just one cell in an HTML table bordered, or just one side of a cell bordered?
CSS Positioning Properties
Free HTML XHTML CSS JavaScript DHTML XML DOM XSL XSLT RSS AJAX ASP ADO PHP SQL tutorials, references, examples for web building.
Ajaxload – Ajax loading gif generator
Ajaxload – Ajax loading gif generator
workalike for top.location.watch(“href”,fn) in IE? – JavaScript
workalike for top.location.watch(“href”,fn) in IE?. Get answers to your questions in our JavaScript forum.
window.onbeforeunload [javascript] [form]
HTML FIELDSET TAG
Free HTML XHTML CSS JavaScript DHTML XML DOM XSL XSLT RSS AJAX ASP ADO PHP SQL tutorials, references, examples for web building.
wg:Bubble Tooltips
web graphics is a compilation of hypertext design resources, links, and commentary.
lixlpixel CSS tooltips
pure CSS pop up tooltips with clean semantic code – valid XHTML – degrades nicely
Simple Round CSS Links ( Wii Buttons )
HTML URL-encoding Reference
Free HTML XHTML CSS JavaScript DHTML XML DOM XSL XSLT RSS AJAX ASP ADO PHP SQL tutorials, references, examples for web building.
Custom error responses
CSS and round corners: Making accessible menu tabs
Find out how to lose the box layout of your CSS pages and make great menu tabs
A List Apart: Articles: Sliding Doors of CSS
Must CSS layouts be flat and boxy? Nope! Bowman shows how to create slick tabbed navigation using CSS and structured XHTML lists.
Light Weight Low Tech CSS Tabs
An example of light weight tabs by combining the Sliding Doors method with the Mountaintop corners idea.
A List Apart: Articles: Mountaintop Corners
Using CSS to create standards-compliant, mountain top corners
Setting a Minimum Body Height:Solving the ‘Height’ Mystery
W3C Markup Validator
W3C’s easy-to-use HTML validation service, based on an SGML parser.
Private RSS Feeds: Support for security in aggregators – silverorange labs
We’ve been experimenting with security options for RSS feeds for our intranet product. However, we found that there weren’t many resources or guidelines for how encryption or authentification should be handled (either in feeds or in readers/aggregators).
The W3C CSS Validation Service
The Art of Web ~ CSS: border-radius and -moz-border-radius
One of the most keenly-anticipated CSS properties is border-radius. It’s not yet available in Internet Explorer, but there is limited support in Firefox (-moz-border-radius) and Safari (WebKit). Discussion and examples.
removeChild
javascript document object javascript document object model sans serif font document object model removechild: Removing objects from a web page.
Adding elements to the DOM
Click here for an introductory tutorial on the DOM of IE 5/ NS 6, and how to program using it
JavaScript tutorial – DOM nodes and tree
HTTP State Management Mechanism [RFC-Ref]
phone number validation
phone number validation
Validating with XML Schema
HTML Color Names
Free HTML XHTML CSS JavaScript DHTML XML DOM XSL XSLT RSS AJAX ASP ADO PHP SQL tutorials, references, examples for web building.
CSS Color Names
Free HTML XHTML CSS JavaScript DHTML XML DOM XSL XSLT RSS AJAX ASP ADO PHP SQL tutorials, references, examples for web building.
CSS Image Opacity / Transparency
Free HTML XHTML CSS JavaScript DHTML XML DOM XSL XSLT RSS AJAX ASP ADO PHP SQL tutorials, references, examples for web building.

JavaScript

O’Reilly Network — Dynamic HTML Tables: Improving Performance
The widespread browser adoption of the W3C Document Object Model (DOM) and other de facto standards have given developers many ways to repopulate a table. So what’s the best approach? Danny Goodman, author of JavaScript & DHTML Cookbook, investigated…
Managing the Dynamic created HTML table thru javascript – SEO Chat
Managing the Dynamic created HTML table thru javascript- HTML Coding. Visit SEO Chat to discuss Managing the Dynamic created HTML table thru javascript
JavaScript Kit- Text Object
Click here for a complete JavaScript Reference, including array, string, document. window, and more.
How do you know what button was pressed in the submit? – JavaScript
How do you know what button was pressed in the submit?. Get answers to your questions in our JavaScript forum.
The JavaScript Source: Forms : Auto Email Link
Automatically creates a new e-mail utilizing the user’s default e-mail client. The script fills in the subject line and adds the URL of the current Web page to the body. Note: May not be compatible with all e-mail clients.
Javascript – Early event handlers
Javascript – The events
Javascript – Introduction to Events
The JavaScript Source: Forms: Form Focus
Places the focus on the first editable field in a form on any web page. Efficient!
JavaScript Help: How to access parent elements from a child window or frame
Popup Window Tutorials
This series of tutorials takes you step by step thrugh thedifferent ways that you can create and modify popup windows.
DevGuru JavaScript PROPERTY: document::forms
Award-winning web developers’ resource: over 3000 pages of quick reference guides, tutorials, knowledge base articles, Ask DevGuru, useful products.
ActiveWidgets • sorting • download data xls

JAVA

java integer to string – Google Search
Java Dynamic Management Kit 5.1 Release Notes
NetBeans IDE 6.0.1 Download
NetBeans IDE 6.0.1 Download
Welcome to JavaWorld.com
Solutions for Java developers
Your Source for Java Information – Developer.com’s Gamelan.com
Get the latest Java news, articles, whitepapers, analyst reports, and more. This is your one stop for information that will help you make decisions related to Java.
String (Java Platform SE 6)
Arrays (The Java™ Tutorials > Learning the Java Language > Language Basics)
StringBuffer (Java 2 Platform SE 5.0)
Throwable (Java 2 Platform SE v1.4.2)
NetBeans Forums – How to use Netbeans packages?
Javadoc Guide
System Properties (The Java™ Tutorials > Deployment > Doing More With Rich Internet Applications)
Adding Classes to the JAR File’s Classpath (The Java™ Tutorials > Deployment > Packaging Programs in JAR Files)
NetBeans Forums – Create manifest, jar and class file in netbeans
Packaging and Deploying Desktop Java Applications
Creating executable JAR files and deploying netbeans projects
Understanding the Manifest
JAR files can support a wide range of functionality, including electronic signing, version control, package sealing, extensions, and others. What gives JAR files the ability to be so versatile? The answer is embodied in the JAR file’s manifest.
http://java.sun.com/j2se/1.3/docs/guide/jar/jar.html#JAR%20Manifest
JAVA Illegal Start of Expression Error – Java
JAVA Illegal Start of Expression Error Java
do-jar-with-main class ‘manifest.available+main.class’ not set. – Google Search
Formatted Printing for Java (sprintf)
The C language utility sprintf is for formatting strings of characters and numbers. This article documents the use of a Java programming language class, PrintfFormat, whose behavior is based on the sprintf specification. Source code is provided.
Java Tips – How to align your components in horizontal or vertical layout
Java Tips — Java, Java, and more Java, How to align your components in horizontal or vertical layout
Java Tips – PointBase Embedded
Java Tips — Java, Java, and more Java, PointBase Embedded
Programming with Java in 24 Hours: Building a Complex User Interface
Questions, corrections and clarifications for hour 16 of the book Teach Yourself Java 6 in 24 Hours by Rogers Cadenhead. The book teaches Java 6 programming for non-programmers, new programmers who hated learning a language, and experienced programmers who want to quickly get up to speed.
Java look and feel Graphics Repository
Welcome to the Java Software Human Interface Group’s Java look and feel Graphics Repository pages.

MySQL

MySQL Stored Procedures / Functions
MySQL AB :: MySQL Forums :: Install :: Comments on my.cnf for high insert volume db
MySQL Configuration
MySQL Performance Blog » Should MySQL and Web Server share the same box ?
MySQL AB :: Index update with update statement
MySQL AB :: MySQL 5.0 Reference Manual :: 18 Stored Procedures and Functions
MySQL Stored Procedures
MySQL Server Tweaking Basics – Admin Zone Forums
This is a basic guide to understanding what the directives in your my.cnf mean, and what they do. We’ll also try to give some general
mySql — CHECK TABLE Syntax
Live Backups of MySQL Using Replication
Russell Dyer, author of MySQL in a Nutshell, walks through the process of using replication for data backups in MySQL.
MySQL AB :: MySQL 5.0 Reference Manual :: B.1.4.1 How to Reset the Root Password
MySQL – best methods for backups
Best way to backup, MySQL/VPS etc Linux
MySQL AB :: MySQL 5.0 Reference Manual :: 12.2.17.1 Troubleshooting InnoDB Data Dictionary Operations
ERROR: database: mysql_error: Can’t connect to MySQL server on ‘10.10.0.7 (110) – Snort Forums Archive
The open source Snort Intrusion Detection and Prevention system is the most flexible and widely deployed solution available.
How to fix error 134 from storage engine – MySQL
How to fix error 134 from storage engine. Get answers to your questions in our MySQL forum.
MySQL AB :: MySQL 5.0 Reference Manual :: 12.2.3 InnoDB Configuration
MySQL Stored Procedures: Part 2
Part 2 of MySQL Stored Procedures covers some more advanced concepts, including conditions and loops.
MySQL AB :: MySQL 5.0 Reference Manual :: 10.11.1 GROUP BY (Aggregate) Functions
Select INTO OUTFILE
MySQL :: stored procedure to list stored procedures
MySQL :: MySQL 5.1 Reference Manual :: 10.4.5 The SET Type
InnoDbEngineStatusAndTuning < Development < TWiki
MySQL :: MySQL 5.0 Reference Manual :: 13.2.11 InnoDB Performance Tuning Tips
Regular Expressions in MySQL
Regular Expressions in MySQL
Dan Winchester – MySQL date_format
MySQL :: MySQL 5.0 Reference Manual :: B.1.2.11 Communication Errors and Aborted Connections
MySQL Bugs: #36910: Mysql Server has gone away
Mysql_User_Add < Development < TWiki
mytop – a top clone for MySQL
Using ON DUPLICATE KEY UPDATE to improve MySQL Replication Performance « Kevin Burton’s NEW FeedBlog
MySQL :: MySQL 5.1 Reference Manual :: 13.11 The FEDERATED Storage Engine
MySQL :: MySQL 5.1 Reference Manual :: 11.4.2 Regular Expressions
MySQL :: MySQL 5.1 Reference Manual :: 18.2 Partition Types

PERL

Proc::ProcessTable – Perl extension to access the unix process table – search.cpan.org
Proc::ProcessTable::Process – Perl process objects – search.cpan.org
perlvar
Perl 5.8 Documentation – Signals
Perl 5.8 Documentation – Signals
khtml2png – Make screenshots from webpages
Sys::Load – Perl module for getting the current system load and uptime – search.cpan.org
The CPAN Search Site – search.cpan.org
Perl HTML::Form
Perl 5.8 Documentation – HTML::Form – Class that represents HTML forms
LWP Cookbook – URL explosion
Reaping Zombies
PERL ‘UTF-16LE’ – MarkMail
urlencode / urldecode in Perl | melecio.org
Regular Expression Examples
how check new URL of redirected page
Getopt::Long – perldoc.perl.org
perl.com: Beginners Intro to Perl – Part 6
Doug Sheppard shows us how to activate Perl’s built in security features.
How can I make one class extend another one?
XML::Generator – Perl extension for generating XML – search.cpan.org
Free Perl source library: unescape
subroutine name: unescape decode URL-encoded string
rami.info » URLEncode And URLDecode For Perl
File Tests in Perl
File Tests in Perl
Page 2 – Build a Perl RSS Aggregator with Templating Tools
Page 2 – Build a Perl RSS Aggregator with Templating Tools
perl.com: Preventing Cross-site Scripting Attacks
Paul Lindner, author of the mod_perl Cookbook, explains how to secure our sites against Cross-Site Scripting attacks using mod_perl and Apache::TaintRequest
libapreq2-2.08: libapreq2: Apache2::Upload
binary data handling – examine a .gif file – Perl example
binary data handling – examine a .gif file – Perl example
PERL — Conversion Functions
LWP::UserAgent
Web user agent class
Fastest XML Parser ?

mod_perl

Practical mod_perl: 6.3.3.6. A third solution
This solution makes use of package-name declaration in the …
mod_perl: HTTP Handlers
mod_perl documentation: This chapter explains how to implement the HTTP protocol handlers in mod_perl.
Installing mod_perl from RPM | O’Reilly Media
It’s easy to install mod_perl using the Red Hat package manager. Configuring it is trickier.
mod_perl: Apache2::RequestRec – Perl API for Apache request record accessors
mod_perl documentation: <code>Apache2::RequestRec</code> provides the Perl API for Apache request_rec object.
How to extract name/value pairs from the query string? | ModPerl | ModPerl
How to extract name/value pairs from the query string? ModPerl ModPerl
mod_perl: Code Snippets
mod_perl documentation: A collection of mod_perl code snippets which you can either adapt to your own use or integrate directly into your own code.

XML

Helpful XML related sites
iWeb Toolkit: XML Validator
XML Schema (REC (20010502) version, as amended) Checking Service
XML Schema Examples
XML::Generator
Perl extension for generating XML
XML DOM – Validate XML
Free HTML XHTML CSS JavaScript DHTML XML DOM XSL XSLT RSS AJAX ASP ADO PHP SQL tutorials, references, examples for web building.

Swish-e

Swish-e :: Re: performance aspects
Swish-e Lightning Talk
Swish-e :: Re: running out of memory during merge
Swish-e :: INSTALL – Swish-e Installation Instructions
Connecting Linux or UNIX system to Network attached storage device
Network attached storage (NAS) allows using TCP/IP network to backup files. This enables multiple servers in IDC to share the same storage for backup at once, which minimizes overhead by centrally managing hard disks. NAS …
How do I access NAS server using automount?
Network-attached storage commonly used to store backup and other shared files over TCP/IP network. For example: i) Corporate e-mail system with multiple, load-balanced webmail servers ii) Load-balanced web servers access the same contents from NAS …
SmallNetBuilder
SmallNetBuilder provides networking and IT news, reviews, help and information for professional and &quot;prosumer&quot; SOHO and SMB users.
IP Subnet Calculator
Online IP Subnet Calculator
10 Steps to Installing PostgreSQL
Remote Network Commands | Linux Journal
Cikul » How to change hostname in CentOS
Thecus User Group – NFS mount failed permission denied
Thecus 1U4500 – Enable root SSH
Wahoo’s Word » Using watch to monitor Javascript
Ascii control codes (control characters, C0 controls)
DOM Based Cross Site Scripting
document.body, doctype switching, and more | evolt.org
A world community for web developers, evolt.org promotes the mutual free exchange of ideas, skills and experiences.
DOM:window.open – MDC
Welcome to Hadoop!
Internationalized domain name – Wikipedia, the free encyclopedia
Code Charts – Scripts
ISO 8859-1 Latin 1 and Unicode characters in ampersand entities
Dig Demonstrations
Re: say if grep can find non-ascii
http://www.ietf.org/rfc/rfc2181.txt
MIL-STD-498,MIL STD 498,MIL-STD,MIL-SPEC,MIL SPEC,Military Standards
MIL-STD-498,MIL-STD,MIL-SPEC,MIL STD,MIL SPEC,Military Standards for ISA
J-STD-016 & MIL-STD-498 vs. DOD-STD-2167A & 7935A
DOD Standards Procedures Collection Document Listing – Page 2
IHS, DOD STANDARDS (MILITARY/FEDERAL SPECS) – General Collection
Fred Morris, project management/distributed systems/practices
Info: (dir) Top
VICNET Help: Online – Web Design – .htpasswd encoder
reminders about programming: Fedora firewall setup is simple using built-in tools
Sample Usability Test Plan
FavIcon from Pics — free favicon.ico for your website (animated, static and marquee icons)
Free and easy to use online tool for creating favicons (.ico, animated, marquee and static) for browser address bars, favorites and tabs, from pictures, logos and other graphics.
How to Obscure Any URL
frames test page for harvesters
IP CIDR Subnet Calculator
Online IP CIDR / VLSM Supernet Calculator
DNS BIND named.conf Parameters
Copy files and directories recursively with tar – Tech-Recipes.com
Copying a directory tree and its contents to another filesystem using tar will preserve ownership, permissions, and timestamps. A neat trick allows using tar to perform a recursive copy without creating an intermediate tar file.
Scrum (development) – Wikipedia, the free encyclopedia
Searching your files with SWISH-E
Disk I/O
Installing and Configuring iptables
How to add an external USB hard drive to your Linux server (Redhat, CentOS, Ubuntu, Gentoo and SUSE) | my-whiteboard
Extenal USB hard disk is really useful (and inexpensive) for backing up your Linux server. Follow these steps to get it to work. 1, Buy a USB hard disk (I have
Which Web Browser is King? – Round 5: JavaScript Library and Framework Tests – OS, Software & Networking by ExtremeTech
Which browser is faster? IE7, Firefox 3, Google Chrome, Safari, or Opera? We run a bevy of tests to determine the king of the Web browser hill.
How to: Debug SSL certificate problems from the shell prompt
OpenSSL is a cryptography toolkit implementing the Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS v1) network protocols and related cryptography standards required by them. It also includes the openssl command, which provides a rich variety of commands You can use the same command to debug problems with SSL certificates. To test the secure connections to a server, type the following command at a shell prompt: openssl s_client -connect ssl.servername.com:443 Where, s_client : This implements a generic SSL/TLS client which can establish a transparent connection to a remote server speaking SSL/TLS. It’s intended for testing …
Ascii Table – ASCII character codes and html, octal, hex and decimal chart conversion
Ascii character table – ascii ascii ascii ascii and ascii…conversions
How to Copy Files Across a Network/Internet in UNIX/LINUX (Redhat, Debian, FreeBSD, etc) – scp tar rsync
HOWTO: Installing DenyHosts – Page 2 – Ubuntu Forums
Page 2- HOWTO: Installing DenyHosts Tutorials & Tips
sendmail Configuration
history–Show status of files and users
using rsync to copy files from one server to another
CVS Commands
FC5 Repositories & Updates
Multiple Bugzilla databases with a single installation
IP Address Lookup (IPv4 & IPv6)
Determines your IP address and shows information (host, location, whois…) about any IP address entered – Up to 10 IP addresses can be looked up at the same time
FC10 network setup repeatedly overwritten – FedoraForum.org
FC10 network setup repeatedly overwritten Installation Help
Interface Configuration Files
CVS revision numbers
CVS Branch and Merge example
CVS–Concurrent Versions System – Merging two revisions
Linux.com :: All about Linux swap space
When your computer needs to run programs that are bigger than your available physical memory, most modern operating systems use a technique called swapping, in which chunks of memory are temporarily stored on the hard disk while other data is moved into physical memory space. Here are some techniques that may help you better manage swapping on Linux systems and get the best performance from the Linux swapping subsystem.
Zone Files
HowTo Setup an NTP Daemon for Time Synchronization – SIPfoundry sipXecs IP PBX, The Open Source SIP PBX for Linux – Calivia
YAML Ain’t Markup Language (YAML) Version 1.1
Trouble editing whine.txt.tmpl – What variables can I use? – mozilla.support.bugzilla | Google Groups
The Cafes » Privacy Tip #3: Block Referer Headers in Firefox
Writing Software Requirements Specifications | A Technical Communication Community
DiG HOWTO
How to use dig to query DNS name servers.
Behavior Driven Development – Wikipedia, the free encyclopedia
Agile software development – Wikipedia, the free encyclopedia
Waterfall model – Wikipedia, the free encyclopedia

Mobile social networking technology – FourSquare

I noted this a couple of weeks ago, that I was starting to play around with foursquare, the mobile social networking tool. It’s primary advantage is as a Smart Phone app. I, of course, prefer the iPhone 3GS (aka The Oracle) platform for my mobile networking, blogging and just plain monkeying around.

One of it’s features, is the ‘Stats’ page. It gives you a rundown of how often you’ve used the app, what cities you spend the majority of your time in, how many check-ins you average on each day of the week. Anyhow, here are my stats (as of this morning). It’s just and example.

My foursquare stats - Easter Sunday 2010

But, it turns out that it’s much more than a neat tool for big-brother to keep an eye on your travels (frankly, they already know where you phone is 24×7, heck, they can even turn on the microphone and listen to you without you’re knowledge, so you tin-foil that types can just simmer on down), I’ve found that it’s one of the most useful, real world source for finding places and things to do, while traveling, or just putting around your local area!

The way it works is very simple. Using the geoLocation capabilities of your smart phone (now I’m assuming your phone is actually smart.. and can do this) it will show you venues nearby that people have entered into the system. But the real benefit comes for the following three neat features!

iPhone foursquare app - Nearby Venues

1) – Displays nearby locations

Based on your location, it will display all sorts of businesses, parks, eateries, medical facilities, you name it.

Now in my sample image here, you see an number of medical facilities, primarily because I was near the hospital when I snapped this screengrab (another thing that the iPhone is amazingly adept at doing). The list can be fairly long, or pretty short. It all depends on how densely populated the area you are in is, and how many locations people have added to the system. Even out here in the sticks, I’ve found a lot of places entered into the system, presumably by developers, since almost all of them had no previous visitors when I checked in.


2) – Tips and things to do

Now, this is the one I find the most useful! Using this, in places that I’ve been frequenting for years, I discovered some NEW businesses and interesting things to do (or eat/drink) at them. I see this as a great way to explore a new town, or even the one you live in.

Traveling, not familiar with the area? Why not try foursquare to see if you can get some good tips from the locals on things to do (or things NOT to do), places with deals, specials, whatever. This is my favorite and most used part of the app.

Now a nice bonus feature is the ability for businesses to add ‘Specials’ that will pop up on the screen that you can unlock. For instance (sorry, no screen shot here, maybe when I get into the office tomorrow I’ll add that one) the hotel next to my work offers a free coffee in the morning and a $20 off certificate on a future stay, simply by checking in there using foursquare and showing it to the barrista in the lobby! See, you really can get something for (nearly) nothing!


iPhone foursquare app - ToDo List

3) – ToDo lists

This is sort of an extension of Feature 2, but I think it’s useful enough on it’s own to be mentioned.

It’s another neat feature that comes in handy for a traveler. Did you hear about something cool to do on foursquare (a tip) and want to recall it next time you are in the area? Mark it as a ‘To Do’ and it will show up on your list. Simple as that.

“Big Deal!”. you might say,“I can do that with a notepad!”. True enough. BUT, you use the foursquare website to poke around in another city location, look at the tips and remember the ones you think sound interesting, by simply clicking and adding to your ToDo list. And guess what, when you are in the local area, the relevant list items are right there at your fingertip. Seems easy enough!

Final thoughts for today – monitization

Being the semi-paranoid data geek I am, what I don’t get about this application, is how the developers and the operator of the website make their money. I can see where a business might pay a little to offer a deal, but how much $$$ is there, for foursquare, in that? I don’t know. Now I also wonder if they are using this neat pile of data they are collecting on your, and re-selling that to potential marketeers! That would be my biggest concern. And one that I’ll be looking into soon.

For now, it’s fun little app. And honestly, it sort of encourages me to get out of the house more and explore places, looking for things people suggest in the tips.

Unless you are one of those tin-foil hat types, you might check it out.