Category Archives: Technology

Aviation, Software Development, Stratux, Technology

Stratux – what’s in that database?

1 Comment

If you’re a hacker, and you’re a data geek.. you probably want to know what’s in that Stratus SQLite database!

Get a SQLite client

I wasn’t able to find a SQLite client on the Straux image, so I installed one with apt-get:


sudo apt-get install sqlite3

Next, run a basic function test to see if it installed and will run:


sqlite3

SQLite version 3.8.7.1 2014-10-29 13:59:56
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.

Cracking open the data file

The sqlist logging / database file is in /var/log, and is named stratux.sqlite.

Cracking open the database and checking the schema I found:


sqlite3 stratux.sqlite

SQLite version 3.8.7.1 2014-10-29 13:59:56
Enter ".help" for usage hints.
sqlite>
sqlite> .tables

dump1090_terminal messages startup traffic
es_messages mySituation status
gps_attitude settings timestamp

Stratux SQLite Tables

Some of these tables contain alot of data:

Table Name Record Count
dump1090_terminal 313290
messages 307
startup 29
traffic 47038
mySituation 0
status 5350
timestamp 252097
settings 0
es_messages 463973
gps_attitude 0
es_messages table

This contains some interesting things. Here are the last 5 records from the database:


select * from es_messages order by timestamp_id desc limit 5;

469502|0001-01-01 00:39:09.91 +0000 UTC|{"Icao_addr":11265096,"DF":0,"CA":0,"TypeCode":0,"SubtypeCode":0,"SBS_MsgType":7,"SignalLevel":0.002207,"Tail":null,"Squawk":null,"Emitter_category":null,"OnGround":false,"Lat":null,"Lng":null,"Position_valid":false,"NACp":null,"Alt":22625,"AltIsGNSS":false,"GnssDiffFromBaroAlt":null,"Vvel":null,"Speed_valid":false,"Speed":null,"Track":null,"Timestamp":"2017-04-24T19:04:54.804Z"}|296790
469503|0001-01-01 00:39:09.97 +0000 UTC|{"Icao_addr":10846969,"DF":0,"CA":0,"TypeCode":0,"SubtypeCode":0,"SBS_MsgType":7,"SignalLevel":0.039045,"Tail":null,"Squawk":null,"Emitter_category":null,"OnGround":false,"Lat":null,"Lng":null,"Position_valid":false,"NACp":null,"Alt":12075,"AltIsGNSS":false,"GnssDiffFromBaroAlt":null,"Vvel":null,"Speed_valid":false,"Speed":null,"Track":null,"Timestamp":"2017-04-24T19:04:54.901Z"}|296790
469498|0001-01-01 00:39:09.53 +0000 UTC|{"Icao_addr":11265096,"DF":0,"CA":0,"TypeCode":0,"SubtypeCode":0,"SBS_MsgType":7,"SignalLevel":0.003431,"Tail":null,"Squawk":null,"Emitter_category":null,"OnGround":false,"Lat":null,"Lng":null,"Position_valid":false,"NACp":null,"Alt":22600,"AltIsGNSS":false,"GnssDiffFromBaroAlt":null,"Vvel":null,"Speed_valid":false,"Speed":null,"Track":null,"Timestamp":"2017-04-24T19:04:54.426Z"}|296789
469499|0001-01-01 00:39:09.53 +0000 UTC|{"Icao_addr":10687795,"DF":11,"CA":1,"TypeCode":0,"SubtypeCode":0,"SBS_MsgType":8,"SignalLevel":0.001740,"Tail":null,"Squawk":null,"Emitter_category":null,"OnGround":null,"Lat":null,"Lng":null,"Position_valid":false,"NACp":null,"Alt":null,"AltIsGNSS":false,"GnssDiffFromBaroAlt":null,"Vvel":null,"Speed_valid":false,"Speed":null,"Track":null,"Timestamp":"2017-04-24T19:04:54.441Z"}|296789
469500|0001-01-01 00:39:09.69 +0000 UTC|{"Icao_addr":10687795,"DF":0,"CA":0,"TypeCode":0,"SubtypeCode":0,"SBS_MsgType":7,"SignalLevel":0.000488,"Tail":null,"Squawk":null,"Emitter_category":null,"OnGround":false,"Lat":null,"Lng":null,"Position_valid":false,"NACp":null,"Alt":28025,"AltIsGNSS":false,"GnssDiffFromBaroAlt":null,"Vvel":null,"Speed_valid":false,"Speed":null,"Track":null,"Timestamp":"2017-04-24T19:04:54.626Z"}|296789

Selected one of the records and decoded the JSON message:

{
	"Icao_addr": 11265096,
	"DF": 0,
	"CA": 0,
	"TypeCode": 0,
	"SubtypeCode": 0,
	"SBS_MsgType": 7,
	"SignalLevel": 0.002207,
	"Tail": null,
	"Squawk": null,
	"Emitter_category": null,
	"OnGround": false,
	"Lat": null,
	"Lng": null,
	"Position_valid": false,
	"NACp": null,
	"Alt": 22625,
	"AltIsGNSS": false,
	"GnssDiffFromBaroAlt": null,
	"Vvel": null,
	"Speed_valid": false,
	"Speed": null,
	"Track": null,
	"Timestamp": "2017-04-24T19:04:54.804Z"
}

These seems to be the 1090 ‘es’ messages, however (at least this example) is missing tail number and squawk information. It’s interesting.., but not very actionable.

The next table I looked at was a completely different story!

traffic table

This looks like a good dataset to mine. With about 47,000 contacts int it.. this might be the basic dataset I’m looking for to use for some visualization.


select * from traffic limit 5;

1|11017168|N621VA|eaVRD947|0|0|0|1|-18.1550731147|6742|1|36.9193725586|-122.0635986328|13075|650|0|7|8|310|287|1|-1536|2016-02-26 01:22:53.426 +0000 UTC|0|0.27|0.16|0001-01-01 00:04:45.29 +0000 UTC|0001-01-01 00:04:45.4 +0000 UTC|0001-01-01 00:04:45.08 +0000 UTC|13075|0001-01-01 00:04:45.08 +0000 UTC|1|0|0.0|0.0|4
2|11017168|N621VA|eaVRD947|0|0|0|1|-19.4002028306|6742|1|36.9202423096|-122.0649414062|13050|650|0|7|8|310|287|1|-1472|2016-02-26 01:22:54.57 +0000 UTC|0|0.22|0.0|0001-01-01 00:04:46.33 +0000 UTC|0001-01-01 00:04:46.55 +0000 UTC|0001-01-01 00:04:46.55 +0000 UTC|13050|0001-01-01 00:04:46.55 +0000 UTC|1|0|0.0|0.0|7
3|11017168|N621VA|eaVRD947|0|0|0|1|-21.540961611|6742|1|36.9209747314|-122.0660552979|13025|650|0|7|8|310|287|1|-1472|2016-02-26 01:22:55.503 +0000 UTC|0|0.23|0.23|0001-01-01 00:04:47.32 +0000 UTC|0001-01-01 00:04:47.32 +0000 UTC|0001-01-01 00:04:47.48 +0000 UTC|13025|0001-01-01 00:04:47.48 +0000 UTC|1|0|0.0|0.0|10
4|11017168|N621VA|eaVRD947|0|0|0|1|-24.8545224734|6742|1|36.9220275879|-122.0676879883|13000|650|0|7|8|310|287|1|-1472|2016-02-26 01:22:56.453 +0000 UTC|0|0.14|0.14|0001-01-01 00:04:48.41 +0000 UTC|0001-01-01 00:04:48.41 +0000 UTC|0001-01-01 00:04:47.48 +0000 UTC|13025|0001-01-01 00:04:47.48 +0000 UTC|1|0|0.0|0.0|12
5|11017168|N621VA|eaVRD947|0|0|0|1|-22.2004294875|6742|1|36.9220275879|-122.0676879883|12975|650|0|7|8|310|287|1|-1408|2016-02-26 01:22:57.573 +0000 UTC|0|1.14|0.0|0001-01-01 00:04:48.41 +0000 UTC|0001-01-01 00:04:49.55 +0000 UTC|0001-01-01 00:04:49.44 +0000 UTC|12975|0001-01-01 00:04:49.44 +0000 UTC|1|0|0.0|0.0|14

Question.. what are the top 10 traffic events by Tail number?
Let’s find out! The simplest way I know of to do this.. and it offers the advantage that the select query against the large dataset with aggregation only runs ONCE.. is to get the parts you want, and the summary count and jam them into a temporary table. With SQLite, you can do it like this:


CREATE TABLE traffic_summary AS select Reg,Tail,count(*) as hits from traffic group by Tail;

That produced a summary table with 423 records in it:


select count(*) from traffic_summary;

423

Now.. who’s the nosiest one of them all?


select * from traffic_summary WHERE Tail>'' order by hits DESC limit 10;

N713FR|ea1435|660
N214NN|CPZ6073|624
N887NN|eaAL2508|564
|eaAL8215|553
N76503|UAL1010|539
N141SY|SKW5982|511
N363VA|VRD1935|501
N204NN|CPZ6074|494
N630VA|VRD941|491
|eaAAR284|478

Using a little sleuthing.. I found that ‘N713FR’ is a Frontier Airlines Airbus 321…

Upcoming Article: Creating some tools to automate acquisition of this information and find out just who’s flying over and how often.

Aviation, Software Development, Stratux, Technology

Stratux – handling a full filesystem

Leave a comment

Running a Stratux in test-bench mode (not in an aircraft, and for days at a time), you’ll likely run into an issue with disk space. The ISO image I acquired from Stratux only provided a 1.8 BG partition for things to live in, and it’s quickly exhausted.

Here is the status of my system after running for about 36 hours… it’s full.

Filesystem Size Used Avail Use% Mounted on
/dev/root 1.8G 1.8G 0 100% /
devtmpfs 459M 0 459M 0% /dev
tmpfs 463M 0 463M 0% /dev/shm
tmpfs 463M 30M 434M 7% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 463M 0 463M 0% /sys/fs/cgroup
/dev/mmcblk0p1 60M 20M 41M 34% /boot

Now to get about the business of increasing the partition and filesystem size without destroying it. First

Locate the disk device

Instructions on the web are not exactly correct, some suggest /dev/sda as the main device, however my testing shows it’s actually this named ‘/dev/mmcblk0’.


root@raspberrypi:~# fdisk -l | grep Disk
[...]
Disk /dev/mmcblk0: 14.5 GiB, 15523119104 bytes, 30318592 sectors

… with the following partitions:

Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 131071 122880 60M c W95 FAT32 (LBA)
/dev/mmcblk0p2 131072 3887103 3756032 1.8G 83 Linux
Running fdisk

With the physical partition located.. start fdisk:


fdisk /dev/mmcblk0

Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

I ended up creating 3 primary partitions. The plan is to delete partition 3 and then re-size the main partition to use up remaining space:


Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 131071 122880 60M c W95 FAT32 (LBA)
/dev/mmcblk0p2 131072 3887103 3756032 1.8G 83 Linux
/dev/mmcblk0p3 2048 8191 6144 3M 5 Extended
/dev/mmcblk0p4 3887104 20664319 16777216 8G 83 Linux

Command (m for help): d
Partition number (1-5, default 5): 3

Partition 3 has been deleted.

Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 131071 122880 60M c W95 FAT32 (LBA)
/dev/mmcblk0p2 131072 3887103 3756032 1.8G 83 Linux
/dev/mmcblk0p4 3887104 20664319 16777216 8G 83 Linux

Write out the partition and… then run to enable it:


Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.

partprobe

Next, put a filesystem on this new partition. Using df to determine the type of filesystem currently in use; I recommend that you stick with it for this most basic of operations:


df -T

Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/root ext4 1815440 1799056 0 100% /

Run mkfs


/sbin/mkfs -t ext4 /dev/mmcblk0p4

Creating filesystem with 2097152 4k blocks and 524288 inodes
Filesystem UUID: e36a8f6c-a457-4531-b67d-bea4885a9583
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information:... this might go on for a bit..

Once completed.. mount this where the logs and databases live. To do this the first thing that needs to happen is to check your current fstab:


cat /etc/fstab
proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults 0 2
/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for that

My first order of business was to mount the new filesystem to a temporary location (/var/log2) and then copy the contents of /var/log to that location, then delete everything in the log directory, and then unmount log2.


/var/log2

mount -t ext4 /dev/mmcblk0p4 /var/logs

cp -R log/* log2/.

cd log
rm -rf *

umount /dev/mmcblk0p4

Edit the fstab file to create a mount point for the new partition where the logs used to be written (added the orange line), and ran mount to verify that it will automount on a restart.


vi /etc/fstab

proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults 0 2
/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
/dev/mmcblk0p4 /var/log ext4 defaults,noatime 0 0

mount -a

df

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 1815440 1768992 0 100% /
devtmpfs 469688 0 469688 0% /dev
tmpfs 474004 0 474004 0% /dev/shm
tmpfs 474004 35972 438032 8% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 474004 0 474004 0% /sys/fs/cgroup
/dev/mmcblk0p1 61384 20400 40984 34% /boot
/dev/mmcblk0p4 8125880 333800 7356268 5% /var/log

Restart and verify

Restart the little box and verify that the mount was preserved.


init 6

Log back in, and run df to check the filesystem health. It should now has the the main filesystem has some breathing room again:

                                                                                              
 ad88888ba  888888888888  88888888ba          db    888888888888  88        88  8b        d8  
d8"     "8b      88       88      "8b        d88b        88       88        88   Y8,    ,8P   
Y8,              88       88      ,8P       d8'`8b       88       88        88    `8b  d8'    
`Y8aaaaa,        88       88aaaaaa8P'      d8'  `8b      88       88        88      Y88P      
  `"""""8b,      88       88""""88'       d8YaaaaY8b     88       88        88      d88b      
        `8b      88       88    `8b      d8""""""""8b    88       88        88    ,8P  Y8,    
Y8a     a8P      88       88     `8b    d8'        `8b   88       Y8a.    .a8P   d8'    `8b   
 "Y88888P"       88       88      `8b  d8'          `8b  88        `"Y8888Y"'   8P        Y8

NOTE TO DEVELOPERS: Make sure that your system has an acceptable clock source, i.e., a GPS
with sufficient signal or enable ntpd (internet connection required).

Everything here comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Type 'stratux-help' (as root) for a few debugging commands.
pi@raspberrypi:~ $ df

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 1815440 1389856 332592 81% /
devtmpfs 469688 0 469688 0% /dev
tmpfs 474004 0 474004 0% /dev/shm
tmpfs 474004 6336 467668 2% /run
tmpfs 5120 16 5104 1% /run/lock
tmpfs 474004 0 474004 0% /sys/fs/cgroup
/dev/mmcblk0p4 8125880 329820 7360248 5% /var/log
/dev/mmcblk0p1 61384 20400 40984 34% /boot

At a later date I’ll work on expanding the main partition, but for now this should stabilize the machine and resolve the main disk consumption issue.

Aviation, Data, Software Development, Stratux, Technology

Monitor local Aircraft (for baiscally free) using Stratux

Leave a comment

An ADS-B listening station has long been on my list of things to build.

Our current residence is located right under the domestic approach to San Francisco International Airport (see picture), so I believed there should be plenty of data for testing and tuning.

Local Air Traffic


What is Stratux

So, what are we talking about here? It’s Stratux, and Open Source complete software package that leverages inexpensive SDRs (Software Defined Radios).

“Stratux is a homebuilt ADS-B In receiver for pilots. It’s easy to assemble from inexpensive, off-the-shelf hardware, and probably already works with your electronic flight bag (EFB) of choice. Even better, if you’re so inclined, the software is open-source and hackable so you can build the system that’s right for you. “

This is some powerful stuff!


The Raspberry Pi 3 (revision b)

Raspberry PI 3
After completing a proof-of-concept residential IP space data acquisition project for a client, I found myself with a Raspberry Pi just sitting on shelf.

The Raspberry Pi 3b is a neat little device. A full Linux computer in a form factor the size of a pack of card, including a graphics chip that drives and HDMI output making it a real (compact and low power) desktop project computer.

SoC: Broadcom BCM2837
CPU: Quad-core ARM Cortex-A53, 1.2GHz
GPU: Broadcom VideoCore IV 3D graphics
RAM: 1GB LPDDR2 (900 MHz)
Networking: 10/100 Ethernet, 2.4GHz 802.11n wireless
Bluetooth: Bluetooth 4.1 Classic, Bluetooth Low Energy
Storage: microSD
GPIO: 40-pin header, populated
Ports: HDMI, 3.5mm analogue audio-video jack, 4× USB 2.0, Ethernet, Camera Serial Interface (CSI), Display Serial Interface (DSI)

My first Raspberry Pi purchase (as requested by the client) was a complete kit that cost me about $75 [ link to super size kit ], but you can certainly get the bare Raspberry Pi for under $40 (assuming you have some spare things like a micro-USB cable and a micro SD card).

Adding ADS-B radios

Adding ADS-B radios to the Raspberry Pi was as easy as ordering a kit form Amazon for under $40. [ Dual-Band ADS-B (978MHz UAT & 1090MHz 1090ES) Bundle For Stratux ]. For some reason, I’d debating buying the radios and building a kit. There are several complete kits with the computer, radios, specialized case, memory card etc. Prices vary between $120 to $250 depending on what parts you want. When I found this little kits with 2 sets of antennas, radios and coax for under $40.. it was just too easy to pull the trigger. So far they have been well worth the very inexpensive purchase!

Assembling the Sysetm

Custom Stratux Pi Case
Being a proof of concept, I didn’t feel like dropping another $20 on a specialized case such as this one (right), because I wasn’t sure if I’d be happy with this project.

The previous projects housing was too small (in my opinion) to provide what I wanted, which was a single item housing all the parts. Again, wanting to minimize costs while building project, I opted to re-purpose an small plastic ammo can into a make-shift housing. The unfortunate side effect of that decisions is that the final product looks like some sort of nefarious device (see final photos somewhere below).

Using the drill press / mill I have setup for another project, I quickly milled some vents to the plastic box to vent out the heat created by the Pi and the two nano radios. And believe me, this is something you want to do. Using the Stratux software, I’m typically seeing CPU temperatures around 140F (toasty), and the radio run a lot hotter.

SDRs installed into Raspberry Pi
Hot enough to blacken the decals I’d put on the bottoms of the radios (this is what they looked like before they were cooked).

Once I had all the milling completed I installed the radios, Pi and coax into the box. The coax are reasonable flexible but still barely looped around inside the box. This photo was before I added another port for an Ethernet cable (that hack to be discussed in a subsequent post). It might not look pretty, but it does work!

Milled ammo box
SDRs and Pi installed in the ammo box.
Final Stratux in Ammo Box project

Making it all work

Once the physical construction was done, the last step was to download the software, burn it to a little MicroSD card and fire it up!

UPDATED: 14-MAY-2017 — I have a new setup procedure documented in this newer article: STRATUX – Hacking together a WiFi connected Ground Station.

What does it look like?

Once you have connected to the ad-hoc stratux WiFi network, navigate to this IP address: http://192.168.10.1 . If your system is up and running you’ll see a page that looks like this:

Stratux landing page at 192.168.10.1

If you have some aircraft overhead (as I almost always do), you should see them listed on the ‘Traffic’ page. This is what mine looked like just a few minutes before writing this article:

Stratux Air Traffic page

Now that you have this up and running, it can provide a GDL 90 data feed to variety of flight planning / monitoring software, including some free apps for iOS and Android. The full current list of software supported on the Stratux main page. Here is a snapshot of software support at this time:

Stratux Software support


Software Development, Technology

casperJS – passing parameters on the command line (cli)

1 Comment

I like using the command line / shell to pass information to scripts and programs. Sure it has limitations but it’s generally pretty effective.

Each language seems to have it’s own method of handing them, and CasperJS does not seem to be any different. What I did find a little different from others, is how CapserJS splits out arguments from parameters (options in CapserJS speak).

Passing Arguments

Passing arguments is a fairly simple affair. His an example of passing some arguments on the command line to a screen capture utility I’m working on:

casperjs generic.screenshot.cjs http://www.someplace.com save_local
[
“http://www.someplace.com”,
“save_local”
]

They are placed into a list (integer indexed array), and can be addressed in the following way:

url = casper.cli.args[0];
mode = casper.cli.args[1];

This would result in ‘url’ = ‘http://www.someplace.com’ and ‘mode’ = ‘save_local’. Pretty straight forward. One major drawback, you must know the order of the parameters, add a new one, get one out of place and the entire house of cards collapses.

There is a better way!

Passing Parameters (options)

Passing options is a fairly simple affair as well. His an example of passing some option on the command line:

casperjs generic.screenshot.cjs –url=http://www.someplace.com –mode=save_local
{
“casper-path”: “/opt/casperjs”,
“cli”: true,
“url”: “http://www.someplace.com”,
“mode”: “save_local”
}

Note: “casper-path” and “cli” are injected automatically

They are placed into an array, and can be addressed in the following way:

url = casper.cli.options[‘url’];
mode = casper.cli.options[‘mode’];

This would result in ‘url’ = ‘http://www.someplace.com’ and ‘mode’ = ‘save_local’. Now, you do not need to worry about ordering of options, and adding more options doesn’t mean that you’ll have to worry about re-coding the variable localization, ordering etc. It’s also CLEAR to any user how the cli is formatted.

Now, go forth and parse, screen capture and automate your testing until your heart is content!

Software Development, Technology

Installing CasperJS 1.1.4 on AWS (CentOS)

Leave a comment

Installing CasperJS to work with PhantomJSs latest version 2.1.1

This is the current status of my test installation. My perviously hacked version of CaserpJS ( instructions are there: Helping CasperJS 1.1.0-beta3 play nice with PhantomJS 2.0.0 ), however it’s time to rev-up to the non-beta version of the code.

casperjs
CasperJS version 1.1.0-beta3 at /usr/lib/node_modules/casperjs, using phantomjs version 2.1.1

As of today:
Screen Shot 2016-02-08 at 10.42.19 AM

Step 1 — Clone CasperJS from Git

Hopefully you already have Git installed, and you are ready to clone:

git clone git://github.com/n1k0/casperjs.git
Cloning into ‘casperjs’…
remote: Counting objects: 14392, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 14392 (delta 0), reused 0 (delta 0), pack-reused 14385
Receiving objects: 100% (14392/14392), 8.50 MiB | 0 bytes/s, done.
Resolving deltas: 100% (8648/8648), done.
Checking connectivity… done.

Step 2 — Perform Installation

Using hints from the Instructions at CasperJS 1.1.0-DEV documentation, I first located my current casper image, moved it aside, then linked the new one into it’s location.

whereis casperjs
casperjs: /usr/bin/casperjs /usr/local/bin/casperjs /opt/n1k0-casperjs-e3a77d0/bin/casperjs /opt/casperjs/bin/casperjs /opt/casperjs/bin/casperjs.exe

mv /usr/bin/casperjs /usr/bin/casperjs.1.1.0-beta3

ln -sf `pwd`/bin/casperjs /usr/bin/casperjs

Step 3 — Verify Casper

Running casper, I checked to ensure it’s on the latest version:

casperjs
CasperJS version 1.1.0-beta5 at /opt/casperjs, using phantomjs version 2.1.1

This looks like it’s good to go.. now CASPER AWAY!!

Entrepreneurship, Software Development, Technology

Installing PhantomJS 2.1.1 on AWS (CentOS)

Leave a comment

phantomjs-logoIt’s a gamble to do this, and according to the build script it’s going to take a long time to complete the compile / install of Phantom 2.1.1.

Note: If you are looking for instructions on building for Ubuntu, the steps are different. I’ve documented that process in this post: Installing PhantomJS 2.1.1 on Ubuntu.

Step 1 — install required dependencies

You may or may not have most of these on your AWS / CentOS system. I found that most of these were required to start the PhantomJS build.Here are the ones that I’ve confirmed I needed:

  • autoconf
  • pkgconfig.x86_64
  • python26-pyudev.noarch
  • python26-twisted.noarch
  • sip.x86_64
  • python27-pyudev.noarch
  • python27-twisted.noarch
  • gcc
  • flex
  • bison
  • xorg-x11-server-Xorg.x86_64
  • xorg-x11-server-devel.x86_64
  • xorg-x11-utils.x86_64
  • xorg-x11-proto-devel.noarch
  • sqlite-tcl.x86_64
  • sqlite-devel.x86_64
  • openssl.x86_64
  • crypto-utils.x86_64
  • openssl-devel.x86_64
  • libfontenc.x86_64
  • libfontenc-devel.x86_64
  • fontconfig.x86_64
  • fontconfig-devel.x86_64
  • libicu-devel.x86_64
  • freetype-devel.x86_64
  • libpng-devel.x86_64
  • libjpeg-turbo-devel.x86_64
  • libXext-devel.x86_64
  • libxcb-devel.x86_64
  • xcb-util.x86_64

Installing the packages went smoothly:

sudo yum install autoconf pkgconfig.x86_64 python26-pyudev.noarch python26-twisted.noarch sip.x86_64 python27-pyudev.noarch python27-twisted.noarch gcc flex bison xorg-x11-server-Xorg.x86_64 xorg-x11-server-devel.x86_64 xorg-x11-utils.x86_64 xorg-x11-proto-devel.noarch sqlite-tcl.x86_64 sqlite-devel.x86_64 openssl.x86_64 crypto-utils.x86_64 openssl-devel.x86_64 libfontenc.x86_64 libfontenc-devel.x86_64 fontconfig.x86_64 fontconfig-devel.x86_64 libicu-devel.x86_64 freetype-devel.x86_64 libpng-devel.x86_64 libjpeg-turbo-devel.x86_64 libXext-devel.x86_64 libxcb-devel.x86_64 xcb-util.x86_64

Step 2 — clone the Git repo to local drive:

git clone git://github.com/ariya/phantomjs.git
Cloning into ‘phantomjs’…
remote: Counting objects: 63695, done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 63695 (delta 16), reused 0 (delta 0), pack-reused 63657
Receiving objects: 100% (63695/63695), 129.05 MiB | 4.08 MiB/s, done.
Resolving deltas: 100% (31013/31013), done.
Checking connectivity… done.

cd phantomjs

git checkout 2.1.1
Note: checking out ‘2.1.1’.
[…]
HEAD is now at d9cda3d… Set version to “2.1.1”

git submodule init
Submodule ‘3rdparty-win’ (https://github.com/Vitallium/phantomjs-3rdparty-win.git) registered for path ‘src/qt/3rdparty’
Submodule ‘qtbase’ (https://github.com/Vitallium/qtbase.git) registered for path ‘src/qt/qtbase’
Submodule ‘qtwebkit’ (https://github.com/Vitallium/qtwebkit.git) registered for path ‘src/qt/qtwebkit’

git submodule update
Cloning into ‘src/qt/3rdparty’…
Cloning into ‘src/qt/qtbase’…
Cloning into ‘src/qt/qtwebkit’…

Step 3 — Hack the QT build

It seemed that I needed to set some different flags for the qtbase build. It was not clear to me if this could be done with the build.py options, so I hacked the qt/qtbase/configure script.

vi src/qt/qtbase/configure

First off, I changed the settings of these two values near the top of the config file:

Then commented out part of the section around Werror, so that the build would not treat warnings as errors. The C++ macro options in the code will generate A LOT of errors, most of them from the flags defined in build.py. I tried the route of disabling those flags and ended up with more errors and more issues.. so changing the flags in the config was my next option:

[…]
#CFG_WERROR=auto
CFG_WERROR=no
[…]
#CFG_DEV=no
CFG_DEV=yes
[…]
warnings-are-errors|Werror)
# if [ “$VAL” = “yes” ] || [ “$VAL” = “no” ]; then
# CFG_WERROR=”$VAL”
# else
UNKNOWN_OPT=yes
# fi
;;
[…]

Step 4 — Build!

python build.py
—————————————-
WARNING
—————————————-

Building PhantomJS from source takes a very long time, anywhere from 30 minutes
to several hours (depending on the machine configuration). It is recommended to
use the premade binary packages on supported operating systems.

For details, please go the the web site: http://phantomjs.org/download.html.

Do you want to continue (Y/n)? Y

Step 5 — check the binary

Once the build has completed, you will find the binary to be built in the local directory bin/

ls -l bin/phantomjs
-rwxr-xr-x 1 root root 56736434 Feb 5 11:33 /usr/sbin/phantomjs

To complete the installation, you’ll need to replace the current phantomjs binary with the new one. To find the location if your current binary (if you have one), this should work:

whereis phantomjs
phantomjs: /usr/bin/phantomjs

Copy the new binary to that location and verify version:

cp bin/phantomjs /usr/bin/phantomjs
cp: overwrite ‘/usr/bin/phantomjs’? y

phantomjs -v
2.1.1

YOU ARE DONE!! It was just that easy

Software Development, Technology

Installing PhantomJS 2.1.1 on Ubuntu

Leave a comment

phantomjs-logoIt’s a gamble to do this, and according to the build script it’s going to take a long time to complete the compile / install of Phantom 2.1.1.

Note: If you run into build problems with some of the required components, such as the fonts, qtbase, etc., you will want to check my previous post Installing PhantomJS 2 on Ubuntu for some help.

Step 1 — install required dependencies

You may or may not have most of these on your Ubuntu system. I found that most of these were required to start the PhantomJS build.Here are the ones that I’ve confirmed I needed:

  • autoconf2.13

  • pkg-config

  • build-essential

  • qt5-qmake

  • g++

  • python

  • ruby

  • perl

  • sqlite

  • flex

  • bison

  • gperf

  • openssl

  • fontconfig

  • xorg

  • xorg-dev

  • xutils-dev

  • xcb-proto

  • libtool

  • libsqlite0

  • libssl-dev

  • libsqlite3-dev

  • libfontconfig1-dev

  • libicu-dev

  • libfreetype6

  • libssl-dev

  • libpng-dev

  • libpng12-dev

  • libjpeg-dev

  • libx11-dev

  • libxext-dev

  • libxcb-xkb-dev

Installing the packages went smoothly:

sudo apt-get install autoconf2.13 pkg-config build-essential qt5-qmake g++ python ruby perl sqlite flex bison gperf openssl fontconfig xorg xorg-dev xutils-dev xcb-proto libtool libsqlite0 libssl-dev libsqlite3-dev libfontconfig1-dev libicu-dev libfreetype6 libssl-dev libpng-dev libpng12-dev libjpeg-dev libx11-dev libxext-dev libxcb-xkb-dev x11proto-core-dev libxcb-render-util0 libqt5webkit5-dev

Step 2 — clone the Git repo to local drive:

git clone git://github.com/ariya/phantomjs.git
Cloning into ‘phantomjs’…
remote: Counting objects: 63695, done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 63695 (delta 16), reused 0 (delta 0), pack-reused 63657
Receiving objects: 100% (63695/63695), 129.05 MiB | 4.08 MiB/s, done.
Resolving deltas: 100% (31013/31013), done.
Checking connectivity… done.

cd phantomjs

git checkout 2.1.1
Note: checking out ‘2.1.1’.
[…]
HEAD is now at d9cda3d… Set version to “2.1.1”

git submodule init
Submodule ‘3rdparty-win’ (https://github.com/Vitallium/phantomjs-3rdparty-win.git) registered for path ‘src/qt/3rdparty’
Submodule ‘qtbase’ (https://github.com/Vitallium/qtbase.git) registered for path ‘src/qt/qtbase’
Submodule ‘qtwebkit’ (https://github.com/Vitallium/qtwebkit.git) registered for path ‘src/qt/qtwebkit’

git submodule update
Cloning into ‘src/qt/3rdparty’…
Cloning into ‘src/qt/qtbase’…
Cloning into ‘src/qt/qtwebkit’…

python build.py
—————————————-
WARNING
—————————————-

Building PhantomJS from source takes a very long time, anywhere from 30 minutes
to several hours (depending on the machine configuration). It is recommended to
use the premade binary packages on supported operating systems.

For details, please go the the web site: http://phantomjs.org/download.html.

Do you want to continue (Y/n)? Y

NOTE: If you want to suppress the warning regarding perils of the long compile, you an use the –confirm flag to bypass the question. This is really helpful if you want to background the process and write it to a log. Where I find this most beneficial is when I want to/need to close the terminal window before the compile completes.

Here is an optional method of running that will background the process, auto-reply to the warning and write to a log file:

nohup ./build.sh –confirm –jobs 1 > build.log &

You might carp about not being able to monitor progress now! Well sure you can.. just do a following tail on the file. Exact command varies with system, I’ll provide the one for typical LINUX and for typical OSX:

For typical LINUX:
tailf build.log

For typical OSX:
tail -f build.log

Step 3 — check the binary

Once the build has completed, you will find the binary to be built in the local directory bin/

ls -l bin/phantomjs
-rwxr-xr-x 1 root root 56736434 Feb 5 11:33 /usr/sbin/phantomjs

To complete the installation, you’ll need to replace the current phantomjs binary with the new one. To find the location if your current binary (if you have one), this should work:

whereis phantomjs
phantomjs: /usr/bin/phantomjs

Copy the new binary to that location and verify version:

cp bin/phantomjs /usr/bin/phantomjs
cp: overwrite ‘/usr/bin/phantomjs’? y

phantomjs -v
2.1.1

YOU ARE DONE!! It was just that easy

Software Development, Technology

Fun with ‘sed’ taking a list of table names and creating a list of T-SQL TRUNCATE commands

Leave a comment

Woo.. I love *NIX, especially ‘sed’ (and awk too for that matter). Wonderful things you can do to automated your programming, speed up development and really REALLY cut down on cut-paste errors.

Today’s task, was to take this list of tables… and generate a single bit fat T-SQL command to truncate them all.. *however* these tables might not exist in a given environment so each Truncate is wrapped in a test to make sure the table is there. If it’s not there, nothing to do, not to mention avoid a potential fatal error trying to truncate a non-existent table.

Here is a snippet of the tables in question:

…
FCT_CRIMINOGENIC_NEED_TOP_DOMAINS
FCT_DETENTION_RISK_RESPONSE_DETAILS
FCT_DETENTION_RISK_RESPONSE_FULL
…

This is the desired T-SQL for these three tables

— Truncate table FCT_CRIMINOGENIC_NEED_TOP_DOMAINS
IF EXISTS (SELECT * FROM sys.objects WHERE NAME = ‘FCT_CRIMINOGENIC_NEED_TOP_DOMAINS’)
BEGIN
TRUNCATE TABLE FCT_CRIMINOGENIC_NEED_TOP_DOMAINS
END

— Truncate table FCT_DETENTION_RISK_RESPONSE_DETAILS
IF EXISTS (SELECT * FROM sys.objects WHERE NAME = ‘FCT_DETENTION_RISK_RESPONSE_DETAILS’)
BEGIN
TRUNCATE TABLE FCT_DETENTION_RISK_RESPONSE_DETAILS
END

— Truncate table FCT_DETENTION_RISK_RESPONSE_FULL
IF EXISTS (SELECT * FROM sys.objects WHERE NAME = ‘FCT_DETENTION_RISK_RESPONSE_FULL’)
BEGIN
TRUNCATE TABLE FCT_DETENTION_RISK_RESPONSE_FULL
END

The Sed Command

sed ‘s#^\(.*\)#– Truncate table \1 \’$’\nIF EXISTS (SELECT * FROM sys.objects WHERE NAME = \’\\1\’) \\’$’\n BEGIN \\’$’\n TRUNCATE TABLE \\1 \\’$’\nEND \\’$’\n#g’ MY_INPUT_FILE

Secret Sauce Ingredients

Setting up the string match. In this case it was simple.. I wanted the entire line which is represented with .*

\(.*\)

This loads the matching string into an internal register in ‘sed’ that can be references as ‘1’ within the output replacement expression. This is what it looks like in the command:

\1

Adding a newline in the output.. this was the big trick, and requires dropping into the shell to generate the desired output. The string in the command looks like this:

\’$’\n

You may notice after the first drop into the shell, the escaping has to be doubled for the substitution and the newline…

\\1 \\’$’\n

I cannot speak to the reason for this, but that is what I encountered and why the substitutions change in the latter part of the string. You may find that is not necessary, or might cause an issue in your environment. Adjust as necessary.

Software Development, Technology

Installing CasperJS the compiled way

Leave a comment

Installing CaspeJS from source is always fun, so I’m going to tell you how.

Step 1 – Get the source code

Getting the source code from CasperJS website. As of 16-OCT-2015 when I write this, the following command worked. When a new version is released.. this link might go away.. so.. a little hunting might be needed on your part.

The CaseperJS website is located HERE ( http://casperjs.org/ ).

wget https://github.com/n1k0/casperjs/zipball/1.1-beta3/n1k0-casperjs-1.1-beta3-0-g4f105a9.zip

Step 2 – unpack the files

Unpack the files and prepare for compiling:

unzip n1k0-casperjs-1.1-beta3-0-g4f105a9.zip

Step 2 – build it

Software Development, Technology

Installing PhantomJS 2 on Ubuntu

1 Comment

phantomjs-logoIt’s a gamble to do this, and according to the build script it’s going to take a long time to complete the compile / install of Phantom 2.0.

Step 1 — locate the source, download and unzip:

wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.0.0-source.zip
Length: 110092872 (105M) [application/zip]
Saving to: ‘phantomjs-2.0.0-source.zip’

unzip phantomjs-2.0.0-source.zip
Archive: phantomjs-2.0.0-source.zip
a2912c216d06df4d8b51f12ad4082a48c5fc7ba6
creating: phantomjs-2.0.0/
inflating: phantomjs-2.0.0/.gitignore
[…]
inflating: phantomjs-2.0.0/tools/preconfig.sh
inflating: phantomjs-2.0.0/tools/qscriptengine.h
inflating: phantomjs-2.0.0/tools/src.pro

Step 2 — install required dependencies

You may or may not have most of these on your Ubuntu system. I found that most of these were required to start the PhantomJS build.Here are the ones that I’ve confirmed I needed:

  • autoconf2.13
  • ruby
  • pkg-config
  • libicu-dev
  • gperf
  • bison
  • libjpeg-dev
  • g++
  • openssl
  • libtool
  • libssl-dev
  • libpng-dev
  • libpng12-dev
  • libjpeg-dev
  • fontconfig
  • sqlite
  • fontconfig
  • libsqlite0
  • qt5-qmake
  • xorg
  • xorg-dev
  • xutils-dev
  • xcb-proto
  • libxcb-xkb-dev
  • x11proto-core-dev
  • libxcb-render-util0
  • libqt5webkit5-dev

Installing the packages went smoothly:

apt-get install autoconf2.13 ruby pkg-config libicu-dev gperf bison libjpeg-dev g++ openssl libtool libssl-dev libpng-dev libpng12-dev libjpeg-dev fontconfig sqlite fontconfig libsqlite0 qt5-qmake xorg xorg-dev xutils-dev xcb-proto libxcb-xkb-dev x11proto-core-dev libxcb-render-util0 libqt5webkit5-dev

Step 3 — Build Freetype from source and link

Following this I grabbed the source code to install freetype2. Although freetype successfully installed, the required header files where not found. I decided it was bet to grab it and build from source:

wget http://download.savannah.gnu.org/releases/freetype/freetype-2.6.tar.gz
gunzip freetype-2.6.tar.gz
tar xvf freetype-2.6.tar
./configure
[…]
configure: creating ./config.status
config.status: creating unix-cc.mk
config.status: creating unix-def.mk
config.status: creating ftconfig.h
config.status: executing libtool commands
configure:
make & make install
[…]
/usr/bin/install -c -m 644 ./builds/unix/ftconfig.h \
/usr/local/include/freetype2/config/ftconfig.h
/usr/bin/install -c -m 644 /usr/local/freetype-2.6/objs/ftmodule.h \
/usr/local/include/freetype2/config/ftmodule.h
/usr/bin/install -c -m 755 ./builds/unix/freetype-config \
/usr/local/bin/freetype-config
/usr/bin/install -c -m 644 ./builds/unix/freetype2.m4 \
/usr/local/share/aclocal/freetype2.m4
/usr/bin/install -c -m 644 ./builds/unix/freetype2.pc \
/usr/local/lib/pkgconfig/freetype2.pc
/usr/bin/install -c -m 644 /usr/local/freetype-2.6/docs/freetype-config.1 \
/usr/local/share/man/man1/freetype-config.1

Now following that build, due to some inexplicable continuous oversight on the part of freetype’s maintainers.. OR.. phantom.. a link has to be make so that the build process can find the actual libraries required:

ln -s /usr/include/freetype2/freetype /usr/include/freetype

Step 4 — Install an updated version of libxkbcommon

Get the latest version.. and build it:

https://launchpad.net/ubuntu/+archive/primary/+files/libxkbcommon_0.5.0.orig.tar.gz
gunzip libxkbcommon_0.5.0.orig.tar.gz
tar xvf libxkbcommon_0.5.0.orig.tar
cd libxkbcommon-0.5.0
./autogen.sh
./configure

Step 5 — hack QtBase

After many times running thorough the process and resolving errors.. I decided to manually build QTBase. To do this I moved to the source directory for it under PhantomJS, and used this configuration command to get past a libxcb error:

cd src/qt/qtbase
./configure -developer-build -opensource -nomake examples -nomake tests -qt-xcb
./configure -opensource
make
make install

Step 6 — build

Now bulid.sh script. NOTE: if you are executing the compile on a VM (or in this case AWS), it’s recommended that the build process Does Not try to run parallel build jobs on the virtual cores. The PhantomJS website was not clear (to me) why.. but it did recommend using the –jobs 1 flag on the build.. which I am doing. You may omit that if you’d like to experiment.

cd phantomjs-2.0.0

./build.sh –jobs 1
—————————————-
WARNING
—————————————-

Building PhantomJS from source takes a very long time, anywhere from 30
minutes to several hours (depending on the machine configuration).
We recommend you use the premade binary packages on supported operating
systems.

For details, please go the the web site: http://phantomjs.org/download.html.

Do you want to continue (y/n)?
y
[…]

NOTE: If you want to suppress the warning regarding perils of the long compile, you an use the –confirm flag to bypass the question. This is really helpful if you want to background the process and write it to a log. Where I find this most beneficial is when I want to/need to close the terminal window before the compile completes.

Here is an optional method of running that will background the process, auto-reply to the warning and write to a log file:

nohup ./build.sh –confirm –jobs 1 > build.log &

You might carp about not being able to monitor progress now! Well sure you can.. just do a following tail on the file. Exact command varies with system, I’ll provide the one for typical LINUX and for typical OSX:

For typical LINUX:
tailf build.log

For typical OSX:
tail -f build.log

Step 7 — check the binary

Once the build has completed, you will find the binary to be built in the local directory bin/

ls -l bin/phantomjs
-rwxr-xr-x 1 root root 56587060 Sep 30 17:16 bin/phantomjs

To complete the installation, you’ll need to replace the current phantomjs binary with the new one. To find the location if your current binary (if you have one), this should work:

whereis phantomjs
phantomjs: /usr/bin/phantomjs

Copy the new binary to that location and verify version:

cp bin/phantomjs /usr/bin/phantomjs
cp: overwrite ‘/usr/bin/phantomjs’? y

phantomjs -v
2.0.0

YOU ARE DONE!! It was just that easy