Category Archives: Data

Data related posts

Creating a Personalized Recreation Dashboard – Installing Software (Part 2)

Rasperian Buster has a working desktop out of the box, but to run NodeJS Roosevelt and Java, some installations will need to be done first.

INSTALLING JAVA JDK / JRE

This is pretty simple. I like to install both the Dev Kit (JDK) and the Runtime (JRE) elements. To do so on Buster, simply request the default packages:

sudo apt install default-jdk default-jre

Let Apt do it’s work. Once done you should be able to verify java is installed and available:

java --version

My installed version reported openjdk 11.0.7 2020-04-14

INSTALLING NodeJS and NVM

Next, NodeJS will be installed and built from git

curl -sL https://deb.nodesource.com/setup_10.x | sudo bash -

Installing NVM follows a similar process

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash

Now the Pi is setup and ready to install other helper applications.

INSTALLING ROOSEVELT SCAFFOLDING

For this project, I selected Roosevelt based on successful prior experience with a complex NodeJS / Java web application built on Roosevelt. Follow the prompts to create the basic application.

I selected a directory called lakedash and an app named app .

npx mkroosevelt

SETTING UP THE BASIC APP FRAMEWORK

Change directories to the path lakedash/app (or whatever you picked).

cd  lakedash/app

Now, run the node install process to download and localize the packages used by Roosevelt. It’s easy.. just do it.

npm i

Starting up the simple little app in development mode.

npm run dev

Next phase will be setting up some swap space by re-partitioning the little Pi’s SD card.

Creating A Personalized Recreation Dashboard – Setting up the Pi – Part 1

With the impending move to one of the state’s premier recreation areas, I’m embarking upon setting up a customized data systems based on the Raspberry Pi, that will collect and display recreation related data. Some of the goals of this project are:

THE PLAN

  • River flow rate — knowing what the current river flow helps decided if it will be a weekend of tubing, kayaking or finding something else to do. River flow is managed by the Army Corps of Engineers and can vary greatly. Knowing the current flow helps plan the day.
  • Lake Level — certain boat ramps, beaches and parks are better at near-full level than not. Having a good idea of what the beaches and marinas might be like before heading down the hill to the lake can be helpful.
  • Area Traffic — this one will be tricky, but the plan is to leverage TXDOT traffic APIs to find out what the traffic might be like heading into the major cities (ex. Austin or San Antonio) should there be a need to do so. There are a lot of existing resources but having it all in a dashboard would be nice to have.
  • WX — of course.. some sort of simple weather forecast dash to show the day’s expected temps, if lightning is in the forecast (you don’t want to be on the river/lake when the heavens open up), etc.

THE RECREATION DEVICE

The plan is to use a Raspberry Pi 3b as the data harvesting and webapp platform. In addition to the standard Pi, I’ve added a 3.5″ touchscreen to display the data in a stand-alone format.

Raspberry Pi 3b with 3.5″ touchscreen

The full plan for the webapp will support both the micro 3.5″ display as well as interface versions for larger displays such as a table or desktop.

CONFIGURING THE PI

Once you have downloaded the Pi Image you want to run, have it burned to the micro SD card and the system booted up, you’ll need to have it either plugged into your network on an ethernet cable, or attached via HDMI to a monitor.

I plugged the stripped down device into the router with short cable:

This image has an empty alt attribute; its file name is stratux-pi-initial.jpg

Once it was powered on, checked the router to locate it’s DHCP IP address, in this case it’s 192.168.1.220

Using the terminal window on my, I ssh’d into the device as the user ‘pi’ using default password ‘raspberry’ . Once connected, I set about updating the PI.

sudo apt-get upgrade

If you face a dpkg error.. this command will resolve it, and you can re-run the command above.

sudo dpkg --configure -a

sudo apt-get upgrade

SETTING UP WIFI

To untether the device from my LAN, you’ll need to configure the Pi to connect to your WiFi. If you are running the desktop version of Raspberian, you can do this with the networking setup dialogs. If you do not, here are the steps for setting up your networking, including defining your network and password.

Locate the wifi network configuration file and add the following data. You can of course use tools to encrypt the password in this file, but I’m not super-concerned about an exploit on my LAN so I’ve let this in plain text.

sudo vi /etc/wpa_supplicant/wpa_supplicant.conf

Add the following block, with your network specifics

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

country=USnetwork={
  ssid="YOUR-NETWORK-SSID"
  psk="your-super-secret-password-goes-here"
  key_mgmt=WPA-PSK
}

Now setup the wlan with a static IP address. You could configure a DNS service to automagically detect the IP address of the system, but for me it was easier for me to pick a static IP and configure the Pi’s network accordingly.

Edit the network/interfaces file:

sudo vi /etc/network/interfaces

Add the following block. This will set a static IP (make sure it’s in a range not managed by your DHCP to avoid future conflicts).

auto wlan0

iface wlan0 inet static
  metric 0
  wireless-power off
  hostname recserver
  wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
  address 192.168.1.100
  netmask 255.255.255.0
  gateway 192.168.1.1
  network 192.167.1.0
  broadcast 192.168.1.255
  dns-nameservers 192.168.1.1 8.8.8.8

SETTING UP THE DISPLAY

The display that I have installed in the Pi’s case is the MHS-3.5 display. Installing the drivers requires only a few commands. While logged in as the user ‘pi’ download the LCD-show package from git and configure execute the display configuration script.

git clone https://github.com/Lcdwiki/LCD-show.git

Move to the cloned directory and run the setup script

cd LCD-show
sudo ./MHS35-show

This will download and install packages required to use the display. Once it completes, you’ll want to reboot the Pi to verify that it is working properly.

sudo init 6

Upon reboot, the screen should become visible and operable.

In Part 2, I’ll cover installing Node, Maven and Node Package Manager.

Relevance. What is Search Relevance?

Working in the Search sector for nearly a decade now, it would seem like this is an easy question to answer. However, to a degree, it is not. Why can it be such a vexing concern? As simply put as I can; it’s all in the context of the search, not so much the content.

You can (and should) build the more beautifully indexed body of content imaginable, but without relevance applied to a search, it’s use is limited at best. Identifying the context of the search is critical to determining the relevancy of the results.

A good example from a book recently read, illustrated the importance of relevancy signals to search. Take the case of two doctors searching the term ‘myocardial infarction’.

Doctor #1 is in his office, looking for recent research into the causes of sudden death from heart attacks. He’s most interested in the latest articles regarding predictive signals or other metrics that might more accurately predict the likelihood of an event in his patients.

Doctor #2 is in the ER, faced with a perplexing acute MI; and is in need of a procedure that will stabilize the patient as rapidly as possible. Perhaps there is a new class of drugs that can help, and he needs to know how to administer them, quickly, or even know if his facility caries the desired drug.

Both of these physicians may be searching on the same term, but they needs are strikingly different. Giving each of these doctors the right (relevant) results quickly can only occur with the application of signals. Signals might include the location of the search? Is it coming from a network in the ER, or the subnet in the offices? Are their other signals such as which doctor may be making the search? Some doctors are assigned to ERs, others may not ever rotate through. These are all signals that can impact the relevancy of a search.

So, how does the search engine know the context? It needs to be fed the signals so it can calculate the relevancy of each result to the search being conducted. These signals need to be identified and provided by an intervening layer that understands the context of the search. That context might be the geolocation or known assignment, or even the past search history of a specific user that includes information on which links they more often explore. All of these are signals that can be expressed to a search engine (or appliance) to improve relevancy.

How might these signals be expressed? Some examples, if I might diverge down a rabbit hole related to my own decade or so experience with Solr, would be applying boost formulas to specific fields containing results stored in certain fields or term sets in the body of content.

Perhaps for the ER doctor, documents with a field property of medical_procedure:myocardial medical_procedure:infarction would boost the score by a major factor, pushing how-to or other information to the top of the list.

In opposition to that, the other doctor’s search might look at boosting publication date (more recent, more boost) and other fields such as a flag indicating the content is research (e.g. research:true article:true). The way relevancy can be identified and improved will always require the application of some signals, of one type or another.

Other examples of signals that can be applied, might originate from the content curator. Any good search application needs to have SMEs (subject mater experts) and content curators continually updating the data and reviewing the results. Something a curator might do is apply a boost factor for products that are currently on sale, or have a ‘most popular’ rank that has recently changed. Maybe the product is over-stocked, or perhaps it was just discontinued. The curator is a critical pat of the relevancy solution. It can’t be left solely on the shoulders of the search engineering team to ‘guess’ that is relevant today, and what is not.

There needs to be a concentric circle of needs, so to speak, when considering the factoring and refactoring of the search (relevancy) solution. The layers (outer to inner) of this concentric feedback look might look like this.

Poor feedback / upset users / lost sales
Business and domain awareness
Content curation
Paired relevance tuning
Test-driven relevance.

The Takeaway

More important to a good search solution, and thus a good customer experience is a TEAM of people looking at the changing relevancy of content, tuning to meet the most urgent business and customer needs, and method of signaling the context of a search to ensure the most relevant results are returned, every time.

Stratux Active Ports

Dictionary of active ports found on a typical STRATUX ADS-B device.

Running nmap, between ports 100 and 48000, the following ports were found to be open on my STRAUX receiver:


Nmap scan report for 192.168.1.200
Host is up (0.0037s latency).
Not shown: 47892 closed ports
PORT STATE SERVICE
8080/tcp open http-proxy
9977/tcp open unknown
30001/tcp open pago-services1
30002/tcp open unknown
30003/tcp open unknown
30004/tcp open unknown
30005/tcp open unknown
30006/tcp open unknown
30104/tcp open unknown

STRATUX Port Mapping

Port Protocol Detected As Data Stream Contents
8080 tcp http-proxy HTTP 400 response
9977 tcp unknown JSON – Pi hardware data
30001 tcp pago-services1 no data seen on port
30002 tcp unknown TEXT – stream of hash/id strings

*20000FBF8811DA;
*8DA8B395E1160600000000E15133;
*8DAAF471591DA3F06975FD4FF289;
 
30003 tcp unknown TEXT – stream of ICAO ids in a message with timestamp and some Lat/Lon data

MSG,3,111,11111,A21F2F,...,10450,,,29.68991,-98.40042,,,,,,0
MSG,7,111,11111,A17D1A,...,3050,,,,,,,,,,0
30004 tcp unknown no data seen on port
30005 tcp unknown BINARY – unknown stream type
30006 tcp unknown JSON – stream of contact informtion with tail, squawk, Lat/Lon etc.

{“Icao_addr”:11064874,”DF”:0,”CA”:0,”TypeCode”:19,”SubtypeCode”:1,”SBS_MsgType”:7,”SignalLevel”:0.000378,”Tail”:null,”Squawk”:null,”Emitter_category”:null,”OnGround”:false,”Lat”:null,”Lng”:null,”Position_valid”:false,”NACp”:null,”Alt”:35000,”AltIsGNSS”:false,”GnssDiffFromBaroAlt”:null,”Vvel”:null,”Speed_valid”:false,”Speed”:null,”Track”:null,”Timestamp”:”2020-10-08T14:07:01.547Z”}
{“Icao_addr”:11425751,”DF”:17,”CA”:5,”TypeCode”:11,”SubtypeCode”:0,”SBS_MsgType”:3,”SignalLevel”:0.000388,”Tail”:null,”Squawk”:null,”Emitter_category”:null,”OnGround”:false,”Lat”:29.654480,”Lng”:-97.743935,”Position_valid”:true,”NACp”:null,”Alt”:17025,”AltIsGNSS”:false,”GnssDiffFromBaroAlt”:null,”Vvel”:null,”Speed_valid”:false,”Speed”:null,”Track”:null,”Timestamp”:”2020-10-08T14:07:02.319Z”}

30104 tcp unknown no data seen on port

Configure Maven pom.xml to build integrated executable .jar (fat jar)

Configuring Apache Maven to build an integrated .jar file took a little research, especially when building with NetBeans.. but it can be done with a little hand editing.

NOTE: Project using this example can be forked/pulled from: IngeniiCode/AvMet

Setting up the basic POM

My project originated as a NetBeans nbproject, but I wanted to convert it over to Maven for a variety of reasons, not the least of which was standardization. To do this I created a dummy Maven project, copied the pom.xml and reconfigured my project. There are many tutorials on that, so I won’t cover that here; but I will cover the pom.xml itself for reference to others, as well as myself.

Main Block

The entire pom.xml is bounded by this tag group:

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> </project>
Basic Properties

This block defines the most basic properties of the application. I’m including the bounding block here one more time just for continuity / reference; even though all blocks are within this bounding block. The ellipsis ( […] ) is not part of the package.. it’s only denoting that there is more of pom than just this section.

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.ingeniigroup.stratux</groupId> <artifactId>AvMet</artifactId> <version>0.1.0</version> <packaging>jar</packaging> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> </properties> [...] </project>
Dependencies Block

For the current iteration of this project, I am importing the JDBC library for accessing SQLite database. This is a fairly heavy-weight chunk of code (resulting jar is over 6 MB). Before using the Maven configuration, project simply included the current copy of the jar. That was OK for a Proof of Concept but, bad for security patching, and keeping updates integrated when they are released. Your own Maven version handling scheme will of course dictate when/if you define later updates.. but this will get you started.

This configuration is latest as of time time of this was originally published (2-NOV-2017).

<dependencies> <!-- https://mvnrepository.com/artifact/org.xerial/sqlite-jdbc --> <dependency> <groupId>org.xerial</groupId> <artifactId>sqlite-jdbc</artifactId> <version>3.20.0</version> </dependency> </dependencies>
The Build Configuration

Finally, the build block will determine how your jar(s) are built.

This configuration will end up generating two Jars (as of original publishing date, the version number was 0.1.0). AvMet-0.1.0.jar is the stripped Jar, and to execute will require the other supporting jars to be available in a ‘lib/’ sub directory. The other jar, AvMet.jar is the integrated (fat) executable jar.

74382 Nov 2 10:57 target/AvMet-0.1.0.jar 6708128 Nov 2 10:57 target/AvMet.jar

This build block will create those two executables. To prevent Maven from appending the string ‘jar-with-dependencies’ to your combined executable, the option ‘<appendAssemblyId>false</appendAssemblyId>’ must be defined in your build configuration.

To create an integrated (single) jar, this ‘<goals>’ block must be defined:

<goals> <goal>single</goal> </goals>

In addition, to generate a specific final jar name (such as not without the version number), the ‘<finalName>${project.name}</finalName>’ tag will enable that action.

This is what my Maven build block looks like:

<build> <plugins> <plugin> <!-- Build an executable JAR --> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <version>3.0.2</version> <configuration> <archive> <manifest> <addClasspath>true</addClasspath> <classpathPrefix>lib/</classpathPrefix> <mainClass>com.ingeniigroup.stratux.AvMet.AvMet</mainClass> </manifest> </archive> </configuration> </plugin> <plugin> <!-- Build a *fat* executable JAR --> <artifactId>maven-assembly-plugin</artifactId> <executions> <execution> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> <configuration> <!-- tell plugin to NO addpend the descriptionRef into target filename --> <appendAssemblyId>false</appendAssemblyId> <!-- define the final assembly name --> <finalName>${project.name}</finalName> <archive> <manifest> <addClasspath>true</addClasspath> <mainClass>com.ingeniigroup.stratux.AvMet.AvMet</mainClass> </manifest> </archive> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> </plugin> </plugins> </build>

Conclusion

One you have a good pom.xml setup, you can build run within NetBeans and end up with an integrated executable that will run with this simple command path:

java -jar target/AvMet-0.1.0-jar-with-dependencies.jar ../sqlite-dbs/stratux.sqlite.11-01 keepdb scrub