Replace Text Recursively using grep and sed on MacOS (osX).

As with many MacOS’isms, you have to change up your old school LINUX to work with the slightly different command syntax of the MacOS tools. Specifically in the this case xargs and sed.

xargs can be an amazingly powerful ally when automating commands line functionality, wrapping the reference command with auto-substitutions using the ‘@’ meta-character.

I’ll get to the meat of this. While trying to automated a grep -> sed pipeline, I encountered this error:

egrep -rl 'version 0.0.1' src/* | xargs -i@ sed -i 's/version 0.0.1/version 0.0.1c/g' @
xargs: illegal option -- i

It turns out that MacOS xargs like -I instead of -i.. (a quick trip to man sorted that out).

     -I replstr
             Execute utility for each input line, replacing one or more occurrences of replstr in up to replacements (or 5 if no -R flag is specified) arguments to utility
             with the entire line of input.  The resulting arguments, after replacement is done, will not be allowed to grow beyond 255 bytes; this is implemented by con-
             catenating as much of the argument containing replstr as possible, to the constructed arguments to utility, up to 255 bytes.  The 255 byte limit does not apply
             to arguments to utility which do not contain replstr, and furthermore, no replacement will be done on utility itself.  Implies -x.

Next I ran into an errors with sed.. when using the ‘edit in place’ flag -i. (yes, that makes for a confusing debug when you have the “offending” switch in two places).

egrep -rl 'version 0.0.1' src/* | xargs -I@ sed -i 's/version 0.0.1/version 0.0.1c/g' @
sed: 1: "src/com/ingeniigroup/st ...": bad flag in substitute command: 'a'
sed: 1: "src/com/ingeniigroup/st ...": bad flag in substitute command: 'a'
sed: 1: "src/com/ingeniigroup/st ...": bad flag in substitute command: 'a'
[...]

With a little testing of the individual command, and another trip to man, it took a little noodly to deduce that the error message was being generated because it was trying the sed commmand as the output file suffix and the filename as the sed command! Adding an empty string designator after sed’s -i solved this.

     -i extension
             Edit files in-place, saving backups with the specified extension.  If a zero-length extension is given, no backup will be saved.  It is not recommended to give
             a zero-length extension when in-place editing files, as you risk corruption or partial content in situations where disk space is exhausted, etc.

Final Command Query
The objective was to simply hammer through the source code and update the version tag from 0.0.1 to 0.0.1c. Running egrep in recursive filename list mode ( egrep -rl ) for the string I wanted ( ‘version 0.0.1’ ) gave me a file list, which was then piped into xargs, which expanded that list into the sed command ( sed -i ” ‘s/version 0.0.1/version 0.0.1c/g’ ). And viola.. 18 files changed with just a little big of effort:

egrep -rl 'version 0.0.1' src/* | xargs -I@ sed -i '' 's/version 0.0.1/version 0.0.1c/g' @

Since this took me more than 5 minutes to figure out, I decided I’d take 5 more and hopefully help someone else down the line.

AvMet Beta Release – STRATUX Database Reporter Project (open source!)

AvMet Beta Available on GitHub


For that last 6 months I have been capturing Aviation Mode-S and ADS-b data transmissions using my own customizd STRATUX Aviation Traffic Receiver running Raspberry Pi 3b. With my modifications, the device automatically shuts down daily for a few seconds to release the current SQLite3 database, and the start up pointing at a fresh database. Once released, the database is compressed to save space and for later download (I use rsync to automate that as well).

These database files can be very big, and contain hundreds of thousands of contact events each day. When these databases are generated every day, it doesn’t take long to get a backlog of them to examine for interesting things, such as NASA aircraft and some mysterious high-flyers using spoofed identities.

Being a professional programmer and data guru, this seemed like a perfect project to open-source and use a marketing tool for my consulting. After a week of design, development and preliminary tests.. it’s ready for comment.

What it does:


The tool is purpose built for my needs, so it might not (yet) be easily used on your own Stratux database files, but I feel I’ve made a reasonable effort in the interests of ease.

Each database file is opened and checked for a number of conditions and data:

  • Verify database contains expected STRATUX data tables.
  • Select first timestamp record and report it out. Since the name of the file might not be self descriptive, this is really helpful to know the dataset’s start time:
    Using Database File: './sqlite-stratux-temp'
    	Start: 2017-10-10 08:11:09.967 +0000 UTC
    	-----------------------------------------
    
  • Check dataset for obvious duplicates and remove them. It’s not uncommon for the message log to repeat a fixed position up to 5 times. Although this does not affect the current metrics, this bloats the dataset. There is an optional flag to disable this feature
  • Once duplicates are removed, each airframes dataset is checked for gross errors, such as changes in altitude or speed that would be possible only with alien spacecraft; records are also removed. There is an optional flag to disable this feature
      10486213 -- Bad Distance  	   24924 --> 51857   
      10617769 -- Bad Distance  	   82039 --> 206496  
      10707621 -- Bad Speed              318 --> 8303
    
  • Following data repair a few metrics are pulled from the dataset, looking for interesting boundary events such as Fastest and Furthest contacts. Each record reports the Callsign or Tail Registration number (when available), and the Mode-S ICAO24 code as well as altitude, speed and distance for the specific event. Example:
    	  FASTEST:   XAOLE [0D0AAE]   43000 ft.  @   529 kts.      23 mi.
    	  SLOWEST:   NDU45 [A6EBAF]    1150 ft.  @    61 kts.    2.26 mi.
    	  HIGHEST:   XAOLE [0D0AAE]   47025 ft.  @   505 kts.      92 mi.
    	   LOWEST:  N41218 [A4DE2A]     550 ft.  @    77 kts.    1.71 mi.
    	  CLOSEST:  N6464R [A87E79]    2525 ft.  @    84 kts.    0.00 mi.
    	 FURTHEST:  N229NN [A203A9]   36000 ft.  @   472 kts.     128 mi.
    

    Here is an example of an interesting contact captured last year along the coast of California; a high-altitude signal from a NASA 747 test aircraft almost 100 miles off the coast.

    	  HIGHEST: NASA747 [AA0DB8]   43025 ft.  @   473 kts.      32 mi.
    	 FURTHEST: NASA747 [AA0DB8]   43000 ft.  @   466 kts.      92 mi.
    
  • During this phase of the process, special Squawk Code events are trapped that might indicate special civil, science or military operations. Example:
    	**ALERT:  N7253N [A9B8ED] 4403     2500 ft.  @   118 kts.  SR-71, YF-12, U-2 and B-57, pressure suit flights
    	**ALERT:  N7272N [A9C038] 4402     2075 ft.  @   116 kts.  SR-71, YF-12, U-2 and B-57, pressure suit flights
    	**ALERT:  N7274N [A9C07E] 4404     1250 ft.  @   100 kts.  SR-71, YF-12, U-2 and B-57, pressure suit flights
    	**ALERT:  N907CH [AC8967] 4442     1800 ft.  @   140 kts.  SR-71, YF-12, U-2 and B-57, pressure suit flights
    

    In the above case, N7253N, N7272N and N7274N are Department of Homeland Security Border Patrol helicopter contacts.

Here is a recent report example:

        Start: 2017-10-10 08:11:09.967 +0000 UTC
        -----------------------------------------
          FASTEST: AFR6721 [3951C1]   32975 ft.  @   531 kts.      14 mi.
          SLOWEST:  N42894 [A51D07]     950 ft.  @    51 kts.    3.21 mi.
          HIGHEST:    JCB1 [43E9D5]   47025 ft.  @   469 kts.      34 mi.
           LOWEST:  N724DP [A9B2C1]     400 ft.  @    72 kts.    1.65 mi.
          CLOSEST:  N605CH [A7DA0C]   16200 ft.  @   400 kts.    0.02 mi.
         FURTHEST:  N9023N [AC7968]   33050 ft.  @   433 kts.     143 mi.

Where can you get it?

You can fork or pull the current source code from GitHub here: IngeniiCode AvMet