Tag Archives: database

Stratux – repartition to use entire sd card

The issue..

It’s been an ongoing tail of woe, when it comes to installing a new version of STRATUX on a new card; out of the box, at least in all the installs I’ve done, it fails to make use of the entire card. With any sort of logging enabled to measure performance, run tests or just keep even a short history of contacts received.. you’re out of disc space… FAST:

Here is the ‘df’ dump of a STRATUX fixed position station running the latest version (released September 2019). As you can see, it’s only allocated 2GB of the 32GB card’s total available space. That is a lot of wasted space on the card.

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       1.8G  1.6G  110M  94% /
/dev/mmcblk0p1   60M   20M   41M  34% /boot

Get total device size:

fdisk -l | grep Disk

Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors
...
Disk /dev/mmcblk0: 29.7 GiB, 31914983424 bytes, 62333952 sectors

Disklabel type: dos
Disk identifier: 0xe6a544c8

Repartitioning the Device

With the physical partition located.. start fdisk:
fdisk -u /dev/mmcblk0

I like to increase the size of the main partition to 6G to leave room for installing more system updates and tools.

To do this you will need to know the starting and ending blocks of the partition. That is available with the ‘print’ command:

Command (m for help): p

Results:

Disk /dev/mmcblk0: 29.7 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Disk identifier: 0xe6a544c8

Device         Boot  Start     End Sectors  Size Id Type
/dev/mmcblk0p1        8192  131071  122880   60M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      131072 3887103 3756032  1.8G 83 Linux

To increase the size, the partition must first be deleted, then re-create at the exact same starting block, or the filesystem will become corrupted.

First, delete the partition with the ‘d’ command, selecting partition #2.

Next, re-create the partition with the same starting block, but now with increased filesystem size:

Command (m for help): d 

  Partition number (1,2, default 2): 2

  Partition 2 has been deleted.

Command (m for help): p
 
  Device         Boot Start    End Sectors Size Id Type
  /dev/mmcblk0p1       8192 131071  122880  60M  c W95 FAT32 (LBA)


Command (m for help): n

  Partition type
     p   primary (1 primary, 0 extended, 3 free)
     e   extended (container for logical partitions)
  Select (default p): p
  Partition number (2-4, default 2): 2

First sector (2048-62333951, default 2048): 131072

Last sector, +sectors or +size{K,M,G,T,P} (131072-62333951, default 62333951): +6G

  Created a new partition 2 of type 'Linux' and of size 6 GiB.

Command (m for help): w
  
  The partition table has been altered.
  Calling ioctl() to re-read partition table.
  Re-reading the partition table failed.: Device or resource busy

  The kernel still uses the old table. The new table will be used
  at the next reboot or after you run partprobe(8) or kpartx(8).

The partition table will have been modified but the kernel will not be able to take that into account as some partitions are mounted.

In theory, the command ‘partx /dev/mmcblk0’ is all that is required.. however I’ve found that rebooting is the only way to really reload the partition, so that the filespace can be increased.

reboot

Once system comes back up, run ‘resize2fs‘ to expand the filesystem.

Fill up drive with current filesystem

Execute `resize2fs` and run an on-line expansion of the filesystem, and finally verify it again with ‘df -h’

resize2fs /dev/mmcblk0p2

  resize2fs 1.43.3 (04-Sep-2016)
  Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing
  required
  old_desc_blocks = 1, new_desc_blocks = 1
  The filesystem on /dev/mmcblk0p2 is now 1572864 (4k) blocks long.

Running df shows that it has in fact resized. STEP 1 COMPLETED

root@STRATUX-FIXED:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       5.9G  1.6G  4.2G  28% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           463M     0  463M   0% /dev/shm
tmpfs           463M  6.2M  457M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           463M     0  463M   0% /sys/fs/cgroup
/dev/mmcblk0p1   60M   20M   41M  34% /boot

STRATUX – Add swap and logging partitions.

It’s back! The STRAUX project lives again. With a fresh build of Stratux, it’s time to re-configure the SD card for swap space (new!) and a large logging partition to keep a full filesystem from crashing the device.

Get Current Partition Information

Once logged into the Straux box, switch to the root user and interrogate the disk partitions.

pi@raspberrypi: sudo su -
root@raspberrypi: fdisk -l
[...]

Device Boot Start End Sectors Size Type
/dev/mmcblk0p1 8192 131071 122880 60M W95 FAT32 (LBA)
/dev/mmcblk0p2 131072 3700592 3569521 17.G Linux

The important aspects you will want to record as the End block ids for each partition, this is important in the next step, creating more partitions.

Creating the Extended Partition

First task is I want to create the extended partition that can be used for swap. This will eventually contain the swap and log partitions.

Running the fdisk tool, start to edit the partition. You will want to reference the base block of the device.. eg: /dev/mmcblk0

root@raspberrypi: fdisk /dev/mmcblk0

Command (m for help): p

Device Boot Start End Sectors Size Type
/dev/mmcblk0p1 8192 131071 122880 60M W95 FAT32 (LBA)
/dev/mmcblk0p2 131072 3700592 3569521 17.G Linux

Command (m for help): n

Partition type
p primary (2 primary, 0 extended)
l logical (numbered from 5)
Select (default p): l

At this point, the tool threw an error adding partition 5…

Partition 5 is already defined. Delete it before re-adding it

Listing the partitions shows this:

Device Boot Start End Sectors Size Type
/dev/mmcblk0p1 8192 131071 122880 60M W95 FAT32 (LBA)
/dev/mmcblk0p2 131072 3700592 3569521 1.7G Linux
/dev/mmcblk0p3 3700593 62333951 5863d359 28G Extended
/dev/mmcblk0p5 2191 4194446 4192256 2G Linux

Re-running ‘n option and letting it setup partition 6, got me where I wanted to be. This process didn’t seem like it worked right.. but the end result matches up with my goal of a huge partition and a smaller 2G to be used for swap:

Command (m for help): n

Partition type
p primary (2 primary, 0 extended)
l logical (numbered from 5)
Select (default p): l

Adding logical partition 6
First Sector: 4198400
Last Sector: 62333951

Created a new partition 6 of type 'Linux and a size of 27.7 GiB

Command (m for help): p
Device Boot Start End Sectors Size Type
/dev/mmcblk0p1 8192 131071 122880 60M W95 FAT32 (LBA)
/dev/mmcblk0p2 131072 3700592 3569521 1.7G Linux
/dev/mmcblk0p3 3700593 62333951 5863d359 28G Extended
/dev/mmcblk0p5 2191 4194446 4192256 2G Linux
/dev/mmcblk0p6 4198400 62333951 58135552 27.7G Linux

Command (m for help): w

.. this is where the Ending block of your p2 partition number comes into play. The starting sector will be the ending number of your last partition (3569521) + 1 for next sector:


First Sector: 3700593
Last Sector: 62333951 (this was the default/max)

Created a new partition 3 of type 'Extended' and of size 28 GiB

Command (m for help): n

Partition type
p primary (2 primary, 0 extended)
l logical (numbered from 5)
Select (default p): l

Adding the largest partition, for log file storage

Now repeat this process to add the 2nd new partition, this one will consume the rest of the device, and eventually will be where the `/log` directory is mounted.

root@raspberrypi: fdisk /dev/mmcblk0

Command (m for help): p

Device Boot Start End Sectors Size Type
/dev/mmcblk0p1 8192 131071 122880 60M W95 FAT32 (LBA)
/dev/mmcblk0p2 131072 3700592 3569521 1.7G Linux
/dev/mmcblk0p3 3700593 7895039 4194447 2G Extended

Command (m for help): n

Partition type
p primary (2 primary, 0 extended)
l logical (numbered from 5)
Select (default p): l

First Sector: 4194448

STRATUX — Database Log Rotator

Space in the /var/log directory of a Stratux device, can quickly become consumed, if you don’t have a log management / rotation strategy.

I use a couple of mitigation methods, including a dedicated filesystem for /var/log (so the system does not become unstable and crash).

On the Stratux device itself, I use the following script to auto-rotate the sqlite database file every 24 hours, using a crontab entry.

crontab entry
PATH=/root/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # Run the Stratux DB Rollover 5 6 * * * /root/stx-rollover.sh >> /var/log/stx-rollover.log
Rotation script

stx-rollover.sh

#!/bin/bash # Startup Settings STARTTIME=`date +%m-%d_%T`; TIMESTAMP=`date +%m-%d`; LOGPATH="/var/log"; OLDFILE="$LOGPATH/stratux.sqlite"; NEWFILE="$LOGPATH/stratux.sqlite.$TIMESTAMP"; echo "$STARTTIME === Stratux Rollover === " echo "Stratux STOP" service stratux stop echo "Check for $OLDFILE" ls -ltr $OLDFILE if [ -e "$OLDFILE" ] then echo "Moving $OLDFILE ==> $NEWFILE" mv $OLDFILE $NEWFILE else echo "ERROR - Unable to locate database $OLDFILE" fi ## Startup Stratux Now. echo "Stratux START" service stratux start if [ -s "$NEWFILE" ] then echo "Moved DB to $NEWFILE" ls -l $NEWFILE echo "Compressing $NEWFILE" gzip $NEWFILE else echo "ERROR - Unable to locate $NEWFILE" fi echo "Rollover Completed"

End result of this process, is a list of gzipped date stamped databases:

[...] 152905988 Nov 4 01:05 stratux.sqlite.11-04.gz 135434058 Nov 6 11:04 stratux.sqlite.11-05.gz 148176518 Nov 6 11:45 stratux.sqlite.11-06.gz 157341677 Nov 7 13:10 stratux.sqlite.11-07.gz [...]

In addition to this logfile rotation strategy, I also pull off these files on a daily basis and archive them to another *NIX based system in my local network. This process is handled using the rsync utility. This process is run every 24 hours, using a cron job on the archival system.

rsync file archiving process

download.dbs.sh

echo '======================================' date rsync -a --remove-source-files -e "ssh -l root" 192.100.0.21:/var/log/stratux.sqlite.*.gz /Development/STRATUX/sqlite-dbs/.

This should be a good starting point for your own Stratux logfile management strategy.