Tag Archives: pi3

Amateur Radio, Aviation, Raspberry Pi, Software Development, Stratux, Technology

Stratux – repartition to use entire sd card

Leave a comment

The issue..

It’s been an ongoing tail of woe, when it comes to installing a new version of STRATUX on a new card; out of the box, at least in all the installs I’ve done, it fails to make use of the entire card. With any sort of logging enabled to measure performance, run tests or just keep even a short history of contacts received.. you’re out of disc space… FAST:

Here is the ‘df’ dump of a STRATUX fixed position station running the latest version (released September 2019). As you can see, it’s only allocated 2GB of the 32GB card’s total available space. That is a lot of wasted space on the card.

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       1.8G  1.6G  110M  94% /
/dev/mmcblk0p1   60M   20M   41M  34% /boot

Get total device size:

fdisk -l | grep Disk

Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors
...
Disk /dev/mmcblk0: 29.7 GiB, 31914983424 bytes, 62333952 sectors

Disklabel type: dos
Disk identifier: 0xe6a544c8

Repartitioning the Device

With the physical partition located.. start fdisk:
fdisk -u /dev/mmcblk0

I like to increase the size of the main partition to 6G to leave room for installing more system updates and tools.

To do this you will need to know the starting and ending blocks of the partition. That is available with the ‘print’ command:

Command (m for help): p

Results:

Disk /dev/mmcblk0: 29.7 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Disk identifier: 0xe6a544c8

Device         Boot  Start     End Sectors  Size Id Type
/dev/mmcblk0p1        8192  131071  122880   60M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      131072 3887103 3756032  1.8G 83 Linux

To increase the size, the partition must first be deleted, then re-create at the exact same starting block, or the filesystem will become corrupted.

First, delete the partition with the ‘d’ command, selecting partition #2.

Next, re-create the partition with the same starting block, but now with increased filesystem size:

Command (m for help): d 

  Partition number (1,2, default 2): 2

  Partition 2 has been deleted.

Command (m for help): p
 
  Device         Boot Start    End Sectors Size Id Type
  /dev/mmcblk0p1       8192 131071  122880  60M  c W95 FAT32 (LBA)


Command (m for help): n

  Partition type
     p   primary (1 primary, 0 extended, 3 free)
     e   extended (container for logical partitions)
  Select (default p): p
  Partition number (2-4, default 2): 2

First sector (2048-62333951, default 2048): 131072

Last sector, +sectors or +size{K,M,G,T,P} (131072-62333951, default 62333951): +6G

  Created a new partition 2 of type 'Linux' and of size 6 GiB.

Command (m for help): w
  
  The partition table has been altered.
  Calling ioctl() to re-read partition table.
  Re-reading the partition table failed.: Device or resource busy

  The kernel still uses the old table. The new table will be used
  at the next reboot or after you run partprobe(8) or kpartx(8).

The partition table will have been modified but the kernel will not be able to take that into account as some partitions are mounted.

In theory, the command ‘partx /dev/mmcblk0’ is all that is required.. however I’ve found that rebooting is the only way to really reload the partition, so that the filespace can be increased.

reboot

Once system comes back up, run ‘resize2fs‘ to expand the filesystem.

Fill up drive with current filesystem

Execute `resize2fs` and run an on-line expansion of the filesystem, and finally verify it again with ‘df -h’

resize2fs /dev/mmcblk0p2

  resize2fs 1.43.3 (04-Sep-2016)
  Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing
  required
  old_desc_blocks = 1, new_desc_blocks = 1
  The filesystem on /dev/mmcblk0p2 is now 1572864 (4k) blocks long.

Running df shows that it has in fact resized. STEP 1 COMPLETED

root@STRATUX-FIXED:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       5.9G  1.6G  4.2G  28% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           463M     0  463M   0% /dev/shm
tmpfs           463M  6.2M  457M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           463M     0  463M   0% /sys/fs/cgroup
/dev/mmcblk0p1   60M   20M   41M  34% /boot
Aviation, Software Development, Stratux, Technology

STRATUX – Filesystem Full; Managing disk space redux

Leave a comment

Checking in with my Starux project this morning, I found it unresponsive. A physical check shows a flashing red light on the Pi… something has gone haywire, and I couldn’t SSH into the little thing, so a really hard cold restart was in order.

Following the restart I quickly shut Stratux back down to start performing diagnostics:


pi@raspberrypi:~ $ sudo su -

root@raspberrypi:~# service stratux stop

Diagnostics 101

Check Filesystem Health

The last time the little Stratux suddenly had problems it was a filesystem space issue ( previous article ). It turns out the boot drive space was OK but the new Logging partition was again, completely consumed:

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        1815440 1391284    331164  81% /
devtmpfs          469688       0    469688   0% /dev
tmpfs             474004       0    474004   0% /dev/shm
tmpfs             474004    6340    467664   2% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             474004       0    474004   0% /sys/fs/cgroup
/dev/mmcblk0p1     61384   20400     40984  34% /boot
/dev/mmcblk0p4   8125880 8109496         0 100% /var/log

Locate the Culprit

There are a number of ways to locate large file on a *NIX system. My favorite tool is find. First thing I want to do is locate any file that is larger than 1 Gigabyte, and sure enough it located a massive sqlite database file. The same one that ate up all the space on the boot drive. So.. this is going to require some more extrodinary measures to maintain 100% 24×7 operational status.


root@raspberrypi:~# cd /var/log
root@raspberrypi:/var/log# find . -size +1G -exec ls -l {} \;
-rw-r--r-- 1 root root 7463211008 Apr 29 21:48 ./stratux.sqlite
Start Solving

First order of business is to move aside the massive database, but try to preserve the data for examination. Since the filesystem is full, I can’t zip this thing in place, so first some space needs to cleared on the device. The things I’m least interested in go first.. like the zipped syslogs and any other ‘archived’ file (those with a .# suffix). 

-rw-r----- 1 root adm      728756 Apr 25 06:25 syslog.5.gz
-rw-r----- 1 root adm     2552845 Apr 26 06:25 syslog.4.gz
-rw-r----- 1 root adm     2447263 Apr 27 06:25 syslog.3.gz
-rw-r----- 1 root adm     2498089 Apr 28 06:25 syslog.2.gz
-rw-r----- 1 root adm    33735478 Apr 29 06:25 syslog.1
-rw-r----- 1 root adm       54307 Apr 30 05:19 debug.1
-rw-r----- 1 root adm      380928 Apr 30 05:19 kern.log.1
-rw-r----- 1 root adm   332365824 Apr 30 06:18 daemon.log.1
-rw-r----- 1 root adm       48723 Apr 30 06:25 auth.log.1
-rw-r----- 1 root adm           0 Apr 30 06:25 syslog.1.gz
-rw-r----- 1 root adm     1032192 Apr 30 06:25 messages.1

root@raspberrypi:/var/log# rm -f *.gz *\.[0-9]

But.. that’s not going to doe the complete trick, especially if after deleting files df still shows 100% utilization. You need to figure out what is holding which deleted file(s).

Normallyh, the best way to do that on *NIX is with lsof. Much to my chagrin, it was not available on the OS… so I had to go grab it. Thankfully I had the main filesystem on a different partition that still had enough space to install more tools! Once lsof was installed, grep through the list of open file handles and find those marked for delete.. and those are the processes that have handles pointing to those files. However, this didn’t help me with Rasperian Jessie. I know that a reboot will recover the space so.. that was the next step.. a brutal warm-boot.


root@raspberrypi:/var/log# lsof
-su: lsof: command not found

root@raspberrypi:/# apt-get install lsof
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libperl4-corelibs-perl
The following NEW packages will be installed:
libperl4-corelibs-perl lsof
...

root@raspberrypi:/var/log# lsof | grep deleted
root@raspberrypi:/var/log#

root@raspberrypi:/var/log# init 6

Archiving Massive SQLite Database

The first thing was to move aside the current database, and then restart stratux to verify it can created a new empty database for it’s purposes.. then shut it right back down again.


root@raspberrypi:/var/log# mv stratux.sqlite stratux.sqlite.1
root@raspberrypi:/var/log# service stratux start
root@raspberrypi:/var/log# ls -l
total 7291480
[...]
-rw-r--r-- 1 root root 4096 Apr 30 14:27 stratux.sqlite
-rw-r--r-- 1 root root 7466160128 Apr 30 14:24 stratux.sqlite.1
[...]
root@raspberrypi:/var/log# service stratux stop

A new empty database file has been created [ 4096 Apr 30 14:27 stratux.sqlite ]. This tells me that moving aside the current database file on a periodic basis, compressing and then archiving it should be sufficient to maintain operational status.

Logging Insanity

Before restarting Stratux, I zerod out these log files. Running for first a few moments these files were already reading up a lot of space. Tailing one of them I see that Stratux, with my current settings is logging A LOT of data to these log files. I feel this was my first error.. enabling too much logging. My settings look like this:

With those settings enabled, there are a lot of GPS and other events that I don’t really have a use for, being dumped into the Statux log.


-rw-r--r-- 1 root root 1361778 Apr 30 14:39 stratux.log

Turning OFF ‘Verbose Message Log’ made that insanity stop.

Replay logging is what is writing to the SQLite database. So the question is. how much of that data do I want to keep, and how much will I lose if I turn off the replay logs. I think that will be research for another day… right now the goal is to recover disk space by compressing the massive database file that was moved aside, and get Stratux stabilized again. Once compressed, check filesystem and file size!


root@raspberrypi:/var/log# gzip stratux.sqlite.1

root@raspberrypi:/var/log# df

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        1815440 1392384    330064  81% /
devtmpfs          469688       0    469688   0% /dev
tmpfs             474004       0    474004   0% /dev/shm
tmpfs             474004   12272    461732   3% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             474004       0    474004   0% /sys/fs/cgroup
/dev/mmcblk0p4   8125880  681792   7008276   9% /var/log
/dev/mmcblk0p1     61384   20400     40984  34% /boot

root@raspberrypi:/var/log# ls -lktr --color --block-size=M stratux.sqlite*

-rw-r--r-- 1 root root 643M Apr 30 14:24 stratux.sqlite.1.gz
-rw-r--r-- 1 root root   8M Apr 30 15:21 stratux.sqlite
-rw-r--r-- 1 root root   1M Apr 30 15:22 stratux.sqlite-shm
-rw-r--r-- 1 root root   5M Apr 30 15:22 stratux.sqlite-wal

With the file compressed, it could be copied elsewhere for analysis.