The Great iPhone Location File Controversy – is it really a problem?

Unless you have been living under a rock, or use a Windows Mobile Device (no difference), you no-doubt have heard about the reports floating around in the last day or so about the infamous consolidated.db location history file that is maintained on each 3G enabled iOS4 devices such as the iPhone and 3G iPad.

In fact, the number of articles discussing this issue (and now this one is also joining the fray) is extensive.

Regardless, it’s an issue of one sort or another, so I decided to delve into this a little bit. My research starts with this site on ReadWriteWeb: Your iPhone Is Tracking Your Every Move.

The article references this page [iPhoneTracker], where you can download and app, or the source code to compile a program to read your consolidated.db file. However, first you must find it! The instructions for finding the file were not completely accurate, there are some references with path misspellings etc. So, I’m going to re-do those pages, show you how I did it, include the Python script I ran to read the database file, and finally the steps I went about to move the file, compare snapshots of the file, and see if turning off location services, as it purported, solves this problem

First order of business is to get the App. Since I’m smart and use a MAC (and LINUX, but I don’t sync my phone to LINUX, so that’s not going to be discussed any further) I grabbed this zip file, extracted the app inside, and dropped it in my Applications pane.

Running the program, I see this:

As you can see, I don’t do much wandering around. Zoomed in you can see some of the places along the West Coast I have wandered since I purchased the current device. The App will display ALL data, or you can have it just display a specific weeks’ full of data, form any time frame in the device. Here is a shot of some travel I did during Thanksgiving 2010.

To say the dataset is full of inaccuracy’s is an understatement. Just look at this map of the last two days of travel. I can assure you, I was not in Sumner, nor in Olympia or any of those other places on the eastern side of the sound in a long time, much less that last two days:

And yet another, from the middle of last year. You can see a lot of ‘hits’ on Vancouver island. However, I have NEVER been there. Ever!

OK, so, I hope you can see that, the use of the data has it’s limitations. It’s not very accurate. In face it’s pretty inaccurate in enough cases to make it’s utility, dubious.

But, I wanted to know more, so I delved deeper into the files and went in search of the nefarious SQLite file itself. First stop was this page, where I grabbed a script, applied some mentioned patches, located the backups directory and find my consolidated location files (I found 4).

Here is the patched Python script:

#!/usr/bin/env python
import sys

def getint(data, offset, intsize):
    """Retrieve an integer (big-endian) and new offset from the current offset"""
    value = 0
    while intsize > 0:
        value = (value<<8) + ord(data[offset])
        offset = offset + 1
        intsize = intsize - 1
    return value, offset

def getstring(data, offset):
    """Retrieve a string and new offset from the current offset into the data"""
    if data[offset] == chr(0xFF) and data[offset+1] == chr(0xFF):
        return '', offset+2 # Blank string
    length, offset = getint(data, offset, 2) # 2-byte length
    value = data[offset:offset+length]
    return value, (offset + length)

def process_mbdb_file(filename):
    mbdb = {} # Map offset of info in this file => file info
    data = open(filename).read()
    if data[0:4] != "mbdb": raise Exception("This does not look like an MBDB file")
    offset = 4
    offset = offset + 2 # value x05 x00, not sure what this is
    while offset < len(data):
        fileinfo = {}
        fileinfo['start_offset'] = offset
        fileinfo['domain'], offset = getstring(data, offset)
        fileinfo['filename'], offset = getstring(data, offset)
        fileinfo['linktarget'], offset = getstring(data, offset)
        fileinfo['datahash'], offset = getstring(data, offset)
        fileinfo['unknown1'], offset = getstring(data, offset)
        fileinfo['mode'], offset = getint(data, offset, 2)
        fileinfo['unknown2'], offset = getint(data, offset, 4)
        fileinfo['unknown3'], offset = getint(data, offset, 4)
        fileinfo['userid'], offset = getint(data, offset, 4)
        fileinfo['groupid'], offset = getint(data, offset, 4)
        fileinfo['mtime'], offset = getint(data, offset, 4)
        fileinfo['atime'], offset = getint(data, offset, 4)
        fileinfo['ctime'], offset = getint(data, offset, 4)
        fileinfo['filelen'], offset = getint(data, offset, 8)
        fileinfo['flag'], offset = getint(data, offset, 1)
        fileinfo['numprops'], offset = getint(data, offset, 1)
        fileinfo['properties'] = {}
        for ii in range(fileinfo['numprops']):
            propname, offset = getstring(data, offset)
            propval, offset = getstring(data, offset)
            fileinfo['properties'][propname] = propval
        mbdb[fileinfo['start_offset']] = fileinfo
    return mbdb

def process_mbdx_file(filename):
    mbdx = {} # Map offset of info in the MBDB file => fileID string
    data = open(filename).read()
    if data[0:4] != "mbdx": raise Exception("This does not look like an MBDX file")
    offset = 4
    offset = offset + 2 # value 0x02 0x00, not sure what this is
    filecount, offset = getint(data, offset, 4) # 4-byte count of records 
    while offset < len(data):
        # 26 byte record, made up of ...
        fileID = data[offset:offset+20] # 20 bytes of fileID
        fileID_string = ''.join(['%02x' % ord(b) for b in fileID])
        offset = offset + 20
        mbdb_offset, offset = getint(data, offset, 4) # 4-byte offset field
        mbdb_offset = mbdb_offset + 6 # Add 6 to get past prolog
        mode, offset = getint(data, offset, 2) # 2-byte mode field
        mbdx[mbdb_offset] = fileID_string
    return mbdx

def modestr(val):
    def mode(val):
        if (val & 0x4): r = 'r'
        else: r = '-'
        if (val & 0x2): w = 'w'
        else: w = '-'
        if (val & 0x1): x = 'x'
        else: x = '-'
        return r+w+x
    return mode(val>>6) + mode((val>>3)) + mode(val)

def fileinfo_str(f, verbose=False):
    if not verbose: return "(%s)%s::%s" % (f['fileID'], f['domain'], f['filename'])
    if (f['mode'] & 0xE000) == 0xA000: type = 'l' # symlink
    elif (f['mode'] & 0xE000) == 0x8000: type = '-' # file
    elif (f['mode'] & 0xE000) == 0x4000: type = 'd' # dir
    else: 
        print >> sys.stderr, "Unknown file type %04x for %s" % (f['mode'], fileinfo_str(f, False))
        type = '?' # unknown
    info = ("%s%s %08x %08x %7d %10d %10d %10d (%s)%s::%s" % 
            (type, modestr(f['mode']&0x0FFF) , f['userid'], f['groupid'], f['filelen'], 
             f['mtime'], f['atime'], f['ctime'], f['fileID'], f['domain'], f['filename']))
    if type == 'l': info = info + ' -> ' + f['linktarget'] # symlink destination
    for name, value in f['properties'].items(): # extra properties
        info = info + ' ' + name + '=' + repr(value)
    return info

verbose = True
if __name__ == '__main__':
    mbdb = process_mbdb_file("Manifest.mbdb")
    mbdx = process_mbdx_file("Manifest.mbdx")
    sizes = {}
    for offset, fileinfo in mbdb.items():
        if offset in mbdx:
            fileinfo['fileID'] = mbdx[offset]
        else:
            fileinfo['fileID'] = ""
            print >> sys.stderr, "No fileID found for %s" % fileinfo_str(fileinfo)
        print fileinfo_str(fileinfo, verbose)
        if (fileinfo['mode'] & 0xE000) == 0x8000:
            sizes[fileinfo['domain']]= sizes.get(fileinfo['domain'],0) + fileinfo['filelen']
    for domain in sorted(sizes, key=sizes.get):
        print "%-60s %11d (%dMB)" % (domain, sizes[domain], int(sizes[domain]/1024/1024))

I placed the script in my home directory for now. Later I’ll move it off to the Applications directory. Setting the script to executable, I then set out to locate the manifest db files. To do this, I leveraged the find utility (don’t worry Windoze users… you’re not missing this utility, you never had it in the first place). I thus located these 3 manifest db files. The one I’m most interested in is dated today:

Now it’s time to run the python script, and grep for the filename I need.

Filename in hand, I check and verify it’s existence. The important part of the data returned is this:

(4096c9ec676f2847dc283405900e284a7c815836)RootDomain::Library/Caches/locationd/consolidated.db:

That number is the real filename. Checking that I see that the file is there. Making a copy of that file available in my home directory (using cp to copy it… that means ‘copy’ for you Windoze users reading from your bunkers).

Word on the street, is that this is a SQLite database file. I should be able to confirm that with a simple strings test, and the rumor is confirmed:

I opened the database file, and selected the first 5 records, and last 30 records from the CellLocation table (rumored to contain the data of interest).

The last 5 records recorded are:

MCC|MNC|LAC|CI|Timestamp|Latitude|Longitude|HorizontalAccuracy|Altitude|VerticalAccuracy|Speed|Course|Confidence
310|410|42979|9468182|325110108.640637|47.24654626|-122.43737727|2164.0|0.0|-1.0|-1.0|-1.0|70
310|410|42997|9468176|325110108.640637|47.24717628|-122.43819308|500.0|0.0|-1.0|-1.0|-1.0|50
310|410|42997|6781206|325110108.640637|47.24570667|-122.43808859|2138.0|0.0|-1.0|-1.0|-1.0|50
310|410|42997|7895312|325110108.640637|47.2472279|-122.43625974|1550.0|0.0|-1.0|-1.0|-1.0|70
310|410|42985|6781206|325110108.640637|47.24575543|-122.43641382|500.0|0.0|-1.0|-1.0|-1.0|50

Now, the time stamping get’s a little tricky.. it’s seconds since January 01, 2001 (I really don’t want to do the GMT / epoch offsets right now), but I don’t need to worry about that, all I really care about is that LAST location the phone recorded for me. If it did, in fact, honor my demand to turn OFF location services, my last point of origin should be in or north of Purdy.

Checking this Latitude and Longitute with Lougle Maps (OK, so it’s a corny movie reference, move on with your big bad selves)… I see….

Well, this does not bode well for the Apple ‘researchers’ that say turning off Location Services solves this problem. As far as I can tell it DOES NOT. I’ll be doing a little more research on this myself later, when I have time to verify the timestamps. But for now, it looks like, regardless if you like it or not.. Big Brother Steve Jobs is WATCHING!!!!

Just a few articles culled from a Google query on iOS4 devices

iOS 4 devices quietly track, store users’ locations | iLounge News

Apple tracking location of iOS4 device users, researchers say …
Apr 20, 2011 … A team of researchers have discovered that iOS4 is secretly obtaining your location and recording it to a hidden file, raising obvious …
www.betanews.com/article/Apple-tracking…iOS4–device…/1303319892

Got an iPhone or 3G iPad? Apple is recording your moves – O’Reilly …
Apr 20, 2011 … Ever since iOS 4 arrived, your device has been storing a long list of ….. Why not just visit http://oo.apple.come from any IOS4 device and …
radar.oreilly.com/2011/04/apple-location-tracking.html