Open-Source Data and Personal Analytics

I considered authoring a lengthy one-sided debate about my personal struggle with the openness of the Twitter era and/or my recent fascination with open-source code, business, and everything, but thought better of it. That’s a lot of typing. Instead, I’ll walk you through my thought progression by showing the highlight reel of relevant Internet discoveries.

  1. A Brief History of Open-Source Code

Following up on an earlier post about open-source business, I discovered this infographic (worth more words/commits than I could count) and thought it was an excellent illustration even for those unfamiliar with the datasets displayed:

A brief history of open-source code


“The 16 languages represented account for 90% of total commits”

2. A 2009 Tim Berners-Lee TEDTalk worth revisiting.

In this impassioned talk, Berners-Lee spoke about the importance of linked data and its role in shaping our future as a society. Through a variety of clear visuals and his eminently persuasive perspective, he ignited a linked data revolution and urged the audience to stop holding their knowledge to themselves and thus limiting the potential of our interconnected world and to apply this philosophy to their personal as well as professional lives. In the slide below, he illustrates the power of data linkages through the example of DBpedia.

The graphic is impressive on its own, but stood out to me in large part because it illustrates the powerful connections that can be drawn between “personal” (e.g., Flickr or MySpace) and academic (e.g., US Census Data and Project Gutenberg).

3. Stephen Wolfram on Personal Analytics

Berners-Lee makes his point clearly: Linked data is the key to unlocking the untapped potential of technology, and we all have a responsibility to contribute. We love to consume data, he points out, but so many of us engage in this practice of “data-hugging,” protecting our own information as though it represented our souls.

The debate over access to personal data is heated, so I won’t deal with that now. But personal data can give us power even if we keep it to ourselves – you manage what you measure, right? Stephen Wolfram, creator of Wolfram Alpha (among many other things), shows us how.

By recording his own activities over nearly two decades, Wolfram has amassed a powerful database from which he can derive real, data-driven meaning about his own activities. Take, for example, this chart of emails sent between 1990 and today:

Pretty cool in isolation, but it’s only one piece of the habit puzzle. Now, take this plot and plot it against data captured over the years illustrating the patterns of other life activities, and you start to see some fascinating trends:

Life habits plot

Stephen Wolfram’s life habits, plotted by hour

As humans, we’re endlessly inclined to make decisions based on our “gut feelings.” Constantly, even – perhaps especially – when we don’t realize we’re doing it, we take in impressions of our surroundings and our own behavior. These impressions then inform our assumptions about our realities that may very well be off-base; by collecting data about ourselves, we equip ourselves to make decisions based on reality rather than our assumptions.

4. Web Timer (Chrome)

I was inspired by Stephen Wolfram’s dataset, so I searched around for some tools that would allow me to mimic his controlled data-collection. Specifically, since my professional life does and will continue to exist predominantly online, I thought I’d start by tracking my digital activity in order to understand my own productivity.

The Chrome app store provides an app that will do just that (or so it claims). It tracks the amount of time I spend on a given page in a session and globally over time. Through this, I will be able to see what kinds of websites I view, when, how much time I spend on each, and how often I flip between pages.

A Chrome extension that tracks how you spend your time on the web (from

A Chrome extension that tracks how you spend your time on the web (from

5. iPhone Tracker (via GitHub)

This is an inevitably controversial but undeniably incredible project: using location data collected from your iPhone on import, iPhone Tracker displays a heat map of your activities based on the number of cell phone towers utilized in a given area. Check mine out:

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s