Visualizing my entire Twitter history in Processing
2nd January 2012
Over the Christmas break I’ve been working on some simple data visualisation using my Twitter data and Processing. This was an exercise to help me learn more and hopefully give me some ideas for other projects. I find that generally I am more sociable when I am most happy and I think that my activity on Twitter is probably a good reflection of this. My eventual idea is to map my tweets to a calendar and then try and map events from my life and other data (such as my running data from nike plus, my sleep data from Sleep Cycle) onto this to see if things tally up. I could also start checking who I mention more frequently when I am tweeting a lot and then only knock about with them.
I have set myself a couple of smaller projects at first to get used to handling the data and also to help me think of ways to display the data usefully.
Project 1: Displaying all Tweets and comparing Frequency
Getting the data
Initially I Googled Twitter and Processing and found Jer Thorpe’s tutorial. This is great for using the Twitter API, and really simple to set up but Twitter will only allow you to query the last 20 posts from a user and I wanted to display my whole history (about 2800 tweets). After searching around I found a script that will download up to your last 3200 tweets:
To get this to work just add your username instead of mine for the name of the xml file. The script took a few minutes to run but soon my terminal window contained everything I had ever written on Twitter. If I was smarter on the command line I could have saved the data to an XML file and edited it so all tweets shared 1 root node (as there are 10 requests there are 10 XML files that need to be joined into 1). As I am not so good no the command line I copied and pasted into Textmate and did this manually- if anyone can suggest how to do this automatically let me know! Now I had a 7mb XML file with about 115,000 lines.
Parsing and displaying the data
As soon as my application launched I parsed the data into custom objects. Amazingly This only took a few seconds.
Once I had all of my tweets in an ArrayList I then cycled through the list to display each one chronologically and also drew a dot for each tweet divided into months. Here’s a video of all the app running start to finish:
Project 2: Interactive app, Cartesian to polar transition
One of my favourite transitions is displaying data around a circle and then transitioning the data to a line. This means you can display the relationships between all data well when in the circle and also show the data in a linear way that you can easily scroll through. Luckily toxiclibs handles this transition beautifully (check the Polar Unravel example included with the library). In this application each month is a different colour and the length of the line shows the index of the tweet (earliest is the shortest, most recent the longest).
Here is a video of the application in use:
This app needs some refinement to make it easy to use and could also display the data more usefully but it does help give an idea of the volume and frequency of tweets and it’s nice to see the polar to cartesian transition in action.
Visualizing my entire Twitter history in Processing
2nd January 2012
Over the Christmas break I’ve been working on some simple data visualisation using my Twitter data and Processing. This was an exercise to help me learn more and hopefully give me some ideas for other projects. I find that generally I am more sociable when I am most happy and I think that my activity on Twitter is probably a good reflection of this. My eventual idea is to map my tweets to a calendar and then try and map events from my life and other data (such as my running data from nike plus, my sleep data from Sleep Cycle) onto this to see if things tally up. I could also start checking who I mention more frequently when I am tweeting a lot and then only knock about with them.
I have set myself a couple of smaller projects at first to get used to handling the data and also to help me think of ways to display the data usefully.
Source for both projects: Fork me on github
Project 1: Displaying all Tweets and comparing Frequency
Getting the data
Initially I Googled Twitter and Processing and found Jer Thorpe’s tutorial. This is great for using the Twitter API, and really simple to set up but Twitter will only allow you to query the last 20 posts from a user and I wanted to display my whole history (about 2800 tweets). After searching around I found a script that will download up to your last 3200 tweets:
To get this to work just add your username instead of mine for the name of the xml file. The script took a few minutes to run but soon my terminal window contained everything I had ever written on Twitter. If I was smarter on the command line I could have saved the data to an XML file and edited it so all tweets shared 1 root node (as there are 10 requests there are 10 XML files that need to be joined into 1). As I am not so good no the command line I copied and pasted into Textmate and did this manually- if anyone can suggest how to do this automatically let me know! Now I had a 7mb XML file with about 115,000 lines.
Parsing and displaying the data
As soon as my application launched I parsed the data into custom objects. Amazingly This only took a few seconds.
Once I had all of my tweets in an ArrayList I then cycled through the list to display each one chronologically and also drew a dot for each tweet divided into months. Here’s a video of all the app running start to finish:
Project 2: Interactive app, Cartesian to polar transition
One of my favourite transitions is displaying data around a circle and then transitioning the data to a line. This means you can display the relationships between all data well when in the circle and also show the data in a linear way that you can easily scroll through. Luckily toxiclibs handles this transition beautifully (check the Polar Unravel example included with the library). In this application each month is a different colour and the length of the line shows the index of the tweet (earliest is the shortest, most recent the longest).
Here is a video of the application in use:
This app needs some refinement to make it easy to use and could also display the data more usefully but it does help give an idea of the volume and frequency of tweets and it’s nice to see the polar to cartesian transition in action.