Blogs Readership Graphs

I have wanted to do this for a while, and now that i am on winter break, i finally did.

So here is the deal: I wanted to create a graphical map of the relationship between all the blogs on Qwaider Planet. More specifically, i wanted to represent each blog with a dot on a map, and draw a line between two blogs if one of them reads the other or links to it. So the end result will be like one of those Internet maps, but localized only for QP.

First, and without further adieu, here is a high-level readership map of all the blogs on QP.


Click on the image for a larger version.

The process was almost entirely automatic. On a high level, i retrieved the most recent posts of all the bloggers via their RSS/Atom feeds. Then iterated over all of them and parsed out comments by other bloggers on QP and the links to them.

Unfortunately it was not that easy. There were many complications that i had to get around. However, i will leave that for another post (or via email to those who might be interested).

Anyway, the picture above of QP as a whole makes sense. There are many bloggers who heavily communicate with many other bloggers; these represent the black blob at the center. Also, there are a few fringe blogs that communicate with less people on QP; these represent the few spikes that only have a few lines (edges) to the other blogs (nodes).

I created a large detailed picture with all the blogs, but it was 90 MB in size, and it crashed the program when i tried to compress it. Here is a link to it if you would like to give it a shot.

I also created a cooler individualized set of graphs of which bloggers read a certain blog, and the relationship between them. So for example, here are a couple of graphs for the blogs that have been recently linked to from my blog and the relationship between them. Again, the edges (lines) represent links.


Click on the images for a larger version.

I also expanded that to show all the blogs that i have had any sort of interaction with. So not only those that are linked to from my blog, but also those on which i had posted a comment on, but i have no recent link or comment from them on my blog.


Click on the image for a larger version.

Finally, i created the readership graphs for many of you! 🙂  Check them out, i think they are pretty interesting. Here are the maps for Qwaider and Maioush, Hani, KJ, Garfan, Batoul, Nizar, Jad Madi, 360 East, Black Iris, 7aki Fadi, Hreega, Moey, Kinzi, Mimi Cooks, Jessyz. and Mayyasi. I hope you like them. Again, each graph shows which blogs have recently commented on your blog, and what are the relations between them. If you want your own graph, please let me know and i will generate it and send it to you.

Leave a comment ?

28 Comments.

  1. Good work!

    I think from the first diagram you can somehow find some raw data about the percentage of activity inside the QP blogsphere for each bloger and how that percentage varies.

    I don’t know hoe possible it i from the technology that you have used but it would be interesting to see some sort of graph or data that shows something similar to:
    8% of bloggers are connected to 40 blogs.
    12% of bloggers are connected to 25 blogs.
    25% of bloggers are connected to 10 blogs.
    58% of bloggers are connected to 5 blogs.

    These studies can help and see how is the connectivity inside QP community changing, increasing or decreasing, and it would be helpful to know what factors and crease it.

    It is really nice to see the diagram of my blogs connectivity with the other QPs, thanks for that!
    although there are some blogs that I expected to see in my diagram like mab3oos and Hareega.

  2. By the way, it seems like Qwaider is always in the center of the web 😀

  3. OH WOW! What a fantastic effort man!
    I still trying to collect my thoughts on this, I’ll dig into it a little deeper and let you know what I think. But so far, that looks fantastic!
    Way to go man!

  4. Nizar: Thanks. I actually have the raw data for all of this. So i can generate these numbers easily (i hope).

    As for your comment on your graph, each graph shows who were the last commentors on that blog and the links between them, and *not* everybody that is linked to them. So for example, if i take everybody that Qwaider is linked to, then i get a 78MB graph that has most of QP. But, his graph only shows people that *read* his blog and are connected to it (i.e. the comment links on his blog). The same goes for your graph.

    hehe .. yeah Qwaider is always near the center because all the blogs that you read (or they read you), he also reads thenm, so he has many connections 😀
    Qwaider: Thanks. I also have all the raw data about all the recent links on these blogs. So as Nizar suggested, i can generate more statistics about our total connectivity ..etc.

    The current personalized graphs are interesting from my perspective because they show all the outgoing links from each blog, and where do they link to.

  5. Something is wrong with that All-qwaider chart. It seems to be a broken link or something
    Sorry, I had to run to have lunch. Ma7shi Kousa and Eggplants. Couldn’t even think with the smell 🙂

    So how did you go about doing this? Did you parse the links going out, and then generated the graphs based on that?
    Did you do this over a period of time, or is this a snap shot?
    Are you going to productize this anytime soon ;-)?

    Wow .. I guess that second post to clarify things is much needed …
    But seriously, fantastic effort man, and I’m really blown away with the results. This is GOOD WORK man.. really!

  6. Qwaider: Hehe .. no problem. Sa7teen on that lunch :-p.

    Which link was giving you trouble ? I checked all of them and they are all fine. The link with all of QP is also working; however that png file is 90MB in size, so it will take a while to load. I haven’t been able to open it in the default viewer (and i haven’t tried anything advanced yet). It seems that basic viewers have problems with really huge png files. I have the raw data, so i will try to recreate the file again at some point.

    I did this as you said: i just iterated over all the posts in each feed, and parsed the pages for all the outgoing links, then graphed the links that landed back in QP. Most of the work involved was in just identifying which links belong to the same blogger. For example, Batoul had an old .blogspot url, then she bought a domain, and when she comments on BlogSpot the url points to a special blogspot profile page. So i had to dynamically identify all the links that point to the same single blogger so i can account for them correctly. I also had some trouble because feeds and urls were often changed multiple times .. so mostly just small technicalities.

    The graphs represent just a snapshot of QP. I actually wrote the script two days ago and ran it last night. So it just represents what it saw as of yesterday night.

    I am not thinking of productizing this yet. Although, there is a market for identifying “hotspots” on the Blogosphere such that if an advertisement is put there, it will reach the largest number of people in the least amount of time. So for example, if a piece of news is advertised on your blog, most likely all the people in your graph will learn about it as well. However, i only did this because i was curious about how did QP look like. I had an earlier intuition that people on QP had a bigger sense of “belonging to a community” than other blog aggregators such as iToot or Dwwen. So i just wanted to do this to compare how QP differs from iToot and Dwwen. (btw .. what are your thoughts on this ? specific issue).

  7. Good to know wallahi. this is really good stuff. It also means that someone has too much time on their hands 🙂
    The problem with Snapshots is that they don’t give the full picture. They will show you a 2-dimensional view of a 4-D landscape.
    I would suggest enhancing the script to collect data over a period of say a month or a week, and then chart the results.

    So how did you convert raw data to a graph? Could there be a way to convert it to a format that can conserve more space than PNG and keep the chart in manageable size?

    Finally, what did you do with blogs that have “no follow” directive for their links? are you following those?

    Oh, and how are you storing the results?

  8. What a fun idea! I can’t read them very well, even enlarged. I’d love one, if you don’t mind.

    Yes, it is a nice solar system in which to belong. 🙂

  9. if you do not mind, please send me mine for http://mimicooks.com
    thanks for all your efforts!

  10. Qwaider: 🙂 ..yeah i admit that i have been doing almost nothing since the semester ended (although those days are numbered).

    You are right, it would be much much nicer if the data took multiple time into consideration. That way we can track the “growth” of each blog in the community 🙂 . Actually, i will see if i can tweak the script a little bit and have it periodically run on my server. That is really a superb idea! and i bet it will lead to many interesting results.

    I am currently using an application called GraphViz to do the graphing. The nice thing about it is that you can send it a textual representation of the graph and it will place the nodes and draw the edges automatically. However, it takes a long time for very dense graphs with more 200 nodes in them (such like the complete graph for your blog). This application can create graphs in multiple formats, however i have just learned it for this task so i am not aware of its full power yet. I just checked online now and it seems it can generate GIFs and JPEGs automatically.

    My script follows all the links it finds interesting (i.e. the ones that land back into QP or a known intermediate). As for storage, i am currently dumping the results into a CSV file and the inverted index into a binary file. However, i believe it is actually very easy to make it interface with a database instead.

    Kinzi, Summer: Thank you. Of course i don’t mind creating a graph for you 🙂 I have uploaded them to the list above.

  11. I think you should start charging money for these charts, or at least link backs to your blog 😀

  12. Blogs Readership Graphs - Qwaider Planet - pingback on December 26, 2008 at 9:03 pm
  13. Excellent work buddy. Let me know if you’re in need of more processing power since I have at my disposal few beefy machines 🙂

    Excellent work indeed

  14. This is truly awesome. I have always thought that QP is some sort of community where some blogs are more active and more connected but seeing it visually really is enlightening. This really is worth becoming a commercial/business product.

  15. Can the line connectors be weighted to show the strength and depth of the relationship between blogs?

  16. Who are you connected to? | Chocolate Mints In a Jar - pingback on December 28, 2008 at 10:26 am
  17. Nizar: hehe.. i doubt it would sell, so i’ll keep it for free :-p

    Qwaider: Thanks buddy.

    7aki: 🙂 Thanks

    Jessyz: Thanks for your kind words and the trackback. I am not sure how to do different line widths yet. I have collected that data, but i do not know how to represent it yet. If i find out i will let you know 🙂

  18. wow

    you impressed Samer

    this is… something admirable! you must be worshiped! *za3tar receives Moogle Thoughts immunity*

    I’ve seen something similar done in flash on some other site (I don’t recall it though) but this is a marvellous effot dude and I believe you need to take it further

  19. btw I attempted opening the 90MB file but photoshop “ran out of scratch disk space”

    I’d say!

    Ask Hani or Qwaider to have it open for you and downsize it – or possibly use an image editing program to split it into multple sections

    Then you could probably print it out on a big cardboard and cut it as a jigsaw puzzle, and sell it to people!

  20. Za3tar! You made what I have been wanting to do since a very very long while now!

    What if i want to see my graph? a3mel eh 😀

    Begad Great Work Dude!!

  21. Man, I just tried openeing it (and succeeded) but the contents are so dense that it looks like the background radiation left by the bigbang 🙂

  22. Za3tar, thanks so much for making mine and sending it!! It was fun to see, even some surprises. Very practically creative, you are. 🙂

  23. KJ: yeppi !! I am Moogle-immune 🙂 . Yeah i think such a think has definitely been done before (nothing magical about it) .. but now it was customized for QP :-p

    gjoez: I don’t have your blog on the list of members of QP so i do not have any data for your site. I will post more details about the process i used to get the data, but essentially i just retrieved all the posts from the blog feed, and parsed the outbound links in each page and determined were they go, and i only counted them if they go back to QP-member. But again, more details later :-p

    Qwaider: Great !! I think the high connection density is actually a good sign that QP is an actually community. Can you compress the file and put it somewhere were i can retrieve it please ?

    Kinzi: Thanks for your kind words, and you are most welcomed.

  24. well i can say : what a great job for i do programming and i know it was not that easy task at all 🙂

    expecting mine soon: http://mayyasi.blogspot.com 😀

  25. Mayyasi: Thanks. I have created your map and added it to the list above.

  26. WOW amazing … I see my blog
    Amazing how some bloggers connect in the QP-Sphere
    🙂

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Trackbacks and Pingbacks: