expecting mine soon: http://mayyasi.blogspot.com

gjoez: I don’t have your blog on the list of members of QP so i do not have any data for your site. I will post more details about the process i used to get the data, but essentially i just retrieved all the posts from the blog feed, and parsed the outbound links in each page and determined were they go, and i only counted them if they go back to QP-member. But again, more details later :-p

Qwaider: Great !! I think the high connection density is actually a good sign that QP is an actually community. Can you compress the file and put it somewhere were i can retrieve it please ?

Kinzi: Thanks for your kind words, and you are most welcomed.

What if i want to see my graph? a3mel eh

Begad Great Work Dude!!

I’d say!

Ask Hani or Qwaider to have it open for you and downsize it – or possibly use an image editing program to split it into multple sections

Then you could probably print it out on a big cardboard and cut it as a jigsaw puzzle, and sell it to people!

you impressed Samer

this is… something admirable! you must be worshiped! *za3tar receives Moogle Thoughts immunity*

I’ve seen something similar done in flash on some other site (I don’t recall it though) but this is a marvellous effot dude and I believe you need to take it further

Qwaider: Thanks buddy.

7aki: Thanks

Jessyz: Thanks for your kind words and the trackback. I am not sure how to do different line widths yet. I have collected that data, but i do not know how to represent it yet. If i find out i will let you know

Excellent work indeed

You are right, it would be much much nicer if the data took multiple time into consideration. That way we can track the “growth” of each blog in the community . Actually, i will see if i can tweak the script a little bit and have it periodically run on my server. That is really a superb idea! and i bet it will lead to many interesting results.

I am currently using an application called GraphViz to do the graphing. The nice thing about it is that you can send it a textual representation of the graph and it will place the nodes and draw the edges automatically. However, it takes a long time for very dense graphs with more 200 nodes in them (such like the complete graph for your blog). This application can create graphs in multiple formats, however i have just learned it for this task so i am not aware of its full power yet. I just checked online now and it seems it can generate GIFs and JPEGs automatically.

My script follows all the links it finds interesting (i.e. the ones that land back into QP or a known intermediate). As for storage, i am currently dumping the results into a CSV file and the inverted index into a binary file. However, i believe it is actually very easy to make it interface with a database instead.

Kinzi, Summer: Thank you. Of course i don’t mind creating a graph for you I have uploaded them to the list above.

Yes, it is a nice solar system in which to belong.

So how did you convert raw data to a graph? Could there be a way to convert it to a format that can conserve more space than PNG and keep the chart in manageable size?

Finally, what did you do with blogs that have “no follow” directive for their links? are you following those?

Oh, and how are you storing the results?

Which link was giving you trouble ? I checked all of them and they are all fine. The link with all of QP is also working; however that png file is 90MB in size, so it will take a while to load. I haven’t been able to open it in the default viewer (and i haven’t tried anything advanced yet). It seems that basic viewers have problems with really huge png files. I have the raw data, so i will try to recreate the file again at some point.

I did this as you said: i just iterated over all the posts in each feed, and parsed the pages for all the outgoing links, then graphed the links that landed back in QP. Most of the work involved was in just identifying which links belong to the same blogger. For example, Batoul had an old .blogspot url, then she bought a domain, and when she comments on BlogSpot the url points to a special blogspot profile page. So i had to dynamically identify all the links that point to the same single blogger so i can account for them correctly. I also had some trouble because feeds and urls were often changed multiple times .. so mostly just small technicalities.

The graphs represent just a snapshot of QP. I actually wrote the script two days ago and ran it last night. So it just represents what it saw as of yesterday night.

I am not thinking of productizing this yet. Although, there is a market for identifying “hotspots” on the Blogosphere such that if an advertisement is put there, it will reach the largest number of people in the least amount of time. So for example, if a piece of news is advertised on your blog, most likely all the people in your graph will learn about it as well. However, i only did this because i was curious about how did QP look like. I had an earlier intuition that people on QP had a bigger sense of “belonging to a community” than other blog aggregators such as iToot or Dwwen. So i just wanted to do this to compare how QP differs from iToot and Dwwen. (btw .. what are your thoughts on this ? specific issue).

So how did you go about doing this? Did you parse the links going out, and then generated the graphs based on that?
Did you do this over a period of time, or is this a snap shot?
Are you going to productize this anytime soon ?

Wow .. I guess that second post to clarify things is much needed …
But seriously, fantastic effort man, and I’m really blown away with the results. This is GOOD WORK man.. really!

As for your comment on your graph, each graph shows who were the last commentors on that blog and the links between them, and *not* everybody that is linked to them. So for example, if i take everybody that Qwaider is linked to, then i get a 78MB graph that has most of QP. But, his graph only shows people that *read* his blog and are connected to it (i.e. the comment links on his blog). The same goes for your graph.

hehe .. yeah Qwaider is always near the center because all the blogs that you read (or they read you), he also reads thenm, so he has many connections
Qwaider: Thanks. I also have all the raw data about all the recent links on these blogs. So as Nizar suggested, i can generate more statistics about our total connectivity ..etc.

The current personalized graphs are interesting from my perspective because they show all the outgoing links from each blog, and where do they link to.