How to improve my Algorithm to find the Hot-Topics like twitter does

I have created a cron job for my website which runs every 2hours and it counts the words in the feeds and then displays the 10 highest count words as the hot topics.

Something that Twitter does on their homepage, is to show the most popular topics that are being discussed.

What my cron job does right now is it counts the words except for the words that i have mentioned, words like:

array('of', 'a', 'an', 'also', 'besides', 'equally', 'further', 'furthermore', 'in', 'addition', 'moreover', 'too',
                        'after', 'before', 'when', 'while', 'as', 'by', 'the', 'that', 'since', 'until', 'soon', 'once', 'so', 'whenever', 'every', 'first', 'last',
                        'because', 'even', 'though', 'although', 'whereas', 'while', 'if', 'unless', 'only', 'whether', 'or', 'not', 'even',
                        'also', 'besides', 'equally', 'further', 'furthermore', 'addition', 'moreover', 'next', 'too',
                        'likewise', 'moreover', 'however', 'contrary', 'other', 'hand', 'contrast', 'nevertheless', 'brief', 'summary', 'short',
                        'for', 'example', 'for instance', 'fact', 'finally', 'in brief', 'in conclusion', 'in other words', 'in short', 'in summary', 'therefore',
                        'accordingly', 'as a result', 'consequently', 'for this reason', 'afterward', 'in the meantime', 'later', 'meanwhile', 'second', 'earlier', 'finally', 'soon', 'still', 'then', 'third');       //words that are negligible

But this does not completely solve the issue of eliminating all the non-required words. And give only the words that are useful.

Can someone please guide me on this, and tell me how can I improve my algorithm.

php cron

Edit Report

What you are looking for is generally called a stop word list. Here's a blog post with a list of them (even in PHP array format for you) and another text file version.

A bit more googling should find you some other examples.

A few more potential ideas for improving the algorithm in general:

Weight the word usage by recency. You already do this by recalculating every 2 hours, but you could also factor in the exact time since the word was used as well. So, rather than having each mention of a word be worth 1 "point", it's point value would be determined by the time in minutes since the message containing it was posted.
Create a database table of words and their average frequency in messages on your site. When you examine the messages created in the last X hours, compare the word frequency to the average frequency in your database. Those words that have a frequency significantly higher than the average would be considered "hot". Make sure you re-calculate the average frequency for words on a semi-regular basis (once a day maybe?)

answered 2009-12-28T21:39:25.867

How to improve my Algorithm to find the Hot-Topics like twitter does

1 Answer

Your Answer