A few weeks ago we were presented a challenge: create a web page that always has the 10 most popular news, pop-culture, entertainment, and sports articles/blogs that is more than just an aggregator news site. The trouble with popularity is that it can be measured in many ways: pageviews, visitors, or even duration. In addition, there is also the challenge of getting the necessary data and the constant concern of the data being current. The answer – Twitter. The rise in social media allows people to freely express their thoughts and creates the ultimate popularity contest. Twitter tracks this popularity and identifies the current trends by the minute with all of it accessible though a lovely API. This seems like the perfect match to my problem. The trouble is it is only part of the solution. Twitter tracks at best eight trends, which is a few dollars short of 10. Twitter is going to need some help.
Hooray, Google Trends comes to the rescue. Google Trends tracks the 100 most current search terms and is updated through an RSS feed. Holy cow batman! Combining Twitter and Google Trends gives 108 trends; way past the goal of 10 with plenty of change to spare. The problem is solved! Not so fast. As you can imagine, there are many duplicate trends between Google and Twitter. Since Twitter is the most current, being updated every minute, and Google is updated every hour, Twitter trends need to have priority. Google Trends will fill in the remaining trends to reach the goal of 10. Now that we have a unique list of trends, I can now move to the other half of the battle: finding articles and posts that relate to each trend.
There are millions of news articles and blog posts. That is why news aggregators were invented. A simple solution is to take each trend, search a news aggregator, and display the latest article. Mhhhhhhhh. Even the biggest news aggregators such as Google or Yahoo don’t have everything, especially in regards to pop-culture. How would I find out about the latest YouTube video percolating about the interweb? The next logical step was RSS. The trouble with RSS is we would need to maintain an unwieldy list of feeds with multiple RSS formats. The back-end code would simply be chaos. After some brainstorming with Jim, our Technical Director, we came to the realization that other people have done this before. (Not sure why this realization took more than a few minutes, but I digress). Google Reader does exactly what we are looking for. Collects RSS feeds that use multiple formats and display them in one place. It also has the beauty of being able to add and remove feeds without any code or database changes. Perfect! Combine that with an API and we have a solution trifecta!
Now we need to connect the dots.
- Get the latest from Twitter Trends
- Get the latest from Google Trends
- Filter out duplicates and create a list of unique leading edge trends
- Get the latest articles and posts from Google Reader
- Search all the articles and posts to get the most current item for each trend
- Display those articles on the web page
- Rinse. Lather. Repeat every 5 minutes.
Now there are other technical hurdles we had to overcome, such as stripping HTML from the RSS feeds, finding the right RSS feeds, devising a search algorithm, and do secondary searches if a trend comes back with no results. No small feat and I congratulate our developers Jon, Ganesh, and Carrie for making all of this possible for a Delta Sky Magazine advertisement that starts circulating today. Take a look at http://www.modernclimate.com/fly to see the final result and enjoy the leading edge of social and search trends.
