Since I started as a drupal developer at Brazen Careerist last fall, one of the projects I couldn't wait to get my hands on was an overhaul of the RSS infrastructure. We were using the (wonderful) FeedAPI module, but running into scaling issues; trying to pull and parse the number of feeds we're importing on drupal's cron wasn't going to cut it. One of our early ideas was to offload the feed fetching and parsing to a custom daemon. While this is still a viable idea, it comes with the overhead of added infrastructure to manage. In the short term we've decided to upgrade to the successor to FeedAPI, the Feeds module.
One of the especially exciting parts of this conversion is the Feeds module built in support for PubSubHubbub (PuSH). PuSH allows near real time notification when the feed URL of interest is updated. Roughly half of the feeds we were interested in aggregating already had support for PuSH, including those from Wordpress, Feedburner, Blogger and Typepad. In addition, by using Superfeedr as a designated hub, we can mimic real time notifications for feeds that aren't already PuSH enabled. This allows us to get notifications within (worst case scenario) 15 minutes of the original post, helping us make the conversations on Brazen Careerist closer to real time. Thankfully, superfeedr's service doesn't require us to distinguish between PuSH and non-PuSH enabled feeds, so it can be used for all feeds.
Anyone who has experience with a data conversion like this knows that it can be a bit tricky and requires quite a bit of testing. Luckily, the wonderful folks at Development Seed have released a module called FeedAPI2Feeds to help get the process started. FeedAPI2Feeds comes with a pair of drush commands that will help migrate FeedAPI content types to Feeds importer configurations. In our case this worked perfectly for creating the importer configurations. The feed configurations are exportable relying on Ctools to provide the exportables API. Ctools defines hook_ctools_plugin_api() which allows a module to define it's own plugin hook. The exported feed importer configuration can then be placed in hook_feeds_imported_default() like this:
<?php
/**
* Implementation of hook_feeds_imported_default.
* Exports of existing feed importers, exported from dev site.
*/
function custom_feeds_importer_default() {
$export = array();
// Copied export from test site.
$feeds_importer = new stdClass;
$feeds_importer->disabled = FALSE;
$feeds_importer->api_version = 1;
$feeds_importer->id = 'feedapi_node';
$feeds_importer->config = array(
//...
);
$export['feedapi_node'] = $feeds_importer;
return $export;
}
/**
* Implementation of hook_ctools_plugin_api().
* This needs to be present for the exports in the hook above to work.
* See http://drupal.org/node/622698 for more details.
*/
function custom_ctools_plugin_api($module = '', $api = '') {
if ($module == 'feeds' && $api == 'feeds_importer_default') {
// The current API version is 1.
return array('version' => 1);
}
}
?>
This whole process is also documented on drupal.org. I, along with the rest of the Brazen Careerist team, am a big fan of exportables and keeping configuration in code (and version control) as much as possible. Ctools provides a handy framework for module developers to support exportables. Thanks for making life easier Earl and Alex!
After our configuration settings had been moved over, accomplishing the full data migration required a bit more additional work. I decided to write a few drush commands to process the migration in stages. Creating your own custom drush commands is quite simple. Drush will find any file named *.drush.inc within your custom module directory. All your drush.inc file needs to do is implement hook_drush_help() and hook_drush_command(). Here's an example:
<?php
/**
* Implementation of hook_drush_help().
*/
function custom_drush_help($section) {
switch ($section) {
case 'drush:custom migrateFeedURLs':
return dt('Runs custom drush command code');
break;
}
}
/**
* Implementation of hook_drush_command().
*/
function custom_drush_command() {
$items['custom mydrushcommand'] = array(
'callback' => 'custom_my_drush_command',
'description' => dt('runs my custom drush command code'),
);
return $items;
}
function custom_my_drush_command() {
// Run my awesome drush code here...
}
?>
The most complicated custom drush command I wrote handled the actual subscription of feeds to superfeedr. If you haven't had a look at drush in a while I'd highly recommend it (especially 'drush mmas').
So what's next? I can't speak with any great certainty, but I'd love to clean up our Activity Streams and start offering them via PuSH. I'd also love to add to the conversation by integrating with other sites using the Salmon protocol (thanks for the inspiration at DrupalCon Brett).
