The 2012 candidate system is going to need a more fault tolerant collection system. Since the Egyptian protest started Twitter has been even more unreliable, and this is affecting the connection to the streaming API. I use a monitoring system to warn me when tweet collection has stopped, but it hasn’t been necessary in the past to automatically restart the connection. I saw some communication about this on the Phirehose mailing list. I need to run the collection script and capture the process id. Then kill the process with the same id and restart it when no new tweets are received for 10 minutes.
Update: This was pretty easy. I can add this code to the collection script to have it write it’s process id to a text file when it starts:
$fp = fopen('process_id.txt','w');
fwrite($fp,getmypid());
fclose($fp);
Then I can modify the monitoring script to kill the current collection script when no tweets are added for 10 minutes, and then restart it:
$process_id = file_get_contents('process_id.txt');
exec('kill -9 ' . $process_id);
exec('nohup php get_tweets.php > /dev/null &');
