Thoughts on Amazon Simple Queue Service (SQS)
tracy — Tue, 12/02/2008 - 14:56
I've moved some of our queues from a MySQL table to Amazon's Simple Queue Service (SQS). The main reason I did this is that we are processing more than a million tasks a day using at least 4 servers and that amount of processing in a database situation was a bit more than I could handle. I haven't been able to move all of our queues over, in part due to the 8k message limit, but it's been great for the queues moved over thus far.
That said, there are a few things to keep in mind about SQS. First, I get a significant number of 500 errors when making various requests. I'm using the Amazon::SQS::Simple module from CPAN and had to wrap it in my own custom module to handle errors and making the request again if it originally failed. Given the frequency of error, it's definitely something to keep in mind.
In addition to 500 errors, it's important to note that Amazon SQS has an issue handling utf-8 characters. According to a bug report for that module, Amazon is aware of this issue and is working on a fix. Apparently it is also an error with some other services, but I'm not sure which ones. One work around is to encode the message text in base64. Fortunately, the messages I need to save are small enough where I can do this encoding and still have room to meet the 8k limit. If I did not have enough room, I could also compress the message text.
Also, while they aren't important for my current project, there are two other handy tidbits regarding the service. First, it's a loose First In, First Out (FIFO) system, which means that messages won't always be returned in the order they were added. Second, due to the system architecture, your information is kept on multiple servers that might not be in complete sync at all times. This means that there is no guarantee you won't get the same message more than once even after you've deleted it or that a request for messages will return all messages you've sent.
A really handy book for learning more about SQS and other Amazon services is Programming Amazon Web Services: S3, EC2, SQS, FPS, and SimpleDB (Programming)
