On 11th May 2015 we held a webinar to discuss the testing that the Black Ops Testing Team had performed on QA Mail.
QA Mail is a small server based application for helping test email sending applications. You can find out more on the web site.
We chose QA Mail because testing emails can prove problematic. You can use the ‘dot’ and ‘plus’ gmail strategies. But sometimes applications strip those out and you can’t use them to create multiple accounts or emails.
You can use services like mailinator or temp-mail.org. But again, some systems blacklist these email addresses. Which is why mailinator has hundreds of alternative domains to use.
Much of the time the issues comes not when testing interactively but when you use automation as part of your process, so ideally, you want a service that makes it easy to add to your automation i.e. offers an API.
QA Mail offers a GUI for interaction, and an API for automation.
We really liked the application, and I (Alan) would certainly use it in my test approach - if I could just overcome the installation hassles.
- James spoke to the author of the tool Vitaly Pryakhin to find ‘what was important’ prior to testing.
- Steve investigated alternative tools to send emails to provide more flexibility.
- Tony turned up some insecurity in the public test environment.
- Alan experimented with some automation abstractions and tested through the API
In the webinar we mention a bunch of tools and links - some of which I have extracted below:
- RFC 2821 (Simple Mail Transfer Protocol) - ietf.org/rfc/rfc2821.txt
- RFC 2822 (Internet Message Format) – ietf.org/rfc/rfc2822.txt
- RFC3696 (Application Techniques for Checking and Transformation of Names) - tools.ietf.org/html/rfc3696
- RFC 5321 (Simple Mail Transfer Protocol) - tools.ietf.org/html/rfc5321
- RFC 5322 (Internet Message Format) – tools.ietf.org/html/rfc5322
- RFC 6531 (SMTP Extension for Internationalized Email) -tools.ietf.org/html/rfc6531
- en.wikipedia.org/wiki/Disposable_email_address - useful overview and pros and cons
- dmoz.org/Computers/Internet/E-mail/Spam/Preventing/Temporary_Addresses/
- bitvise.com/ssh-client
One interesting challenge was trying to isolate if the problems we found resided in the application or in the infrastructure - sending email app, routing servers, server email receiving app. Many of the issues we thought were in QA Mail were from the postfix email server rejecting email addresses. We could only really spot this by monitoring the log files from QA Mail.
We identify other issues and approaches in the webinar.
You can watch the video replay on Youtube.
Our notes are listed below:
- Steve has released his testing notes his notes as a pdf
- Steve has released his test scenarios as an .xls file
- Alan has released his notes as a pdf
- Alan has a detailed blog post on the testing over on EvilTester.com