Again, I’ve just tested Thundra against the mentioned scenarios.I have to admit that since the original post has been written, Thundra team has been regularly in contact with me to improve their service. I appreciate their willingness to improve their product.
I have used Thundra 1.6.0 and have performed performance testing for each test scenario. To enable Thundra detecting errors, users need to increase time out of their function to 30s and also to specify the environmental variable thundra_lambda_timeout_margin and set it to 1000 (it’s based on milli seconds). Otherwise, Thundra may fail to detect those errors. Having this configurations, Thundra passes almost all test scenarios (Thumbs up for them!) . However,
In test scenario #1 (i.e. time out): Thundra detects the errors but categorises them into two different errors:
- “Lambda is timed out.”
- Httperror: “The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.”
Even though error #2 is cause of error #1. This is something that Thundra team can take a further look into. I don’t know what’s the reason for separating these two errors? What makes Thundra to categorise a buggy trace into #1 and categorise the other one trace into #2?
In test scenario #2 (i.e. bad format): In Audit metric Function page, Thundra just detects that there is an error but doesn’t explain what has caused it. And if user looks into individual buggy traces, he/she just see that trace is marked as buggy, but there is no error explanation, and indeed the buggy trace is marked with 201 status code (with indicates successful trace). Thundra team needs to work more on this.
Furthermore, UI of Thunder is really user unfriendly (sometimes it gets really annoying). I hope they work further on it and make it better.
Thanks to Thundra team for their willingness to improve!