AWS SES Rate Limiting & Bounce Handling System

[00:00]
If you're watching this, I've been where you are trying to figure out a way to rate [00:04]
limit your AWS SES email system from going on a control and sending to many emails. [00:11]
It sucks. [00:12]
I've been there. [00:13]
Since 2002, our company has completed more than 30,150 web development projects in [00:19]
the marketing and web development space. [00:21]
So I've seen quite a bit of team hacked accounts that start sending out a control or [00:26]
balancing catastrophic leading AWS's SES team to suspend your account. [00:32]
Or it's a system that wasn't configured properly and starts sending out 50 emails every [00:37]
three minutes starting out a Friday afternoon. [00:39]
And by the time you notice, your account is over quota and you can't send anything else. [00:44]
So we asked the SES team at AWS. [00:47]
We hired a former senior level AWS solutions architect. [00:51]
We were told that no AWS doesn't have any of the tools that allow you to rate limit [00:56]
how many emails a specific email account can send in one day. [01:00]
It's just one big bucket. [01:01]
So good luck with it and do your best. [01:05]
So after dealing with one too many outages for several of our clients, we decided to [01:10]
see if we could develop something that would work. [01:12]
Using only AWS tools to track and manage the SES outbound process. [01:18]
And after about six months of fine tuning and working on it, we've developed a really [01:23]
cool system, I guess really cool in my end, that allows you to rate limit your AWS SES outbound [01:30]
emails and automatically pull the keys if any specific email account sends out too many [01:35]
emails or starts bouncing too much out of control. [01:38]
So we built a self service portal. [01:40]
I'm going to walk you through that next that allows you to test out the process and have [01:44]
a full documentation process for that. [01:47]
So why are we putting this out there? [01:49]
Because we were really frustrated by not having a solution that would protect our clients. [01:54]
And we can't be the only ones out there that are affected by something like this. [01:58]
So it would have been easier to keep this as an internal tool or just give it to our clients. [02:03]
But we wanted to turn it to our experience in frustration into something that can help others [02:08]
out there. [02:09]
So we hope this is a good system for you. [02:12]
We've priced us in a way that allows us to cover our costs of setting up the system. [02:17]
And we have an option for self service installation. [02:19]
In case you're not able to let us into your AWS infrastructure. [02:23]
So there's two options for working with us. [02:26]
Full service. [02:27]
We set up everything in your AWS environment, including the specific server was lambda [02:31]
functions and my SQL database allows your system to rate limit SES outbound emails. [02:36]
We'll set up up to 10 different email addresses in the system and additional ones we can [02:41]
just do it in an hourly rate. [02:43]
We'll test it. [02:44]
And make it sure that you're able to send from various systems will use the SES rate [02:48]
limiting portal that we've built and put this in your infrastructure to allow you to set up [02:54]
and test rate sending in bounce limits within the system. [03:00]
The price for full service includes the complete setup of the AWS SES rate limiting system. [03:06]
And we'll set it up for you. [03:08]
Rekey email accounts will include the PDF manual for setup and configuration. [03:13]
We'll give you a zip file of the SES rate limiting portal. [03:17]
And we'll do at least an hour of screen share support in 12 months of free updates to the [03:23]
system. [03:24]
Additional support is available on an hourly basis. [03:28]
Option two self service with included screen share support during US hours. [03:33]
So if you don't want us messing around in your production AWS environment and no [03:37]
offense taken, I get it. [03:39]
We offer a self service option that gives you the blueprints to put everything into place. [03:43]
We'll show you how to do it. [03:45]
And we get it. [03:46]
Sometimes it's not so easy to allow outsiders into the system. [03:49]
So we've created to do it yourself manual or approach it allows you to set it up and [03:54]
configure it. [03:55]
But because it's sometimes is complicated, we've included an hour of screen share support [04:00]
where we can help troubleshoot any issues you might have. [04:03]
So the self service option includes the PDF manual for how to set up the AWS SES [04:07]
rate limiting system in your own environment. [04:10]
As well as a zip file for setting up the optional rate limiting portal. [04:15]
And the portal allows you to configure great limits and bounce limits for each individual [04:20]
email address. [04:22]
So the couple of questions because a lot of people want to know how does this work. [04:28]
So everything happens within your AWS environment. [04:32]
You're not connecting to any external communications or connections to other systems. [04:38]
Everything is done that way. [04:40]
And we did it this way because it's the most secure way to handle the email security [04:44]
and setup. [04:45]
So we've created a system that safeguards your SES system by each email is set up and configured [04:52]
as its own IAM user in your AWS infrastructure. [04:56]
We add IP level restrictions to the email account to safeguard sending. [05:01]
And anytime the system sends out an outbound email via Amazon SES, we use serverless [05:06]
land-to-functions and other systems to track what email was sent, including the subject [05:11]
line recipients and whether or not it bounced. [05:14]
We store all of this at a MySQL database which is also hosted in your AWS environment. [05:20]
And you're able to use this online portal to configure track, test and repeat email accounts. [05:26]
And if any email account is compromised, hacked, starts sending out a control, bouncing [05:31]
more than a specific threshold, the system yanks the SMPP keys and disables sending right [05:36]
away. [05:37]
You'll also receive an email from AWS letting you know why. [05:42]
In the portal, you're able to re-key an email account and regenerate the user and even [05:46]
password. [05:47]
You'll need that to update your application separately. [05:50]
We offer this as a full service setup or self service as I described before. [05:56]
And one question that frequently comes up is how much will this cost in your AWS environment. [06:02]
The additional cost of running this on a medium volume system is somewhere between $10 [06:07]
to $20 a month in additional AWS server costs. [06:10]
That includes the Lambda functions, the MySQL database, and a small micro-ec2 or light-tail [06:16]
instance to run the rate limiting portal. [06:19]
So let me run you through a quick walkthrough of how the portal works. [06:24]
So first let's go and let's send a test email out of the system. [06:29]
So I'm going to choose from the drop down one of these and then I'm going to send it [06:34]
to myself in this case and we're going to say test number one, test number one. [06:41]
This is going to send out. [06:43]
The test has said email successfully sent. [06:45]
Okay. [06:46]
So now let's do test number two and we'll do test number two and we'll send that and that also [06:54]
sent out which is fabulous. [06:57]
We are now going to send out test number three and we'll send that and that went through. [07:05]
We're going to do test number four and just so you know the threshold is five emails. [07:11]
So at a certain point we're going to hit that threshold and that was four. [07:16]
So let's do five here. [07:18]
Here's test number five that also went out and then the next test test number six. [07:25]
This should have pulled the credentials. [07:29]
So let's see if that works. [07:33]
Maybe know it. [07:34]
Oh, here we go. [07:36]
Test number six. [07:37]
S. M. T. P. error could not authenticate and this gives us an indication that the system has now been revoked. [07:46]
And all this drag this over so you can see it on here and it says, hey, this test user test one one. [07:53]
One at cp-demo.com, which is a domain of the own. [07:56]
There's a daily email send quote of 15 hourly send of five. [08:00]
We sent 11 today. [08:02]
We sent five in the last hour. [08:04]
We just reached the hourly quota and this is the action that's required. [08:09]
If you want to see one of the tests, we can see here is test number five that went out. [08:15]
And we've got all of those guys right there so that we can see how this stuff works. [08:23]
So what do we do? [08:25]
Now this is done. [08:27]
So now we have to rotate the S. M. T. P. keys for this. [08:31]
So I'm going to go to this page in the system and say, okay, test one one one. [08:37]
Rotate the keys. [08:38]
This is going to regenerate the username and password in here. [08:43]
And in this case, this is going to treat a new username and password. [08:50]
This is the actual real legit stuff. [08:53]
Of course, I'm going to do this again. [08:54]
So these will be revoked pretty quickly. [08:57]
So now let's go back and let's send one more test email to see if we can send this through [09:04]
the system. [09:05]
Test after re-heating the credentials. [09:13]
Sesting one two three. [09:17]
Okay, we're going to send another test. [09:20]
And you can see the send send it successfully, which is very cool. [09:26]
Now if I want to send to a bounced email account, I can send to a email account that will [09:36]
likely bounce and it'll do the same type of basic configuration. [09:41]
So I wanted to go through that example. [09:43]
So it's good to trust things, but it's also very good to verify how this works and how it all comes [09:51]
together. [09:52]
And I have a lot more comfort in the system knowing that that's how it works. [09:56]
So let's go in and look at some of the configuration. [09:59]
So if you want to configure email addresses in the system, this allows you to send to test or to [10:06]
configure the accounts for bouncing and laying. [10:10]
It's sometimes easier to look at an existing process than one that we have to set it from scratch. [10:16]
So for the test one one one user that we just did, we have a daily send quota of 15. [10:22]
So you can only send 15 emails out per day. [10:26]
Great for a test system when you're trying to see if this system works. [10:29]
Such if this is a production system, so you'd send it a little bit higher. [10:33]
Again, hourly send quota, we're saying if there's more than five emails being sent to an hour, [10:40]
just take it out of the system. [10:42]
Next daily bounce quota. [10:44]
There's three bounces that are allowed in the system per day or per hour. [10:51]
And the reason for this is that when you are going through a system like this and you have an email [11:01]
account that goes way out of control, there's a couple things that happen. [11:04]
One is it's going to send rapidly. [11:06]
A spammer or attacker is going to send out or try to send out hundreds or thousands of emails [11:12]
right away. [11:13]
And typically the list that the attacker uses is probably older. [11:17]
It's out of date email addresses that bounce right away. [11:20]
So using the combination of these two to say, okay, let's set the bounce rate as a little bit lower, [11:28]
the daily and hourly send rates a little bit higher. [11:32]
This will help you really control it very, very quickly. [11:37]
We also have something for foreign language quotas. [11:39]
That's not available in all SES regions, so we're not supporting it and putting it in this case. [11:47]
So for right now, the bounce rate quota is set for 15 per day and that's from midnight to 1159 [11:55]
PM, that same day, same thing with daily and hourly bounce rates. [12:03]
So from there, if we wanted to add a new email, what we can do is you can select an identity through [12:13]
the system and we can say this is this one and we are going to set it up as test one one one or [12:19]
test two two two and I have a bunch in there that we can see. [12:22]
Now let's say you don't want to do that for every single account, which is totally fine and [12:27]
reasonable and if you don't set any, you can set default settings so it's a catch-off for everything. [12:33]
So you could set the send quota for everything to 20 per day for sending or 10 per hour. [12:40]
Again, on a production system, you would have this much higher, same thing with bounce quotas. [12:45]
So you don't have to set it up for each and every one, which is very, very nice. [12:49]
So the next thing that we'll look at are the email archive. [12:55]
So this is a really nice way and you can see this is the test after resending the credentials, [13:00]
test six, test five, four, three, two, one. [13:04]
These are the ones that went through and if you've had the pleasure of being in the AWS [13:14]
environment in the SES system where it's really tough to find stuff and somebody's like hey, [13:21]
I'm looking for something that was sent out for this really important AR invoice and we don't [13:27]
seem to have sent it. Can you check? Well, this allows you to do this. This is in a mySQL database [13:33]
in your system. So if you'd like to use this search system, awesome. You can search by sender, event type, [13:40]
crowds or send, recipient from date, all these things are fabulous or you can search directly [13:46]
in your mySQL database. You can also export to Excel or PDF. Block history. Any time the system [13:54]
blocks people and you can see I've been testing and other things are going on the system. You can see, [14:00]
hey, this sender quote, it was exceeded. You can see it was six out of five because we [14:05]
receded and resent more through. But this allows you to figure out, hey, what's going on? What was [14:11]
blocked? What was there? On the dashboard itself, you can see SES insights. We built a local AI agent [14:21]
that runs strictly on your system. It's not connected to anything else. Basically, it's a Python [14:26]
script that scrapes through all the data from the email sends, the block history and sender config [14:31]
systems and refreshes it hourly compute it and says, okay, the last 24 hours we've sent. 10 messages, [14:39]
nothing's balanced. One block, no complaints, which is all good. You can see how this goes over time, [14:45]
top senders, here the senders near quota, here are the recent blocks and other kinds of things. [14:51]
And then if you want, optionally, you can say, hey, and I know, do you really want to get an email [14:59]
about an email system? Well, it kind of beats going to a specific portal and trying to remember [15:06]
it and all the rest. So you can basically say, okay, I want a digest of all of what the system is [15:13]
doing once a day, once a week, once a month, and you can set it this way and handle it that way. [15:21]
This is kind of a basic overview of the system. You can see what are your SMPP settings. You can change [15:27]
your passwords for admin, add other users or go through the system. So anyway, I hope this is helpful [15:35]
and I wish you luck and we built this out of frustration. We built this as a way where you can take [15:44]
back control over this massive bucket of emails that you have for the AWS SES system and really [15:51]
generate something that's positive for you and your organization and not get that dreaded. [15:57]
TUS service notice from AWS because some developers screwed up and put something in there [16:04]
or left the keys to an email account in a public repo and it got hacked and now you're sending [16:10]
and now you're begging to get back onto the system. So this is a demo system. So in a production [16:17]
system, there would be more. If you have any questions, I'd love to talk with you. My phone number [16:22]
is 303-473-4400. Jeff, I've found it customer repair time. I designed the system out of frustration [16:30]
but we're releasing it out there two people so that they have a better way to control this. I hope [16:37]
this helps. Give us a shout and I'd love to help out with your AWS SES system. Thanks!