How We Saved Over $1,000 By Building CloudForecast With Serverless And AWS Lambda
In a previous blog post, we described our journey to launch CloudForecast and
some of the questions every startup face. In this post, we will answer the question “What’s the right way to build this product with our resources?”.
The “$100 Startup” concept
I’ve been obsessed with a book called “$100 Startup” by Chris Guillebeau for over a year now. In his book, Chris advises his readers to get their product out fast while being cost-effective to limit the damage if your idea fails (definitely possible, unfortunately!). We try to apply this advice on every side of our business including our technical decisions.
With the $100 startup concept in mind, we started by listing our requirements that were important to us while building CloudForecast:
- Ship it out fast: We were excited to build CloudForecast to help companies save money on AWS. We wanted to put it in front of our clients ASAP to keep our excitement going. Spending time debugging config, deploy process, etc. will not make the end product better. We were looking for an out of the box solution that we can grow with and allowed us to focus on our MVP the right away.
- Cost effective: As an early startup, we wanted to avoid fixed costs by building a system that will scale seamlessly based on our client base.
AWS Lambda + Serverless = Easy + Focus + Cost Effective
With our next steps, we brainstormed on how can we achieve our goals and requirements. We discussed managing our own instances, using containers and half a dozen other ideas but we wanted something simple so we decided to use a Serverless solution (or FaaS). While there are few downsides (e.g cold starts, …), we believe going Serverless better suited our use case (zero administration, pay-per-execution with no idle cost and auto-scaling).
For the dev setup and deploy process, we decided to use Serverless
Framework with AWS Lambda for the following reasons:
- We can focus on writing our product and let Serverless manage the rest
(permission, event management)
- Easy to configure and easy to deploy
- Serverless supports multiple platforms (GCP, AWS, … ) which could prevent
headaches in the future. We picked AWS Lambdas to start with since our clients
data will be stored in S3 bucket. It was a logical decision in order to reduce the
reduce the network cost
- Great documentation and community (Github, Gitter, Slack)
Our original requirements were fairly simple: We needed 4 AWS lambda functions
each on their own cron job, and each lambda would need to talk to various AWS
products (RDS, DynamoDb, SQS et al.). All that with an easy way to manage
multiple environments (dev vs prod) easily with an easy/effective way to manage
Here is how we do it:
- We used 4
with external JSON
files variables based on the environment:
- Used the
AWS Lambda functions code snippet to configure all the permissions:
With these two code snippets, we were able to set up most of our architecture. 4
functions that will interact with each other through SNS. Two functions will run
check on a cron schedule (via the ‘schedule.rate’ params) to check a file need
to be reprocessed and trigger another function via an SNS. This configuration
will be able to scale effortlessly while keeping our cost under control. We are
able to fully silo-ed our environments using iamRoleStatements to configure the
permissions. We used an external variable to manage environment variables from a
simple YAML file.
We considered running a couple of small instances to do a job which would cost
us at least $1k for the year. Meanwhile, the AWS Lambda cost is meaningless
since we are only running tens of thousands of functions a day.
Using Serverless helped us get
off the ground quickly. We saved our first clients thousands of dollars by
giving them visibility into their AWS spend.
We are in the process of integrating AWS Athena into CloudForecast for our next
project. Integrating with AWS Athena will enable new features that will allow
end users to dive deeper into their AWS spend. Every company has unique spending
patterns, and this will allow them to fully understand where their money is going.
Check out our new blog post on how to save money on AWS by using Parquet on AWS Athena.
If you have any questions related to this post or what we do at
feel free to reach out to me at
[email protected]. We would love to
hear from you!
-Francois & the CloudForecast Team
Haven’t tried our features? Sign up today and get started with a risk-free 30 day free trial. No credit card required.
Special Thanks to Kyle Galbraith for reviewing this draft. Check out his weekly
Newsletter, Learn By Doing.
Learn by Doing is a free weekly curated cloud, blockchain, and coding newsletter
delivered to your inbox every week.
Manage, track, and report your AWS spending in seconds — not hours
CloudForecast’s focused daily AWS cost monitoring reports to help busy engineering teams understand their AWS costs, rapidly respond to any overspends, and promote opportunities to save costs.
Monitor & Manage AWS Cost in Seconds — Not Hours
CloudForecast makes the tedious work of AWS cost monitoring less tedious.
AWS cost management is easy with CloudForecast
We would love to learn more about the problems you are facing around AWS cost. Connect with us directly and we’ll schedule a time to chat!