FinOps: Automating Monthly AWS Cost Reports - Biz x Dev Roundtable
In this article we sit down with the business development team to discuss how to continuously optimize cloud costs, which fluctuate due to various factors such as service growth and exchange rates.
Continuously Optimizing AWS Costs
Sakata:
Today I’d like to talk about cloud resource optimization from a cost perspective. At Morph, we primarily use AWS, but factors like currency fluctuations cause monthly variations in charges.
I check the AWS usage details every month, but since I'm not an engineer, I have to ask the engineers to look into any significant cost increases.
Shibata:
Yes, that's right. From the development team's perspective, each AWS service has a different person in charge. Therefore, the reasons for cost increases or decreases and whether they are appropriate need to be confirmed by the person responsible.
Sakata:
There is a communication cost to confirm with engineers every month, and the finance department can't grasp every detail.
Ideally, I want to create a state where we can continuously and clearly understand the cost situation.
Finance + DevOps = FinOps
Sakata:
While researching these challenges, I came across the FinOps approach. By effectively involving the development department in cost management, I feel we can create an ideal operational structure.
Shibata:
I see. Indeed, if we try to manage costs thoroughly, we tend to solve it with dedicated personnel which increases costs...
Sakata:
It becomes a situation where managing costs incurs further costs.
Shibata:
For resources directly linked to sales, it's easy to justify additional costs, but for development cloud resources it’s a bit of a bitter pill to swallow. We’d need to involve the development team to optimize it.
Gathering AWS Usage Data with AI-generated Reports
Sakata:
I've created a one-shot report on my end. I imported the AWS usage details CSV into Morph, and it handled pre-aggregation, data visualization, and report creation.
The creation process was very simple. It also recognizes cost increases and decreases and creates detailed reports, which I found more convenient than just creating charts.
Ideally, if we can automate this analysis and distribute the results to the responsible engineers, that would be great.
Shibata:
The fact that reports are created is different from the AWS dashboard. Did the content of the created report meet your expectations?
Sakata:
Yes, dashboards and usage details take time to parse what each metric means. Compiling it into a report makes the content easier to understand.
Also, it compares past data and explains it, and the ability to customize report content through prompts is a good feature.
Distributing Created Reports via Email
Shibata:
Understood. The AWS Cost Explorer seems to have an API, so we could automate the detail retrieval. By scheduling the flow you created, we could automate up to the report creation.
As an improvement idea, customizing the report by AWS resource category and distributing it to the engineers responsible might make the process smoother.
I think if you specify it through prompts, it might suggest actions as well as provide status reports.
Specialized Content Based on AWS Characteristics
Sakata:
While building this, I felt it was unique to generative AI that it reports considering the characteristics of AWS services.
For example, "This service has these characteristics, so these factors might be causing cost increases" were output.
Shibata:
Right!
We are currently using GPT-4, and it ‘knows’ more than I imagined. With this, we can also expect action proposals.
Let's try a bit more and move forward with building the automation.
Sakata:
Thank you. This was very helpful.
Shibata:
Thanks!