AWS Billing and Cost Management Tools. Cloud Academy, AWS Billing and Cost Management Tools

Transcription

1 AWS Billing and Cost Management Tools Cloud Academy,

2 Introduction Organizations choose the cloud over large data centers for many reasons. Minimal upfront costs and its easy, pay-as-you-go operating model make the cloud affordable. The ease of resizing and repurposing the wide variety of available computing resources makes it a more However, despite these initial advantages, companies can still fall into a familiar trap post migration. We call it the infrastructure trap. Here, the optimal performance of resources empty servers, underutilized RDS instances or unattached storage volumes may go overlooked in favor of day-to-day operations. For example, after products go on the market, keeping the infrastructure available well after the R&D phase or application life cycle has ended As a result, your infrastructure costs may end up being higher than they need to be, so it is important to practice good housekeeping to gain the best advantage from the pay as you go model. Fortunately, Amazon Web Services (AWS) and third party vendors offer an array of tools that can help you manage, analyze, forecast and optimize cloud infrastructure costs (and avoid the infrastructure trap). In this ebook, we ll introduce you to a range of tools and resources to help you understand: The resource types that are incurring the largest share of costs (EC2, RDS, Redshift, etc.). How to read the trends in your monthly bills and use them to forecast future expenses How to prevent billing surprises Where to look for savings How to estimate costs before a resource is provisioned And more Using the insight from these tools, infrastructure managers can decide: Which AWS resources or service can be shut down or retired Which AWS resources can be resized based on their usage pattern How to apportion the total bill to various customers based on their resource usage. 2

3 Part 1: AWS Billing Management Let s start with the basics of Amazon Web Services billing management. Located in the AWS console drop-down menu, the Billing and Cost Management option shows overall spending for the current month, the billing forecast based on the current usage pattern, and costs for the previous month. Another widget on the screen breaks down the current month s spending by service type. As we can see in the image, the majority of costs for this billing cycle are associated with the Elastic Compute Cloud (EC2) service. 3

4 Getting Started with AWS Billing & Reports All billing information is available from the Bills menu, or you can refer to the PDF copy sent via (if chosen in your billing preferences). The PDF option is useful because you can forward it to people without giving them access to the console. One benefit of the console report is that it allows you to drill down into the billing components by service and region. On the Bills menu, we would also recommend enabling the Receive Billing Reports option under the Preferences section at the bottom of the page. This will allow you to receive billing once or more daily, and you can to refer to them anytime rather than waiting for the monthly bill. In this sample report, we will start by taking a closer look at the biggest cost drivers, which in this case is the Elastic Compute Cloud service. Click the arrow beside Elastic Compute Cloud to see how each region is performing in terms of usage and costs. This is useful information, but as we will see, there are better ways to find it. 4

5 AWS Cost Explorer AWS Cost Explorer shows exactly how your money is being spent. It uses the data from each month s spending for up to the last 13 months and can forecast a spending pattern for the next three months, or may be tailored for specific time periods. AWS Cost Explorer shows spending by: AWS regions: Helpful for large, global companies that run workloads from multiple regions. Availability Zone (AZ): Useful when you are using an AZ as a backup location. Service: Which service type costs more? EC2 instance types: Which EC2 instances cost the most? Are they supposed to run with a large configuration? Linked account: If multiple AWS accounts are linked to a parent payer account that pays the bill, this report can show which linked account is accruing the highest costs. The customer can then be billed back if necessary. API calls: Which type of API calls are most expensive? Purchase options: Look at how much is being spent for reserved instances and on-demand instances. If on-demand instances are accruing a larger portion of the costs, it may be worthwhile to think about reserved instances. Cost allocation tags: Attach tags to AWS resources to map a cost analysis against each tag. Once enabled, the AWS the Cost Explorer dashboard comes with a few standard reports under the tab Preconfigured Views : 5

6 You can also create a custom report like the one below for RDS instance spending. In this sample report, we are providing: Report name, analysis period and forecasting period, and filter spend by service (in this case, RDS). Although not shown here, we could also estimate costs by grouping together RDS spending from the last three months by region. After saving your report criteria, you will be able to access it from the drop-down menu in the Launch Cost Explorer console, where you will also be able to view and manage all of your reports. 6

7 Cost and Usage Reports AWS customers can also create custom reports based on their usage patterns. These reports are more detailed and information may be customized by frequency (hourly, daily). To use this feature, select the Reports tab in the Billing and Cost Management console. To get started, click on the Create report button. This will initiate a wizard where you can specify various options as shown below: You will need to enable or select an Amazon S3 bucket and set permissions on that bucket for AWS to use to store your reports. The cost and usage reports tool provides a sample bucket policy: 7

8 Here, you can see all of the report details. Click Review and Complete to finalize your report. Once defined, a report cannot be modified. You will need to delete the existing report and create a new one. Cost and usage reports are automatically generated and periodically saved in an Amazon S3 bucket as comma separated value files (CSV) with a large number of columns. With large AWS footprints, these documents can be fairly large. You can load them into thirdparty tools for further analysis. 8

9 Customized Reports The AWS Cost & Usage Reports also include some pre-configured reports. To utilize the Amazon EC2 Instance Usage Report, for example, simply click the link and it will open another window where you can provide custom search criteria. In the sample report below, we are looking at the daily spending from our Amazon EC2 m3.large instance types; the report has also been divided by platform. From this report, you can see: Most of our Amazon EC2 m3.large instances are using Red Hat Enterprise Linux, while some are using other forms of Linux, and the remainder are Windows boxes. The average daily spend is $500. 9

10 We can also look at Amazon EC2 spending over a year s time. Billing Reports Like the Cost and Usage Report, Billing Reports must also be saved in an Amazon S3 bucket in CSV format. Billing Reports can be enabled from the AWS Billing preferences screen. Once the Amazon S3 bucket has been verified, you can choose from the following types of reports: Monthly report Detailed report Cost allocation report Detailed billing report with resources and tags As you can see, the monthly AWS bill provides only a high level picture of the last billing cycle. You can take advantage of different types of reports and analyze the historical data to better understand your spending patterns. 10

11 Part 2: Optimizing AWS Costs Now that you understand your spending patterns and biggest cost drivers, you can use this information to optimize costs. Cost management in the cloud is an ongoing exercise for operation managers who are often expected to answer questions on cost reduction such as: How can we keep our spending within a reasonable limit? How can we optimize costs? Where do we start? How can we forecast future spending? In this section, we will introduce the AWS cost management tools that can help answer some of these important questions. Let s take a closer look. Billing Alerts A common account management best practice is to enable billing alerts. After setting a threshold for monthly spend, the system will notify you when a spend threshold has been reached, ensuring that there are no surprises at the end of the billing cycle. This is known as proactive cost monitoring and it can provide clues for how to better manage your resources. For example, if you see a consistent pattern of alerts each month, it may mean that the threshold (and perhaps the infrastructure budget) is too low for the expected workload. Or it may indicate that you can scale back on your current resource consumption. To get started, enable billing alerts from the AWS Billing and Cost Management console Preferences tab. Once enabled, it cannot be deactivated. Now, you can create an alert from the Amazon CloudWatch console. 11

12 Before creating your alerts, there are a couple of important details to note: 1. Billing alerts can only be created from the US East (N. Virginia) region. However, monitoring will include costs from resources running in any region. If you try to create a billing alert from Amazon CloudWatch in any other region, you will receive a message like this: 2. Before creating the alarm, we must create an Amazon Simple Notification Topic (SNS) topic called AWS_Billing_Notification and confirm its subscription. In this case, the Amazon SNS topic is configured to send notifications to the payer account address. Now, we re ready to create the alarm. Billing alarms can be based on: Estimated total charge for the current account Estimated total charge for one or more linked accounts Estimated charges for one or more AWS services consumed by the current account Estimated charges for one or more AWS services consumed by one or more linked accounts 12

13 In this example, we are interested in the total estimated charges in USD for the current account. The alarm will tell us when the monthly budget equals or exceeds $1,000. With Amazon CloudWatch, we can use Amazon Auto Scaling operations to manage costs in certain use cases. For example, if we are running a busy site during a particular campaign or promotion season, we could use multiple servers in a farm to keep up with any spikes in traffic. Because this would also incur greater costs, you can use auto scaling actions in the billing alarms to reduce the number of servers once spending hits $1,000. Once set up, Amazon CloudWatch monitoring is automatic. Each time the spending threshold is reached, an like the following would be sent to the specified address. 13

14 Budgets AWS allows you to monitor actual usage against your budget. Go to the Billing and Cost Management console to enable budgets. As with Billing Alerts, once enabled, it cannot be turned off. An AWS account budget is easily created using a wizard. In this example, the wizard walks us through the following selections: The budget includes two services: Amazon EC2 instances and Amazon EBS volumes Covers a six-month period (June to December) Allocated spending of $1,500/month Threshold notification at 2% over budget Notifications sent to the specified address (behind the scenes, AWS creates a CloudWatch alarm for this) 14

15 Once the budget is created, it is visible from the Budget console. The status bar shows current spending against your budget. You can create budgets for the following categories: Specific API operations against an AWS service: For example, you can budget for Amazon S3 PutObject requests. Availability Zones: Allows you to adjust your budget based on zone. Linked accounts: Separate AWS accounts within the company may be linked to a master payer account that pays the consolidated bill at the end of the month. You can create budgets for each linked account. Purchase option: Decide how much you want to spend for on-demand instances and reserved instances. Tag: Use tags to more clearly identify resources and usage. Some things to keep in mind about budgets: By default, only two budgets are allowed per account. Once a budget is created, it cannot be modified, only deleted. In the notification options (actual or forecasted spending against budgeted value), AWS creates the necessary Amazon CloudWatch alarms and the associated Amazon SNS topics. However, these alarms are subject to AWS account limits. By default, an AWS account is allowed 10 free alarms per month. If you already have a large number of alarms configured, the budget alarms may actually come at an additional cost. 15

16 Cost Allocation Tags While budgets and alerts can help keep costs down, cost allocation tags allow IT departments to apportion the total AWS bill to various internal and external customers. The traditional IT shop usually hosts a number of resources for different cost centers (sales, marketing, finance, corporate apps, HR, etc.), projects or teams (application developers, business analysts, service desk, or training). Tags can be added to each of these resources to identify or associate them with a specific business owner or purpose. For example, an Amazon EC2 instance can have two tags: Owner and Environment with values of HR and Production, respectively. When cost allocation tags are enabled, AWS will generate a cost allocation report with the total monthly bill broken down by the enabled tags in the form of a CSV file created in an Amazon S3 bucket. In this example, if both the Owner and Environment tags are enabled, the report can show what IT should be charging back to HR for hosting their production server. This image shows that three tags are enabled for the cost allocation report. Next, let s look at more sophisticated tools that can help us optimize costs and take advantage of price reductions from AWS. 16

17 Part 3: Reducing Costs with AWS AWS billing reports, alarms, and budgets are just a few tools that can help limit your cloud footprint. We have seen how billing reports provide insight into current and historical spending patterns and what tools are available to warn us when the spending crosses a limit. But how can you limit what you are spending? For companies with a large AWS footprint, this issue can be daunting. How do you know which services are no longer needed? How do you decide if an instance needs to be terminated or resized? Fortunately, AWS provides some tools to help answer these critical questions. AWS Trusted Advisor AWS Trusted Advisor is a multi-purpose tool that helps AWS clients run efficient, secure, and fault tolerant workloads. It looks at the AWS resources currently being used, compares their configurations against best practices, and highlights any ideas for improvement. The Trusted Advisor app is located in the AWS console under the Management Tools section. Once in the dashboard, customers can choose to optimize for any one of the following: Cost Optimization, Performance, Security, or Fault Tolerance. To identify potential savings, customers should aim to run the Cost Optimization wizard at least once a month. The Cost Optimization report can be downloaded as a spreadsheet. The AWS Cost Optimization tool audits a number of areas, including: Low utilization EC2 instances: These are your idle servers. Perhaps the applications they once hosted have been migrated, upgraded, or retired. Look closely at the disk, CPU, and network usage of these instances over a reasonable period of time (three to six months). If you know they are no longer needed, terminate them. If the instances are still being used, resize them to a lower spec. 17

18 Idle Amazon RDS instances: These are database instances that are probably empty, infrequently accessed, or simply over provisioned. Check the number of connections against each instance and their sizes. Check what databases they host. If they don t contain any user databases, terminate them. If the instances are running for non-production purposes in multiple Availability Zones, recreate them in single-az configuration. This can reduce costs significantly. Unassociated Amazon Elastic IP Addresses: Unassigned Elastic IP addresses or EIPs associated with offline EC2 instances incur costs. Audit these Elastic IP addresses and release them if not needed. Underutilized Amazon EBS Volumes: You may have a large number of unattached EBS volumes. Check why they are there in the first place, and delete them if possible. In the sample report below, a Trusted Advisor audit revealed potential savings of more than $32,000 every month. Expanding each area of the audit will show a more detailed cost analysis. The overall or individual audit reports can be downloaded as CSV files. 18

19 AWS Price List API When designing cloud-based applications and infrastructures, architects need to consider the pricing model of the services involved. Multiple factors can affect the total cost of ownership. A database hosted in a multi-az RDS instance could potentially cost more than running it in a two-node EC2 cluster. Then again, the price of EC2 instances depends on their instance types, and even those prices are different for on-demand and reserved instances. Traditionally, this involves a lot of manual price look-ups from different service pages. Fortunately, AWS now offers programmatic access to its price list. This is called the AWS Price List API. This is great news for system designers because you can dynamically query the AWS price list. Queries can be built into custom code written in any language. Use these links to get dynamic pricing information: The Offer Index: This is a JSON file available at ( offers/v1.0/aws/index.json) that contains a list of services and offer code. The following code snippet shows an excerpt from this file: 19

20 Price List URL: This URL has the form: Here, the offer code for a specific service comes from the offer index file mentioned above. The format at the end of the URL can be either JSON or CSV. For example, if we want to download the pricing information for RDS in a JSON format, the URL will is: The JSON pricing file has two sections for each region: The product information, identified by a unique code or SKU The pricing information related to that unique code The following code snippet shows the product information for an RDS service in the apsoutheast-2 (Sydney) region. This is the product information for: Oracle Enterprise Edition RDS instances On the db.m3.2xlarge instance type In multi-az configuration Where the customer will be using its on-premise license (BYOL). 20

21 Here is the pricing information related to this particular SKU. The first is an on-demand price per hour ($1.93) and the second is the price for reserved instances with a one-year term and no upfront fees ($1.36 per hour): 21

22 Download these files and programmatically parse them with supplied parameters for up-to-the minute pricing information. A few points about the dynamically generated price list: It does not contain free tier pricing It does not contain EC2 spot instance pricing The pricing information is not available in China Price Update Notification AWS Price Update Notification is yet another way to keep on top of AWS price changes. Set up an Amazon Simple Notification Service (SNS) alert for AWS price changes and, when AWS makes changes to its price list, the Amazon SNS topic will send an alert to the subscriber. Price changes can happen for a number of reasons. For example, AWS may decide to lower its existing prices, or it can introduce a new instance type or even a new service. Staying on top of these prices will help you budget for the future. The following images show how to create the Price Update Notification Simple Notification Service (SNS) topic. Here, we have to choose our region as US East (N. Virginia) (AWS billing data for all your resources is stored here). We are creating a new subscription and choosing the endpoint for an existing Amazon SNS topic. This Amazon SNS topic has already been created by AWS, and its ARN is: arn:aws:sns:us-east-1: :price-list-api We would like to be notified by when the SNS topic sends an alert, so we have provided an address as the subscription endpoint. From now on, an automated will be sent to this address each time there is a price change from AWS. 22

23 AWS Simple Monthly Calculator Finally, the AWS Simple Monthly Calculator is another great tool for estimating resource costs for your project or application. This is an online tool that you can access at s3.amazonaws.com/. Simply specify a number of parameters for one or more AWS services and calculate the estimated monthly cost. The estimation is based on the pricing at the time of calculation. In the images below, we are estimating the cost of the following infrastructure components in the ap-southeast-2 (Sydney) region: 2 on-demand m3.xlarge type Linux-based Amazon EC2 instances 2 500GB general purpose SSD Amazon EBS storage volumes for each Amazon EC2 instance 1 500GB Amazon S3 standard storage with 2000 GET requests and 1000 PUT/LIST requests per month 1 multi-az db.m3.large type MySQL Amazon RDS instance with 300GB general purpose storage and same storage space for backup 1 100GB Amazon DynamoDB instance with provisioned throughput and strong read consistency 23

24 24

25 As you can see, the cost estimations are added as you include resources in your calculation. In the last image, we can see the monthly expenditure for this infrastructure can be up to $2, Third Party Tools There are also numerous third party tools for running analytics, breakdowns, and estimations for your AWS spending. Some of these tools are open source, while others are commercial but offer free trials. Here are a few programs that you can try: Netflix Ice: A free, open source tool offered by Netflix. You can start by downloading and running it in your EC2 environment. Cloudcheckr: A unified security and cost management solution with a 14-day free trial. Cloudability. A popular SaaS solution used by Uber, Atlassian, Cisco, and others. They also offer a 14-day free trial. This is by no means an exhaustive list, and more services can be found in the AWS Marketplace. 25

26 Part 4: Putting it all together: 5 things you can do right now On-demand pricing is one of the biggest advantages of cloud deployment. However, despite all the available tools, anticipating your monthly costs can still be a challenge.. Here are 7 easy steps that you can take right now. 1. Pay less when you reserve AWS instances If you re currently using an on-demand Amazon EC2 or Amazon RDS instance, take a few minutes to do a review. If it turns out that your usage is fairly steady over a long period, you may find that reserve pricing costs much less. One upfront payment that covers a longer time period (say, 1-3 years), can offer a significant discount over the hourly usage for the same instance. 2. Bid on the Spot market for unused AWS Resources When you launch an instance, you re given the option of bidding for unused Amazon EC2 capacity on the Spot market. When your bid is equal to or higher than the current Spot price, you will automatically be given use of an Amazon EC2 instance of the specified type for as long as the price stays within your bid. Spot bidding can save you up to 90% on the cost of EC2 instances. Spot instances are useful for heavy lifting tasks such as complex analytics, Big Data processing, scientific computing, and media processing. You just have to be confident that your application can survive unexpected terminations should the Spot price rise above your bid. It may be worthwhile to review the Spot price history in your availability zone. 3. Take proactive steps to monitor your AWS billing and usage Reduce the risk of unanticipated usage costs by setting up daily alerts for billing and usage. In the Billing and Cost Management console, enable the Receive Billing Reports option under Preferences to get daily billing information. Use Amazon CloudWatch to notify you when your monthly charges for using an AWS product reach a pre-configured threshold. Rather than having to log in to the AWS Billing Console to check for yourself, these push notifications proactively notify you of activity on your AWS accounts. When your usage crosses the pre-set threshold, you will receive a message from Amazon CloudWatch advising you to take the necessary action. To get started with Amazon CloudWatch, log into the AWS Billing Dashboard, go to Preferences, and select Receive Billing Alerts. 4. Tag your resources When you tag your AWS resources to identify their function or association (QA, Dev, Prod, etc.) you ll be able to quickly know which environments are incurring the highest costs and which 26

27 business unit may be exceeding its AWS cost allocations. This can be a particularly effective way to monitor and control costs. 5. Analyze AWS billing reports AWS provides monthly billing reports that highlight the costs incurred by individual AWS services and the number of hours used per month. You may also want to see a forecast of your future expenses in a format you can share with a customer. To get this kind of clarity, you can export your billing reports in CSV format or even build a custom application with its own analytics. You could also enable programmatic access to your AWS account and specify an Amazon S3 bucket into which you want your billing data copied. AWS will then generate estimated monthly bills several times a day and save them to your Amazon S3 bucket. You can also use the Amazon S3 bucket data as input for your application. 6. Use the Simple Monthly Calculator and Total Cost of Ownership Calculators The Simple Monthly Calculator makes it easy to estimate usage costs. The Total Cost of Ownership Calculator is a useful tool for calculating infrastructure costs and savings Use the AWS Price List API for Dynamic Cost Data You can query current prices of AWS Services using the AWS Price List API. This is very useful for creating up to the minute pricing budgets outside of the Simple Monthly Calculator. Price changes can be monitored using an SNS topic. The AWS Price List API can present pricing as JSON and CSV. The API URLs to query Amazon RDS prices would be: In conclusion, the best strategy for cost management is generally to be as proactive as possible. In addition to the tools and tips introduced in this ebook, we encourage you to also stay up to date on the latest AWS releases and announcements. We hope this ebook has been a useful resource, and we welcome any questions or feedback at support@cloudacademy.com. A few additional resources outlined below. 27

28 Additional Resources Amazon Web Services Cost Optimization web page AWS Trusted Advisor Best Practices AWS White Paper: Cost Optimization with AWS Cloud Academy Course: Governance on AWS 28