Manage Azure OpenAI and ChatGPT/GPT-4 costs

In my last post I showed how to get started with Azure OpenAI, the service which is effectively "Microsoft-hosted ChatGPT" but also includes other OpenAI models. In particular I showed how to integrate ChatGPT with the Power Platform via a custom connector that talks to your instance in Azure. If you're doing anything with Azure OpenAI it's a good idea to monitor costs, particularly if the service will be open to a few consumers - it's not the cheapest Azure service, and costs could accumulate quickly in some usage patterns. The good news is that it is possible to monitor Azure OpenAI costs just like any other Azure service - this means using Azure Cost Management, and you just need to the right configuration to zero in on this service in particular. As with any Azure service it's not possible to put a hard limit in place by default - you could with some custom automation, but only if you implement that. 

What you can do of course, is use Azure Cost Management to implement a Budget which will notify you when costs are passing defined thresholds. This can alert an operations team that consumption is higher than expected, at which point some investigation and/or intervention action can occur. With many Azure services this is better than shutting down production use, which would leave teams scrambling to work out why a critical process is not working.

This article focuses on how to put such a Budget and alerting mechanism in place. 

Indeed, that's one of the benefits of using Microsoft's Azure OpenAI instead of the API provided by the OpenAI organisation - in addition to the service meeting all usual Azure standards related to security, support for private networking, regional support, and compliance, you can work with Azure OpenAI through regular Azure service conventions and management tools too. 

Creating an Azure Budget to monitor Azure OpenAI costs

If you've ever created an Azure Budget to monitor another service, the process is no different - it's simply a question of finding the right filter for Azure OpenAI. An important note on this:

Potential challenge - Azure OpenAI options may not appear in the filter list

Something you may run into (I did) when creating a Budget for Azure OpenAI is that there are no relevent filters in the filters list - meaning it's not available for selection. Part of this is expected - if you try to filter on 'ServiceName' expecting to see an option for OpenAI, you won't since it falls under Azure Cognitive Services (and the documentation helpfully calls this out). However, sometimes you won't see anything specific to OpenAI - in particular, when trying to configure the filter for your budget using attributes such as ServiceName, Product, Meter, MeterCategory, MeterSubcategory etc. If this happens, most likely it's because you are trying to create the Budget at a scope which has no OpenAI usage yet. Filters are dynamically created based on usage, so you may need to incur some (and wait a few hours) before your Budget can be created. One option could be to raise the Budget scope (e.g. create at the subscription level rather than resource group) if you have usage elsewhere in your organisation. Either way, once you have valid usage at the scope you're creating for you should see the options in the list.

I provide valid filters you can use below. 

Given the varying scopes a Budget can be created at, there are a couple of ways to do it. The process I outline below creates it at the subscription level (i.e. covering all resource groups in that sub) - but creating at the RG level or the overarching management group level are alternatives,

The process

We start by heading into Cost Management in the Azure portal, and then into 'Budgets'.

Once there, hit the 'Add' button to create a new one. The key step is to add a filter for one of the Azure OpenAI fields - in the image below I'm using MeterSubCategory = Azure OpenAI, but there are several possible options (detailed below):

From there, enter the other details you require from your budget - for example the amount and frequency:

On the next page, define the alert conditions - for example in the image below, I'm asking to be alerted both when Azure forecasts we'll hit 80% of the budget and when the actual spend hits 80%:

 

That's it - your budget should now be created and Azure will start monitoring shortly. The key thing is really understanding what to filter on for Azure OpenAI. There are some options here in fact, so let's explore them.

How granularly do you want to manage costs? Filter options relevant to Azure OpenAI 

Depending on how granular you want to get with monitoring specific Azure OpenAI usage, the following filter options could be used in your Azure Budget:

Filter	Effect
MeterSubCategory = 'Azure OpenAI'	Monitor all OpenAI usage
Product = 'Azure OpenAI - Text-Davinci'	Monitor the specified model only (Text-Davinci in this case)
Meter = 'Text-Davinci Tokens'	I believe filtering on this will allow you to monitor specified model consumption only (as opposed to model training, which would have a different value)
Service Family = 'AI + Machine Learning'	[Slightly different but including for completeness] - this would allow you to monitor ALL AI usage, including Azure Cognitive Services

In short, you can configure your Azure Budget with varying degrees of granularity. In most cases you probably want to use "MeterSubCategory = 'Azure OpenAI'" to measure all Azure OpenAI usage, but hopefully that gives you a sense of the other options.

Using Azure Cost Analysis to monitor OpenAI usage

Creating an Azure budget will help you become aware if costs are beyond what is expected (via the alerts), but sometimes you just want to do some ad-hoc analysis of what's happening. This can be done by heading into the 'Cost analysis' area:

Now go into the 'Services' view (still in preview at the time of writing):

In the list of services displayed. if you expand Azure Cognitive Services you'll find your OpenAI usage listed there:

As ever, you can use the controls in Cost Analysis to change the period so you can compare costs for different periods to establish trends.

Sometimes Excel is better though. Cost analysis will also let you download an Excel file, and the image below shows what your line(s) for Azure OpenAI will look like - see the highlighted row:

As a final note on this, you can configure these Excel files to be automatically generated on a schedule and copied to an Azure BLOB storage container you have. This is done in the 'Exports' area - if you need more detail, I covered how to do it for Microsoft Syntex costs in my Syntex Pay As You Go - how it helps and how to use it article and the process is exactly the same. This will save you the job of manually generating Excel files, and can form the basis of departmental cross-charging, weekly/monthly analysis, or any number of similar FinOps scenarios.

Summary

Creating an Azure budget to monitor costs is likely to be an essential step for any organisation starting to work with Azure OpenAI, whether it's for ChatGPT or something else. One of the benefits of Azure hosting is that all the typical service constructs apply, including Cost Management functionality like Azure Budgets. Defining the correct filter is the key step, and if you don't see options in the list then my suggestion is to reference the explanation in the highlighted box above.

In addition to cost management, you may wish to monitor other aspects of usage such as the number of calls made and how many tokens are used. See the 'Monitoring Azure OpenAI services' link below for more information on this.

Useful documentation pages

• Monitoring Azure OpenAI Service
• Plan to manage costs for Azure OpenAI Service 
• (note this page currently has a small but important inaccuracy, since it suggests a filter of "Service tier: Azure OpenAI" but this doesn't align with the Budget filter options - use one of those in the table above instead)