The best resource you have to understand your users is the search box on your site or app – every time they search for something, they are telling exactly what they want. Search not only makes it easier for your users to find what they need in your app, but it can also make it possible for you as the owner of the app to understand what your users are looking for and whether your app is delivering on it.

Imagine you have an e-commerce app, if you could know what the most common queries are, you would get valuable information about the products popular with your users, or if you have queries that return no results at all, it would give you a hint about products you are not currently offering that you could add to your inventory.

You told us you'd like to be able to know what's happening inside your search service and we are happy to announce that we have rolled out the first phase of Search Traffic Analytics that will let you gain visibility into your search service and unlock insights about your users and their behavior.

So, how does it work? Every time a request is made to your search service, we can optionally log that request together with some useful information and we do aggregations to calculate some key metrics. When you enable Search Traffic Analytics, those two streams of data are copied to your own Azure Storage account. In particular, you’ll have two blob containers:

• Operation logs: These are your search service logs, all of your requests will be available in this container. This will let you answer business questions like: “What are your most-searched-for products” or “Which search terms are returning empty results,”as well as operational aspects like indexing rate or API version usage.
• Metrics: These provide you aggregated calculations for operational aspects. The only available metric today is Latency but we are working to add others like QPS soon.

Search domains can be vastly different and it’s partly because of that that we decided to separate the data access layer from the consumption and analytics layer. We want to give you the flexibility to manipulate and analyze the data in the way that works best for you.

Analyzing your data with Power BI

We wanted to make it very easy to start, and that’s why we partnered with Power BI to offer a Content Pack that will let you have a basic dashboard in minutes, and that you will be able to customize to your own needs, creating new visualizations or sharing reports with your team.

When you connect Power BI to your Azure Search analytics storage, the service will collect data for the last 30 days and generate four reports for different dimensions of your service.

1.    Search

A quick glance at how your users are searching. You can see the search volume over time, the most common search queries and queries without results. Also included are two visualizations that show the search distribution across indices and the number of documents returned over time.

2.    Indexing

Similar to the previous report but in this case it focuses on indexing operations. You will be able to see how many documents you have indexed over time and, particularly interesting for services with a high indexing load, we also included a graph that shows the number of 503s which can help you understand if you are putting too much pressure on your system.

3.    Service Stats

This report contains information about all the operations that have been executed against your service. You can drill down on API Version or Operation Category, and you can see which queries caused the search service to return a particular result. This last part is very useful to track down bugs that result in failed requests.

4.    Latency

This report will give you detailed information about the latency metric. You can examine the search and indexing latency separately, as well as the latency for the overall system which includes all operations to help identify if additional replicas are needed to support your increased query load.

Remember that you can edit and modify the reports and dashboards to suit your needs, highlighting the data that is important to you. Stay tuned as we are working on an Azure HDInsight sample that will be able to handle more advanced scenarios, such as very high data loads or more specific business questions.

Enabling search traffic analytics

To enable this feature, go to your Azure Search blade in the Azure Portal and under Settings, you will find Search Traffic Analytics. Select the storage account you’ll use to store the data and make sure to select Logs and Metrics. You will need a storage account in the same region as your search service. Standard costs apply for this storage account.

Once enabled, the data will start flowing to your storage account within 5 to 10 minutes. The data is stored as JSON blobs in blob storage, and there will be one file per hour per container. You can find more information on how to enable Search Traffic Analytics and the full description of the logs and metrics schemas in the docs.

Final thoughts

We encourage you to use all of this new available data, as we had very satisfied customers during our preview testing phase. We are confident that you will gain new information about your service that will help you provide a better experience for your users and obtain a much better grasp on how your search service is doing from an operational perspective. Thanks to this data, one of our customers found and fixed a bug in their indexing code and another one updated all their codepaths to the latest API version.

We can’t wait to know how this feature will help you. Connect it to your Azure Search service and let us know what you think!