We will be making some changes to improve the performance and reliability of Visual Studio Application Insights. We're consolidating the set of dimensions you can use to segment charts in metrics explorer, removing rarely used metrics and properties, and reducing the retention period for aggregated metrics.

We spent the past few months reviewing how developers are using Application Insights and are continuously improving the service based on the performance, diagnostics and usage data we collect using Application Insights itself. We’ve already made significant improvements to our UI performance, data freshness and service availability, and are committed to making your experience faster and smoother.

On Wednesday, March 16, 2016, we will make changes to further improve service performance and reliability, while at the continuing to provide you the powerful diagnostics experience you can expect from Application Insights. We’re also working on some cool new things, some of which Brian Harry referred to in his recent blog post, so stay tuned for these announcements as well!

Consolidated dimensions for grouping metrics

When we first designed Application Insights, our vision for metrics was that you should be able to slice and dice by a very wide range of dimensions. But as we all gained experience, it turns out quite a lot of these combinations are not used very often. The trouble is enabling this many combinations of aggregated metrics means storing a huge amount of data, nearly all of which is never used, and which unfortunately impacts overall performance.

Application Insights provides two ways to explore the telemetry from your application:

• Diagnostic Search: Powerful diagnostic tool enabling you to explore specific events, exceptions and the telemetry related to them.
• Metrics Explorer: Produces time charts of aggregated data, which you can filter and segment.

To improve our processing performance and simplify the user experience for picking dimensions to group by, we are consolidating the list of dimensions available out-of-the-box. There will be no change in the data sent to the service from your app. You’ll still be able to do all the diagnostic searches you do today in addition to continuously exporting all that telemetry to Azure Storage or Power BI for further processing.

In practice, this means if you want to create or customize a chart in Metrics Explorer, you’ll have a much simpler list of metrics to choose from. And if you want to segment a chart, you’ll also see a cleaner list in the Grouping menu.

What if we cut your favorite metric or dimension? Well, there are workarounds. You can take advantage of custom instrumentation and send your own TrackMetric() calls or TrackEvent() with the metric as a parameter. If you need this for all metrics sent out-of-the-box by the SDK web module, you can write a telemetry processor to intercept the metrics it generates, and copy the metric values to your custom calls.

As we said, this doesn’t affect the raw telemetry or exported telemetry. For example, if you’re using Power BI or other tools to display your data, the change won’t affect that. Going forward, we’re planning a more powerful search solution with the ability to perform aggregations over your raw telemetry. Stay tuned!

Consolidated usage metrics

Many of our customers have been using Application Insights to monitor usage of their applications with user and session metrics and triage issues based on their real user impact. We are working on a set of powerful capabilities to make this experience even better! For now, to optimize your core diagnostics experience and enhance performance, we are removing some of the rarely used metrics and properties: Events per Session, Exceptions per Session, Pages per Session, Time between sessions, Time between sessions (authenticated users) and Session Duration.

If you’re using any of these metrics or properties, you will be able to calculate them from the exported data using your favorite tools.

Reduced data retention for aggregated metrics

Currently Application Insights retains aggregated metrics at an hourly level for 13 months or more depending on your pricing plan. While going through our usage telemetry we discovered the vast majority of custom queries are under 30 days with more than half being under 24 hours. In order to significantly enhance service reliability and provide optimized data performance, we’re changing the retention period for aggregated metrics from the current levels to 90 days (for all pricing plans).

Application Insights is mainly used to detect, triage and diagnose issues in web apps and services (see recent announcement on transitioning device apps to HockeyApp). It’s rare that CPU utilization or exception rates from several months ago would have much meaning for current deployment and issues. Since the new time range represents over 99.9% of all queries,  we believe optimizing queries in these time intervals will be more valuable than working on improving the behavior of long term retention. Again, if maintaining some metric data for longer periods of time is a key scenario for you, you can continue to get data from Metrics Explorer in Excel or you can use continuous data export.

Reduced search query period

Finally, as we called out in our blog post this past December, we’ve been making changes to how data is managed in our service to improve query performance and overall system reliability. To enable this change, we restricted the period you can perform search queries to seven days. As mentioned above, we’re planning a more powerful search solution that will help you search over raw data for longer periods.

In the interim, Standard and Premium customers can continue to use continuous export to access data older than seven days. Metrics Explorer will continue to provide detailed metrics data including the ability to filter on and group by standard and custom properties beyond seven days.

Feedback

We are learning a lot about how developers like you are using Application Insights. We value your feedback and want to keep making improvements to help you solve problems faster than ever. Please keep the feedback coming on our UserVoice and MSDN forums and watch for these great new capabilities to be announced at Build 2016.