This post was co-authored by the QnA Maker Team.
With Microsoft Bot Framework, you can build chatbots and conversational applications in a variety of ways. Whether you’re looking to develop a bot from scratch with the open source Bot Framework, looking to create your own branded assistant with the Virtual Assistant solution accelerator, or looking to create a Q&A bot in minutes with QnA Maker. QnA Maker is an easy-to-use web-based service that makes it easy to power a question-answer application or chatbot from semi-structured content like FAQ documents and product manuals. With QnA Maker, developers can build, train, and publish question and answer bots in minutes.
Today, we are excited to reveal the launch of a highly requested feature, Active Learning in QnA Maker. Active Learning helps identify and recommend question variations for any question and allows you to add them to your knowledge base. Your knowledge base content won’t change unless you choose to add or edit the suggestions to the knowledge base.
How it works
Active Learning is triggered based on the scores of top N answers returned by QnA Maker for any given query. If the score differences lie within a small range, then the query is considered a possible “suggestion” for each of the possible answers. The exact score difference logic is a function of the score root of the confidence score of the top answer.
All the suggestions are then clustered together by similarity and top suggestions for alternate questions are displayed based on the frequency of the particular queries by end users. Therefore, active learning gives the best possible suggestions in cases where the endpoints are getting a reasonable quantity and variety in terms of usage queries.
QnA Maker learns new question variations in two possible ways.
- Implicit feedback – The ranker understands when a user question has multiple answers with scores which are very close and considers that as implicit feedback.
- Explicit feedback – When multiple answers with little variation in scores are returned from the knowledge base, the client application can ask the user which question is the correct question. When the user selects the correct question, the user's explicit feedback is sent to QnA Maker with the Train API.
Either method provides the ranker with similar queries that are clustered. When similar queries are clustered, QnA Maker suggests the user-based questions to the knowledge base designer to accept or reject.
How to turn on active learning
By default, Active Learning will be disabled for everybody. Please follow the below steps to enable the Active Learning.
1. To turn active learning on, go to your Service Settings in the QnA Maker portal, in the top-right corner.
2. Find the QnA Maker service then toggle Active Learning.
Once Active Learning is enabled, the knowledge suggests new questions at regular intervals based on user-submitted questions. You can disable Active Learning by toggling the setting again.
How to add Active Learning suggestion to the knowledge base
1. In order to see the suggested questions, on the Edit knowledge base page, select Show Suggestions.
2. Filter the knowledge base with question and answer pairs to only show suggestions by selecting Filter by Suggestions.
3. Each question section with suggestions shows the new questions with a check mark to accept the question or an x mark to reject the suggestions. Click on the checkmark to add the question.
You can add or delete all suggestions by selecting Add all or Reject all.
4. Select Save and Train to save the changes to the knowledge base.
To use Active Learning efficiently, one should have higher traffic on the bot. Higher the number of end-user queries, the better will be quality and quantity of suggestions.
QnA Maker active learning Dialog
The QnA Maker active learning Dialog does the following:
- Get the Top N matches from the QnA service for every query above the threshold set.
- If the top result confidence score is significantly more than the rest of the results, show only the top answer.
- If the Top N results have similar confidence scores, then we prompt the user asking which of the following question he meant.
- Once the user selects the right question that matches intent, show the answer for that corresponding question.
- This selection also triggers feedback into the QnA Maker service via the Train API.
Migrating knowledge bases from the old preview portal
You may recall at //Build last May 2018, we announced the general availability (GA) of QnA Maker with new architecture built on Azure. As a result, knowledge bases created with QnA Maker free preview will need to be migrated to QnA GA, as the QnA Maker preview will be deprecated January 31, 2019. Learn how to migrate existing ones on the documentation, “Migrate a knowledge base using export-import.”
Below is a screenshot of the old QnA Maker preview portal for reference:
For more information about the changes in QnA Maker GA, see the QnA Maker GA announcement blog post, “Announcing General Availability of QnAMaker.”
QnA Maker GA highlights:
- New architecture. The data and runtime components of the QnAMaker stack will be hosted in the user’s Azure subscription. Learn more on the documentation, “What is QnA Maker?”
- No more throttling. Pay for services hosted, instead of transactions. See pricing information.
- Data privacy and compliance. The QnA data will be hosted within your Azure compliance boundary.
- Brand new portal experience to create and manage your knowledge base. Check out the new portal.
- Scale as you go. Scale different part of the stack as per your needs. See upgrading your QnA Maker service.