Responsible machine learning (responsible ML)
Azure Machine Learning capabilities that empower data scientists and developers to innovate responsibly.
Understand, protect and control your data, models and processes to build trusted solutions.
State-of-the-art technology enabling responsible ML development, deployment and use. Put responsible AI principles into practice and build trust throughout the ML lifecycle.
Understand
Gain visibility into your models, explain model behavior and detect and mitigate unfairness—all with out-of-the-box visualisations.
Protect
Apply differential privacy techniques to protect sensitive data and prevent leaks. Encrypt data and build models in a secure environment to maintain confidentiality.
Control
Use built-in lineage and audit trial capabilities and enable responsible process by documenting model metadata to meet regulatory requirements.
Understand your models and build for fairness
Explain model behavior and uncover features that have the most impact on predictions. Use built-in explainers for both glass-box and black-box models during model training and inferencing. Use interactive visualizations to compare models and perform what-if analysis to improve model accuracy. Test your models for fairness using state-of-the-art algorithms. Mitigate unfairness throughout the ML lifecycle, compare mitigated models and make intentional fairness versus accuracy trade-offs as desired.
Protect data privacy and confidentiality
Build models that preserve privacy using the latest innovations in differential privacy, which injects precise levels of statistical noise in data to limit the disclosure of sensitive information. Identify data leaks and intelligently limit repeat queries to manage exposure risk.
Use encryption and confidential machine learning (coming soon) techniques specifically designed for machine learning to securely build models using confidential data.
Control and govern through every step of the ML process
Access built-in capabilities to automatically track lineage and create an audit trial across the ML lifecycle. Obtain full visibility into the ML process by tracking datasets, models, experiments, code and more. Use custom tags to implement model datasheets, document key model metadata, increase accountability and ensure responsible process.
Resources
Documentation and samples
See responsible ML in action
Get better visibility into models to explain their behavior
Get better visibility into models to explain their behavior
How to build trusted models?
Susan, a data scientist, works for an online streaming platform. Her company is launching a new show next month and she has been asked to build a model to identify customers who would be interested in watching it and thus should receive a promotion email inviting them.
Her first model shows an accuracy of 90% on historical data. It seems too good to be true, so she wants to understand the model behavior to ascertain its trustworthiness.
Susan is a data scientist tasked with creating a model to identify customers for a new show. Her model shows an accuracy of 90% on historical data. She would like to understand the model behavior to ascertain its trustworthiness.
First, get the most recent model explanations
Azure ML effortlessly surfaces model explanations via the InterpretML SDK. Either from the ‘Models’ tab in her Azure ML workspace or from her notebook, Susan can fetch model explanations from the most recent experiment run and visualise them.
Examine the strongest predictor variable/feature
On the global importance view, she sees LOCATION_NEWYORK as the strongest predictor for the promotion. This is unexpected - she knows that viewing habits can be materially different for New York compared to other locations.
Validate the predictor and assess model behavior
On the local importance view, she can inspect the results with more granularity. She finds the model has drawn the wrong conclusions about locations other than New York and realises that the model may not perform well across their wider customer base.
Improve the training data for accuracy and rerun the experiment
Susan’s hypothesis is that the bias may be coming from her training data. On reviewing, she finds that records for New York are introducing a skew in her training set. She works with her team to remove this skew and re-run the experiment. The new model is published in Azure ML, where its experiment results, including model explanations, are available for review.
Model interpretability encourages a glass box approach to your models
With model interpretability, machine learning models are no longer a black box. It helped Susan understand her model behavior at training time and identify a fault. This enabled her to avoid releasing a model which would have performed poorly in production.
With interpretability supported at inferencing time as well, it helps make model predictions transparent to customers - a key principle of responsible AI.
Model Interpretability in Azure ML InterpretML Github repo for this demo
With model interpretability, machine learning models are no longer a black box. It helped Susan understand her model behavior at training time and identify a fault. This enabled her to avoid releasing a model which would have performed poorly in production.
With interpretability supported at inferencing time as well, it helps make model predictions transparent to customers - a key principle of responsible AI.
Model Interpretability in Azure ML InterpretML Github repo for this demo
Make critical decisions with models that are without hidden biases
Make critical decisions with models that are without hidden biases
How to fairly identify the right job applicants?
Joe, a data scientist, works for an IT company. Their recruiting team is swamped with job applications for developer roles and has asked him to build a model to recommend which applicants should be interviewed. His first model shows accuracy at over 85%. Joe is worried though - his company has traditionally recruited college graduates with degrees and he want to ensure his model does not discriminate against other candidates with diplomas, certifications or relevant experience.
He turns to Azure ML, to see if there is any bias in his new model.
Joe, a data scientist, works for an IT company. Their recruiting team is swamped with job applications for developer roles and has asked him to build an unbiased model to recommend which applicants should be interviewed.
Choose a potential bias factor in the model
Azure ML provides capabilities to detect bias and ensure fairness within models via the Fairlearn SDK. From within his Jupyter notebook, Joe initiates the Fairlearn toolkit and selects HAS_DEGREE as a sensitive feature, to determine if having a degree has any impact on the final recommendation from the model.
Select the model metric for analysis
The Fairlearn toolkit offers many metrics to analyse against potential bias, like accuracy, precision, recall etc. Joe opts accuracy as the metric to determine if there is any impact on the accuracy of the model while adjusting it for fairness. This will help Joe maintain a good balance between the accuracy of the model and its fairness.
Identify if there is inherent bias
The Fairlearn toolkit offers visualisations that show the relevant details of the model. Joe observes that the accuracy is relatively good for both, degree holders and non-degree holders, however, the model is far more likely to recommend interviewing a degree holder. Joe’s concerns are validated - the model is showing bias.
Mitigate bias in the model
Within Azure ML, the Fairlearn toolkit shows more versions of the model with varied levels of bias mitigation. Joe can see the tradeoff between the accuracy and the disparity across predictions. This provides Joe the options to reduce bias and increase fairness within his model. He selects one of the improved models and publishes it for release.
Model fairness ensures that there is no bias within the model’s recommendations
With Fairlearn in Azure ML, Joe was able to quickly identify and mitigate an inherent bias in his model before it went to production. This bias could have risked the company losing valuable candidates, not to mention their reputation. With the inbuilt mitigation options, he was able to address the bias transparently.
Fairlearn enables AI systems to be inclusive and treat all people with fairness, a key principle of responsible AI.
Model fairness in Azure ML Fairlearn Toolkit Github repo for this demo
With Fairlearn in Azure ML, Joe was able to quickly identify and mitigate an inherent bias in his model before it went to production. This bias could have risked the company losing valuable candidates, not to mention their reputation. With the inbuilt mitigation options, he was able to address the bias transparently.
Fairlearn enables AI systems to be inclusive and treat all people with fairness, a key principle of responsible AI.
Model fairness in Azure ML Fairlearn Toolkit Github repo for this demo
Using differential privacy techniques to protect sensitive datasets
Using differential privacy techniques to protect sensitive datasets
How to ensure data privacy while building models?
Anna, a data scientist, has been working on a model to predict patient readmission rates but feels restricted by the redacted or masked data she can access due to the patient data privacy regulations.
With her initial models lacking the predictive rigor needed for deployment, she works with her internal compliance team to expose a more complete dataset, while still maintaining the privacy, by using the WhiteNoise differential privacy toolkit within Azure ML.
Anna, a data scientist, is trying to predict patient readmission rates, but does not have access to patient data due to privacy regulations. Using SmartNoise in Azure ML, Anna gets a more useful dataset, while still maintaining privacy.
Leverage SmartNoise to protect patient privacy while querying data
Anna queries the dataset using SQL in her Jupyter notebook within Azure ML, but now, via the SmartNoise toolkit. SmartNoise ensures that the result sets are carefully injected with statistical noise and the exposure risk is managed by tracking the information exposed by individual queries and limiting them as appropriate.
Leverage this differentially private dataset for model training
Anna incorporates the dataset into her existing training script. It does not require her to change anything in her code except pointing to the new dataset. She can seamlessly execute the training job on the same Azure ML compute target. Throughout the process, her compliance team can validate that her queries remain within budget and thus prove patient privacy is protected.
Validate the model for efficacy
Using the results widget, Anna can see her new model is now showing better accuracy due to the added features in her training data. She has not violated any privacy rules and her compliance team has full visibility into her querying behavior. This new model is now ready to move to production, for better predictions. She publishes it to Azure ML for release.
SmartNoise in Azure ML, protects data privacy while building models
With SmartNoise in Azure ML, Anna was able to confidently build her machine learning model without compromising privacy of sensitive data. This enabled her to use machine learning effectively and materially improve outcomes, while allowing the compliance team to rest assured they could prove sensitive data was not exposed.
SmartNoise uses differential privacy techniques to enable building machine learning while respecting privacy, a key principle of responsible AI.
SmartNoise in Azure ML SmartNoise Toolkit Github repo for this demo
With SmartNoise in Azure ML, Anna was able to confidently build her machine learning model without compromising privacy of sensitive data. This enabled her to use machine learning effectively and materially improve outcomes, while allowing the compliance team to rest assured they could prove sensitive data was not exposed.
SmartNoise uses differential privacy techniques to enable building machine learning while respecting privacy, a key principle of responsible AI.
SmartNoise in Azure ML SmartNoise Toolkit Github repo for this demo
Customers using responsible ML
Alex Mohelsky, Partner and Data Analytics Leader at EY Canada"Azure Machine Learning and its Fairlean capabilities offer advanced fairness and explainability that have helped us deploy trustworthy AI solutions for our customers, while enabling stakeholder confidence and regulatory compliance."
Daniel Engberg, Head of Data Analytics and Artificial Intelligence, Scandinavian Airlines"Azure Machine Learning helps us build AI responsibly and build trust with our customers. Using the interpretability capabilities in the fraud detection efforts for our loyalty program, we are able to understand models better, identify genuine cases of fraud, and reduce the possibility of erroneous results."
