{"id":577,"date":"2020-06-25T00:00:00","date_gmt":"2020-06-25T07:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/the-next-frontier-in-machine-learning-driving-responsible-practices"},"modified":"2025-06-30T03:59:50","modified_gmt":"2025-06-30T10:59:50","slug":"the-next-frontier-in-machine-learning-driving-responsible-practices","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/the-next-frontier-in-machine-learning-driving-responsible-practices\/","title":{"rendered":"The next frontier in machine learning: driving responsible practices"},"content":{"rendered":"\n

Organizations around the world are gearing up for a future powered by artificial intelligence (AI). From supply chain systems to genomics, and from predictive maintenance to autonomous systems, every aspect of the transformation is making use of AI. This raises a very important question: How are we making sure that the AI systems and models show the right ethical behavior and deliver results that can be explained and backed with data?<\/p>\n\n\n\n

This week at Spark + AI Summit, we talked about Microsoft\u2019s commitment to the advancement of AI and machine learning driven by principles that put people first.<\/p>\n\n\n\n

Understand, protect, and control your machine learning solution<\/h2>\n\n\n\n

Over the past several years, machine learning has moved out of research labs and into the mainstream and has grown from a niche discipline for data scientists with PhDs to one where all developers are empowered to participate. With power comes responsibility. As the audience for machine learning expands, practitioners are increasingly asked to build AI systems that are easy to explain and that comply with privacy regulations.<\/p>\n\n\n\n

To navigate these hurdles, we at Microsoft, in collaboration with the Aether Committee and its working groups<\/a>, have made available our responsible machine learning<\/a> (responsible ML) innovations that help developers understand, protect and control their models throughout the machine learning lifecycle. These capabilities can be accessed in any Python-based environment and have been open sourced on GitHub to invite community contributions.<\/p>\n\n\n\n

\"The<\/figure>\n\n\n\n

Understanding the model behavior includes being able to explain and remove any unfairness within the models. The interpretability and fairness assessment capabilities powered by the InterpretML<\/a> and Fairlearn<\/a> toolkits, respectively, enable this understanding. These toolkits help determine model behavior, mitigate any unfairness, and improve transparency within the models.<\/p>\n\n\n\n

Protecting the data used to create models by ensuring data privacy and confidentiality, is another important aspect of responsible ML. We\u2019ve released a differential privacy toolkit<\/a>, developed in collaboration with researchers at the Harvard Institute for Quantitative Social Science and School of Engineering. The toolkit applies statistical noise to the data while maintaining an information budget. This ensures an individual\u2019s privacy while enabling the machine learning process to run unharmed.<\/p>\n\n\n\n

Controlling models and its metadata with features, like audit trails<\/a> and datasheets<\/a>, brings the responsible ML capabilities full circle. In Azure Machine Learning, auditing capabilities track all actions throughout the lifecycle of a machine learning model. For compliance reasons, organizations can leverage this audit trail to trace how and why a model\u2019s predictions showed certain behavior.<\/p>\n\n\n\n

Many customers, such as EY<\/a> and Scandinavian Airlines<\/a>, use these capabilities today to build ethical, compliant, transparent, and trustworthy solutions while improving their customer experiences.<\/p>\n\n\n\n

Our continued commitment to open source<\/h2>\n\n\n\n

In addition to open sourcing our responsible ML toolkits, there are two more projects we are sharing with the community. The first is Hyperspace<\/a>, a new extensible indexing subsystem for Apache Spark. This is designed to work as a simple add-on, and comes with Scala, Python, and .Net support. Hyperspace is the same technology that powers the indexing engine inside Azure Synapse Analytics. In benchmarking against common workloads like TPC-H and TPC-DS, Hyperspace has provided gains of 2x and 1.8x, respectively. Hyperspace is now on GitHub. We look forward to seeing new ideas and contributions on Hyperspace<\/a> to make Apache Spark\u2019s performance even better.<\/p>\n\n\n\n

The second is a preview of ONNX Runtime’s<\/a> support for accelerated training. The latest release<\/a> of training acceleration incorporates innovations from the AI at Scale<\/a> initiative, such as ZeRO optimization<\/a> and Project Parasail<\/a>, which improves memory utilization and parallelism on GPUs.<\/p>\n\n\n\n

We deeply value our partnership with the open source community and look forward to collaborating to establish responsible ML practices in the industry.<\/p>\n\n\n\n

Additional resources<\/h2>\n\n\n\n