Este vídeo no está disponible en Español. El vídeo no está disponible en English (US).
Columbia Sportswear's CI practices, processes, and automation to accelerate Azure PaaS adoption
2 years ago we decided to become a "cloud first" organization with a heavy emphasis on Azure PaaS. We wrote a DSL (domain specific language) based on PowerShell and ARM (json) templates so that we could accelerate the repeatable, governed, self-service, and safe consumption of cloud resources. ### Columbia Sportswear is a high profile company in the middle of an enterprise transformation to aggressively move to Azure PaaS (Service Bus, APIM, APIs, Functions, Logic Apps, Cosmos DB, Azure Data Warehouse, Azure Data Lake Store, Databricks, etc.) while also implementing Dynamics 365 Finance & Operations for their Retail footprint. ### This talk will provide context, examples, demos, and discussion of the DSL we built to manage Azure at Columbia Sportswear including our focus on testing, continuous integration, and continuous deployment for the DSL. I will share both a tool maker's perspective as well as the perspective of someone consuming the tooling. ### The overall goal of this effort was to accelerate the repeatable, governed, self-service, and safe consumption of cloud resources. ======================================================================================================================= During the early days of the Open Data Platform (mid 2017) we evaluated a range of options (three listed below) to facilitate automation of Columbia's Azure footprint while achieving the principles of Continuous Delivery and DevOps. It wasn't our intention to write our own framework! In fact, we very much wanted to use Terraform to implement Columbia's pipelines. ### One option was to use CI/CD Pipeline plugins from the [VSTS Marketplace](https://marketplace.visualstudio.com/). Superficially this had many appealing characteristics such as having someone else do the heavy lifting and possibly parameterize complex implementations. We identified several disadvantages such as: * Marketplace extensions are typically created by 3rd party developers. One concern is that these developers may not be responsive to address bugs or to pursue changes that Columbia wants * Because the extensions only exist within a pipeline context, builds are difficult if not impossible to test or debug locally * Columbia would find it difficult to maintain governance or uniformity of how services have been implemented ### A second option, [Terraform](https://www.terraform.io/docs/providers/azurerm/index.html) was being explored within Columbia's Platforms Engineering team to provision on-premises Virtual Machines. * We liked many aspects of Terraform including: * High levels of abstraction in some resources * Commercially developed and supported * "What If" simulation via Terraform Plan * Well-documented use of the Azure resources * Open source licensing model * We noted several disadvantages in Terraform in 2017 versus Columbia's SaaS/PaaS Azure implementation strategy * Terraform requires `terraform.tfstate` files to be stored locally or in a blob storage. Running terraform against multiple build servers introduces complexity in ensuring consistent `tfstate` files. In addition, if the blob were to be destroyed for whatever reason that could have a profound impact on activities as Terraform would want to build everything again * At the time Terraform's energy was mostly centered on AWS and releases for Azure weren't very frequent * The available Azure resources were mainly Infrastructure as a Service (IaaS) and many of the types of services we needed to implement (APIs, Functions, Web Apps, Azure SQL, etc.) were not available * If Columbia wanted to contribute to the Azure PaaS resources on Terraform we'd need to work in [Go Lang](https://golang.org/) and wait for Pull Requests to be merged * If Columbia used Terraform we might have to wait multiple months for Terraform resources to be created for new PaaS services. Enhancements for existing Azure services might mean waiting several weeks for changes to be implemented ### We ultimately selected [Azure Resource Management](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-quickstart-create-templates-use-the-portal) **(ARM)** templates * ARM templates are essentially JSON documents. They are universally used on Azure for both IaaS and PaaS. In many cases, an ARM template can be generated as part of a manual implementation. These templates may be subsequently generalized for additional deployments * Editing ARM templates can be quite difficult and error-prone. In most cases, parameters are passed into ARM templates from yaml files ### Our intentions for our DSL, `csc-azure` 1. Easy for other teams to get started 2. Well-behaved Continuous Integration / Continuous Delivery (CI/CD) Pipelines 3. Extensible framework / patterns](#extensible-framework-patterns 4. Delivery Engineering (or those submitting PRs) can safely make changes ### Writing our own DSL ended up providing some unanticipated benefits * We can decide the pace at which we implement new features from specific Azure API releases * We can regression test our entire footprint to understand the impact of breaking changes from Microsoft * We can 'hydrate' an entire Azure subscription with required components (Service Bus, API Management, Event Grid, Data Lake, Gateways) with relatively low effort