Monday, September 9, 2019

Sitecore.com Continuous Deployment on Azure (2019 Edition)

It's been a while since my previous post about continuous deployment with Sitecore, lots of things have changed since then, www.sitecore.com is now hosted on Azure PAAS, the team is using Azure DevOps instead of TeamCity and much more. Over the past few years we completely revamped our deployment infrastructure and processes, and in this blog post I'll share how our current setup looks like after all of those iterations. Hint: it is now extremely simply and basically configurable in point-and-click manner.

Preface
One of the major requirements for us was being able to test the features and fixes in isolation, that's where the release flow and pull requests came handy. Here's how we do branching in GIT (read more about this strategy):



This way we always have "ready to release" master branch, as well as can easily merge and deploy hotfixes.

We use the following environments:
  1. QA (on-prem)
    • This is where "topic" (aka "feature") branches are deployed (usually, as pull requests)
    • Automated UI tests run after each deployment
  2. Pre-production (Azure)
    • This is where "release" branch will be deployed before the release
    • Automated UI tests run after each deployment
  3. Production (Azure)
    • Once we validate the build in pre-production environment, we will deploy the same artifact to production
    • Automated UI tests run after each deployment
What I refer to as "build artifact" is:
  • WebDeploy package with custom files (built by TDS or custom script)
  • Item .update package (GIT DeltaDeploy or Sitecore Courier)

On-prem deployment
In QA environment, we test each pull request separately from others. We basically install clean Sitecore instance (XP0) from scratch and deploy custom code on top. All of that takes less than 30 minutes and includes production content restore.

Specifically, we do the following:
  1. Deploy new Sitecore instance: using with Sitecore Deploy module from Azure DevOps marketplace
  2. Copy over custom code using standard "Unzip zip files to a direcory" task
  3. Install items using Sitecore.Ship (version compatible with Sitecore 9.x)
azure devops qa deployment

The Install step basically consists of two task:


"Manage IISWebsite" is a standard Azure DevOps task where we set correct bindings for the website. Since we deploy pull requests (PR), we use wildcard domain, i.e. test instance name will be:

pr#id#.#server#.domain, for example: pr1143.server15.yourcompany.com.

Single Windows 2019 server can easily handle 5-10 instances running simultaneously, and this approach scales to multiple servers.

Azure deployment
Azure is slightly different. Obviously, it does not make sense to deploy scaled Azure instance each time before the release, because it would take hours to set up, even if we take Azure Search, Analytics, etc. out of equation.
However, it is still simple: we use App Service Slots and deploy the updated files into the copes of CD and CM instances. The databases are shared between the main app and the slot, therefore we don't need to copy them

Here are the pipeline stages:
  1. Create a zip archive of production instance (CM, CD)
  2. Create a slot and deploy the website copy there (CM, CD)
  3. Deploy custom files using standard "Azure App Service deploy" task (CM, CD)
    • It is important to note, that we deploy our updates to the slot, so even if something goes wrong - this will not affect production website
  4. Install items using Sitecore.Ship (version compatible with Sitecore 9.x)
    • Since we used shared databases, we make sure that the changes in items are not breaking, i.e. the website should work with "old" files and "new" items
  5. Review the website deployed to slot (as well as run different kinds of tests)
  6. Publish the items and then "swap" the slots in Azure
    • "Slot" content is seamlessly moved to production by Azure, and new www.sitecore.com becomes available (without downtime)
azure devops production deployment

Pre-production environment is mirroring production setup, the only difference is that production environment has slightly better specs and all those bindings and certificates assigned. It is particularly useful when you setup or update automation so that you don't break your production website along the way.

By and large
This setup proved to be robust and manageable. There is definitely no "one size fits all" solution, but I'd like to highlight a few important points that make sense for every project if you want to deploy the changes frequently and safely:

  • Test features and fixes in isolation, otherwise you will greatly complicate the maintenance and deploying often will be painful
  • Automate as many deployment steps possible and run them hundreds times to polish the process
  • Use automated UI tests that run after each deployment on any stage to identify the issues early

Don't hesitate to share your thoughts on deploying Sitecore to Azure, as well as your own stories and hints.

13 comments:

  1. Nice post. Can you articulate how you go about making "sure that the changes in items are not breaking?" Wouldn't that result in technical debt over time? e.g. you remove a field from a template because a new deployment doesn't need that any more and it is replaced by another field. Are you saying you'd still keep that deprecated field on the template just for backwards compatibility with the old code? I feel like this part of the whole process along could be a lengthy blog post. The biggest challenge we've seen with blue/green deployments using slots is having duplicated master and web DBs and the time it takes to sync changes between them once they are stale.

    Also, what tool(s) do you use for automated UI testing?

    ReplyDelete
    Replies
    1. Good question! We use Delta Deploy packages which do not remove anything (only add and change items). For anything that requires deletion we delete it in Got and add manual "production cleanup" task. Luckily, such tasks account for less than 1% of the backlog.

      As for UI testing - we use Selenium (C# driver). Quite important to run them as a part of CI too, not from Dev/QA personal machines.

      Delete
  2. This one is really nice post.

    Some Notes form my side:
    One more reson why not to deploy scaled environment by Azure Pipelines into Azure PaaS: an agent job times out after 60 min.
    Deployment of the Sitecore 9.1 XP extra small takes way more than 60 minutes, so that means your Release Pipeline will fail (or you can run PS script with -AsJob argument and then guess when the deployment is ready)
    Small tip - Sitecore 9.1 XP Large and Extra Large are deployed much faster, sometimes it makes sense to deploy Extra Large and then downscale using PS commands.
    About Slots deployment: there is also nice possibility to do a backup of Prod Web App (New-AzWebAppBackup) and then restore that backup to a slot (Restore-AzWebAppBackup) which is pretty easy to do from Azure Pipeline (Azure Powershell Task) and is relatively fast.

    ReplyDelete
    Replies
    1. Thanks Ivan! Will try that out, I was looking for more robust/native way to create staging slots than Kudu.

      Delete
  3. This comment has been removed by the author.

    ReplyDelete