Improve your Flow: Multi-Account, Multi-Region CodePipeline

A fundamental principle of successful software development is that flow is the currency of discovery. A theme I’ll work to develop over my blog posts is that the more flow you achieve, the more you discover. When you are requested to build an AWS ecosystem that spans not only regions but accounts – always a complicated situation! – you’ll probably struggle with maintaining flow on this path.

We at Indellient have walked these paths, and this blog post’s goal is to help make the problems and gotchas a little easier to understand – let’s improve your flow!

Table of Contents

• A Word About CodePipeline
• Where IAM Fits into the Picture
• Visualizing Roles Working Together
• CodePipeline: Live Example
• Learn More: Resources

Preface: IAM is complicated to write about. Multi-account adds more complication, and multi-region adds even more. CodePipeline adds its own pieces to the puzzle. To round out the kitchen sink are tools like CodeDeploy and ECS – the real world is often complicated at best! Hopefully, you at least come away from this post with a more organized mental model of IAM in the multi-account/multi-region thought space. Why this specific set of tools? The alternative ways of solving for multi-account/multi-region deploy pipelines can often involve a lot of code duplication (i.e. risk) and manual work (i.e. flow friction).

A Word About CodePipeline

For this post we’ll visualize the multi-region and multi-account issues inside the framing of IAM and CodePipeline. IAM, a service AWS provides for us to manage Identity and Access, is often a tricky subject that touches essentially every single part of AWS. CodePipeline is essentially a customizable pipeline orchestration framework. You can take a code repository’s changes all the way through to your production account with the specific process that works for you. To word it more along the lines of the documentation…an artifact is passed through sequential stages made up of actions that you can order as you need.

As with all tooling, there are opinions baked in to CodePipeline, but my initial experience is that it has some great potential. An example of the opinionated nature is the expectation that you can only have one app per pipeline – so a microservice mono-repo (the code for multiple services lives inside a single code repository) will need to have multiple pipelines. The value proposition here is both that it’s a native AWS tool, and also that you have less build/deploy infrastructure to manage, while providing lots of customizability. So balance the opinions with the value prop – the same situation you’d be in with any tool, really!

Where IAM Fits into the Picture

Anyways, back to IAM and CodePipeline…it’s probably easier to start with a diagram…

Uhhh, okay, maybe a simpler diagram…let’s just focus on build artifact flow from the CI tool’s S3 bucket (we are using GitLab for CI in this case).

Now, let’s consider IAM’s scope.  The green circles specify where IAM comes into play…

Our example here illustrates a simple need made a bit more complicated with IAM: Deploy a build artifact into both AccountA and AccountB. To do this, we need to have a good grasp on how IAM works, specifically, how IAM roles work.

In IAM parlance, a ‘role’ is rather like a very specific (well-defined) hat an AWS resource can put on (assume), and the hat grants superpowers (permissions/access)! The awkward part comes when we understand that IAM, while region-agnostic (e.g. a role is an account-global object), is limited to a single account (e.g. while AccountA and AccountB can contain identical roles – the role only works in the account it was created in). So while IAM can span ‘multi-region’, it cannot span ‘multi-account’. How will we ever deploy across accounts??

The answer is trusted entities! You can configure the IAM role to trust some other entity (e.g. another account). An IAM role can be configured to allow other roles to assume it. A role wearing a hat putting on another hat, if you will. Stack o’ hats.

Visualizing Roles Working Together

In our case, what we’re looking at is the need for CodePipeline to assume a role in another account and another region, so that it can pull in build artifacts to create a CodeDeploy deployment.  Here is a visualization of hats roles talking to one another.  (you would talk to yourself via hats, too, if you’d spent this much time in IAM!)

The policy pseudocode looks like this…

# IAM policy attached to role
# Let anything with this policy attached do the following…
{
  "Statement": [
    {
      "Effect":"Allow",
      "Action": [
        "Actions we need for our resources."
      ],
      "Resource": [
        "AccountA-Region1 resources we need."
        "AccountA-Region2 resources we need."
      ]
    }
  ]
}

# IAM policy attached to role
# Let anything with this policy attached do the following…
{
  "Statement": [
    {
      "Effect":"Allow",
      "Action": [
        "Actions we need for our resources."
      ],
      "Resource": [
        "AccountA-Region2 build artifact resources we need."
      ]
    },
    {
      "Effect":"Allow",
      "Action": [
        "Actions we need for our resources."
      ],
      "Resource": [
        "AccountB-Region2 resources we need"
      ]
    }
  ]
}
# Trusted Entities policy from role
# Things from AccountA can assume this role…
{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "AccountA-AccountID"
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

So simple!  You’re done! 😀

CodePipeline: Live Example

Great, now we have a less-than-vague idea of how on earth you allow AccountA to work inside AccountB in a way that won’t get you flagged by the security team. To help tie the bow, let’s look at a real (and deceptively complicated) example of needing multi-account, multi-region IAM access.

Let’s look at the flow of artifacts in CodePipeline again…

1. Your CI tool puts build artifact into S3 bucket
2. CodePipeline ingests this into its own S3 bucket(s) called “artifact stores”
3. CodePipeline formats the artifact appropriately (its own internal magical formula)
4. CodePipeline actions farther down the pipeline pull from the artifact store matching their region

Caveat: If you need to flow into multiple regions at any stage of the journey, the far-end deploy action calls back to this artifact store  in its own region.  Let’s state that again: in CodePipeline, if you configure an action with a role other than the one the service is running under, the artifact store CodePipeline uses is the one that matches the region in which you’re trying to act.

One more time: A us-east-2 action (CodeDeploy to ECS) pulls from the us-east-2 artifact store.

That’s a lot of words…here’s a simple visual illustrating that relationship. 

CodePipeline’s hub-and-spoke design model is a bit clearer in the above diagram, too.  A subtle requirement of a CodePipeline configuration is that the account you run CodePipeline from is forever the source for all subsequent accounts.  As in, an artifact goes from CP -> Dev, CP -> Prod…not CP -> Dev -> Prod.  This has security ramifications – one account has access to dev and prod.

What allows CodePipeline to work here is role assumption: RoleA putting on RoleB’s hat.  What allows the action in AccountB to call back to AccountA’s ‘ArtifactStore-us-east-2’ bucket is the policy attached to RoleB.  Yes, you read that right, we have allowed RoleA from AccountA to put on (assume) RoleB from AccountB in order to retrieve files from the bucket in AccountA.  Yo dawg.

Let this post be a reminder of the hidden costs of going multi-region/multi-account!  Budget time for the inevitable gotchas….  I recognize that there is much I have left out in this post – please scroll down for more reading to help fill in the gaps.

Well, hopefully now you understand IAM role assumption in a multi-account/multi-region CodePipeline context a little better, and you will be able to accomplish complicated design work without sacrificing flow!

P.S. You might be wondering why some of the drawings are so…artisanal… This is extremely intentional on my part. Our world today has become obsessed with only ever portraying the highlight reel, the perfect, the Instagram filtered…how are we ever to learn if we can only do ‘perfect’? Perfect is not just the enemy of good, but full-on anti-learning. So I model this through my drawings – which I am slowly improving at!…honest! – how will you model your learning to others today?

Want to Know More?

Are You Ready for a DevOps Transformation?

While software continues to eat the world at an ever-increasing pace with DevOps, the challenges and struggles of companies implementing DevOps is very real. We all can overcome these challenges by working together, improving our tools, processes, knowledge, and training our workforce.

Indellient is a Software Development Company that specializes in Data Analytics, Cloud Services, Managed IT Solutions, DevOps Services, and Document Process Automation.