AWS IAM Service Troubleshooting Playbook

This technical note provides a step-by-step tips to troubleshoot AWS IAM Service.

Detail:

Check whether IAM role/policy for AWS Accounts (1) Primary and (2) Secondary are configured properly

Check Point 1:

Option 1: Check Account Audit Status from Aviatrix Controller

Option 2: Check whether AWS IAM role and policy are updated to the latest version from AWS portal

Probable Causes:

  • Company manages permission through AWS Organization
  • AWS IAM is out of sync
  • Customized instance profile name
  • Outdated IAM policy
  • Customized IAM policy
  • IAM role does not attach to Controller/Gateway properly
  • Trust relationship is not established properly between primary and secondary account

Suggestions:

Check whether InstanceProfileArn has the exact string “instance-profile/aviatrix-role-ec2”

Check Point 1: Run Aviatrix Diagnostic report for Controller from Aviatrix Controller

  • https://docs.aviatrix.com/HowTos/Troubleshooting_Diagnostics_Result.html

  • Expect the string “aviatrix-role-ec2” is exactly be named in the report of InstanceProfileArn as below example:

    InstanceProfileArn "  \"InstanceProfileArn\" : \"arn:aws:iam::XXXXX:instance-profile/aviatrix-role-ec2\",\n"
    
  • If the string in InstanceProfileArn field is not “aviatrix-role-ec2”, Aviatrix Software might not function the AWS IAM role/policy properly.

Check Point 2: Run Aviatrix Diagnostic report for Gateways from Aviatrix Controller

  • https://docs.aviatrix.com/HowTos/Troubleshooting_Diagnostics_Result.html
  • Option 1:
    • Repeat Check Point 1 for all gateways
  • Option 2:
    • Expect to view “Passed” status in the field of “GatewayIamRole” as below example:
      • “GatewayIamRole”: “Passed”
    • If the string is not “Passed”, Aviatrix Software might not function the AWS IAM role/policy properly.

Probable Causes:

  • There is a customized instance profile name which is not “aviatrix-role-ec2”

Suggestions:

  • Users need to clean up the customized instance profile name first.
  • Users need to rename the InstanceProfile to “aviatrix-role-ec2”. The customized name might need to be updated in Terraform or CloudFormation script.

Check whether IAM role is attached to Aviatrix Gateway and IAM policy is associated to IAM role properly

Check Point 1: Check Aviatrix Gateway Audit Status from Aviatrix Controller

Probable Causes:

  • gateway’s aviatrix-role-ec2 is detached from the instance profile
  • aviatrix-role-app does not have associated policy

Suggestions:

  • Toggle IAM role on Aviatrix Gateway
    1. Attach “No Role” to Aviatrix Gateway and click the button “Apply” in AWS portal
    2. Wait for a few seconds
    3. Attach “aviatrix-role-ec2” to Aviatrix Gateway and click the button “Apply” in AWS portal
  • Update the Aviatrix IAM role/policy

Check whether trust relationship is not established properly between primary and secondary account

Check Point 1: Check the primary account in AWS portal

  1. Check the aviatrix-role-app
  2. Expect to grant
  1. the primary (Controller) AWS account itself access to the aviatrix-role-app in this primary account

Check Point 2: Check the secondary account in AWS portal

  1. Check your aviatrix-role-app in all the secondary account
  2. Expect to grant
  1. the primary (Controller) AWS account access to the aviatrix-role-app in this secondary account
  2. the secondary (Gateway) AWS account itself access to the aviatrix-role-app in this secondary account

Check whether company centrally manages & governs permission/policies across accounts through AWS Organization

Check Point 1: Check “Service Control Policies” for “Root” has the right permissions by following the steps below:

  1. Go to “AWS Console > AWS Organizations > Organize Account”
  2. Click on “Root” on the left panel, followed by a click on “Service Control Policies” on the right panel.
  3. Check all attached “Service Control Policies”.

Check Point 2: Check “Service Controller Policies” for “Organization Unit” has the right permissions by following the steps below:

  1. Go to “AWS Console > AWS Organizations > Organize Account > Find”
  2. Click on the “Oranization Unit” (which the account belongs to) on the left panel > Click on “Service control policies” on the right panel.
  3. Check all attached “Service Control Policies”.

Check Point 3: Check “Service Controller Policies” for the account:

  1. Go to “AWS Console > AWS Organizations > Account > Find”
  2. Click on the account from the list. Click on “Service Control Policies” on the right panel.
  3. Check all attached “Service Control Policies”.

Expectation:

  • allowing us-west-1 region in your AWS organization policy
  • at least the same permission as Aviatrix IAM policy to all attached “Service Control Policies”

Suggestions: