AWS Secrets Manager
Lifecycle Management
A customer-deployable companion tool for enterprise compliance audit evidence generation
ℹ️ This PRD is adapted from a real cloud infrastructure engagement. Customer-identifying details, direct quotes, and account-specific configuration have been anonymized or generalized. Technical requirements, architecture decisions, and delivered outcomes reflect the actual solution built.
| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0 | Day 1 | Ray Elkins / TPM | Initial draft |
| 1.1 | Day 18 | Ray Elkins / TPM | Customer feedback incorporated: manual secret creation clarified; Identity Center cross-account promoted to core requirement; output format made interface-agnostic; Persona 2 demoted to post-MVP; PDF export downgraded from P0 to P2 |
| 1.2 | Day 28 | Ray Elkins / TPM | MVP delivered. PRD updated to reflect implemented state: --region, --management-profile, --last-accessed opt-in, --versions, --quiet, --allow-partial, --ic-region. IC instance caching bugfix. 213 tests passing. |
Background & Current State
Customer / Segment
A large enterprise customer operating dozens of AWS accounts across multiple regions, subject to recurring external compliance audits. The primary stakeholder is a Senior Infrastructure Architect serving as the senior technical escalation point for complex infrastructure challenges.
Current Environment
- AWS Secrets Manager deployment: Single production account, multi-region with cross-region replication enabled
- Account architecture: Two-account model — production account (secrets reside here) + management account (AWS Identity Center hosted here)
- Secret creation: All secrets created manually by authorized engineers via AWS CLI or Terraform. Native rotation automation not in use
- Identity management: AWS Identity Center (SSO) for human access; IAM roles for EKS service accounts via External Secrets Operator
- Naming convention: Hierarchical —
<service>/<environment>/<username> - Secret content types: RDS credentials, data warehouse credentials, SaaS API keys, Kubernetes workload credentials
- Consumption pattern: Machine/process credentials consumed by EKS pods via External Secrets Operator
How the Product Is Used Today
AWS Secrets Manager serves exclusively as a machine/process credential store. There is no centralized, auditor-presentable view of who has access to any given secret.
Problem Statement
Primary Problem
AWS Secrets Manager provides no native, consolidated interface to answer the audit question: "Who has IAM permissions to access this specific secret, and what is the business justification for their access?"
The current process forces engineers to:
- Script
aws iam simulate-policycalls across all IAM users and roles (10–15 minute execution time) - Parse results to identify entities with Secrets Manager permissions
- Resolve IAM roles back to Identity Center permission sets
- Cross into the management account to query Identity Center and map permission sets to human users and groups
- Document each step with screenshots for auditors
This produces a large volume of screenshots requiring detailed step-by-step explanation to auditors — creating an intractable audit burden compared to peer tools like commercial password vaults, which provide a single "Access" tab showing resolved human identities.
Why Now
The current audit scope expanded to explicitly include Secrets Manager access attestation for the first time. If auditing remains intractable, the customer will evaluate replacing Secrets Manager with an alternative secrets management platform.
Who Feels the Pain Most
- Primary: Infrastructure engineers responsible for audit evidence preparation
- Secondary: Compliance/audit teams who must interpret and validate engineering-generated evidence
- Tertiary: AWS account administrators lacking visibility into over-permissioned or under-utilized secrets
Goals (Outcomes)
- Audit-ready access visibility: An authorized engineer can run a command, specify a secret, and immediately receive a resolved, human-readable report of who has access — at a fidelity that satisfies external auditors without manual scripting.
- Interface-agnostic access reporting: Deliver well-structured, auditor-acceptable output — human-readable table format and structured YAML/JSON for programmatic use.
- Auditor-acceptable evidence artifacts: Generate timestamped output that auditors accept as authoritative evidence without requiring technical walkthroughs.
- Open-source reference implementation: Publish a GitHub-hosted template that customers can fork and adapt to their corporate identity providers or ITSM tooling.
- Rapid delivery: Deploy a functional MVP within the delivery window to support the customer's current audit cycle.
Out of Scope
Always Excluded
- No changes to existing AWS Secrets Manager service API or console UI
- No changes to pricing, packaging, or billing
- No data migration from legacy systems
- No changes to authentication or access control mechanisms
- No deprecation of existing AWS Secrets Manager features
Transcript-Stated
- ACM certificate lifecycle management (not yet in audit scope)
- Full multi-account secrets federation (management account IC read is in-scope; monitoring many independent workload accounts is out-of-scope for MVP)
PM-Proposed
- Native secret rotation configuration or Lambda function management
- Real-time secret usage telemetry beyond CloudTrail API call history
- Secret content editing or bulk import/export
Personas / Target Users
Primary need: Rapidly generate audit-compliant evidence demonstrating who has access to specific secrets without manual scripting or multi-step IAM policy traversal across two AWS accounts.
Success looks like: Runs a CLI command, specifies a secret name or ARN, and receives a clean structured report showing all IAM principals with access, resolved to human users and groups via Identity Center, with a timestamp. Task completes in under 2 minutes. Report submitted to auditors without explanation.
Primary need: Create secrets with appropriate naming conventions, assign initial access policies, track which workloads consume the secret.
Customer confirmed the existing workflow is sufficient for the compliance use case. Deferred to Phase 2.
Primary need: Receive clear, verifiable evidence that access controls follow least-privilege principles and that business justifications exist for all access grants.
Success looks like: Receives a single-page report per secret showing: secret name, list of users/roles with access, Identity Center group memberships, and justification metadata — in a format matching established patterns from peer password vault tools.
User Journey
Context: The Infrastructure Architect receives a request from the audit team to provide an access report and user attestations for a specific production secret by end of week.
- Engineer runs a custom Python script calling
aws iam simulate-principal-policyfor every IAM user and role — runs for 10–15 minutes - Parses JSON to extract role ARNs with allowed access
- Script assumes a cross-account role into the management account to query Identity Center APIs
- Aggregates findings back in the production account context
- Compiles results into a 15+ page document with screenshots of every step
- Adds narrative explanations, including justification for cross-account calls
- Sends to audit team; auditors reply with follow-up questions and schedule a call
- Engineer runs the CLI tool:
secrets-audit --secret rds/prod-example/app_user \
--region us-east-1 \
--management-profile management-account \
--versions --last-accessed --expand-groups --output table- Tool validates all inputs and immediately validates cross-account credentials before the expensive IAM simulation step
- Progress messages appear on stderr:
Simulating principals... (42/594) - Tool auto-detects IC region, caches IC instance, resolves all IAM principals to human identities automatically
- Outputs a clean formatted table to terminal
- Engineer submits the report to the audit team
- Auditors review the report and accept it without follow-up questions
Requirements (Prioritized)
Queries the AWS production account to determine which IAM users, roles, and groups have permissions to perform secretsmanager:GetSecretValue, PutSecretValue, or UpdateSecret. Resolves Identity Center roles to human identities via cross-account access. Outputs in table, YAML, and JSON formats.
Acceptance Criteria
- ✓Given a secret ARN or name, identifies all IAM principals with Allow effect via identity-based policies, resource-based policies, or SCPs
- ✓Detects whether IAM roles were created by Identity Center (checks trust policy for
sso.amazonaws.comor role path) - ✓For Identity Center-managed roles, resolves to human users and groups via REQ-002
- ✓Labels service roles (e.g., EKS pod service accounts) as "Service Account" with trust relationship displayed
- ✓Table output includes: Principal Type, Principal Name, IC User/Group, Access Level, Last Accessed (opt-in)
- ✓YAML/JSON includes all table fields plus: secret ARN, region, report timestamp (ISO 8601), tool version
- ✓If no principals have access, outputs "No IAM principals have access to this secret" in all formats
- ✓v1.2:
--region,--versions,--last-accessed(opt-in),--quiet, progress messages on stderr
Implementation Notes
Uses iam:SimulatePrincipalPolicy with adaptive retry (max_attempts=5, mode=adaptive) and batching sleep for rate limiting (~5 req/sec). CloudTrail LookupEvents queries scoped to the target secret via ResourceName filter — reducing enrichment time from hours to seconds.
Assumes a cross-account IAM role into the management account to call Identity Center APIs. Supports two access methods and two output tiers (complete vs. partial/degraded).
Acceptance Criteria
- ✓Accepts
--management-profile(named AWS CLI profile) OR--management-account-id+--cross-account-role-arn(mutually exclusive) - ✓Cross-account credentials validated early (before expensive IAM simulation step). Default is fail-fast;
--allow-partialopts into graceful degradation - ✓Auto-detects IC region by trying common deployment regions;
--ic-regionallows explicit override - ✓IC instance (ARN + identity store ID + region) discovered once and cached for all role resolutions (bugfix from v1.1)
- ✓Calls
sso:ListAccountAssignmentsto retrieve user and group assignments;identitystore:DescribeUserfor display name and email - ✓Output distinguishes direct user assignments from group-based assignments (e.g., "Jane Doe via group DatabaseAdmins")
- ✓If IC instance not found, outputs: "No Identity Center instance found in management account. Verify account ID configuration."
Tier 1 — Delivered
- ✓Every report includes: secret name/ARN, region, ISO 8601 timestamp, operator identity (IAM user ARN or IC email), tool name and version
- ✓Output saved to file via
--output-fileflag or standard shell redirection - ✓If operator identity cannot be determined, outputs assumed role ARN
Tier 2 — Post-MVP (PDF export)
- ○Optional
--output pdfflag; formatted tables suitable for printing; footer with version and timestamp
Customer confirmed existing workflow is sufficient. Deferred to Phase 2.
Searchable, sortable list of all secrets with lifecycle metadata (creation date, last accessed, replication status, rotation status) to support operational hygiene.
Post-MVP per customer feedback. Basic version metadata partially addressed by --versions flag in v1.2.
Comprehensive metadata and version history for a selected secret. Secret values are never displayed.
Post-MVP per customer feedback. Existing CLI/Terraform workflow is sufficient.
Web form for creating secrets that enforces the hierarchical naming convention and validates inputs before calling Secrets Manager API.
Queries CloudTrail logs for GetSecretValue API calls and displays requesting resources. For EKS workloads using External Secrets Operator, attempts to resolve IAM role to Kubernetes namespace and service account.
Detects and displays key names for JSON key-value secrets without revealing values. Uses secretsmanager:DescribeSecret by default; optional --show-keys flag for top-level key enumeration.
No workflow detail in source material. Requires customer-specific ITSM configuration.
Optional approval ticket field on secret creation form; ticket number stored as secret tag. Full ITSM integration deferred pending workflow definition.
MVP Scope — Delivery Status
Delivered in MVP (v1.0.0)
| Requirement | Priority | Status | Notes |
|---|---|---|---|
| REQ-001: Secret Access Resolution Report | P0 | Delivered | All three output formats. IAM Policy Simulator with adaptive retry. Resource-based policy parsing. EKS service account classification. |
| REQ-002: IC Cross-Account Resolution | P0 | Delivered | Two access methods. IC region auto-detection. IC instance caching. --expand-groups. --allow-partial. |
| REQ-003: Structured Output (Tier 1) | P2 Tier 1 | Delivered | Table, YAML, JSON with full metadata header. --output-file flag. |
Post-MVP Enhancements (v1.2)
| Enhancement | Description |
|---|---|
--region | Target a specific AWS region for the audit session without requiring a full ARN |
--management-profile | Named AWS CLI profile shortcut for management account access |
--last-accessed | CloudTrail enrichment moved from default-on to opt-in; significantly improves performance on high-volume accounts |
--versions | Include secret version metadata (version IDs, staging labels, creation dates) via ListSecretVersionIds |
--quiet | Suppress progress messages on stderr for scripted/automated use |
--allow-partial | Opt-in graceful degradation on cross-account failure (default is now fail-fast) |
--ic-region | Override IC region auto-detection |
| IC instance caching | IC instance discovered once and reused for all role resolutions (bugfix: was re-discovered per role) |
| Progress messages | Real-time status on stderr during long-running steps |
| Early cross-account validation | Credentials validated before expensive IAM simulation step |
Success Metrics
| Metric | Target | Actual (v1.2) | Method |
|---|---|---|---|
| Time to generate audit evidence per secret | < 2 minutes | ~2 min for 594 principals with IC resolution | Customer field test |
| Report accepted by auditors without follow-up | Yes | Pending next audit cycle | Auditor feedback |
| Test coverage | Comprehensive | 213 tests (unit, property-based, integration) | pytest suite |
| Graceful degradation coverage | All failure modes handled | Verified: cross-account failure, CloudTrail unavailable, deleted users, version access denied, no principals | Test suite + customer validation |
Timeline & Milestones
| Milestone | Description | Target | Status |
|---|---|---|---|
| PRD v1.1 approved | Customer feedback incorporated; PRD reviewed and approved | Week 1 | Done |
| MVP architecture finalized | CLI tool (Python package); cross-account IAM role design; CloudFormation templates for both accounts | Week 2 | Done |
| REQ-001 + REQ-002 complete | Secret access resolution and cross-account IC resolution implemented and tested; all three output formats validated | Week 3-4 | Done (ahead) |
| Beta deployment | Solution delivered to customer; tested against real secrets (594 principals, 18 IC roles) | Week 4 | Done (ahead) |
| MVP launch | REQ-003 Tier 1 complete; GitHub repository published with setup docs and CloudFormation templates | Day 45 | Done (ahead) |
| PRD v1.2 | PRD updated to reflect delivered implementation state | Day 28 | Done |
| Audit deliverable | Customer uses solution to generate access reports for current audit cycle | ~6 weeks post-launch | Pending |
Risks & Dependencies
CloudFormation templates provided in cfn/ for both the management account role and the production account role. Setup documented in cfn/README.md.
Adaptive retry config (max_attempts=5, mode=adaptive) for all Identity Center API calls. IC instance cached — discovered once, not per-role.
Not applicable. Implementation uses iam:SimulatePrincipalPolicy exclusively, not Access Analyzer. Adaptive retry handles the ~5 req/sec rate limit.
CloudTrail enrichment is opt-in via --last-accessed. When enabled but unavailable, the tool degrades gracefully with "Unknown (CloudTrail unavailable)".
Tool logs all operations to CloudTrail in the production account. Customer will submit reports during the next audit cycle. GitHub repository includes version tagging for pinned releases.
Designed as a reference implementation that customers can fork. If AWS ships native access reporting, the solution may be superseded — this is an accepted risk.
MVP strictly limited to Secrets Manager. Extensibility patterns documented in README.
IC region auto-detection tries common deployment regions automatically. --ic-region flag provides explicit override.
Constraints
- Read-only design: Solution will NOT modify IAM policies, secret values, or Identity Center assignments
- Cross-account scope limited: Only the specific Identity Center and Identity Store APIs listed are required
- GitHub open-source: Must be generalizable beyond a single customer
- Never calls GetSecretValue: Only needs metadata and policy information
Open Questions
| # | Section | Question | Status |
|---|---|---|---|
| 1 | Success Metrics | What is the target time reduction for audit evidence preparation? | Answered: ~2 min for 594 principals (down from 4–6 hours) |
| 2 | Success Metrics | What is the acceptable latency for report generation? | Answered: ~2 min for 594 principals is acceptable |
| 3 | Success Metrics | What adoption target constitutes success? | Open — pending next audit cycle |
| 4 | REQ-002 | What is the ARN format for the cross-account role? | Answered: CloudFormation templates provided; role name: SecretsAuditReadOnly |
| 5 | REQ-002 | Are there SCPs blocking Identity Center API calls? | Answered: No SCPs blocking IC APIs. IC region mismatch was the actual issue — resolved via auto-detection |
| 6 | REQ-001 | What specific metadata fields do auditors require beyond access list and timestamp? | Open — pending auditor feedback from next audit cycle |
| 7 | REQ-001 | Should the solution evaluate custom IAM policy conditions? | Answered: IAM Policy Simulator evaluates conditions natively |
| 8 | REQ-001 | How should the solution handle secrets with both identity-based and resource-based policies? | Answered: Union of permissions; policy_source field in structured output indicates source |
| 9 | Architecture | What is the preferred deployment model? | Answered: CLI tool (Python pip-installable). No web interface for MVP |
| 10 | Dependencies | What features are on the AWS Secrets Manager service team's roadmap? | Open — no native access reporting feature announced |
| 11 | Security | What IAM permission boundaries apply to the solution's execution role? | Answered: No restrictive boundaries. Standard IAM read permissions sufficient |
| 12 | REQ-009 | What is the complete ServiceNow/Jira approval workflow? | Open — deferred to Phase 3+ |
| 13 | REQ-007 | For EKS pod callers, should the solution query the Kubernetes API? | Open — deferred to Phase 2 |
Quality Bar Checklist
- ✓Problem statement is clear, specific, and evidence-based
- ✓User Journey reflects corrected current state (cross-account traversal already performed by existing script)
- ✓User Journey Future State updated to reflect implemented CLI with all flags
- ✓Every MVP requirement (REQ-001, REQ-002, REQ-003) has measurable/testable acceptance criteria including edge cases
- ✓All MVP acceptance criteria checked off as delivered with v1.2 annotations
- ✓Cross-account Identity Center requirement promoted from edge case to core P0 requirement
- ✓Output format requirements are interface-agnostic (table, YAML, JSON); web/PDF not prescribed for MVP
- ✓Persona 2 and requirements REQ-004 through REQ-006 clearly marked Post-MVP with rationale
- ✓Success Metrics updated with actual measurements from customer field testing
- ✓Timeline updated with actual delivery dates and Done status
- ✓Risks updated with mitigation status (resolved, mitigated, pending, accepted)
- ✓Open Questions updated with resolution status and answers where available
- ✓213 tests passing (unit, property-based, integration)
END OF PRD v1.2 (Anonymized Portfolio Version) — Authored by Ray Elkins