AWS Well-architected Framework

Evaluate external and internal customer needs involving key stakeholders, including business, development, and operations teams, to determine where to focus efforts. This will ensure that you have a thorough understanding of the operations support required to achieve your desired business outcomes. Use your established priorities to focus your improvement efforts where they will have the greatest impact. This might mean, for example, developing team skills, improving workload performance, reducing costs, automating runbooks, or enhancing monitoring.

Evaluate governance requirements. Governance is the set of policies, rules, or frameworks that a company uses to achieve its business goals. Conformance is the ability to demonstrate that you have implemented governance requirements.

Evaluate compliance requirements. Apply due diligence if no external compliance frameworks are identified. Generate audits or reports that validate compliance. If you advertise that your product meets specific compliance standards, you must have an internal process for ensuring continuous compliance.

Evaluate threat landscape such as competition, business risk and liabilities, operational risks, and information security threats. Maintain current information in a risk registry.

Evaluate the impact of trade-offs between competing interests or alternative approaches.

Manage benefits and risks. For example it may be beneficial to deploy a workload with unresolved issues to make significant new features available to customers. It may be possible to mitigate associated risks, or it may become unacceptable for a risk to remain, in which case you will take action to address the risk.

Executive Sponsorship.

Empower team members to take action.

Escalation.

Timely, clean and actionable communications.

Encourage experimentation.

Promote learning to support new technologies. Cross train to promote knowledge transfer and reduce the risk of significant impact when you lose skilled and experienced team members with institutional knowledge.

Encourage diverse opinions

Implement application telemetry as the foundation for observability. Telemetry should provide internal metrics as well as reflect the state of the solution as compared to the business goals (like number of logins, transactions and so on).

Implement dependency telemetry configuring the workload to emit information about resources it depends on (like databases, DNS and network connectivity).

Implement transaction traceability to be able to trace the flow of data within the workload.

Use version control.

Test and validate changes before production. testing should also include infrastructure, configuration, security and operations procedures besides unit and integration.

Use configuration management systems.

Use build and deployment management systems.

Perform automated patch management.

Document and share design standards and best practices across teams

Implement practices to improve code quality and minimize defects, suche as TDD, code reviews, standards adoption.

Use multiple environments, such as dev, staging, production.

Make frequent, small, and reversible changes can reduce the scope and impact of a change.

Fully automate integration and deployment.

Plan for unsuccesful changes: revert to a known good state, or remediate in the production environment if a change does not have the desired outcome. This preparation reduces recovery time through faster responses.

Test and validate changes.

Use deployment management systems.

Test using limited deployments alongside existing systems to confirm desired outcomes prior to full scale deployment. E.g.: canary or one-box deployments. See Amazon Builder’s Library - Automating safe, hands-off deployments.

Deploy using parallel environments: mplement changes onto parallel environments, and then transition over to the new environment. Maintain the prior environment until there is confirmation of successful deployment.

Deploy frequent, small, reversible changes to reduce the scope of a change.

Fully automate integration and deployment.

Automate testing and rollback.

Ensure personnel capability on the platform and services that support your workload and that they’re an adeguate number.

Consistently review operational readiness: Use Operational Readiness Reviews (ORRs), to validate that you can operate your workload.

Use runbooks to perform procedures.

Use playbooks to investigate issues.

Make informed decisions to deploy systems and changes and perform pre-mortem excercises.

Facilitate support plans for production workloads: make sure to support any software and services that your production workload relies on. Select an appropriate support level to meet your production service-level needs. Support plans for these dependencies are necessary in case of service disruptions or software issues. Document support plans and how to request support for all service and software vendors. Implement mechanisms that verify that support points of contacts are kept up to date.

Identify Key Performance Indicators (KPIs) based on desired business outcomes and customer outcomes, to determine workload success. E.g.: abandoned shopping cart, orders placed, cost, price and allocated workload expense. Adjust workload metrics over time as business needs change.

Define, collect and analyze workload metrics performing regular, proactive reviews of these metrics to identify trends and determine if a response is necessary and validate the achievement of business outcomes.

Establish workload metrics baselines so that you can identify under-performing and over-performing applications and components. This adds to your ability to mitigate issues before they become incidents.

Learn expected patterns of activity for workload so you can respond appropriately if required.

Raise an alert when workload outcomes are at risk or anomalies are present.

Validate the achievement of outcomes and the effectiveness of KPIs and metrics.

Identify KPIs

Define operations metrics such as Mean Time To Detect (MTTD), Mean Time To Recovery (MTTR).

Collect and analyze operations metrics.

Establish baselines.

Learn expected patterns of activity for operations.

Alert when operations outcomes are at risk.

Alert when operations anomalies are detected.

Validate the achievement of outcomes and the effectiveness of KPIs and metrics.

You need processes for each to mitigate the impact of these events on your business and make sure that you respond appropriately.

Prioritize operational events based on business impact.

Define escalation paths in your runbooks and playbooks, including what sets off escalation, and procedures for escalation.

Define and test a communication plan for system outages for both clients and stakeholders.

Communicate status through dashboards.

Automate responses to events to reduce errors caused by manual processes, and to ensure prompt and consistent responses.

Have a process for continuous improvement: Evaluate your workload against internal and external architecture best practices. Conduct workload reviews at least once per year. Prioritize improvement opportunities into your software development cadence.

Perform post-incident analysis.

Implement feedback loops.

Perform Knowledge management: this helps team members find the information to perform their job.

Identify drivers for improvement.

Validate insights. Review your analysis results and responses with cross-functional teams and business owner.

Regularly perform retrospective analysis of operations metrics with cross-team participants from different areas of the business.

Document and share lessons learned from the operations activities.

Allocate time and resources to make continuous and incremental improvements possible.