🔒 Guarding Production with Actions

At the core of any successful software development process is the delicate balance between speed and security. Teams are encouraged to release frequently, innovate rapidly, and deliver value to customers. However, safeguarding production environments against unintentional mistakes that could impact service reliability and customer data security is just as crucial. This article explores how structuring GitHub Actions rules and permissions helps to strike this balance and maintain operational integrity.

The Challenge: Moving Fast Without Breaking Things 🚀💥

In my experience, a few key incidents have underscored the importance of carefully managing access to production-like environments and taught me valuable lessons in doing so. One particularly nerve-wracking moment involved the accidental destruction of multiple services in a Kubernetes cluster that was part of our staging environment. Thankfully, this didn’t happen in production—but it could have with just one mis-click. 😬

Another time, a critical deployment faltered due to a misconfigured environment variable. A developer, aiming to quickly fix the issue, inadvertently introduced a bug that caused a partial outage 🚨. These incidents underscored the necessity for stricter controls and better safeguards around our deployment processes. By implementing more rigorous review processes and automated checks, we can significantly reduce the risk of such incidents, ensuring a more stable and reliable production environment.

It became clear that broad admin-level permissions posed too great a risk. Even with the best intentions, mistakes can happen. We needed a system that made it nearly impossible for any one person to accidentally cause major disruptions while still allowing us to push value to customers swiftly.

Guarding Environments 🛡️

Environment Secrets Management

Strict guidelines for managing secrets in production-like environments should be enforced. For instance, repository-level secrets such as DEPLOYMENT_PRIVATE_KEY are not permitted. Instead, secrets are stored using GitHub "Environments," ensuring that each environment (e.g., production, staging) has its own distinct set of secrets.

Clear, specific names like development, staging, or production must be assigned to these environments. This allows for immediate recognition of the required security level and helps prevent confusion regarding the environment to which deployments are made.

Deployment Protection Rules

Deployment protection rules in GitHub are crucial for maintaining the integrity of production-like environments, such as staging environments, which are designed to mirror the stability and security of production. These rules prevent unauthorized or potentially harmful changes from being deployed, thus reducing the risk of disruptions and ensuring a smoother deployment process. Here’s how they help mitigate risks:

1. Configure Reviewers 🧑‍💻👩‍💻
   Deployment protection rules allow you to specify required reviewers for deployments. Before code changes are deployed, they must be reviewed and approved by designated team members. This added layer of scrutiny helps catch potential issues, bugs, or security vulnerabilities before they impact the environment. For production-like environments, a reviewer from the CloudOps team should sign off.

2. Set Deployment Timers ⏰
   Timers can delay the deployment of code changes, giving teams a buffer period to catch any issues that might not be immediately apparent. This allows time for additional testing and verification, helping to prevent last-minute problems from causing disruptions.

3. Custom Rules 🛠️
   Custom rules tailored to specific needs or requirements for your environments can include automated checks, security scans, or compliance verifications. These proactive measures ensure that deployments meet specific criteria relevant to your environment's security and stability needs.

Overall Benefits:

• Reduced Risk of Errors: By enforcing review and testing processes, deployment protection rules help catch mistakes early and prevent them from reaching critical environments.
• Improved Security: Security-focused rules, such as mandatory code reviews and compliance checks, reduce the risk of introducing vulnerabilities into your production-like environments.
• Enhanced Stability: Deployment timers and custom rules help ensure that changes are well-vetted and thoroughly tested, contributing to the overall stability of the environment.

Branch Protection Rules: Safeguarding the Release Process

Restructuring permissions alone isn’t enough. Ensuring the integrity of our deployment pipelines is also critical. Branch protection rules play a key role in safeguarding the release process, preventing unauthorized changes from being merged into protected branches. Here are some essential rules to enforce:

Required Branch Protection

2. Approval Requirements:
   All changes must go through pull requests (PRs) with enforced reviews from designated team members. This ensures that no code lands in a critical branch without proper scrutiny. All PRs must be approved by number of reviewers before merging.

3. Code Owner Reviews:
   The "Require review from Code Owners" rule in the CODEOWNERS file ensures that changes affecting specific files or directories are reviewed by their designated owners, adding another layer of expertise.

4. Preventing Bypasses:
   The ability for administrators to bypass protection rules should be disabled, preventing anyone—even those with elevated permissions—from merging code without meeting the required checks.

5. Disabling Force Pushes:
   Force pushes to protected branches are strictly disabled, preserving the integrity of the deployment pipeline.

6. Status Checks:
   Status checks ensure that all required checks, such as automated tests and code quality scans, pass before a PR can be merged, maintaining the quality and reliability of the codebase.

Share Your Stories and Feedback

Have you faced similar challenges or discovered innovative solutions for managing permissions and deployments? Share your stories and feedback in the comments 🚀

References:

• GitHub Branch Protection Rules
• Best Practices for GitHub Secrets Management