> For the complete documentation index, see [llms.txt](https://engineering.umh.app/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://engineering.umh.app/engineering/development-process/how-to-ship.md). # 3. How to ship ## Stages and the feature flag ### Why a feature flag **Merging code changes nothing in production.** Anything that changes behaviour goes behind a feature flag. Then we can release quickly without the risk of breaking something. Until the flag is removed from the code, it can be switched off at a customer if the change breaks something in their environment. A large bug fix goes behind a flag too. A bug fix where adding the flag would mean a lot of changes does not need one. Because "a lot of changes" should not become a shortcut a developer can use, agree with the [Code Owner](/engineering/product-engineer/code-owner.md) whether your fix needs a flag. ### When a release breaks something **You can always do a release from staging.** Because new behaviour sits behind a feature flag, a release changes nothing for a customer until the flag goes on. So do not be afraid of breaking something. If you make a release and something breaks, it is not your fault. Do not hold back a release over a discussion that a feature flag can settle. If it breaks at a customer, tell their [Value Engineer](/culture/working-with-the-rest-of-umh.md#value-engineer). The Value Engineer switches the flag off for that customer. For umh-core, that means setting the flag's environment variable to off and restarting umh-core. Then you fix the bug. In the worst case, the Value Engineer rolls the customer back to the previous umh-core version. That has not been necessary so far. Our [Service Level Agreement](https://terms.umh.app/) says how quickly we respond when something breaks at a customer. Switching the flag off does not bring lost data back. So a [one-way door](/engineering/development-process/how-to-build.md#one-way-doors) gets the extra care described in How to build before the flag goes on. ### The four stages **The rollout runs in four stages.** The [rollout ticket](/engineering/development-process/how-to-build/tickets/rollout.md#stages-and-evidence) has a box for each. 1. In Stage 1, the code goes to production with the flag off. 2. Stage 2 comes before the flag goes on for any customer. [Discoverability](/engineering/development-process/how-to-build/tickets/rollout.md#template) is ready. Telemetry and error reporting are live. The Value Engineer and the [Product Engineer](/engineering/product-engineer.md) have tested it with the flag on. The blockers are resolved, there is a plan to roll it back, and every target has its measure. Between Stage 2 and Stage 3, the Value Engineers turn the flag on for single customers. They align this with the Product Engineer. 3. In Stage 3, the Product Engineer turns the flag on by default. That needs the sign-off of the [Product Owners](/engineering/product-engineer/product-owner.md) and the Code Owners, and the CTO's approval. The feature is shown in the next [all hands](/culture/our-week.md#all-hands). After that, the Product Engineer keeps checking the targets in the [rollout plan](/engineering/development-process/how-to-build.md#the-rollout-plan) and works on what is left to meet them. 4. Stage 4 is done when the flag is removed from the code, the targets for during the rollout are met or their miss is accepted in a comment, and no customer still switches back to the old behaviour. If a rollout ticket misses a target, see [When a target is never met](#when-a-target-is-never-met). Do not quietly change the targets to fit what was built. For more on shipping many small changes safely, see [Shipping to Production](https://newsletter.pragmaticengineer.com/p/shipping-to-production) by The Pragmatic Engineer. ## Releases **The Management Console, umh-core and benthos-umh reach customers differently.** The Management Console and umh-core are our two products. benthos-umh is a tool inside umh-core. We want to release all three from one branch, `staging`, by tagging a release there. We also want all three to use [changie](https://changie.dev) for the [changelog](/engineering/development-process/how-to-ship/changelog.md). With changie, each pull request adds its own changelog file. The Management Console and benthos-umh use changie. umh-core does not yet. ### Management Console **The Management Console is SaaS.** Customers get each release as soon as we make it. Ideally we release every day. Each pull request that merges into the staging branch deploys the staging environment. A release is a dated tag on staging. The tag runs the end-to-end tests again, deploys to production and publishes the changelog on [umh.app/insights/changelog](https://www.umh.app/insights/changelog) and in Linear. ### umh-core **Customers run umh-core themselves and upgrade it by hand.** A release is a new version on GitHub. Each release tells the Management Console which version to offer as an update. ### benthos-umh **benthos-umh ships inside umh-core.** Each release opens two pull requests: one in umh-core that bumps the version, and one in the Management Console that adds the new version's schemas. A benthos-umh change reaches a customer once they upgrade to the next umh-core release. ### Ticket states in Linear **Linear moves a ticket through its states by itself, based on the linked pull requests** ([Linear's GitHub integration](https://linear.app/docs/github)). Merged means the pull request is merged to staging but not released yet. For sub-tickets and bug tickets, Done means the ticket was part of a [Linear release](https://linear.app/docs/releases). A release only includes a ticket if its ID is in a commit message. Reference the sub-ticket or the bug ticket in the commit message. Linear's [magic words](https://linear.app/docs/github#magic-words) explain how a reference moves or closes a ticket. The Product Engineer closes the rollout ticket by hand, as described in [When it is done](#when-it-is-done). Move benthos-umh tickets to Done by hand once the umh-core release with them is out. Linear closes a ticket as Canceled after six months without an update. It archives the ticket six months later. If one is still relevant, reopen it. ## When it is done **A rollout ticket is done when there is impact.** That can come before every planned sub-feature is built. Once the impact, adoption and stability targets are met, the Product Engineer can move the features that are not built yet into a new rollout ticket idea. That idea can be pitched again later. The Product Engineer's decision stands as long as nobody argues against it in the product meeting. **Before the ticket closes, the Product Owners read the evidence in the comments on the targets.** [The rollout ticket](/engineering/development-process/how-to-build/tickets/rollout.md#stages-and-evidence) says what that evidence is. **Known defects and** [**tech debt**](/engineering/development-process/how-to-build/tickets/tech-debt.md) **stay in the ticket, even after it is done.** A follow-up rollout ticket only extends the scope of the first one. The Product Engineer removes the feature flag from the code before the ticket closes. ### When a target is never met **If a target is never met, the Product Engineer decides in the product meeting whether to revise it, roll back, or accept the miss.** Once an accepted miss is recorded as a comment in the ticket, the ticket can close. ### After the ticket is done **After the ticket is done, the person who built the feature keeps** [**owning it**](/engineering/development-process/how-to-build/tickets/monitoring.md#rules)**.** The [dashboard](/engineering/development-process/how-to-build/tickets/rollout.md#dashboard) keeps tracking the targets for after the rollout, for as long as the feature ships. **The feature's** [**monitoring ticket**](/engineering/development-process/how-to-build/tickets/monitoring.md) **becomes its home.** A Sentry issue, a defect found later or a problem the dashboard shows becomes a sub-ticket of it. When the rollout ticket is done, its feature tags move to the [Features line](/engineering/development-process/how-to-build/tickets/rollout.md#header-and-sign-offs) of the monitoring ticket. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://engineering.umh.app/engineering/development-process/how-to-ship.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.