Making edits to production databases
Need to make change to a production database that's "live"? Stop right now and read this page first.
What you need to do first
- Make it an issue - open a GitHub issue on the Wego Engineering GitHub pages project. Include as many details as possible in your issue: database name, table(s) name, the change you want to make, reasons for the change, link to the Jira issue.
- Let other developers know - CC the relevant developers using GitHub mentions (e.g. @weynsee).
- Plan for zero downtime - Formulate a plan for how to make your change without impacting production. Some strategies:
- Work on a temporary table that is a copy of the table you want to change, then rename tables. This avoids locking the table.
- Remove all writes to the database (if that's possible), make changes to a replication slave, promote the slave to master.
- Downtime is not always avoidable - In cases where you can't avoid downtime, please discuss it with Chu Yeow and RJ so we can properly plan it within a maintenance window.
Example situations
Some example situations where the process described above applies:
- Adding an index to a table.
- Updating > 1000 records in a table.
- Schema changes (e.g. changing column types, adding a new column).