This is the first full week I have been at work for 5 full days *and* have not traveled, in 9 weeks. I felt extremely productive this week because of it! The database team also got a bit of a boost this week when Dustin Mitchell from Release Operations started lending us a hand and increasing his own database skills.
- Building a redundant db cluster in a second data center for our Bugzilla cluster.
- Resynced the backup for TokuDB.
- Adjusted the swappiness in puppet so that all of our MySQL servers using our standard MySQL puppet module will have a swappiness of 30 (0 is too small for our purposes, only because we have a process that kills processes using memory, when available memory gets low. If we set swappiness to 0, MySQL would be killed when it gets busy…)
- Worked on migrating our Addons slave after it crashed and had db corruption.
- Upgraded the databases running the development/stage versions of affiliates, reps and De Todos. Para Todos.
- Crash-stats Postgres databases had the pg 9.0.9 security update applied.
- Converted MyISAM tables to InnoDB so foreign keys would work in our Mozilla Ignite application.
- Updated all but two of the databases that use Fusion I/O cards to the Fusion IO driver version 3.1, which is faster with MySQL.
- Finished upgrading the Bugzilla cluster to Percona 5.1. After a short while, we will start upgrading to MariaDB 5.5. This involved doing a switchover of the master, which was a success.
- Applied a few sets of data changes to the Graphs databases.
- Investigated a problem where a getpersonas table locks up intermittently. The table locks up because it is MyISAM, but it needs to be MyISAM due to a full-text requirement. No resolution has been determined yet.
- Migrated a database cluster that serves 40 services to our standard puppet module.
- Changed some tables from MyISAM to InnoDB to support foreign keys in the Affiliates Facebook App (and noted the need for an audit of MyISAM tables).
- Changed the MIXED binary log format/replication to test if it fixed some bugs, and see if there is any negative impact.
- Worked on debugging deadlock issues on Bugzilla. pt-deadlock-logger has the distinction of the first tool in the Percona toolkit that I could use without filing a bug (although I almost did, since the “time” parameter is undocumented, e.g. –runtime time, but they did have one example that showed ’4h’ being an example of time, so I used 24h and it indeed ran the tool for 24 hours).
- Queried the list of Mozilla’s viewvc intermittently not showing queries. It turns out to be a networking issue, but we tested to see if it was a database/load balancer issue.
Next week sees the unofficial end of summer in the US, as Labor Day is just about 10 days away. I hope everyone in the Northern Hemisphere has had a great summer!
This is the first full week I have been at work for 5 full days *and* have not traveled, in 9 weeks. I felt extremely productive this week because of it! The database team also got a bit of a boost this week when Dustin Mitchell from Release Operations started lending us a hand and increasing his own database skills.
- Building a redundant db cluster in a second data center for our Bugzilla cluster.
- Resynced the backup for TokuDB.
- Adjusted the swappiness in puppet so that all of our MySQL servers using our standard MySQL puppet module will have a swappiness of 30 (0 is too small for our purposes, only because we have a process that kills processes using memory, when available memory gets low. If we set swappiness to 0, MySQL would be killed when it gets busy…)
- Worked on migrating our Addons slave after it crashed and had db corruption.
- Upgraded the databases running the development/stage versions of affiliates, reps and De Todos. Para Todos.
- Crash-stats Postgres databases had the pg 9.0.9 security update applied.
- Converted MyISAM tables to InnoDB so foreign keys would work in our Mozilla Ignite application.
- Updated all but two of the databases that use Fusion I/O cards to the Fusion IO driver version 3.1, which is faster with MySQL.
- Finished upgrading the Bugzilla cluster to Percona 5.1. After a short while, we will start upgrading to MariaDB 5.5. This involved doing a switchover of the master, which was a success.
- Applied a few sets of data changes to the Graphs databases.
- Investigated a problem where a getpersonas table locks up intermittently. The table locks up because it is MyISAM, but it needs to be MyISAM due to a full-text requirement. No resolution has been determined yet.
- Migrated a database cluster that serves 40 services to our standard puppet module.
- Changed some tables from MyISAM to InnoDB to support foreign keys in the Affiliates Facebook App (and noted the need for an audit of MyISAM tables).
- Changed the MIXED binary log format/replication to test if it fixed some bugs, and see if there is any negative impact.
- Worked on debugging deadlock issues on Bugzilla. pt-deadlock-logger has the distinction of the first tool in the Percona toolkit that I could use without filing a bug (although I almost did, since the “time” parameter is undocumented, e.g. –runtime time, but they did have one example that showed ’4h’ being an example of time, so I used 24h and it indeed ran the tool for 24 hours).
- Queried the list of Mozilla’s viewvc intermittently not showing queries. It turns out to be a networking issue, but we tested to see if it was a database/load balancer issue.
Next week sees the unofficial end of summer in the US, as Labor Day is just about 10 days away. I hope everyone in the Northern Hemisphere has had a great summer!