Next week I will be at Nagios World in St. Paul and then going directly to San Francisco for MySQL Connect and Oracle OpenWorld (I have ACE Director responsibilities for OOW), so this week was frantically finishing up a lot of projects before finalizing my talks. All but one of the MySQL 3rd quarter goals have been met (there are 2 Postgres goals that are still in progress, but our Postgres DBA is not going to conferences next week so there’s plenty of time for those.
And of course, there are the spring conferences to submit to. I have already submitted conference proposals to Confoo and RMOUG Training Days, and am working on proposals for SCALEx11 and Percona Live (calls for papers close Saturday October 12th!).
And somewhere in all that, the database team managed to get a bunch of stuff done this week, even though it was a short week for me (due to the Jewish New Year, and also my birthday!):
- Tweaked our puppet configuration for our backup servers – our backup servers are running several instances of MySQL through sockets only, and some of the puppet configs assumed a running MySQL on port 3306.
- Dealt with a Mozilla Labs experiment running crazy queries against Bugzilla, twice.
- Assessed a problem with one of our development clusters, where several databases could not be dropped due to MySQL crashing. The undo log was corrupt on the master and one slave, but not another slave. We could not determine the cause of the corruption, though we believe a sync on Sept 6th caused the corruption to spread from the master to the one slave. The solution was to restore from the non-corrupt slave (which was also our backup slave, so that was pretty easy). We could have also set innodb_crash_recovery and exported and imported, but restoring from hot backup was faster.
- Opened network flows to get slow query logs copied from the new version check databases for the Addons cluster, and updated SSH keys in puppet.
- Upgraded one of our staging databases from MySQL 5.0 to Percona 5.1, added it to puppet, and converted it from innodb_file_per_table. (phew!)
- Extracted data from Bugzilla about who is watching Mozilla Localizations components.
- Debugged a DHCP issue on one of our Addons staging cluster machines.
- Started compressing binary logs on our backup server as space was starting to get a bit low.
- Debugged an issue where one of the machines in our Support dev/staging database cluster fell off the internet due to an IP conflict.
- Replaced some text in a Bugzilla bug when the profanity filter failed to obscure some profanity.
- Came up with a process to regularly extract data from an appliance that has MySQL embedded in it, and did a one-off of the process to get a database started.
- Gave a developer an SQL backup of the production Buildbot database for analysis.
Next week is sure to be hectic, with two conferences. Happy autumn for those in the Northern Hemisphere, and happy spring for those in the Southern Hemisphere!
Next week I will be at Nagios World in St. Paul and then going directly to San Francisco for MySQL Connect and Oracle OpenWorld (I have ACE Director responsibilities for OOW), so this week was frantically finishing up a lot of projects before finalizing my talks. All but one of the MySQL 3rd quarter goals have been met (there are 2 Postgres goals that are still in progress, but our Postgres DBA is not going to conferences next week so theres plenty of time for those.
And of course, there are the spring conferences to submit to. I have already submitted conference proposals to Confoo and RMOUG Training Days, and am working on proposals for SCALEx11 and Percona Live (calls for papers close Saturday October 12th!).
And somewhere in all that, the database team managed to get a bunch of stuff done this week, even though it was a short week for me (due to the Jewish New Year, and also my birthday!):
- Tweaked our puppet configuration for our backup servers our backup servers are running several instances of MySQL through sockets only, and some of the puppet configs assumed a running MySQL on port 3306.
- Dealt with a Mozilla Labs experiment running crazy queries against Bugzilla, twice.
- Assessed a problem with one of our development clusters, where several databases could not be dropped due to MySQL crashing. The undo log was corrupt on the master and one slave, but not another slave. We could not determine the cause of the corruption, though we believe a sync on Sept 6th caused the corruption to spread from the master to the one slave. The solution was to restore from the non-corrupt slave (which was also our backup slave, so that was pretty easy). We could have also set innodb_crash_recovery and exported and imported, but restoring from hot backup was faster.
- Opened network flows to get slow query logs copied from the new version check databases for the Addons cluster, and updated SSH keys in puppet.
- Upgraded one of our staging databases from MySQL 5.0 to Percona 5.1, added it to puppet, and converted it from innodb_file_per_table. (phew!)
- Extracted data from Bugzilla about who is watching Mozilla Localizations components.
- Debugged a DHCP issue on one of our Addons staging cluster machines.
- Started compressing binary logs on our backup server as space was starting to get a bit low.
- Debugged an issue where one of the machines in our Support dev/staging database cluster fell off the internet due to an IP conflict.
- Replaced some text in a Bugzilla bug when the profanity filter failed to obscure some profanity.
- Came up with a process to regularly extract data from an appliance that has MySQL embedded in it, and did a one-off of the process to get a database started.
- Gave a developer an SQL backup of the production Buildbot database for analysis.
Next week is sure to be hectic, with two conferences. Happy autumn for those in the Northern Hemisphere, and happy spring for those in the Southern Hemisphere!