This is the first full week I have been at work for 5 full days *and* have not traveled, in 9 weeks. I felt extremely productive this week because of it! The database team also got a bit of a boost this week when Dustin Mitchell from Release Operations started lending us a hand and increasing his own database skills.
- Building a redundant db cluster in a second data center for our Bugzilla cluster.
- Resynced the backup for TokuDB.
- Adjusted the swappiness in puppet so that all of our MySQL servers using our standard MySQL puppet module will have a swappiness of 30 (0 is too small for our purposes, only because we have a process that kills processes using memory, when available memory gets low. If we set swappiness to 0, MySQL would be killed when it gets busy…)
- Worked on migrating our Addons slave after it crashed and had db corruption.
- Upgraded the databases running the development/stage versions of affiliates, reps and De Todos. Para Todos.
- Crash-stats Postgres databases had the pg 9.0.9 security update applied.
- Converted MyISAM tables to InnoDB so foreign keys would work in our Mozilla Ignite application.
- Updated all but two of the databases that use Fusion I/O cards to the Fusion IO driver version 3.1, which is faster with MySQL.
- Finished upgrading the Bugzilla cluster to Percona 5.1. After a short while, we will start upgrading to MariaDB 5.5. This involved doing a switchover of the master, which was a success.
- Applied a few sets of data changes to the Graphs databases.
- Investigated a problem where a getpersonas table locks up intermittently. The table locks up because it is MyISAM, but it needs to be MyISAM due to a full-text requirement. No resolution has been determined yet.
- Migrated a database cluster that serves 40 services to our standard puppet module.
- Changed some tables from MyISAM to InnoDB to support foreign keys in the Affiliates Facebook App (and noted the need for an audit of MyISAM tables).
- Changed the MIXED binary log format/replication to test if it fixed some bugs, and see if there is any negative impact.
- Worked on debugging deadlock issues on Bugzilla. pt-deadlock-logger has the distinction of the first tool in the Percona toolkit that I could use without filing a bug (although I almost did, since the “time” parameter is undocumented, e.g. –runtime time, but they did have one example that showed ’4h’ being an example of time, so I used 24h and it indeed ran the tool for 24 hours).
- Queried the list of Mozilla’s viewvc intermittently not showing queries. It turns out to be a networking issue, but we tested to see if it was a database/load balancer issue.
Next week sees the unofficial end of summer in the US, as Labor Day is just about 10 days away. I hope everyone in the Northern Hemisphere has had a great summer!
This is the first full week I have been at work for 5 full days *and* have not traveled, in 9 weeks. I felt extremely productive this week because of it! The database team also got a bit of a boost this week when Dustin Mitchell from Release Operations started lending us a hand and increasing his own database skills.
- Building a redundant db cluster in a second data center for our Bugzilla cluster.
- Resynced the backup for TokuDB.
- Adjusted the swappiness in puppet so that all of our MySQL servers using our standard MySQL puppet module will have a swappiness of 30 (0 is too small for our purposes, only because we have a process that kills processes using memory, when available memory gets low. If we set swappiness to 0, MySQL would be killed when it gets busy…)
- Worked on migrating our Addons slave after it crashed and had db corruption.
- Upgraded the databases running the development/stage versions of affiliates, reps and De Todos. Para Todos.
- Crash-stats Postgres databases had the pg 9.0.9 security update applied.
- Converted MyISAM tables to InnoDB so foreign keys would work in our Mozilla Ignite application.
- Updated all but two of the databases that use Fusion I/O cards to the Fusion IO driver version 3.1, which is faster with MySQL.
- Finished upgrading the Bugzilla cluster to Percona 5.1. After a short while, we will start upgrading to MariaDB 5.5. This involved doing a switchover of the master, which was a success.
- Applied a few sets of data changes to the Graphs databases.
- Investigated a problem where a getpersonas table locks up intermittently. The table locks up because it is MyISAM, but it needs to be MyISAM due to a full-text requirement. No resolution has been determined yet.
- Migrated a database cluster that serves 40 services to our standard puppet module.
- Changed some tables from MyISAM to InnoDB to support foreign keys in the Affiliates Facebook App (and noted the need for an audit of MyISAM tables).
- Changed the MIXED binary log format/replication to test if it fixed some bugs, and see if there is any negative impact.
- Worked on debugging deadlock issues on Bugzilla. pt-deadlock-logger has the distinction of the first tool in the Percona toolkit that I could use without filing a bug (although I almost did, since the “time” parameter is undocumented, e.g. –runtime time, but they did have one example that showed ’4h’ being an example of time, so I used 24h and it indeed ran the tool for 24 hours).
- Queried the list of Mozilla’s viewvc intermittently not showing queries. It turns out to be a networking issue, but we tested to see if it was a database/load balancer issue.
Next week sees the unofficial end of summer in the US, as Labor Day is just about 10 days away. I hope everyone in the Northern Hemisphere has had a great summer!
As a consultant working with MySQL, I learned a lot about MySQL. I got deep into MySQL. But I did not often get a broad sense of the entire application ecosystem. Now that I work in-house, I can focus on the breadth. And especially working at Mozilla, I am in contact with many many developers working on many different applications. One Mozilla developer whom I respect greatly is James Socol, and his blog series on web application security is an excellent example of why he has earned my respect.
For those who want an overview, the articles (which are not all yet written) range many topics:
Basics: locking your car doors.
Password Storage
XSS: Cross-Site Scripting
CSRF: Cross-Site Request Forgeries
Injections, SQL and Otherwise
Access Control
Session Fixation and Hijacking
Server Configuration
Click-jacking and a little Phishing
Stay Up to Date
Advanced: Some gotchas from my experience and some things you may well see.
Mass Assignment
Cache Poisoning
Bots: Spam, Brute-force, and User Experience
PCI-DSS
CEF Logging
What browsers are doing to help.
Content Security Policy
X-Frame-Options
Do Not Track
Sandboxing
I think everyone involved in a web application should understand this series!
As a consultant working with MySQL, I learned a lot about MySQL. I got deep into MySQL. But I did not often get a broad sense of the entire application ecosystem. Now that I work in-house, I can focus on the breadth. And especially working at Mozilla, I am in contact with many many developers working on many different applications. One Mozilla developer whom I respect greatly is James Socol, and his blog series on web application security is an excellent example of why he has earned my respect.
For those who want an overview, the articles (which are not all yet written) range many topics:
Basics: locking your car doors.
Password Storage
XSS: Cross-Site Scripting
CSRF: Cross-Site Request Forgeries
Injections, SQL and Otherwise
Access Control
Session Fixation and Hijacking
Server Configuration
Click-jacking and a little Phishing
Stay Up to Date
Advanced: Some gotchas from my experience and some things you may well see.
Mass Assignment
Cache Poisoning
Bots: Spam, Brute-force, and User Experience
PCI-DSS
CEF Logging
What browsers are doing to help.
Content Security Policy
X-Frame-Options
Do Not Track
Sandboxing
I think everyone involved in a web application should understand this series!
This week, Mozilla IT is on-site at one of our offices for an IT meetup. Back in March, I chose to take part in helping physically move some of our infrastructure. When we were not in the data center, it was great to meet team members in person, but there was very little time for heads-down work. I was skeptical that a week together would be worth it, because it is a lot of expense to send the IT department to one location (we’re scattered all over the globe) and not get too much work done. However…….
As it turns out, the week was fantastic. The culture was conference-like – we had a big room and many breakout rooms, so we were able to address everyone, as well as groups.
Monday was a travel day. Tuesday we had lightning talks, where all of our teams had 10-15 minutes to explain what they do, what they have done, and what their upcoming projects/goals are. There was also a group activity (which I missed due to flight timing so I can’t comment on). The teams are:
Storage/Virtualization
Database
Data Center Operations
Network Operations
Operation Security
Services
Release Operations
Web Operations
Addons/Marketplace
Developer Services
Infrastructure
Special Operations
Desktop
Wednesday through Friday (today) involved breakout sessions. For example, we had a work session where we audited the monitoring against Crash Stats. We had learning sessions such as “Secure Cookie Solutions for Admin Sites” and informational sessions such as “State of the Mozilla Network: Past, Present, and Future” and “Addons/Marketplace Architecture”. And there were strategic sessions, such as “Defining our First Principles (systems team)” and “Brainstorming a Mozilla IT Advisory Board”.
And of course, there’s the hallway track – at this meetup, the “hallway track” consists of finding someone in person whom you want to talk to about an important topic. For example, I was able to sit down with Gerv and hammer out a plan to start optimizing queries for Bugzilla.
And of course there are some evening activities – some touristy events, and there has been copious playing of Cards Against Humanity.
I was skeptical at the beginning of the week, and the actual meetup has blown away my expectations. This week has been totally worth it, and I hope that we can continue to have meetups like this, preferably more than once yearly.
This week, Mozilla IT is on-site at one of our offices for an IT meetup. Back in March, I chose to take part in helping physically move some of our infrastructure. When we were not in the data center, it was great to meet team members in person, but there was very little time for heads-down work. I was skeptical that a week together would be worth it, because it is a lot of expense to send the IT department to one location (we’re scattered all over the globe) and not get too much work done. However…….
As it turns out, the week was fantastic. The culture was conference-like – we had a big room and many breakout rooms, so we were able to address everyone, as well as groups.
Monday was a travel day. Tuesday we had lightning talks, where all of our teams had 10-15 minutes to explain what they do, what they have done, and what their upcoming projects/goals are. There was also a group activity (which I missed due to flight timing so I can’t comment on). The teams are:
Storage/Virtualization
Database
Data Center Operations
Network Operations
Operation Security
Services
Release Operations
Web Operations
Addons/Marketplace
Developer Services
Infrastructure
Special Operations
Desktop
Wednesday through Friday (today) involved breakout sessions. For example, we had a work session where we audited the monitoring against Crash Stats. We had learning sessions such as “Secure Cookie Solutions for Admin Sites” and informational sessions such as “State of the Mozilla Network: Past, Present, and Future” and “Addons/Marketplace Architecture”. And there were strategic sessions, such as “Defining our First Principles (systems team)” and “Brainstorming a Mozilla IT Advisory Board”.
And of course, there’s the hallway track – at this meetup, the “hallway track” consists of finding someone in person whom you want to talk to about an important topic. For example, I was able to sit down with Gerv and hammer out a plan to start optimizing queries for Bugzilla.
And of course there are some evening activities – some touristy events, and there has been copious playing of Cards Against Humanity.
I was skeptical at the beginning of the week, and the actual meetup has blown away my expectations. This week has been totally worth it, and I hope that we can continue to have meetups like this, preferably more than once yearly.
Whoops! I saved this draft last weekend during Northeast PHP but forgot to click publish….
Hypothesis: Teams we admire need innovation, leverage a trusted process, master teamwork, and enjoy what they do.
Douglas Ross, @getonvelocity
Think of a team you admire, and think about what the characteristics are that make that team successful.
Agile manifesto allows us to share values and principles…
shared values/principles
motivated individuals
sustainable processes
self-organizing teams
(and a few more I didn’t get)
Do more, faster, with less stuff. 
An example – wikispeed
They had innovation
They relied on trusted procs
They mastered teamwork
They have fun working
Secret 1: Innovate
Necessity is the mother of invention innovation.
Lean – Entrepeneurs are everywhere
Be more innovative
Stop wasting people’s time [start saving people time]
Be more successful
Innovate means “to make change, hopefully to improve” not to come up with something ingenious.
How to innovate?
Enable creativity – share the experience, remove communication barriers, “have at it”, JUST SAY YES
Foster participation – Move CEO out of his office for the project room, Enable more ppl to see what’s going on, well-designed space helps team stay focused
Deliver innovation – Innovation is the creation and delivery of new ideas.
About Trusted Process –
Trusted Process is to Team as lighthouse is to ship
Don’t innovate on the trusted process (e.g. don’t nix the daily standup), but innovate how you use that process (e.g. split it up if >10 people).
About Teamwork –
Teamwork is an individual Skill (book recommendation)
Turn dead meetings into productive ones
Treat all team members as leaders. He linked to Virtual Team Productivity – looks interesting.
Teamwork = Leadership. Leading + Following = Teamwork “teach to learn”
Bhutan measures gross national happiness (GNH) – to be more productive!
About having fun working
Talks about Brian Marick’s Ease and Joy – What the Agile Manifesto Left Out
Love, Energy, Audacity, Proof (LEAP) Leadership – Steve Farber
Happiness Metric – Jeff Sutherland (a founder of SCRUM) – www.happinessmetrics.com
and http://scrum.jeffsutherland.com/2010/11/happiness-metric-wave-of-future.html
HBR article just published, happy employees are more productive
http://youtu.be/GXy__kBVq1M – The happiness advantage
positive minded ppl are more productive – Shawn Anchor
Having fun doing work is different from having fun *at* work. e.g. make your demo like an “expo hall” – simultaneous demos, etc. Opportunity to have fun DURING work.
5 routines to add to your workday for max productivity – rediculouslyefficient.com:
batch process email
focus periods
weekly automation audit
(2 more)
“If it isn’t fun, it isn’t working.”
(going to a brainstorming, do an empathy map….must look up what that is)
We expect more from teams:
Harvard Business review:
– Getting more out of your team
– Lean Knowledge Work
Forbes
– Agility: the ingredient defines the next generation
– Wikispeed’s secret sauce: agile
InformationWeek
– time to do more with less
Teams maximize potential by….
– Innovating how they work and what they make
– Leverage a trusted process to guide them
– Master teamwork for team’s benefit
– Enjoy successes and failures along the way
QED.
Celebrate your failures too (by getting excited about what you learned).
High performance of a team cannot happen without you!
That’s it!
Whoops! I saved this draft last weekend during Northeast PHP but forgot to click publish….
Hypothesis: Teams we admire need innovation, leverage a trusted process, master teamwork, and enjoy what they do.
Douglas Ross, @getonvelocity
Think of a team you admire, and think about what the characteristics are that make that team successful.
Agile manifesto allows us to share values and principles…
shared values/principles
motivated individuals
sustainable processes
self-organizing teams
(and a few more I didn’t get)
Do more, faster, with less stuff.
An example – wikispeed
They had innovation
They relied on trusted procs
They mastered teamwork
They have fun working
Secret 1: Innovate
Necessity is the mother of invention innovation.
Lean – Entrepeneurs are everywhere
Be more innovative
Stop wasting people’s time [start saving people time]
Be more successful
Innovate means “to make change, hopefully to improve” not to come up with something ingenious.
How to innovate?
Enable creativity – share the experience, remove communication barriers, “have at it”, JUST SAY YES
Foster participation – Move CEO out of his office for the project room, Enable more ppl to see what’s going on, well-designed space helps team stay focused
Deliver innovation – Innovation is the creation and delivery of new ideas.
About Trusted Process –
Trusted Process is to Team as lighthouse is to ship
Don’t innovate on the trusted process (e.g. don’t nix the daily standup), but innovate how you use that process (e.g. split it up if >10 people).
About Teamwork –
Teamwork is an individual Skill (book recommendation)
Turn dead meetings into productive ones
Treat all team members as leaders. He linked to Virtual Team Productivity – looks interesting.
Teamwork = Leadership. Leading + Following = Teamwork “teach to learn”
Bhutan measures gross national happiness (GNH) – to be more productive!
About having fun working
Talks about Brian Marick’s Ease and Joy – What the Agile Manifesto Left Out
Love, Energy, Audacity, Proof (LEAP) Leadership – Steve Farber
Happiness Metric – Jeff Sutherland (a founder of SCRUM) – www.happinessmetrics.com
and http://scrum.jeffsutherland.com/2010/11/happiness-metric-wave-of-future.html
HBR article just published, happy employees are more productive
http://youtu.be/GXy__kBVq1M – The happiness advantage
positive minded ppl are more productive – Shawn Anchor
Having fun doing work is different from having fun *at* work. e.g. make your demo like an “expo hall” – simultaneous demos, etc. Opportunity to have fun DURING work.
5 routines to add to your workday for max productivity – rediculouslyefficient.com:
batch process email
focus periods
weekly automation audit
(2 more)
“If it isn’t fun, it isn’t working.”
(going to a brainstorming, do an empathy map….must look up what that is)
We expect more from teams:
Harvard Business review:
– Getting more out of your team
– Lean Knowledge Work
Forbes
– Agility: the ingredient defines the next generation
– Wikispeed’s secret sauce: agile
InformationWeek
– time to do more with less
Teams maximize potential by….
– Innovating how they work and what they make
– Leverage a trusted process to guide them
– Master teamwork for team’s benefit
– Enjoy successes and failures along the way
QED.
Celebrate your failures too (by getting excited about what you learned).
High performance of a team cannot happen without you!
That’s it!
In the past two days I was fortunate enough to speak to two different groups of people at the Ministry of Science, Technology and Tertiary Education in the Parliament building in Port of Spain, Trinidad – yes, I spoke at Parliament! The PDF slides for my talk are available: The Art of Cat Herding: How to Manage Geeks and Ideas for DBA’s – not “best practices”, but ideas you may or may not want to implement.
I was in Trinidad as part of the Latin America Oracle Technology Network tour of Latin America (North leg). I also spent time in Cali, Colombia and Quito, Ecuador (including visiting the Equator!). Today I am in Guatemala, and I will give talks on more MySQL-specific subjects: MySQL Security and Get Rid of Cron Scripts Using MySQL Events. Tomorrow I travel to Honduras, and on Sunday is Costa Rica, then I go home, which I haven’t seen since the 24th of June. I will have spent 43 hours on a plane in one month, and I am excited to finally go home next week…but I still have a few more countries on the tour!
Copying Brian’s blog post so it’s fed to Planet MySQL:
I gave my first presentation at MySQL/NoSQL/Cloud Latin American Conference today, in fact, it was my first presentation ever. I’ve spoken in front of small groups for a few minutes before, but never a presentation complete with slides. The talk was so successful that even the Percona guys wanted to know how Mozilla manages MySQL with Puppet!
Thanks to Santiago Lertora of Binlogic for putting together the conference. If you’re interested in the slides they can be downloaded here.
Copying Brian’s blog post so it’s fed to Planet MySQL:
I gave my first presentation at MySQL/NoSQL/Cloud Latin American Conference today, in fact, it was my first presentation ever. I’ve spoken in front of small groups for a few minutes before, but never a presentation complete with slides. The talk was so successful that even the Percona guys wanted to know how Mozilla manages MySQL with Puppet!
Thanks to Santiago Lertora of Binlogic for putting together the conference. If you’re interested in the slides they can be downloaded here.
It is a short week for me this week, because I have tomorrow off as I prepare for the 1st Latin American MySQL, NoSQL and Cloud conference in Buenos Aires next week. However, this week in database, we have done a lot, mostly within the Bugzilla property.
- Backed up several database servers that were not previously backed up
- Dealt with more puppet dashboard disk space issues
- Repaired a few tables after changing ft_min_word_length on some bugzilla slaves
- Upgraded bugzilla slaves to Percona 5.1
- Rebuilt our secondary bugzilla permissions
- Added alerts to our backups to make sure we have current backups
- Upgraded one of the bugzilla slaves’ Fusion IO card to driver 3.1, as per Percona’s blog post about it. We went from timing cached reads from 9346.48 MB/sec to 10851.80 MB/sec – an increase of 1.47G/sec, or 16.11%.
- Restored Bugzilla staging from a production restore to test an upcoming upgrade
It is a short week for me this week, because I have tomorrow off as I prepare for the 1st Latin American MySQL, NoSQL and Cloud conference in Buenos Aires next week. However, this week in database, we have done a lot, mostly within the Bugzilla property.
- Backed up several database servers that were not previously backed up
- Dealt with more puppet dashboard disk space issues
- Repaired a few tables after changing ft_min_word_length on some bugzilla slaves
- Upgraded bugzilla slaves to Percona 5.1
- Rebuilt our secondary bugzilla permissions
- Added alerts to our backups to make sure we have current backups
- Upgraded one of the bugzilla slaves’ Fusion IO card to driver 3.1, as per Percona’s blog post about it. We went from timing cached reads from 9346.48 MB/sec to 10851.80 MB/sec – an increase of 1.47G/sec, or 16.11%.
- Restored Bugzilla staging from a production restore to test an upcoming upgrade
I have been at Mozilla nearly three months, and I used to blog a lot more than I currently do. A lot of the content I used to blog about I end up blogging and talking about in OurSQL: The MySQL Database Community Podcast. And I have also been getting used to the Mozilla firehose, as well as my own firehose of database projects that need to be done.
There are two very large projects that are time-sensitive that I am working on: migrating databases from an older data center to a newer one, and the impending public launch of the Mozilla Apps Store.
That being said, this week in Mozilla databases we have:
– migrated/improved/built our dev/stage databases for xtrabackup.
– turned off our Scalarc software, as we now have an appliance for our proof-of-concept test.
– retired 2 machines that were not in use, and exist in our older data center. I am always paranoid when I shut a machine down, triple-checking that I am on the right server when I type “shutdown now”.
– did a test migration of the production database for Mozilla QA, from the old data center to the new one.
– added new custom fields to Bugzilla for the release of Thunderbird 10.
– Created new databases and access for:
– Case Conductor, a replacement for Litmus for the QA team
– De Todos Para Todos, Mozilla’s outreach project to Latin America.
There is much much more to come in the weeks ahead!
I have been at Mozilla nearly three months, and I used to blog a lot more than I currently do. A lot of the content I used to blog about I end up blogging and talking about in OurSQL: The MySQL Database Community Podcast. And I have also been getting used to the Mozilla firehose, as well as my own firehose of database projects that need to be done.
There are two very large projects that are time-sensitive that I am working on: migrating databases from an older data center to a newer one, and the impending public launch of the Mozilla Apps Store.
That being said, this week in Mozilla databases we have:
– migrated/improved/built our dev/stage databases for xtrabackup.
– turned off our Scalarc software, as we now have an appliance for our proof-of-concept test.
– retired 2 machines that were not in use, and exist in our older data center. I am always paranoid when I shut a machine down, triple-checking that I am on the right server when I type “shutdown now”.
– did a test migration of the production database for Mozilla QA, from the old data center to the new one.
– added new custom fields to Bugzilla for the release of Thunderbird 10.
– Created new databases and access for:
– Case Conductor, a replacement for Litmus for the QA team
– De Todos Para Todos, Mozilla’s outreach project to Latin America.
There is much much more to come in the weeks ahead!
