Rusty Divine

Live, Love, Learn, Teach

We Scrubbed our QA/Prod Move

It was quarter-after eleven Friday morning and I noticed my teammate’s brow was furrowed. It was her turn this iteration to coordinate the move to QA and Production, which involved enough steps that we have created several process documents and an iteration calendar to help keep the move on track.

The seven of us were in the basement for our weekly team technical training meeting – two hours set aside to explore new technologies and discuss how to improve our code base. It’s a time we look forward to each week as a fun way to start a Friday and a time to talk openly about our ideas and interests.

Two of the team were discussing or debating something while the rest of us worked through a tutorial on SignalR on our Azure VMs.

I asked the developer in charge of the QA move how it was going and she told me there was a problem.

“Could you open the QA website and check the obligation transaction page? We are getting a work phase not found error,” she said.

I checked and found the same error. It was strange and took a minute to process what could be going on to cause this error message. Neither our developers nor our very thorough QA tester had encountered this in our development branch during the iteration.

After a few minutes of conferencing about it, we realized one of the stories we sent to QA was failing because the table it was expecting to update did not have any data loaded yet, and would not until we turned on this multiple-iteration release in production, converted the data, and took over the functionality we were replacing in the main frame system.

Our application is complicated by having some parts that are used in production, and some parts that are technically in production but are turned off and hidden until we can complete the entire set of features and turn them off in the current main frame system. This particular feature bridged both worlds, which is why it caused a problem when we got to QA – in demo, we have data loaded from production, but QA matches production exactly and does not have that data loaded.

What to do. We could scrub moving to QA and Prod this iteration, roll back the problem feature, and do a normal QA/Prod move next iteration. We could populate the empty tables with production data so that our cross-over feature would work, but the data would become stale as soon as someone used the main frame system, so that was out. We decided our best option was to scrub the move because there was only one minor feature that would have been visible in production.

The QA mover and I went upstairs to talk to the product owner. We explained the situation and she agreed, the minor feature wasn’t important enough to do an emergency fix for and we should just scrub the move.

To me, this issue and how our process handled it so smoothly was awesome. Sometimes its easy to overlook something like this that appears trivial – it was no more than 20 minutes between the time we realized the issue to when we decided to scrub the move. We didn’t need to debate or discuss, it was clear from our process what we should do – and that included both developers and the product owner.

Days like this make me so happy to be working with an agile team!