peppermintspencer

April 3, 2013 · 9:46 pm

Candlemas Eve

It’s been a long time since I wrote anything here. I’d like to say it’s because I’ve been super busy, and in fact that has been the case at times, but the truth is that I have two very specific problems: one, I try to do too many things, and two, I need a lot of sleep. But at least it’s not quite a full year since the last post.

I’m struck by how different things are now compared with a year ago. Back then, I was worried about the way the project was going, concerned that we were going to blow our deadlines, and that we would implement wrong functionality because our BAs were spread too thinly between too many projects. Well, all that happened. There followed a period of intense, stressful and very un-Agile firefighting, but we did eventually come out the other side, and the Big, Important Projects that depended on our technical changes got out the door not too far behind schedule. All’s well that ends well.

Perhaps I’ll write more about what I learned from that later, but this week I’ve moved on to a new project, and indeed a new company, and now is a time for focusing sharply on the task in hand, because, as a contractor, you’re only ever as good as your last week’s work.

The new gig is something of a watershed for me. Having taught myself Ruby, Cucumber and WATIR alongside my exploratory testing, I finally got my dream and landed a full-time Ruby job. The new client has a reasonably mature suite of Watir-webdriver tests, all nicely organised, categorised and wired into Jenkins. They’re now looking to add some browser compatibility testing (everything’s currently done in Firefox) and I’ve spent my second day looking into Watirgrid, with a view to running the tests on several virtual machines simultaneously, with a different browser on each.

The good news is that Watirgrid itself is pretty simple and seems to work rather well. By the end of the day I had irb controlling Chrome on one machine, Firefox on another, and running arbitrary WATIR methods on both together. So the first point of this post is to encourage anyone who hasn’t had a go at this yet to try it. There is a fair amount of help out there if you’re willing to do a bit of Googling and I’ll try and add a “cheat sheet” to this site when I’m more confident with it.

The second thing I wanted to say is that I’d forgotten what a brilliant concept virtualisation is. I know I’m probably preaching to the converted here, but I hadn’t really touched a VM in a good couple of years before this week, and one thing I’ve really appreciated today is that when something fails on Windows (I’m using Windows 7 VMs) I can try it on the host machine (a Mac) and thus narrow down where the problem is. If it works on MacOS, it’s likely either a compatibility or configuration issue. If you can’t get it to work on IE9, have a crack at it on another VM running IE8.

It’s also been quite an eye-opener: I’ve been a Windows user all my life, and I’m used to the sharp intake of breath from Those In The Know (devs) when I tell them I’m programming in Ruby on Windows. I’ve always put this down to good old OS bias, but actually I found today that I could get Watir-webdriver to drive Chrome fine on Mac OSX, whereas on Windows it just produced a string of impenetrable errors. Anecdotal and unscientific this evidence may be, but I’m starting to think the devs may have had a point.

Don’t get me started on the Mac app dock though: suffice to say it’s made me realise that the Office 2007 Ribbon isn’t the last word in poor UI design after all.

Anyway, if you’re a tester, you have a reasonably powerful Windows box, and you have admin rights (a tall order in some companies, I know) then I urge you to slap a hypervisor on your machine and get some VMs running. It’s generally free unless you need Windows licenses (in fact, you can download a 90-day trial version of Windows 8 from Microsoft, but using it for testing might be a licensing grey area, so always read the label) . If nothing else, it’s an extra avenue to explore when you start to feel like you’re out of your depth with the particular random exception you’re grappling with. If you can prove that the thing you’re trying to do works on Linux, you can be that much more specific with your question on StackOverflow.

That’s today’s ramble done. Oh, and the title, in case you were wondering, is the name of my favourite Christmas song. It’s from the album Sweet Bells by the astonishingly talented Kate Rusby, and it actually isn’t really a Christmas song, it just happens to be on a Christmas CD, so it’s OK to mention it in April and you don’t have to throw sharp things at me. Candlemas is a Christian festival that falls on 2 February (40 days after Christmas), and apparently the night before was when people traditionally took down Christmas decorations. Nowadays, of course, we do this earlier, to give the confectionery manufacturers more time to flog us Valentine’s treats and chocolate eggs:

Down with the rosemary and bay,
Down with the mistletoe,
Instead of holly, now up-raise
The greener box, to show

The latter part of the song continues the narrative right through to Whitsuntide (in late Spring) which is honoured with birch and assorted flowers. It’s a song about turning points in the year, and, for me, it’s an ode to changes in general:

Thus times do shift, thus times do shift,
Each thing its time doth hold,
New things succeed, new things succeed,
As former things grow old

I’m an incurably sentimental twerp who clings onto memories with an iron grip and can’t sit through the end of Stardust without crying like a bilge pump. I hate giving up things I like, and view the good old days through spectacles so rose-tinted they’re practically opaque. I have to remind myself frequently that change is both inevitable and often productive, and this song, in addition to being rather beautiful, helps me do that. It reminds us that as seasons change, so must lives. Saying goodbye to so many dear friends at the old office was an unspeakably tough experience, but moving on gives me the chance to learn and grow. Besides, as the seasons are cyclical, so sometimes we can come back to places and people we’ve left behind.

Especially us contractors: we’re particularly fickle like that.

1 Comment

Filed under Uncategorized

April 22, 2012 · 12:19 pm

Does Agile equal fast?

This is more a question than an opinion or a statement of fact. In physical or sporting terms, the word agile has pretty obvious connotations of fast. In fact, the first definition of the word on dictionary.com is “quick and well-coordinated in movement”. But it occurred to me recently that the Agile Manifesto doesn’t contain any reference to speed at all. This is an interesting discrepancy.

For me, Agile has always been about manoeuvrability: being able to react to changing requirements, and having the freedom to make wholesale changes to ones approach if things don’t turn out right on the first attempt. But now I’m starting to wonder whether there’s a tendency among managers to view Agile as a way of increasing the speed of software development. I’d love to know what you think about it, so feel free to let rip in the comments section below.

Now, Agile makes it easy to track a team’s pace accurately: the process of sizing stories relative to each other makes for a pretty reliable measure of the team’s velocity.

It’s also true to say that Agile, done properly, should always help to meet deadlines. The iterative approach makes it far easier to produce software on any given date which, though it might be missing nice-to-have features, is still safe to release to customers (and note that I say “done properly” there — it isn’t always the case in my experience). Compare this with traditional waterfall-style development, which often won’t yield something that can be released until the project is completely finished.

But this doesn’t imply that the pace of development is actually any quicker with Agile: it’s simple a trade-off of time against scope. Developers have their own speed of coding; testers have their own speed of testing. This, as far as I can see, doesn’t change according to which project methodology is being used. If Agile works out well for the team, speed might improve as a function of rising morale (and I certainly don’t have any figures to back that up) but that’s an indirect result of the methodology being used, not a direct one.

If there is, as I suspect, an implicit association in managers’ minds of the words Agile and fast, it could actually jeopardise successful adoption of Agile methodologies. There could be a very strong temptation to cut corners on the grounds that Agile has to mean quick. This could result in rushing the all-important sprint planning and backlog grooming sessions, and skimping on the process of creating story cards and conditions of satisfaction. And, if that’s happening, you can forget about any chance of getting executable specifications done.

If you’re familiar with Agile working, you probably don’t need me to tell you that failing to spend enough time on these things is really bad news. Developers and testers must take the time to discuss each story thoroughly with the BA or Product Owner before work starts (the whole team doesn’t have to be in on these discussions, but at least the ones who “own” the story must be present). Otherwise, how can you ensure a common understanding of the requirements when there’s no formal spec document to fall back on? Equally, skimping on the process of breaking down stories into subtasks in the sprint planning session is likely to lead to important pieces of work being forgotten. I wince when I see post-it notes on the task board that are either blank or just say “do it”, and it happens surprisingly often.

All of this leads to falling quality and rising risk, as tasks get missed and mistakes go unspotted. Perhaps I’m wrong, and this isn’t anything to do with people’s interpretaion of the word Agile. Perhaps it’s all down to business deadlines and business analysts’ time being spread too thinly between multiple projects. Let me know what you think.

1 Comment

Filed under Agile

Tagged as Agile

February 16, 2012 · 2:04 pm

IRB, and when automation is easier than testing it manually

This post was originally published on Tech-a-Porter, the internal tech blog for Net-a-Porter staff, on 10 February 2012. It assumes a basic working knowledge of Ruby and WATIR.

I spend a lot of time thinking about test automation. We all know that, when it’s done right, it can save everyone time, increase test coverage, and eliminate a lot of tedious, repetitive grunt work. The trouble is, when it’s done wrong, it becomes just as much of a millstone as the manual testing it replaces. Witness:

The massive maintenance burden of keeping automated tests working when the products they are hard-coded to look for suddenly disappear from the site under test because they’re sold out.
The pain of debugging a test that’s given you an incomprehensible stack trace instead of a sensible error message.
The constant fine-tuning of automated tests to cope with slight changes to the user interface that prevent them from “seeing” the objects they are looking for.

…and so on. I’m sure you can think of some more examples. So, to sum up, if you’re going for test automation in a big way, you’ve got to plan carefully. And you have to employ the same kind of programming best-practices as the developers do. And, of course, you must be prepared to spend a significant time on maintenance. This is daunting, and, if you’re just dabbling in automation rather than being a full-time Test Automator, you probably aren’t prepared to make that kind of commitment for fear that it won’t deliver enough value to justify the time invested. Game over.

It’s a shame to give up at this point, because, as valid as those concerns are, you can still make good use of automation techniques and achieve real efficiency gains in this situation. The trick is to “think small” and break things down into little pieces. When we think about test automation, there’s a tendency to try and tackle complete test cases in one go. But is you can automate just one part of the case that is fairly simple but still repetitive, it’s often possible to score a significant time saving, even if you’re still completing the rest of the case manually.

I’ve recently been doing some testing of the search functionality on an ecommerce website. The change was technical only: the idea was that the end users wouldn’t notice any difference after it was deployed. Clearly, the easist way to test this is to hook up two sites to the same database, put the new code on one and the old code on the other, then check that any search you run returns the same results on both sites.

IRB is a great tool for this. When you’re viewing search results, there’s almost certainly a repeating html element that contains the product ID. All you have to do is work out how to identify these elements, and you can get a list of every product ID in a couple of seconds. Identifying the elements is pretty easy with a tool like Firebug. In this example, I’m assuming we’re shopping on New Look.

As you can see in the screenshot, the product ID can be found in a hyperlink that resides in a list item with class “c5-1″. Bear in mind that I don’t work for New Look, so I had to work this out. If I did work for New Look, I could get this information from a developer. Anyway, all I have to do is get Ruby to iterate through every element on the page with class “c5-1″ and print the relevant part of the link address. The following snippet does just that:

require 'rubygems' require 'watir' include Watir $browser=IE.new # at this point you can go to your browser, navigate to the site under test, and run the search $browser.elements(:class_name,'c5-1').each do |el| puts el.links[0].href.match(/\d+($|\?)/).to_s.chomp('?') end

Let’s just go through the lines that do the real work and explain how it’s done:

$browser.elements(:class_name,'c5-1')
Dead simple: tells Watir to look at the page and pick out any html element with the class ‘c5-1′.

$browser.elements(:class_name,'c5-1').each do |el|
Tells Ruby to iterate through each of the matching elements, passing each one in turn to the next code block as an object called el.

puts el.links[0].href.match(/\d+($|\?)/).to_s.chomp('?')

el.links[0] just picks out the first link in the element. There’s only one, anyway.
.href gets the target URL of the link.
.match(/\d+($|\?)/) finds the product ID within the link. What’s that regex doing? Well, \d+\ looks for one or more digits (i.e. the bit of the string we want) and ($|\?) just specifies that we only want sequences of digits that are immediately followed either by the end of the line (represented by $) or a question mark (which we must escape using a backslash, otherwise it would be treated as a wildcard). I noticed that some of the links had an argument like “?productFind=search” after the ID, hence why the ‘?’ is needed.
.to_s.chomp(‘?’) extracts the string from the match results, and removes the ‘?’ from the end, if there was one.

Actually, the chances are I could have got away with a much simpler regex that simply returned any string of digits, regardless of what followed them. This would probably have worked, but since I don’t know the site very well, it seemed safer to be a bit more specific.

So, we run that once in IRB, return to our browser and run the search on the other site, then repeat the last three lines in IRB to get another list. Copy/paste the lists into Excel, write a couple of swift vlookup functions, and you can instantly spot any differences between the two results.

Boom! Job done. And we don’t have to bother about source control, nor do we have to do any complicated setting up of environments if we want to share it with colleagues: a simple email containing a few lines of code will do the trick. Even better, if we later decide to automate this test case fully, we’ve already written the most tricky bit of the algorithm.

The trick here is to treat test automation not as a way of life, or a paradigm that must be followed totally or not at all, but as a tool in your armoury that you can use as little or as much of depending on what the situation demands. If you don’t know Ruby, I’m sure you could do something similar with Selenium IDE: the point is to recognise the situations where a bit of innovation can help you work smarter, not harder.

Android apps, document addiction, and the most important skill a tester can have

Peppermint Spencer has been on something of a hiatus since that last post about the seaside. We have been terribly busy taking a well-earned excursion to France, not to mention some crucial top-level parenting. Oh, and trying to push a large test department in a direction it doesn’t want to go.

The cure to the perennial problem of how to find time to blog amongst all this comes in the form of an Android app. Yes, this post comes direct to you from our trusty HTC Desire, where we’re currently enjoying a coach ride up the M40 prior to beer and curry in Oxford.

Anyway, enough of such pleasantries. My topic today is a matter of great pith and gravity: document addiction. You might never have heard the term, but it’s my attempt to hang a name on something that has blighted at least one or two of the test teams I’ve worked in.

The meaning of the term should, I hope, be fairly obvious. The background is an article I read a while back which coined the term “test addiction” to refer to the situation, no doubt familiar to a lot of us, where a team continually runs large numbers of manual tests, most of which always pass, and so add little or no value. Worse, they leave no time to spend on more useful QA activities. These tests are often just there so that the daily status report can have a lot of green in it. They also cover our back if it all goes south when the release is deployed, because the test team Did Its Job and Has The Metrics To Prove it.

Document addiction, then, is a variation on the same theme, only instead of running tests that don’t find defects, we’re writing documents that, bluntly, no one needs. They might be snazzy, breathtakingly erudite, and stored with loving care in a well-organised industry-standard version control system – hell, people might even read them occasionally – but they don’t fulfil a document’s sole raison d’etre: to give the reader the information he needs.

So how does this happen? Well, here’s my story, which serves as an example.

Like many entering the world of testing, I soon realised that many employers won’t give your CV a second look unless it has the word ISEB on it. So I spent three pleasant days at a very nice venue in rural Cheshire taking the Foundation course. It had a steam room and self-service ice cream machine, so I was happy. The training manager got to spend some of her budget, so she was happy. I eventually got a better job that paid more, so my girlfriend was happy. So far, so good.

But of the things we covered that week, I’d say probably only one quarter was new material I could apply to the job I was doing. For the rest, a quarter consisted of things I already knew from studying Computer Science at university, another quarter was irrelevant because it dealt with things like code coverage, which is more the developer’s problem than the tester’s, and the final and most dangerous quarter was all about the IEEE 829 standard: aka test documentation. It’s this last section that led me to bad habits that have taken five years and half a dozen different jobs to kick out of my system. Ironically, and I doubt I’m alone in this, it’s also the part that I took on board most thoroughly. After all, it’s a darn sight easier to produce a template, paint-by-numbers document than it is to think seriously about ways to improve your testing.

Let’s not beat about the bush here: documentation is important, vital even, to the health of any project, but there is a big difference between documentation and documents. Documents for their own sake are worse than useless. For that reason I have serious beef with 829. Here’s why:

First, it teaches us to express our plans in prose. Prose is ambiguous, imprecise, and it’s all too easy for poor content to be hidden by pretty formatting and judicious use of the correct buzzwords. The fact remains that if you have a complex system or test to explain, you can usually do it better with a mindmap, flowchart, input-output grid or annotated screenshot. There’s no rule that says testers aren’t allowed to use their creativity, so use your imagination and remember: just because prose is the default option doesn’t mean it’s the best; incidentally, this applies just as much to business analysts as testers.

Second, 829 lays out a large suite of documents, most of which are very similar, while others are obscure to the point where it took me years to work out what they were actually for: Test Item Transmittal Report, anyone? I can see why it’s tempting to follow the standard to the letter: on one hand, we all instinctively want to stick to standards, because they are good insurance against failure (if you did what the book told you to do but it still went wrong, nobody can blame you, right?) and on the other hand, there’s an implicit “more is more” principle: big thick documents can’t fail to impress the management. Trust me: I’ve been there. The fact is, though, that most of the time either no one’s reading your 40-page Master Test Plan, or they are only reading it to pass the time because their own job bores them witless.

But most of all, I’m still mildly surprised that the Foundation course, at least back then, taught 829 as the only method for documenting test work. No alternatives were mentioned, and it wasn’t really made clear that, in most projects, only a little of it is useful.

Trying to “tick all the boxes” by producing all the prescribed documents (particularly on brownfield projects where the team already has well-established routines and methods) either results in a highly detailed but totally pointless description of the status quo, or an attempt to document away problems by producing a set of aspirations for how we’d do things in an ideal world with unlimited budget. But if all you’re doing is writing down how things are already being done, why bother? And if your test plan starts to look like a manifesto for a brave new world in which you can finally get down to business with that expensive automation tool that management will never fork out for, you’re wasting your time. If nuggets of genuinely new and useful information are buried amongst acres of boilerplate text, your audience will miss them.

I’ll go further: in today’s IT departments, a text-based test plan is no place for things like dates and milestones: these belong on a Gantt chart or an Agile release roadmap, where they are highly visible and easily altered when deadlines move or scope changes. Nor is it the right place for information about who is in the team and what the reporting structure looks like: that kind of thing will be out-of-date as soon as you have a leaver or a joiner.

I have worked for a couple of teams that didn’t use textual test plans at all. Now, one of them was a car crash, but that was because of poor management and ludicrous deadlines, so we’ll ignore it. But the other doesn’t need them because all the information is already available in Jira, on the departmental wiki, or in the test cases themselves. And that brings me to the key point: when you’re writing something, keep asking yourself these questions: first, is this content already available somewhere else? If the answer is yes, leave it out or just include a reference or link to it. Second, does my target audience really need to know this? If it’s a no, well, you know what to do. Be ruthless, because the less noise you produce, the more likely you are to be heard when you say something really important.

If you really must produce a master test plan, use a format that expresses the facts cleanly and minimises waffle. Don’t be afraid to use diagrams, or even avoid Word completely and use a spreadsheet or a mind map.

Remember that you add value by breaking software, not by churning out fat Word documents to try and prove to managers that you’re doing your job.

So go forth, be inquisitive, think of the crazy edge cases others miss. Most of all, don’t ever stop asking “what if?” Because, for me, that’s the most important talent a tester can have.

2 Comments

Filed under Documentation

Tagged as Documentation

April 18, 2011 · 7:59 pm

So what’s the header image about?

Ah, well, that’s a very good question. I spent a few minutes this morning trying to retrofit some logic to it, and convince myself that it relates somehow to software quality assurance, which, after all, is what this site’s about. It’s a photo I took on a late-summer afternoon nearly five years ago, looking out to sea at Dawlish Warren in Devon. It’s one of my all-time favourite places to spend a lazy Sunday, and I miss it more than any other aspect of Devon life now that I’m living in the next county. But what does it have to do with testing? Here are the ideas I came up with:

1. The sea represents software, and the breakwater represents the tester, delving in and uncovering bugs.

Gullibility rating: weapons-grade nonsense.

2. The pole at the end of the breakwater with the marker on top represents the tester, warning approaching vessels (aka stakeholders) that they are approaching underwater rocks (aka bugs) and in danger of getting stuck.

Gullibility rating: ridiculous. You can’t even see the marker because it’s cropped out of the top of the frame, and there are no rocks here anyway.

Cheerful Crayola-coloured cabins

3. The breakwater is there to limit the effects of longshore drift. This is when the waves approach the beach at an angle and move the sand along the coast, eventually washing the beach away (I did my year 7 geography project on this, so I should know). The image here is of a tester stopping bugs drifting through the development lifecycle into the software release.

Gullibility rating: as close to being utter drivel as it’s possible to be without sacrificing correct punctuation.

I don’t know about you, but I don’t find any of those particularly compelling: perhaps metaphors aren’t my thing, but we’ll find out as this blog continues.

That leaves me with no alternative but to veer off-topic and confess that the picture is actually only there because I think it’s a nice picture of a place I love to be. But maybe that’s an valid point in itself: it’s important sometimes to appreciate things for what they are, not what they represent. In fact, Dawlish Warren is far from perfect: much as I love it, parking is a little pricey in the summer and the place isn’t short of an ugly caravan park or two. And software projects have flaws too. As testers, I think we get hung up far to often on these flaws: the spec reviews that were glossed over because of tight deadlines, the corners that were cut in the API design, the regression tests that weren’t run, the bugs that — horror of horrors — couldn’t be fixed before the live date. But sometimes that blinds us to the fact that what we’ve produced (and I say “we” because it’s not just developers but designers, business analysts, database adminstrators and testers too) is something rather magnificent, which, by and large, provides a great service to its users (and, hopefully, makes moolah to pay our wages).

If nothing else, as a tester, you can feel proud that the end product is better than it would have been if it had been released as soon as the code was written. After all, that’s why we have a career!