Testing GUIs Part I: RoR.
Posted by Uncle Bob on 01/13/2007
Testing GUIs is one of the the holy grails of Test Driven Develoment (TDD). Many teams who have adopted TDD for other parts of their projects have, for one reason or another, been unable to adequately test the GUI portion of their code.
In this series of article I will show that GUI testing is a solved problem. Over the years the TDD community has produced and accumulated tools, frameworks, libraries, and techniques that allow any team to test their GUI code as fully as any other part of their code.
Testing GUIs Part I: Ruby on Rails
In the world of web development, no community has solved the problem of GUI testing better than the Ruby on Rails community. When you work on a rails project, testing the GUI is simply de-rigeur. The rails framework provides all the necessary tools and access points for testing all aspects of the application, including the generation of HTML and the structure of the resulting web pages.
Web pages in rails are specified by .rhtml files that contain a mixture of HTML and ruby code similar to the way Java and HTML are mixed in .jsp files. The difference is that .rhtml files are translated at runtime rather than being compiled into servlets the way .jsp pages are. This makes it very easy for the rails environment to generate the HTML for a web page outside of the web container. Indeed, the web server does not need to be running.
This ease and portability of generating HTML means that the rails test framework merely needs to set up the variables needed by the ruby scriptlets within the .rhtml files, generate the HTML, and then parse that HTML into a form that the tests can query.
A typical example.
The tests query the HTML using an xpath-like syntax coupled with a suite of very powerful assertion functions. The best way to understand this is to see it. So here is a simple file named: autocomplete_teacher.rhtml.
<ul class="autocomplete_list">
<% @autocompleted_teachers.each do |t| %>
<li class="autocomplete_item"><%= "#{create_name_adornment(t)} #{t.last_name}, #{t.first_name}"%></li>
<% end %>
</ul>
You don’t have to be a ruby programmer to understand this. All it is doing is building an HTML list. The Ruby scriptlet between <% and %> tokens simple loops for each teacher creating an <li> tag from an “adornment”, and the first and last name. (The adornment happens to be the database id of the teacher in parentheses.) A simple test for this .rhtml file is:
def test_autocomplete_teacher_finds_one_in_first_name
post :autocomplete_teacher, :request=>{:teacher=>"B"}
assert_template "autocomplete_teacher"
assert_response :success
assert_select "ul.autocomplete_list" do
assert_select "li.autocomplete_item", :count => 1
assert_select "li", "(1) Martin, Bob"
end
end
The post statement simply invokes the controller that would normally be invoked by a POST url of the form: POST /teachers/autocomplete_teacher with the teacher parameter set to "B".
The first assertion makes sure that the controller rendered the autocomplete_teacher.rhtml template.
The next makes sure that the controller returned success.
the third is a compound assertion that starts by finding the <ul> tag with a class="autocomplete_list" attribute. (Notice the use of css syntax.)
Within this tag there should be an <li> tag with a class="autocomplete_item" attribute,
and containing the text (1) Martin, Bob.
It should not come as any surprise that this test runs in a test environment in which the database has been pre-loaded with very specific data. For example, this test database always has “Bob Martin” being the first row (id=1) in the Teacher table.
The assert_select function is very powerful, and allows you to query large and complex HTML documents with surgical precision. Although this example give you just a glimpse of that power, you should be able to see that the rails testing scheme allows you to test that all the scriptlets in an .rhtml file are behaving correctly, and are correctly extracting data from the variables set by the controller.
An example using RSpec and Behavior Driven Design.
What follows is a more significant rails example that uses an alternate testing syntax known as Behavior Driven Design (BDD). The tool that accepts this syntax is called RSpec.
Imagine that we have a page that records telephone messages taken from teachers at different schools. Part of that page might have an .rhtml syntax that looks like this:
<h1>Message List</h1>
<table id="list">
<tr class="list_header_row">
<th class="list_header">Time</th>
<th class="list_header">Caller</th>
<th class="list_header">School</th>
<th class="list_header">IEP</th>
</tr>
<%time_chooser = TimeChooser.new%>
<% for message in @messages %>
<%cell_class = cycle("list_content_even", "list_content_odd")%>
<tr id="list_content_row">
<td id="time" class="<%=cell_class%>"><%=h(time_chooser.format_time(message.time)) %></td>
<td id="caller" class="<%=cell_class%>"><%=h person_name(message.caller) %></td>
<td id="school" class="<%=cell_class%>"><%=h message.school.name %></td>
<td id="iep" class="<%=cell_class%>"><%=h (message.iep ? "X" : "") %></td>
</tr>
<% end %>
</table>
Clearly each message has a time, caller, school, and some kind of boolean field named “IEP”. We can test this .rhtml file with the following RSpec specification:
context "Given a request to render message/list with one message the page" do
setup do
m = mock "message"
caller = mock "person",:null_object=>true
school = mock "school"
m.should_receive(:school).and_return(school)
m.should_receive(:time).and_return(Time.parse("1/1/06"))
m.should_receive(:caller).any_number_of_times.and_return(caller)
m.should_receive(:iep).and_return(true)
caller.should_receive(:first_name).and_return("Bob")
caller.should_receive(:last_name).and_return("Martin")
school.should_receive(:name).and_return("Jefferson")
assigns[:messages]=[m]
assigns[:message_pages] = mock "message_pages", :null_object=>true
render 'message/list'
end
specify "should show the time" do
response.should_have_tag :td, :content=>"12:00 AM 1/1", :attributes=>{:id=>"time"}
end
specify "should show caller first and last name" do
response.should_have_tag :td, :content=>"Bob Martin", :attributes=>{:id=>"caller"}
end
specify "should show school name" do
response.should_have_tag :td, :content=>"Jefferson", :attributes=>{:id=>"school"}
end
specify "should show the IEP field" do
response.should_have_tag :td, :content=>"X",:attributes=>{:id=>"iep"}
end
end
I’m not going to explain the setup function containing all that mock stuff you see at the start. Let me just say that the mocking facilities of RSpec are both powerful and convenient. Actually you shouldn’t have too much trouble understanding the setup if you try; but understanding it is not essential for this example. The interesting testing is in the specify blocks.
You shouldn’t have too much trouble reading the specify blocks. You can understand all of them if you understand the first. Here is what it does:
The first spec ensures that <td id="time">12:00 AM 1/1</td> exists in the HTML document. This is not a string compare. Rather it is a semantic equivalence. Whitespace, and other attributes and complications are ignored. This spec will pass as long as there is a td tag with the appropriate id and contents.
HTML Testing Discipline and Strategy
One of the reasons that GUI testing has been so problematic in the .jsp world is that the java scriptlets in those files often reach out into the overall application domain and touch code that ties them to the web container and the application server. For example, if you make a call from a .jsp page to a database gateway, or an entity bean, or some other structure that is tied to the database; then in order to test the .jsp you have to have the full enabling context running. Rails gets away with this because the enabling context is lightweight, portable, and disconnected from the web container, and the live database. Even so, rails applications are not always as decoupled as they should be.
In Rails, Java, or any other web context, the discipline should be to make sure that none of the scriptlets in the .jsp, .rhtml, etc. files know anything at all about the rest of the application. Rather, the controller code should load up data into simple objects and pass them to the scriptlets (typically in the attributes field of the HttpServletRequest object or its equivalent). The scriptlets can fiddle with the format of this data (e.g. data formats, money formats, etc.) but should not do any calculation, querying, or other business rule or database processing. Nor should the scriptlets navigate through the model objects or entities. Rather the controller should do all the navigating, gathering, and calculating and present the data to the scriptlets in a nice little package.
If you follow this simple design discipline, then your web pages can be generated completely outside of the web environment, and your tests can parse and inspect the html in a simple and friendly environment.
Conclusion
I’ll have more to say about RSpec in a future blog. BDD is an exciting twist on the syntax of testing, that has an effect far greater than the simple syntax shift would imply.
I hope this article has convinced you that the rails community has solved the problem of testing HTML generation. This solution can be easily extrapolated back to Java and .NET as future blogs in this series will show.
Clearly the problem of testing Javascript, and the ever more complex issues of Web2.0 and GTK are not addressed by this scheme. However, there are solutions for Javascript that we will investigate in future blogs in this series.
Finally, this technique does not test the integration and workflow of a whole application. Again, those are topics for later blogs.
I hope this kickoff blog has been informative. If you have a comment, question, or even a rant, please don’t hesitate to add a comment to this blog.
Comments
David Chelimsky about 7 hours later:
Very happy to see that you’re enjoying using RSpec.
There are some things that I’ve started to approach differently. I posted on this next door and would love to get your feedback.
Anthony Bailey about 9 hours later:
Thanks for the article,
(I assume RSpec plans to support some kind of should_css_selectequivalent to the elegant assert_select sometime soon. It’s already distressing to go back to the noisier tag style assertions.)
Will you be discussing the testing of direct manipulation (e.g. mouse-driven inputs, highly graphical output) user interfaces at all? I have found those to be the ones where automated testing is the most difficult, although not impossible with the right kind of intermediate representations in place.
Uncle Bob about 15 hours later:
Anthony,
Yes, I plan on discussing the testing of thick client GUIs, and GUIs with lots of JavaScript.
David Chelimsky about 15 hours later:
Anthony – yes, RSpec on Rails already unofficially supports assert_select for tags (not yet for rjs). If you are using the 0.7.5 plugin, take a look at spec/expecations/should_have_spec.rb to see what you can do. We used should_have because should_select doesn’t talk about the behaviour of the system under test. It sounds more like the behaviour of rspec, which “should_select” or “should_find” an element. What we’re saying is that the response should have some material in it, so “should_have” makes sense to me.
At this point there are several things holding me back from making it officially supported. For one, the syntax is likely to change subtly. We also need to support rjs. And the error messages are completely useless. Right now we’re just wrapping assert_select, and its error messages just say “expected 1 element(s), found 0”. That is not all that helpful to me.
Once these are resolved, we’ll make it official. In the mean time you are free to use what’s there – just be aware that there will be some API changes and you’ll have to change your specs for a future release.
Andy Dent 10 days later:
GUI testing is a solved problem
Uhh, do you just mean HTML Forms Web App GUI testing is a solved problem or is your series going to include
rich internet apps using Flash
desktop apps
mobile apps?
I’m really hoping for a good answer on desktop apps because, surprisingly, there’s still a lot of work happening developing them, both for retail and inhouse. We could really use a good solution for wxPython-based GUIs!
Aaron 17 days later:
It is informative, and it is the most effective way I can find as well. Actually I am just finding a solution to develop a better java testing tool to help my team to test faster, easier and in better efficiency. The best idea seems to use annotations and xml configurations to decorate test cases. A typical framework of this kind is TestNG. It seems the advantages of this kind of testing framework is:
1> Stronger control in the workflow of test cases running. You can either use specific annotation syntax to put them into groups, or identify some executing sequences. 2> More flexible. Testing classes don’t have to extend some base classes as JUnit, or have some rules to restrict method defining. 3> Accelerate procedure of developing test cases in some specific situations.
Aha, this is not bad comparing with JUnit. But if it comes to rails, annotation kind of things would not be that competent. Because using syntax way to test, it must be much more flexible & powerful.
prudhvi@30gigs.com 6 months later:
Hi
I am a newbie to Ruby. I am interested in testing the url’s placed in a web-page to find the ‘Page not Found’ error in specific and other link issues like passing the required params in generic using Ruby.
Could you pl guide me on the aforementioned. Maybe through some pdf and examples.
Regards, Prudhvi
sohbet over 3 years later:
You can’t do this alone you need other people with like minded views or at least other people who are willing to listen. There’s nothing worse than someone who wants things to be better but does nothing about it.
craigslist Taylor over 4 years later:
GUI software testing is the process of testing a product to ensure it meets its written specifications or not. Rails community has solved the problem of testing soft ware. Working with the Ruby rail is no more problematic to test soft ware. Great…......
craigslist Beaumont over 4 years later:
I am in favor of this soft ware but there is a problem also. Every new day bring almost a new software and people are double minded which one they can use? Which one is more helpful for them? Ruby Rail is good addition but users may be confused about the efficiency of it. Its my thinking may not be true or it may be…
craigslist Toronto over 4 years later:
It is a software that confirms that product is according to its requirements or not. In simple it is a testing software and i feel proud to that how nice blog it is. Previously it was a major problem to test software but now it is solved…
craigslist los angeles over 4 years later:
There are always opponents as well as those who are in favor of that product… but i think it is such a splendid soft ware that no one can deny from its importance… these kinds of innovations attract encourage the work of technology. Awesome really,,,
Ray Cruz over 4 years later:
At the very 1st, I’d prefer to say thanks to you for this informative post. Second, I’d prefer to doubt wheresoever I can identify a lot more info concerning your post. I arrived right here through Yahoo & can not find out any associated web internet sites on this subject. How do I subscribe for your web blog? I had choose to stick to your updates as they come along! I had a query to interrogate but I forgot what it absolutely was… anyways, thnx. Author of how to cook beef tenderloin
XoX, Ray Cruz
GHD Austalia over 4 years later:
GHD australia have fairly very rated for rather a few of elements just like pattern durability and ease of use.
Criminal Records over 4 years later:
Testing classes don’t have to extend some base classes as JUnit, or have some rules to restrict method defining.
Tenant Screening over 4 years later:
Web pages in rails are specified by .rhtml files that contain a mixture of HTML and ruby code similar to the way Java and HTML are mixed in .jsp files. The difference is that .rhtml files are translated at runtime rather than being compiled into servlets the way .jsp pages are. This makes it very easy for the rails environment to generate the HTML for a web page outside of the web container. Indeed, the web server does not need to be running.
Macripper over 4 years later:
The technique described here can be used to test virtually any static web page, or portion thereof outside of a container, and without a webserver running. It is relatively simple to set up; and then very easy to extend. With it, you can spin around the edit/compile/test loop very quickly, and can easily follow the rules of Test Driven Development
Anthony over 4 years later:
Great blog; Those code examples really helped my project.
bob over 5 years later:
Testing GUIs is one of the the holy grails I have to disagree with this comment but all in all nice article.
beats by dre store over 5 years later:
rails I have to disagree with this comment but all in all nice article.high quality headphones new design headphones
beats by dre store over 5 years later:
and informative, I really enjoyed reading it and will certainly share this post with my friends . Learn everything about high quality headphones new design headphones
billy_corter@yahoo.com over 5 years later:
overall I have not read bnyak articles on this site. but I believe this will provide many benefits for many people. and I also liked this article. It became the lesson material for me to forward his. Billy –event planning jobs
Jeanna Bryner over 5 years later:
Write in a good way is not an easy. But your writing style is mind blowing. I appreciate your work and thank to share this. Wax Tips
William Joseph over 5 years later:
Your article took my several minute but it is not going waste. Your style of writing is marvelous. You have Conway your point in a very well way. Thanks for this http://www.wageaverage.com/
Federal Consolidation Loan over 5 years later:
Wow… easy to stand and well defined post. You wrote in awesome style and not go out from topic. Your writing style is totally new for me. I will bookmark your site.
Brent Nicholson over 5 years later:
Art of writing is God gifted and u can prove it in this post. Sensible and mature post, writes in a good style. I will check your site on a regular basis.
Official Tips over 5 years later:
I don’t understand what words I choose for you this precious post. It is look like a nice piece of article writing. It is such an amazing post. Thanks to share this
Danny's Collection over 5 years later:
Nice style and absolutely different. I always prefer to read good information and your article is a example of this. This is very useful information. Thanks to share with us. I will check your site regularly.
Personal Blog over 5 years later:
Well, your post shows how much time you take to do this and efforts to furnish this. I really impressed by your writing skills. Thanks to share this good piece of writing.
Prince Charles over 5 years later:
I simply nominate three words for your article…Simple, Good, complete. Your power of writing is very good and you chose a different topic for discussion. Keep it up.
Gate Valve over 5 years later:
Thanks for sharing! please allow me to twitter it.
car rental service delhi over 5 years later:
If only I had found this blog before. The advice in this post are very helpful and I will certainly read the other posts in this series too. Thank you for posting this.
banyo dekorasyon over 5 years later:
Yes, I plan on discussing the testing of thick client GUIs, Banyo Dekorasyon and GUIs with lots of JavaScript.
Software Testing Services over 5 years later:
The article very surprised to me! Your writing is good. In this I learned a lot! Thank you! Share with us the good times.<Software Quality Assurance
dentist in houston over 5 years later:
First I must tell you we appreciate the great and informative entry.Need to admit that I never learned about this review, I noticed many new facts, in my opinion. God bless you for sharing this information useful and interesting and I also will probably anticipate other interesting posts you closer to the inside future.keep.
Urdu poetry over 5 years later:
Thanks for sharing these wonderful comic videos with us. They are really funny. Will look after for some more updates.
led displays over 5 years later:
I have to say this is huge work.I have read the whole article and it is really superbly written article.Testing guis part 1 is really an informative article and wanna say we the readers wanna see some more like this.
Direct Cash over 6 years later:
Even 5 years later, this explains a lot. Testing is so tedious so big thanks to you.