Blog‎ > ‎


posted Oct 16, 2011, 10:51 AM by Isaac Lopez   [ updated Dec 15, 2011, 2:14 PM by Victor Reina ]
By Jose Ignacio Gonzalez De La Fuente
What is it?
TestNG is a Java testing framework that allows us to do all kinds of tests (unit, functional, end-to-end, integration, etc.) with a lot of flexibility and easy configuration.
How it started
Cedric Beust wasn’t happy about how
JUnit forces you to use static methods in your code to run tests, a TestCase getting instantiated for every test method and other design flaws. He wasn’t going to stay with his arms crossed so inspired from JUNIT and NUnit he started TestNG, implementing new features like the use of groups, tests for multithreading, dependent methods and others.
Why use TestNG?
TestNG makes a good use of annotations from using @Test to indicate that a method is a test to @DataProvider that marks a method as supplying data for a test method.
TestNG can separate your tests in groups for a more convenient testing; you can place a test in more than one group and have different setups for each group, you can also run just a group of tests if you want.
TestNG is pretty flexible and easy, you can configure a lot of things but if you don’t want to there’s a default for everything. Let’s say that you’re using a data provider, if you don’t give the annotation a name it uses the method name to refer it.
It’s awesome, just try it!

Using TestNG
You can invoke TestNG in many different ways:
  1. From the command line
  2. With ant
  3. With Eclipse using it’s plugin
  4. With IntelliJ’s IDEA
As you can see everything you need to know about TestNG is right in its site, from the basics like setting up a test and it’s different annotations to using listeners to modify TestNG’s behavior

Building TestNG

The first thing I thought when I was looking at the TestNG webpage was: “Hey, they must have their project on GitHub” and they did! The only problem was this:

Welcome to TestNG

“Please note that even though the .zip distribution contains the TestNG sources,
you will not be able to build the software with them because we decided
not to include the external jar files in order to keep the size down.

If you want to build TestNG, please read the instructions at


And I panicked, I searched their webpage, searched Google, searched under my desk, damn I searched everywhere! But I suck at searching and I know that so I tried something different and cloned the project anyway, turns out it was only the lib directory missing and all I had to do was to download the next files: ant-1.6.5.jar, bsh-2.0b4.jar, guice-2.0.jar, jcommander-1.13.jar, junit-3.8.1.jar, snakeyaml-1.6.jar and place them in the lib directory in the testing project.
After checking the files I noticed and it says:

You need to install ivy in order to build.
You can either download it from the ivy site or more simply, 

copy ivy-2.1.0.jar included in the root directory to ~/.ant/lib.
After this, "ant" will build the distribution and run the tests.
Yes, I feel stupid.

Building TestNG (For dummies like me)
First of all download eclipse for java here after that open it and go to the eclipse marketplace.

After that look for the git plugin called egit Install it and select import


You’ll get a message like this, click on the Clone… button

You should get an image like this, copy this URI and paste it there, use your GitHub 

User and password to enter.

Egit will show you some branches, just pick the master and click on next

Then you’ll be asked for a location for your repository, choose one and click on Finish

And that should leave you something like this:

Select it and click on next to get to this window

After that you’ll be asked if you want to import a project from a repository, just click on next and we should be over.

But wait! Do you see that red exclamation point? That means trouble

No biggie, all you have to do is download these files ant-1.6.5.jar, bsh-2.0b4.jar, guice-2.0.jar, jcommander-1.13.jar, junit-3.8.1.jar and snakeyaml-1.6.jar, place them in a lib folder and move them to your project (you can drag it to the workplace).

After that you’ll just get a lot of warnings but that’s normal . 

Ivy resolve dependences

Could also resolve all the dependencies using ivy, you would have to build ivy.xml, and download the library ivy.jar, but as testing using ivy to resolve their dependencies, both files are already at the root of the project:

File ivy.xml

<ivy-module version="2.0">
<info organisation="org.testng" module="testng" revision="5.14.3beta"/> <dependencies> <dependency org="ant" name="ant" rev="1.6.5" /> <dependency org="junit" name="junit" rev="3.8.1" /> <dependency org="org.beanshell" name="bsh" rev="2.0b4" /> <dependency org="" name="guice" rev="2.0" /> <dependency org="org.yaml" name="snakeyaml" rev="1.6" /> <dependency org="com.beust" name="jcommander" rev="1.13" /> </dependencies> </ivy-module>

This is the archive ivy.xml of testing, as you can see here are all the dependencies of the project, would only have to install the ivy plugin to eclipse (also found in the marketplace), right click on the project and find the resolve button on the ivy menu With this ivy will download the necessary dependencies of the project.

Running the build To run it from its build you have to know how TestNG works so after a lot of reading I found the core of TestNG, it’s located in and all the magic happens right there. To run it I did something pretty simple:
public static void main (String [] args){ TestNG runTest = new TestNG(); Class [] tests = new Class[1]; tests[0]=TestingSomething.class; runTest.setTestClasses(tests);; }
What I’m doing here is telling the build which classes I want to test and running the tests, here’s the testfile:
import org.testng.Assert; import org.testng.annotations.Test; public class TestingSomething { @Test public void testThis(){ Assert.assertEquals(0, 1); } }

Just a simple test
And this is what I got:
[TestNG] Running: Command line suite =============================================== Command line suite Total tests run: 1, Failures: 1, Skips: 0 ===============================================
As you can see I’m just telling TestNG which classes I want to test and running it with run() and the first thing it does its initialize the suites, configuration, default listeners, checks for tags and makes sure that test names aren’t the same in the same suite. After that it starts counting the time in milliseconds Example whit groups
First we prepare the test classes with different groups: Showing Test1 class:
import org.testng.annotations.Test; import org.testng.Assert;. public class Test1 { @Test(groups = { "functest", "checkintest" }) public void testMethod1() { } @Test(groups = {"functest", "checkintest"} ) public void testMethod2() { Assert.assertEquals(true, false); } @Test(groups = { "functest" }) public void testMethod3() { } }
Once we have the class we can execute the command:
Java org.testng.TestNG -groups functest -testclass Test1 Or you can create a class that calls the main method of TestNG: Here’s TestMain.class:
import org.testng.TestNG; public class TestMain { public static void main(String[] args) { String [] arg= new String[4]; arg[0]="-groups"; arg[1]="functest"; arg[2]="-testclass"; arg[3]=""; TestNG.main(arg); } }
Note: You could also use this example to start debugging this library. JCommander
As you can see we’re passing arguments at console level, but how does TestNG conver those methods in instructions for the class? It uses the JCommander library, with this library you can easily create a configuration class, JCommander receives the parameters sent by console and send them to your configuration class, this way you only have to call the attributes that are in said class and manipulate them without parsing any String array (args[]) Example of a configuration class for JCommander:
import com.beust.jcommander.Parameter; public class JCommanderExample { @Parameter public List<String> parameters = Lists.newArrayList(); @Parameter(names = { "-log", "-verbose" }, description = "Level of verbosity") public Integer verbose = 1; @Parameter(names = "-groups", description = "Comma-separated list of group names to be run") public String groups; }
This way you only have to declare an instance of said class and pass it as a parameter of a new object of jcommander next to the args[]
JCommanderExample jct = new JCommanderExample(); new JCommander(jct, argv);
With this simple code you’ll have an object called “jct”,this have , it has kept all the parameters. The community The community isn’t really big, with 2696 members on the TestNG users Google group and 210 in the dev group and a GitHub repository; most of the fixes are done by the author, he also checks and responds in record time (I sent him a message and he responded me in 12 minutes). Probably that was part of the TestNG success, listening to their users, even though he’s working on other projects he keeps helping the TestNG community and working on it, be it with a simple question or a difficult fix Cedric will help. The homepage has a pretty good documentation and they even have Javadocs, help on how to migrate from JUnit and small tutorials for different plugins. There’s a miscellaneous section with a lot of articles and presentations about TestNG. The only thing that I didn’t like was that the homepage was a little bit old (the last document was from 2008) but that’s it. Things that I learned When I was trying to run it I started reading a lot of the source code, looking for a class to use in my main method but if you’re smart (I proved again that I’m not) all you have to do is install the TestNG plugin and debug a test with it to see all the methods it uses, that way you’ll save a lot of time trying to understand how something works. What did I learn then? Debug is your friend; I had used it before to fix errors and things like that but never to understand a program. I was hearing an interview with Cedric Beust and some guy said: “Why didn’t you try to fix JUnit?” and he replied: “Do you really think I didn’t?” At first I thought he made TestNG because he hated JUnit but after hearing this it made me realize that he did it because he couldn’t change it; He couldn’t adapt to it nor change it but he could make a better one and that reminded me of: “The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man.” - George Bernard Shaw What did I learn then? I learned that innovation isn’t only to fix problems; it’s also to adapt things to your needs. During the same interview he talked about TDD and unit testing about how many programmers worried more about code coverage than delivering quality code, he also talked about the definition of unit testing and its rules. What did I learn then? There’s no real need to test everything (and Cedric isn’t the only one saying this). But knowing what to test and what not is something pretty difficult and I’m pretty sure that I’ll learn it over time. AND! That sometimes you have to break the rules (a good unit test doesn’t talk to the database, etc.) to get better tests. I also learned about annotations because even thought I used them before I didn’t really bothered in checking what where they and how they work until now. I learned that annotations are like tags (they are meta-tags) and you can pass them parameters, use them at source level (ignored by the compiler), at compile time (these are the usual java annotations like @Override) or at runtime (wich is pretty much what we want) by stating it with the retention policy:
@Retention(RetentionPolicy.RUNTIME) public @interface Test_Retention { String doTestRetention(); }
You can even inherit annotations!