Top Tools and Techniques to Enhance Your Test Suite

[t-richards]

Automated testing is a fundamental aspect of the development workflow for Rubyists. But how do you decide what to test? What happens when your tests run slowly and feel like they're just getting in the way? The following links are some resources which I have found to be helpful in answering these questions.

Minimal testing​

I am a testing minimalist. I want the lowest-cost testing option that can still provide me with some confidence.

In The Magic Tricks of Testing, Sandi Metz covers the secrets of writing stable tests and the philosophy of how to find the right testing balance.

Maximal testing​

I am a testing maximalist. Every line of code I write must have 100% test coverage, and then some. ...and not just the phony kind. I mean real honest coverage.

Mutation testing is the process of introducing intentional mistakes into your code to see whether your tests can catch them. As you continue down the path of mutation testing, you'll experience improvements for both your codebase and your test suite: you can eliminate code that is doing too much, as well as improve tests that aren't doing enough.

The mutant gem provides this functionality for Ruby, and is free for open source use.

GitHub - mbj/mutant: Automated code reviews via mutation testing - semantic code coverage.

Automated code reviews via mutation testing - semantic code coverage. - mbj/mutant

github.com

Faster tests​

My test suite is running slow. Like seriously many many hours. I have to parallelize it in CI otherwise it would take a full day to finish. What gives?

Slow tests are always frustrating, and there typically isn't a one-size-fits-all fix for this.

The TestProf set of tools can help you identify bottlenecks in your test suite, as well as provide some guidance for how to fix the problems once you figure out where they are.

TestProf

Ruby tests profiling and optimization toolbox

test-prof.evilmartians.io

 

[brb3]

Very valuable rundown and resources as always, @t-richards .

How do you see fuzzing fitting into tests?
Do you use fuzzing at all?
Do you consider that a separate component completely?
Do you incorporate fuzzing into a recurring task (maybe part of CI?) or as a one-off measure?

 

[ryanmaynard]

High value, high signal posts like this are why I am beyond excited about the launch of this forum (Thank you again @StrangeWill!). I hope we get 1000 more posts like this. Thank you, @t-richards!

+1 for Mutant. Adding in a few details of some of its helpful options:

1. Target Specific Classes or Methods

bundle exec mutant --include lib --require your_file --use rspec YourClass#your_method

2. Exclude Certain Mutations

bundle exec mutant --include lib --require your_file --use rspec YourClass#your_method

3. Parallel Execution

bundle exec mutant -j4 --use rspec YourClass

Where -j4 specifies to run 4 jobs in parallel. Adjust this number based on your CPU's capabilities.

4. Utilize Custom Matchers
Create custom matchers for your tests to make them more expressive and handle mutations better. They can provide clearer tests and capture more edge cases.

5. Leverage Factory Methods Carefully
When using factories (e.g. FactoryBot), ensure they are not too complex. Simplify factories so that the state and behavior being tested can be easily understood and managed within your tests.

6. Iterative Approach
Adopt an iterative approach for mutation testing:
- Initial Run: Execute Mutant on your critical code paths
- Review Mutations: Analyze survived mutants and identify where your tests need improvement.
- Improve Tests: Enhance your tests and refactor your code
- Repeat: Rerun Mutant to check the effectiveness of the improvements

Just a few things that may be obvious to some but hopefully helpful to others, and of course if anyone sees corrections or improvements in anything I said, do please speak up and correct/improve. Loving this dialogue!

 

[t-richards]

@brb3 fuzzing is definitely a useful strategy to ensure the robustness of your software. However, the landscape of fuzzing in Ruby is still evolving compared to some other languages, where support for fuzzing is more mature.

Personally, I am a big fan of fuzzing. I believe it to be most helpful when you are writing a low-level parser or interpreter of some kind, and you want it to behave well (that is, not crash the entire process) even in the face of the most untrusted, adversarial inputs. Of course, you could still use fuzzing to validate higher level logic and ensure that malformed inputs don't cause your code to raise unnecessary exceptions, but this is less common in my opinion.

One of the more promising Ruby fuzzing tools that I've seen recently is ruzzy, which uses libFuzzer under the hood:

GitHub - trailofbits/ruzzy: A coverage-guided fuzzer for pure Ruby code and Ruby C extensions

A coverage-guided fuzzer for pure Ruby code and Ruby C extensions - trailofbits/ruzzy

github.com

I've only had a chance to try using this tool on a handful of my favorite methods. These are the selection criteria I used to determine which ones to try:

• Find a method where you are handling raw, untrusted user input. Let ruzzy generate some garbage data and see how it holds up.
• Find a method where you absolutely cannot afford to tolerate a crash or an unexpected exception of any kind. Again, what happens if you feed that method some junk?

As for how this fits into the overall Ruby automated testing ecosystem, I certainly hope to see more adoption of ruzzy and tools like it in the near future. Generally speaking, I think a good CI strategy for the most robust Ruby apps and Rubygems might mimic the patterns that we've seen with other language level fuzzers:

1. For each change that you intend to merge into your main branch, it must survive a fuzz test for 5 minutes without crashing.
2. On some designated schedule, perhaps every week or every day, run a more extensive fuzz test with a longer duration. Any unique crashes should be tracked and fixed with an elevated sense of urgency.