Writing tests - Clippy Documentation

# Testing

Developing lints for Clippy is a Test-Driven Development (TDD) process because our first task before implementing any logic for a new lint is to write some test cases.

## Develop Lints with Tests

When we develop Clippy, we enter a complex and chaotic realm full of programmatic issues, stylistic errors, illogical code and non-adherence to convention. Tests are the first layer of order we can leverage to define when and where we want a new lint to trigger or not.

Moreover, writing tests first help Clippy developers to find a balance for the first iteration of and further enhancements for a lint. With test cases on our side, we will not have to worry about over-engineering a lint on its first version nor missing out some obvious edge cases of the lint. This approach empowers us to iteratively enhance each lint.

## Clippy UI Tests

We use UI tests for testing in Clippy. These UI tests check that the output of Clippy is exactly as we expect it to be. Each test is just a plain Rust file that contains the code we want to check.

The output of Clippy is compared against a `.stderr` file. Note that you don't have to create this file yourself. We'll get to generating the `.stderr` files with the command `cargo bless` (seen later on).

### Write Test Cases

Let us now think about some tests for our imaginary `foo_functions` lint. We start by opening the test file `tests/ui/foo_functions.rs` that was created by `cargo dev new_lint`.

Update the file with some examples to get started:

    #![warn(clippy::foo_functions)] // < Add this, so the lint is guaranteed to be enabled in this file

    // Impl methods
    struct A;
    impl A {
        pub fn fo(&self) {}
        pub fn foo(&self) {} //~ ERROR: function called "foo"
        pub fn food(&self) {}
    }

    // Default trait methods
    trait B {
        fn fo(&self) {}
        fn foo(&self) {} //~ ERROR: function called "foo"
        fn food(&self) {}
    }

    // Plain functions
    fn fo() {}
    fn foo() {} //~ ERROR: function called "foo"
    fn food() {}

    fn main() {
        // We also don't want to lint method calls
        foo();
        let a = A;
        a.foo();
    }

Without actual lint logic to emit the lint when we see a `foo` function name, this test will just pass, because no lint will be emitted. However, we can now run the test with the following command:

    $ TESTNAME=foo_functions cargo uitest

Clippy will compile and it will conclude with an `ok` for the tests:

    ...Clippy warnings and test outputs...
    test compile_test ... ok
    test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.48s

This is normal. After all, we wrote a bunch of Rust code but we haven't really implemented any logic for Clippy to detect `foo` functions and emit a lint.

As we gradually implement our lint logic, we will keep running this UI test command. Clippy will begin outputting information that allows us to check if the output is turning into what we want it to be.

### Example output

As our `foo_functions` lint is tested, the output would look something like this:

    failures:
    ---- compile_test stdout ----
    normalized stderr:
    error: function called "foo"
      --> tests/ui/foo_functions.rs:6:12
       |
    LL |     pub fn foo(&self) {}
       |            ^^^
       |
       = note: `-D clippy::foo-functions` implied by `-D warnings`
    error: function called "foo"
      --> tests/ui/foo_functions.rs:13:8
       |
    LL |     fn foo(&self) {}
       |        ^^^
    error: function called "foo"
      --> tests/ui/foo_functions.rs:19:4
       |
    LL | fn foo() {}
       |    ^^^
    error: aborting due to 3 previous errors

Note the failures label at the top of the fragment, we'll get rid of it (saving this output) in the next section.

> Note: You can run multiple test files by specifying a comma separated list: `TESTNAME=foo_functions,bar_methods,baz_structs`.

### `cargo bless`

Once we are satisfied with the output, we need to run this command to generate or update the `.stderr` file for our lint:

    $ TESTNAME=foo_functions cargo uibless

This writes the emitted lint suggestions and fixes to the `.stderr` file, with the reason for the lint, suggested fixes, and line numbers, etc.

Running `TESTNAME=foo_functions cargo uitest` should pass then. When we commit our lint, we need to commit the generated `.stderr` files, too.

In general, you should only commit files changed by `cargo bless` for the specific lint you are creating/editing.

> Note: If the generated `.stderr`, and `.fixed` files are empty, they should be removed.

## `toml` Tests

Some lints can be configured through a `clippy.toml` file. Those configuration values are tested in `tests/ui-toml`.

To add a new test there, create a new directory and add the files:

- `clippy.toml`: Put here the configuration value you want to test.
- `lint_name.rs`: A test file where you put the testing code, that should see a different lint behavior according to the configuration set in the `clippy.toml` file.

The potential `.stderr` and `.fixed` files can again be generated with `cargo bless`.

## Cargo Lints

The process of testing is different for Cargo lints in that now we are interested in the `Cargo.toml` manifest file. In this case, we also need a minimal crate associated with that manifest. Those tests are generated in `tests/ui-cargo`.

Imagine we have a new example lint that is named `foo_categories`, we can run:

    $ cargo dev new_lint --name=foo_categories --pass=late --category=cargo

After running `cargo dev new_lint` we will find by default two new crates, each with its manifest file:

- `tests/ui-cargo/foo_categories/fail/Cargo.toml`: this file should cause the new lint to raise an error.
- `tests/ui-cargo/foo_categories/pass/Cargo.toml`: this file should not trigger the lint.

If you need more cases, you can copy one of those crates (under `foo_categories`) and rename it.

The process of generating the `.stderr` file is the same as for other lints and prepending the `TESTNAME` variable to `cargo uitest` works for Cargo lints too.

## Rustfix Tests

If the lint you are working on is making use of structured suggestions, `rustfix` will apply the suggestions from the lint to the test file code and compare that to the contents of a `.fixed` file.

Structured suggestions tell a user how to fix or re-write certain code that has been linted with `span_lint_and_sugg`.

Should `span_lint_and_sugg` be used to generate a suggestion, but not all suggestions lead to valid code, you can use the `//@no-rustfix` comment on top of the test file, to not run `rustfix` on that file.

We'll talk about suggestions more in depth in a later chapter.

Use `cargo bless` to automatically generate the `.fixed` file after running the tests.

## Testing Manually

Manually testing against an example file can be useful if you have added some `println!`s and the test suite output becomes unreadable.

To try Clippy with your local modifications, run from the working copy root.

    $ cargo dev lint input.rs