test: Add a way to detect a test that breaks another

When running unit tests, some may have side effects which cause a
subsequent test to break. This can sometimes be seen when using 'ut dm'
or similar.

Add a new argument which allows a particular (failing) test to be run
immediately after a certain number of tests have run. This allows the
test causing the failure to be determined.

Update the documentation also.

Signed-off-by: Simon Glass <sjg@chromium.org>

1 files changed, 69 insertions, 0 deletions

diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst
index 8e42a32afb9..bfd3bdb9270 100644
--- a/doc/develop/tests_sandbox.rst
+++ b/doc/develop/tests_sandbox.rst
@@ -143,6 +143,75 @@ For example::
    Test dm_test_rtc_reset failed 3 times
 
 
+Isolating a test that breaks another
+------------------------------------
+
+When running unit tests, some may have side effects which cause a subsequent
+test to break. This can sometimes be seen when using 'ut dm' or similar.
+
+You can use the `-I` argument to the `ut` command to isolate this problem.
+First use `ut info` to see how many tests there are, then use a binary search to
+home in on the problem. Note that you might need to restart U-Boot after each
+iteration, so the `-c` argument to U-Boot is useful.
+
+For example, let's stay that dm_test_host() is failing::
+
+   => ut dm
+   ...
+   Test: dm_test_get_stats: core.c
+   Test: dm_test_get_stats: core.c (flat tree)
+   Test: dm_test_host: host.c
+   test/dm/host.c:71, dm_test_host(): 0 == ut_check_delta(mem_start): Expected 0x0 (0), got 0xffffcbb0 (-13392)
+   Test: dm_test_host: host.c (flat tree)
+   Test <NULL> failed 1 times
+   Test: dm_test_host_dup: host.c
+   Test: dm_test_host_dup: host.c (flat tree)
+   ...
+
+You can then tell U-Boot to run the failing test at different points in the
+sequence:
+
+   => ut info
+   Test suites: 21
+   Total tests: 645
+
+::
+
+   $ ./u-boot -T -c "ut dm -I300:dm_test_host"
+   ...
+   Test: dm_test_pinctrl_single: pinmux.c (flat tree)
+   Test: dm_test_host: host.c
+   test/dm/host.c:71, dm_test_host(): 0 == ut_check_delta(mem_start): Expected 0x0 (0), got 0xfffffdb0 (-592)
+   Test: dm_test_host: host.c (flat tree)
+   Test dm_test_host failed 1 times (position 300)
+   Failures: 4
+
+So it happened before position 300. Trying 150 shows it failing, so we try 75::
+
+   $ ./u-boot  -T  -c "ut dm -I75:dm_test_host"
+   ...
+   Test: dm_test_autoprobe: core.c
+   Test: dm_test_autoprobe: core.c (flat tree)
+   Test: dm_test_host: host.c
+   Test: dm_test_host: host.c (flat tree)
+   Failures: 0
+
+That succeeds, so we try 120, etc. until eventually we can figure out that the
+problem first happens at position 82.
+
+   $ ./u-boot  -T  -c "ut dm -I82:dm_test_host"
+   ...
+   Test: dm_test_blk_flags: blk.c
+   Test: dm_test_blk_flags: blk.c (flat tree)
+   Test: dm_test_host: host.c
+   test/dm/host.c:71, dm_test_host(): 0 == ut_check_delta(mem_start): Expected 0x0 (0), got 0xffffc960 (-13984)
+   Test: dm_test_host: host.c (flat tree)
+   Test dm_test_host failed 1 times (position 82)
+   Failures: 1
+
+From this we can deduce that `dm_test_blk_flags()` causes the problem with
+`dm_test_host()`.
+
 Running sandbox_spl tests directly
 ----------------------------------