[PATCH v2 2/2] ioctl console test v2

[Eugene Syromiatnikov]

On Tue, Mar 17, 2020 at 10:11:38PM +0530, Bran S wrote:
> On Mon, 9 Mar 2020 at 19:19, Eugene Syromiatnikov <esyr at redhat.com> wrote:
> >
> > On Mon, Mar 09, 2020 at 07:06:49PM +0530, Bran S wrote:
> > > I am interested in the Comprehensive test suite project.
> >
> > I'd suggest building strace with LCOV with different kernel headers
> > on different arches (./configure --enable-code-coverage && make check &&
> > make code-coverage-capture) and look at parts of the code (and branches)
> > that are not covered by tests.  Other possibility is to check out
> > codecov[1], but it's vaguely representative (it's built against rather
> > old kernel and only on x86_64).
> >
> > [1] https://codecov.io/gh/strace/strace
> 
> On running `make code-coverage-capture` I am getting below error.
> ----------
> lcov  --directory . --capture --output-file
> "strace-5.5.0.67.269b-coverage.info.tmp" --test-name
> "strace_5_5_0_67_269b" --no-checksum --compat-libtool  --rc
> lcov_branch_coverage=1 --gcov-tool "gcov"
> Capturing coverage data from .
> Found gcov version: 8.2.1
> Using intermediate gcov format
> Scanning . for .gcda files ...
> Found 313 data files in .
> Processing libmpers_m32_a-fetch_struct_stat64.gcda
> geninfo: ERROR: GCOV failed for
> /home/mybox/strace/libmpers_m32_a-fetch_struct_stat64.gcda!
> make: *** [Makefile:9957: code-coverage-capture] Error 29
> ----------
> 
> Searching about I found a suggestion to make sure that the gcc and
> gcov versions are same.
> My gcc and gcov version are same i.e. gcc (GCC) 8.2.1 20180801 and
> gcov (GCC) 8.2.1 20180801 respectively.
> Any ideas ?

Interesting.  What is the environment you're using?

> In the project "Comprehensive test suite" one would need to go through
> the files at codecov and find out untested portion of the code and
> then write tests to test those.

More or less, yes.  The spirit is to check various traces and use cases
(like, how strace behaves, what decoders print in various cases),
but the coverage is a good baseline for that; at least, lack of coverage
signals the lack of testing (the opposite, meanwhile isn't true:
the presence of line/branch coverage doesn't mean by itself that the code
is thoroughly tested, so the test cases that cover not previously covered
code usage patterns are very much welcome as well).

> For example: bpf.c is 61% covered

It seems that the system you've used for the testing skips
bpf-obj_get_info_by_fd* tests.

> , within bpf.c lines 126 and 127 are
> untested. So the task at hand would be to write tests so that those
> lines are executed.

Yes, the bare minimum is like that.  Note, however, that not all code
is easily reachable (for example, conditions for some out of bounds/memory
handling routines are rather difficult to set up).

> Is my understanding correct ? Please correct me where I am wrong and
> please point out what I have missed in understanding.

Yes, you're correct.  Meanwhile. I would suggest to also look into some
lower-hanging fruits, like untested ioctl decoders (ubi.c, fs_x_ioctl.c,
kvm.c, and some others are good candidates), unwinder-related stuff
(unwind*.c and mmap_cache.c), some syscall decoders, even (prctl.c and,
funnily, process.c, for example).