GSOC [RFC]: Support for alternative tracing backends

[Harsha Sharma]

On Mon, Mar 26, 2018 at 1:38 PM, Eugene Syromiatnikov <esyr at redhat.com> wrote:
> On Sun, Mar 25, 2018 at 12:39:08PM +0530, Harsha Sharma wrote:
>> Strace Proposal (Google Summer of Code 2018)
>
>> ->Title of Proposal:
>> Support for alternative tracing backends
>> ->Suggested By:
>> Eugene Syromyatnikov
>> ->Abstract of Proposal:
>> 1. Add backend interface to allow for multiple alternative backends.
>> 2. Using gdbserver:
>> With implementation of catch syscalls in gdbserver which adds a new
>> QCatchSyscalls packet to enable 'catch syscall', and newstop reasons
>> "syscall_entry" and "syscall_return" for those events.
>> GDB can catch some or all of the syscalls issued by the debuggee, and
>> show the related information for each syscall. If no argument is
>> specified, calls to and returns from all system calls will be caught.
>> Basic implementation idea for gdbserver backend:
>> strace talks to gdbserver via gdbserver backend
>> strace sends packet: $vCont;c  (continue)
>> strace receives packet:T05syscall_entry:
>> 16;06:b0e2ffffff7f0000;07:68e2ffffff7f0000;10:27a9b0f7ff7f0000;thread:p2162.2162;core:5;
>> strace sends packet: $g (get registers)
>> strace receives packet: daffffffffffffff0000000000000000...
>>
>> I plan to use previous patches:[1], [2], [3]
>
> Note that the current state of the patch set is available at [1]. I think
> it is worth checking out various XXXs and TODOs there.

OK.Sure.

> [1] https://github.com/stanfordcox/strace/commits/gdbserver0
>
>> 3. Using ftrace:
>> kprobe, uprobe and kernel tracepoint scripts make use of ftrace - This
>> allows tracing kernel functions, stack tracing and debugging crash.
>> write to files in /sys/kernel/debug/tracing and reading output from
>> /sys/kernel/debug/tracing.
>> Detailed description:  [4].
>>
>> 4. Perf events:
>> Call  perf_event_open syscalls , kernel writes events to ring buffer
>> in user-space, read tracepoints from ring buffer.
>> Various ioctl like PERF_EVENT_IOC_ENABLE and PERF_EVENT_IOC_DISABLE
>> also act on perf_event_open() file descriptors, allowing enabling and
>> disabling the individual counter or event group specified by the file
>> descriptor argument respectively. [5]
>
> The main problem with ftrace/perf is that they can overflow the buffer
> and lose events. Do you plan to handle those situations?

Yes. Here are some of the ways this can be done:

1. Increase ring buffer size.
2. stop tracing with tracing_off for some error conditions. By writing
0 in trace_fd file descriptor,  recording events in ring buffer can be
stopped.
3. add trace event with TRACE_EVENT macro and call this new trace
function during system call handler inside kernel (to trace only the
syscalls)
4. Condensing events with DECLARE_EVENT_CLASS . It allows tracing
events which record same structured data.

However, I'll need to do good testing on above points.

>> [1]. https://lists.strace.io/pipermail/strace-devel/2017-February/005985.html
>> [2]. https://lists.strace.io/pipermail/strace-devel/2017-February/005986.html
>> [3]. https://lists.strace.io/pipermail/strace-devel/2017-February/005987.html
>> [4]. https://lwn.net/Articles/366796/
>> [5]. http://man7.org/linux/man-pages/man2/perf_event_open.2.html
>
>> ->Timeline
>> 1. 27th March - 22nd April : Getting familiar with various backends (ptrace, gdbserver, ftrace, perf and eBPF)
>> 2. 23rd April: Student Projects announced
>> 3. 23rd April - 14th May: Community bonding, discuss various possibilities of implementing alternative backends
>> 4. 14th May: Coding period starts
>> 5. 14th May - 24th May: Add backend interface to allow for multiple alternative backends
>> 6. 24th May - 10th June: Add gdbserver backend
>> 7. 10th June - 15th June: Add tests and test for multiple architecture and multiple personality support (First evaluation)
>> 8. 15th June - 20th June: Brushing up code for gdbserver backend and getting it merged (if possible)
>> 9. 20th June - 5th July: Work on adding support for ftrace backend
>> 10. 5th July - 12th July: Add tests and  similarly test for multiple architecture and multiple personality support (Second evaluation)
>> 11. 12th July - 27th July: Add support for perf backend
>> 12. 27th July -3rd August: Adding tests
>> 13. 3rd August - 14th August: Getting code merged and write some documentation for available backends with a TODO list
>> 14. 14th August: Final evaluation
>
> I would say that the schedule is overly optimistic.

Yes, you are right. This is my updated timeline:

5. 14th May - 24th June: Add backend interface to allow for multiple
alternative backends and add code for gdbserver and ftrace backend
6. 24th June - 14th July: Develop relevant test infrastructure for
testing with alternative backends (I think this should be a priority,
since proper testing is necessary.)
7. 14th July - 30 th July: Add code for perf backend
8. 30th July - 14th August: Getting code reviewed and merged.
9. 14th August: Final evaluation


> Regarding tests: one possibility is to run all (or the most of) the testsuite
> with a backend under test.  However, the relevant infrastructure has to
> be developed first.

Thanks for the suggestions and review.
If there is something more that I can add to my proposal, then let me
know as the deadline is close.
Thanks.

Regards,
Harsha Sharma