Sending signals on syscalls

[Seraphime Kirkovski]

Hello straces devs !

Recently, I had to do some reverse engineering on a malware for 
a somewhat exotic platform. As the malware had its .text encrypted my 
only possibility was strace. As always, it helped me to
understand the binary, but after I knew what it did, I couldn't do much 
more because I couldn't see the decrypted code section. What I would 
have liked to do is send a coredump-ing signal when I think the code is 
completely decrypted, i.e. before a call to munmap, after an open call 
or something like this, or simply stop the process in order to attach 
gdb. (This isn't always possible: often, malware writers fork() before
the main routine, which makes it more difficult to attach a debugger, as 
the pid changes, furthermore, if the text section is not decrypted the 
debugger would mess up checksums/keys/whatever.)

So I thought of extending strace like this:

	strace -e sigonsys=<before|after>:<SYSCALL>:<SIG> ./a.out

sigonsys: specifies the signal SIG to be sent before or after a syscall 
SYSCALL is done.

Example:

	strace -f -e sigonsys=after:open:SIGSEGV ./a.out

This sends a SIGSEGV after a call to open(2).

I've already taken a shot at it. And I've identified some limitations
that
	1) probably cannot be overcome from userpace
	2) are due to the racy nature of what I'm trying to do
	3) show some flaws in the kernel

First, the before parameter doesn't change anything in practice. In most 
cases the offending syscall will be executed, checking at the very end 
of the kernel procedure whether there are any pending signals. This 
yields some strange results. For instance,

	int main(void)
	{
		puts("hello");
	}

run with strace -e sigonsys=before:write:SIGSEGV, gives the following 
result:

	...
	write(1, "hello\n", 6) hello

	= -ERESTARTSYS

That it is, the syscall succeeds, it writes "hello" to stdout and before
returning to userspace it checks for pending signals, there is one, so 
it returns ERESTARTSYS, which is apparently stupid.

Another problem I found is related to the fact that signals are not 
delivered immediately. Consider the following program

	int main(void)
	{
		puts("aaaa");
		puts("bbbb");
	}

Strace outputs:

	..
	write(1, "aaaa\n", 5) aaaa = -ERESTARTSYS
	bbbb
	--- SIGNAL SIGSEGV ---

Or even worse,

	int main(void)
	{
		puts("aaaa");
		_exit(0);
		puts("bbbb");
	}

when run with

	strace -e sigonsys=before:write:SIGSEGV ./a.out

yields as expected:

	write(1, "aaaa\n", 5) aaaa = -RESTARTSYS
	--- SIGNAL SIGSEGV ---

But when piped like so

	strace -e sigonsys=before:write:SIGSEGV ./a.out | less

gives:
	group_exit(0)  = ??? (no write at all)


( I ran those examples on x86_64 and 4.7.0-1 kernel )

That being said, I think this option may help kernel developers as well.

What are your thoughts on extending strace like this ? Is it worth it ?
Do you have any ideas how I may overcome some of these difficulties ?
Currently, I modified the sources so the signal is send through 
ptrace(<GET|SET>SIGINFO... and ptrace_restart afterwards. I tried adding
an additional kill(2), but that didn't change anything.

Have a good Christams Eve,
Seraphime Kirkovski