ptrace is very commonly used in software protections schemes, because it can be very effective. I see 3 levels of ptrace based protection, 2 of which can be addressed with gdb in some way.
This is an almost trivial anti-debugging method. There is a single ptrace() call in the executable. Consider this detect_ptrace.c
#include <stdio.h>#include <sys/ptrace.h>int main() {    if (ptrace(PTRACE_TRACEME, 0, 1, 0) == -1) {        printf("Debugger\n");    } else {        printf("Normal\n");    }    return 0;}If the application is being debugged, ptrace returns -1 and is detected in that way by the above code.
$ gcc detect_ptrace.c $ chmod +x a.out $ ./a.out Normal$ gdb a.outGNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git[ ... ](gdb) rStarting program: /home/jan/Downloads/a.out Debugger[Inferior 1 (process 16137) exited normally](gdb) quit$ This is easy to circumvent in gdb:
$ gdb a.outGNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git[ ... ](gdb) catch syscall ptraceCatchpoint 1 (syscall 'ptrace' [101])(gdb) commands 1Type commands for breakpoint(s) 1, one per line.End with a line saying just "end".>set $eax = 0>continue>end(gdb) rStarting program: /home/jan/Downloads/a.out Catchpoint 1 (call to syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:4545  ../sysdeps/unix/sysv/linux/ptrace.c: No such file or directory.Catchpoint 1 (returned from syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:4545 in ../sysdeps/unix/sysv/linux/ptrace.cNormal[Inferior 1 (process 16262) exited normally](gdb) What happens here is that catch syscall ptrace sets a breakpoint on ptrace. gdb allows for execution of a list of commands upon entering a breakpoint. Since the catch command created the first catchpoint, we can enter these commands for catchpoint 1 with commands 1. set $eax = 0 sets the return value. The ptrace function call would set $eax to 0 if successful and -1 if it fails. continue means just that - continue program execution. end tells gdb that we are done entering commands.
Note that we actually break twice, once on entering the ptrace syscall and once on exit.
This is tougher, especially if the code is obfuscated. Some binaries execute ptrace more than once, checking that the first time the return code is 0 and that future calls are -1. Often this check is obfuscated to various degrees. Consider this detect_ptrace.c
#include <stdio.h>#include <sys/ptrace.h>int main() {    if (ptrace(PTRACE_TRACEME, 0, 1, 0) == -1) // first call        {        printf("Debugger (first check)\n");        }     else         {        if (ptrace(PTRACE_TRACEME, 0, 1, 0) == -1) // second call            {            printf("Normal\n");            }         else             {            printf("Debugger (second check)\n");            }        }return 0;}$ gcc detect_ptrace2.c jan@jan-HP-ENVY-17-Notebook-PC:~/Downloads$ ./a.out Normaljan@jan-HP-ENVY-17-Notebook-PC:~/Downloads$ gdb a.outGNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git[ ... ](gdb) rStarting program: /home/jan/Downloads/a.out Debugger (first check)[Inferior 1 (process 16818) exited normally](gdb) quit$ gdb a.outGNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git[ ... ](gdb) catch syscall ptraceCatchpoint 1 (syscall 'ptrace' [101])(gdb) commands 1Type commands for breakpoint(s) 1, one per line.End with a line saying just "end".>set $eax = 0>continue>end(gdb) rStarting program: /home/jan/Downloads/a.out Catchpoint 1 (call to syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:4545  ../sysdeps/unix/sysv/linux/ptrace.c: No such file or directory.Catchpoint 1 (returned from syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:4545 in ../sysdeps/unix/sysv/linux/ptrace.cCatchpoint 1 (call to syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:4545 in ../sysdeps/unix/sysv/linux/ptrace.cCatchpoint 1 (returned from syscall ptrace), 0x00007ffff7afb93f in ptrace (request=PTRACE_TRACEME) at ../sysdeps/unix/sysv/linux/ptrace.c:4545 in ../sysdeps/unix/sysv/linux/ptrace.cDebugger (second check)[Inferior 1 (process 16856) exited normally](gdb) This can be addressed by first determining the address of the first ptrace call, and then writing a script that checks for that address. Here an example for an executable that has the first ptrace call at 0x4d55dc which then proceeds to print out the addresses of all the other ptrace calls:
catch syscall ptracecommands 1 if ($rip) == 0x4d55dc  print "yep"  print $rip  set $rax = 0  continue else  print "nope"  print $rip  set $rax = -1  continue endendIn that particular example, I was able to take a list of the addresses where ptrace was called and tell ghidra that that was code:
Some advanced protection systems fork a thread which then tries to attach to the main thread as a debugger. This will fail if gdb is already attached as a debugger. And gdb will not be able to attach to a thread that has done this already. nanomites protection goes one step further, replacing some of the code with instructions that trip the debugger. This debugger then performs some work before returning control to the child. This type of protection needs a more advanced approach, manual unpacking, PANDA tracing or some such thing. But that is a topic for another time.