Problems remote debugging with gdbserver
I am having trouble getting remote debugging to work with CLion. First, the issue I have is that the program I am trying to debug must run with root privileges. Second, it is started as a suite of other programs that together form a functional node, so it's not really possible in this scenario to start it by itself. Because of these two limitations, and the limitation that CLion cannot attach to a running process with root privileges unless CLion itself has root privileges, I decided to use gdbserver running via sudo and attaching to the running process. This is running on the same machine, however, having the ability to attach to a node in a remote cluster would be extremely useful.
To further complicate the situation, the method where I need to debug is located in a .so (all of this is on Linux). The main program is an executable that daemonizes itself and periodically calls the method to be debugged.
The setup is this
Process runs and daemonizes itself. The executable is run from /opt/PROJECT/bin. The .so is located in /opt/PROJECT/lib (their install directories)
I start gdbserver with this command: sudo gdbserver --attach <ip>:5432 PID
I setup a GDB Remote Debug configuration with
target: <ip>:5432
symbol file: tried several options here from pointing to the built .so on the "local" machine to the "remote" .so in the install directory (given above)
sysroot: tried everything I can think of here
path mappings: even added map from the remote /opt/PROJECT/lib to the local source location
I start gdbserever w/o any issues. I then start debugging and see that it connects. From that point, I cannot break. I have tried the set solib-search-path to the path to the .so on the remote system by creating a .gdbinit file. However, it is really not clear where this file should go, especially in the above configuration with as many different directories involved. Should it be in one of the directories specified by the sysroot property?
Any help would be appreciated. I know I can run CLion as root, but I don't want to do that if I don't have too.
Thanks.
Please sign in to leave a comment.
Hi!
>CLion cannot attach to a running process with root privileges
Feel free to comment or upvote https://youtrack.jetbrains.com/issue/CPP-7067 in order to get updates. See https://intellij-support.jetbrains.com/hc/en-us/articles/207241135-How-to-follow-YouTrack-issues-and-receive-notifications if you are not familiar with YouTrack.
>Should it be in one of the directories specified by the sysroot property?
By default the gdb in CLion uses default
~/.gdbinit.About the settings for remote debug via GDB/gdbserver (https://www.jetbrains.com/help/clion/remote-debug.html) for a general case:
Symbol file is a file separate from the executable itself which contains program’s debugging information. If you have the binary with the DWARF debug info on the target, you don't need to specify anything in the Symbol File field. If you have the stripped ELF on the target, you must have the Symbol File specified in CLion.
Sysroot (http://visualgdb.com/gdbreference/commands/set_sysroot ) : GDB needs to have access to the shared libraries that are used by your project; this can be accomplished either by providing copies of the libraries on the host system, or by asking GDB to automatically retrieve the libraries from the target. In the second case you don't need to specify anything in the Sysroot field. In the first case (when you want the local copies of the libraries to be used), in the Sysroot field you need to specify the local directory that contains target libraries in the subdirectories corresponding to target paths.
Path mappings is the same as GDB substitute-path - https://sourceware.org/gdb/onlinedocs/gdb/Source-Path.html. If you cross-compile on the host and them move it to the target, you don't need to specify "Path mappings".
So if you build the non-stripped executable and library on the same machine where you run them, you can leave the described fields empty, I think.
Thank you! Your comments were helpful and it now works. I believe the sysroot entry with everything else blank is what the missing piece was. I do have some .gdbinit files sprinkled amongst different directories, so I am going to systematically remove them to see if they are needed or not. Once I have a repeatable scenario I am going to share this with the team.
Follow up:
So, I was able to break and step through the code. After I was done, I let it run. After a bit, a "Command timed out" message appeared on the debug console in CLion. gdbserver says the remote side, CLion, has terminated the connection.
Any explanation or way to prevent?
Thanks.
Please enable debugger logging as described here, reproduce the issue and send the resulting idea.log file to clion-support at jetbrains.com. Do not forget to disable debug logging after that. Note that logs might contain private user's information (like file paths and names).