May 23, 2012
Crisis Management
It is around 2 AM when I am writing this but through some marvelous intervention and some persistent help, I handled three separate crisis today involving my computer.
The trouble started around 3:20PM when I figured I had a few minutes this afternoon to sort a few files in my Downloads directory. While I was the 2000 or so images and files in the directory, Ubuntu Unity pretty much choked. The mouse could still move, but everything else was frozen.
Crisis 1: The Deep Freeze
Linux has a very good record of never crashing. The applications that use Linux (especially Ubuntu's) is another story.
When in the course of events your computer freezes, DON'T PANIC! As those words of Douglas Adams hold true, your first step is to switch to another pseudoterminal, or pts. The current pts you are using is likely pts/7. This is where X or Gnome or KDE usually sets up shop. You switch ptses using CTRL+ALT+1 (you could also use CTRL+ALT+2 or CTRL+ALT+3 or etc.). You then login with your user name and password.
The next step is you need to find the troublemakers. Ideally, using ps -AHF would list all the processes that are going on the computer and with detail. To a lesser detail, you could use ps -AH. Either way, this list will spout off a long list of processes neatly structured thanks to the -H option but more complicated no-thanks to the -F option should you use it.
A couple of other general hacks for any Linux/UNIX/BSD newbie would be to use a pipe--that is, a chained command--to view things as a scrollable list (ps -AHF | less -eFMXR) or to look for the things you are looking for by using a pattern search (i.e. ps -AHF | grep "chromium-browser"). Knowing these is also helpful for other commands.
Anyway, back to the trouble makers. Zombies (or rather Zombie Processes) are a good place to start. If you use -F in the ps command, you will see a column marked S for "status". Anything that is marked R is running and S for sleeping is normal. But if you see Z, that is a sign of a zombie process. You can start by killing those using sudo kill pid where pid is the process ID number that corresponds with the PID column. But be warned! If their is a process you are trying to save, killing the zombies may not be the best of ideas. (See Crisis #2 later.)
The next ones you'll want to 86 would be the heavy users. Web Browsers have a nasty habit of this. Often the web browser will have a list of parallel processes going. To kill all of those at the same time use sudo killall -9 chromium-browser for example. You'll have to restart the browser when you get back to the regular pts, but it's much easier than opening the GUI program of the Memory manager or something like that.
Woo! Crisis solved right? Well, for most people yes. However, I my case, no. You see, when I typed in CTRL+ALT+7 to get back to the regular pts, it was about 4:00 PM. The clock on the terminal was still stuck at 3:20 PM.
Crisis 2: The Rescue Mission
Often times, we are tempted to reach for the power button to start things over. But, that would be foolish if you left things open, especially things that you didn't save yet in vim on whatever console emulator you are using.
Someone has finally fixe this problem. A young chap by the name of Nelson Elhage created a program for moving running processes to another pts called reptyr.
Now I know some of the more experienced programmers reading this are like "You could also use injcode or neercs" but we're not using those applications in this post.
In order for this important hack to work, you will also need a screen multiplexer program like screen or--my preference--tmux. Either program will do, but the latter has colors and can be run inside another instance of the program. Knowing both of these program is a must for anyone who wants to use these programs when there isn't a crisis going on. I'd love to explain how they work, but that is for another time.
If you don't have any of these programs installed, consider sudo apt-get install reptyr tmux or whatever else floats your boat (i.e. screen, injcode and/or neercs for the last two arguments).
Let's do a ps -AHF | less -eFMXR and look for the program we want retrieve. In the TTY column, lets say the program tmux is holding like five different windows that have bash going and within them maybe a few vim sessions.
In a normal situation, you could CTRL+Z and disown program from the current terminal. However, these programs that are running are not in the current terminal. So you will want to temporarily hack the kernel. That's right! I said the kernel. That thing you really shouldn't touch unless you are developing upon it or if there is an emergency.
The guy who developed this program had though offered the adivce of using sudo echo 0 > /proc/sys/kernel/yama/ptrace_scope would be a good work around for why distributions like Ubuntu won't let you move the set of processes. Firstly, the reason you can't change this is for security reasons. Making this permissible will open a hole in your kernel that could be exploited by some nay-do-well who could inject code and hijack the computer. (Hence the name of that one program, injcode.) But since this is an emergency and a problem with your own computer, you'll want to set things back to normal later when you are done. At any rate, the proper description of this issue will be addressed in Crisis #3
So you've identified the process you want to move. You've turned off the ptrace_scope for the moment. Now it's time to move it. reptyr -s pid will change the pts of that process, which we'll say was at pts/0 but is now at pts/12 or some other place.
Many of the resolutions I've found explained how to bring these processes back to life with screen using screen -ls followed by screen -x pid where the pid is the Process ID of the rescued screen, ssh, bash, vim, or whatever. (Notice, I didn't say tmux because putting a tmux inside a screen doesn't work. Another advantage to using tmux over screen.)
How I fixed it took a little bit of forethought. I first opened tmux using, well, tmux. I then turned off ptrace_scope temporarily, then I ran reptyr -s the_pid_I_wanted. I used tmux's console feature to run the following commands: CTRL+B then c for a new window; ps -AHF|less -eFMXR to look up the PIDs of the old parents that didn't have a pts (? was in the TTY column.); Killed those old processes.); then CTRL+B then :choose-session -t the_window_number_I_was_in then picked the set of windows that were not attached that I wanted.
If that was too complicated, I have decided to write a more terse explaination after this next sub-section.
So you've identified the process you want to move. You've turned off the ptrace_scope for the moment. Now it's time to move it. reptyr -s pid will change the pts of that process, which we'll say was at pts/0 but is now at pts/12 or some other place.
Crisis 3: The tee Party
As I stated in the previous section, sudo echo 0 > /proc/sys/kernel/yama/ptrace_scope did not work. I tried everything I could think of. I tried sudo sed -i "s/0/1/" and I tried using sudo vim. But the kernel would not allow this change, which would have reset itself on restart of the computer if by chance I didn't reset it later. (There's a more permanent way to set it, but I would not advise doing it.)
The work around used the tee command. Instead of pushing data in to a file through a standard output method (>), I used a pipe instead.
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
A recap summary
That was a long explaination so let's recap.
CTRL+ALT+1 # Switch to a blank terminal and log in. $ ps -AH | less # Look up the PID and TTY (pts) of the process you want to save and the PIDs you want to kill. $ tmux # Open the multiplexer program $ echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope # Temporarily disable ptrace_scope in the kernel. $ reptyr -s save_me_pid # Have reptyr move the process you want to save. (This PID's child processes will also be saved if there are any.) CTRL+B c # Create a new window in tmux $ sudo kill kill_bad_pids # Kill the old PIDs that previously manipulated the pid you want to save. CTRL+B :choose-session -t this_window_number # Envoke a menu to choose the detached set of windows the saved process represents. (Arrows move up and down the list. Return selects. Esc cancels.) $ echo 1 | sudo tee /proc/sys/kernel/yama/ptrace_scope # Re-enable ptrace_scope for security.
And you are back in business! Or at least get a chance to save stuff before rebooting the computer.
The Lifelines
One of the reasons this took solong was because an old piece of software called Privoxy prevented me from using lynx from looking up a solution. Privoxy sort of gummed up everything by setting a proxy to route Lynx to 127.0.0.1:8118 because there was an environment variable set. If this is a problem for you like it was for me, consider unset http_proxy if by chance http_proxy shows up when you env | grep "http_proxy". At any rate, Lynx is a must have piece of software when you can't use Chrome, Firefox, or Opera.
Secondly, it helps to seek outside help. IRC is the place to do that if you are a geek. The best IRC software hands down is IRSSI. I think I may have mentioned that in the past, but it's worth repeating. ##linux is more helpful than #bash in case you are wondering.
Well, that's it. I stood up all night to write this piece. But before I go I want to mention, I'm on Google+ now. So in case LinkedIn wasn't enough to snoop around on what I do when I'm not working on stuff, there you go.. Hopefully, the rest of the day will be better.
