jonEbird

February 19, 2012

Automatically positioning and sizing windows with Devilspie

Filed under: adminstration,blogging,linux — jonEbird @ 1:48 pm

I often talk about my media PC in the living room and how I like to stream music via Pandora in the house. Since I am using Pithos, I can maximize the desktop real estate and enjoy whatever wallpaper suits my mood. In order to best accomplish this, I tend to resize Pithos so that it only shows the current album and song as well as move it off to the corner of the desktop. Although this only takes a moment to accomplish, I’ve grown tired of doing it each and every time I need relaunch Pithos. What I really want to be able to do is automatically resize and position the window immediately when I launch Pithos. Someone has to have solved this before and it’s bugging me enough that I need to solve this once and for all.

I started looking at wmctrl to solve my problem per the recommendation of Patrick Shuff. While familiarizing myself with the utility, I ironically stumbled upon a wmctrl tips page which included a list of other similar tools and the description of devilspie sounded exactly like what I needed:

devilspie is a window-matching utility. It can be configured to detect windows as they are created, and match the window to a set of rules. If the window matches the rules, it can perform a series of actions on that window.

My needs are modest. A resize and a positioning is probably accomplished with a simple declaration to set the window geometry. Let’s install it and start playing.

Fortunately I could install devilspie directly from yum and while the manpage is quite sparse it is packaged with a sufficient README that after a few trial and error iterations, I got a basic configuration working which just ran devilspie in debug mode and could report on the current windows opened and new ones being launched.

sudo yum -y install devilspie
[ ! -d ~/.devilspie ] && { mkdir -m 0755 ~/.devilspie; echo "(debug)" > ~/.devilspie/debug.ds; }
devilspie

That will continue to run in your terminal, so you may want to open another tab/terminal as we explore.

Since I am a devoted Emacs user, I certainly wasn’t disappointed to learn that devilspie uses s-expressions for it’s configurations. I also found a good unofficial devilspie documentation page written up by Gina Häußge which helps round out the education. Indeed, it looks like I’ll be able to solve my problem with devilspie. After meticulously positioning my Pithos window, I used xwininfo to record my current window geometry. I then created the simplistic config file “~/.devilspie/pithos.ds” as the following:

(if (is (application_name) "Pithos") (begin (geometry "481x163--20-17") (unshade) ))

Some more background on setting up your own configurations. If you notice when you launched devilspie, after the initial install, you can see the information on the current windows and their associated application names. For example, here is what I see for Pithos:

Window Title: 'Pithos - Bankrupt On Selling by Modest Mouse'; Application Name: 'Pithos'; Class: 'Pithos'; Geometry: 483x197+797+523

That should make it easy for coming up with rules to match application names, window names or classes when targeting your applications.

The frustrating thing for me was how each utility was reporting a different geometry value for the windows currently launched. Between xwininfo, devilspie and wmctrl they all had differing opinions on what I should use. You’d think that I could use what devilspie was telling me since that is the utility I’m indented on using but it didn’t work out for me. I’ve used xinwinfo for over a decade and I guess I’m sticking with it.

Finally, I added a final few touches to my desktop configuration before calling it complete. I created a simple shell script to relaunch Pithos indefinitely because occasionally I need to relaunch it but I actually never intend to leave it off: (See installing pithos with virtualenv if you want to see how I installed Pithos)

#!/bin/bash

PITHOS_HOME=~/pithos
PITHOS_VENV=~/pithos_venv

#--------------------------------------------------
source ${PITHOS_VENV}/bin/activate
cd $PITHOS_HOME
while :; do
    pithos
    sleep 1
done

My cheap way of launching apps indefinitely is to run them within my main screen session. I’ll create one for the Pithos restart script and another for devilspie. Here is an excerpt from my ~/.screenrc:

screen -t emacs     0  /usr/bin/emacs -nw
screen -t local     1  /bin/bash
screen -t local     2  /bin/bash
screen -t local     3  /bin/bash
screen -t local     4  /bin/bash
screen -t local     5  /bin/bash
screen -t synergy   10 /usr/bin/synergyc -f 192.168.1.23:6700
screen -t devilspie 11 /usr/bin/devilspie
screen -t pithos    12 ~/bin/run_pithos.sh

And all is well in my household again. Should something get borked with Pithos, perhaps due to being paused for too long or whatever, I can simply kill the window which triggers the restart script to launch another copy and devilspie will position and resize it perfectly for me. How will you use devilspie?

February 7, 2012

Python Memoize Decorator with TTL Argument

Filed under: blogging,linux,python — jonEbird @ 8:51 pm

I have been working on a troubleshooting effort on and off over the past couple of weeks. In the process of troubleshooting, I ended up writing a script to query and report on un-read buffered data for sockets across the system and today I wanted to correlate the sockets to a particular set of processes. That turned out to be not so bad knowing I could look down /proc/<pid>/fd/ for a list of the current file descriptors the particular process has open. Here is my winning Python implementation.

def get_pid_socket_nodes(pid):
    """Return a list of socket device numbers for the given process ID (pid)
    pid may be an integer or a string
    Akin to: ls -l /proc/
/fd/ | sed -n 's/^.*socket:\[\([0-9]*\)\]$/\1/p'
    """
    nodes = []
    for fd in os.listdir('/proc/%s/fd' % pid):
        link = os.readlink('/proc/%s/fd/%s' % (pid, fd))
        if link.startswith('socket:['):
            nodes.append(link[8:-1])
    return nodes

What I haven’t told you is the overall program may be scanning /proc/net/{tcp,udp} a lot and with each scan I’ll want to correlate values found with a particular process’s sockets. That means a straight forward implementation could mean calling my helper function get_socket_devices() at each interval for each process of interest. Also, the interval is currently controlled by a sleep interval specified by the caller. If the caller wants to monitor for socket information at an 0.1s sleep interval for five processes, I’ll be scanning /proc/<pid>/fd/ entries 50 times a second. Sure, it won’t crash the machine but it’s certainly a waste of resources assuming the processes you are interested in aren’t closing and opening sockets at an alarming rate.

I wanted to reduce the amount of /proc/<pid>/fd/ scans but I also did not want to clutter the code. “Ah ha, what I want is a memoize decorator”, I told myself, but I want to be able to specify a time-to-live (ttl) value to my decorator. Admittedly, I have only ever needed to write decorators without arguments, so this was a first for me. I also don’t write enough decorators to be able to write one correctly without looking at a sample implementation for a refresher. My implementation is basically a merge between Memoize and Cached Properties from the Python Decorator Library wiki page. (I also spent some time re-reading Bruce Eckel’s Decorator Arguments writeup as well as Elf Sternberg’s Decorators With Arguments writeup.)

class memoized_ttl(object):
    """Decorator that caches a function's return value each time it is called within a TTL
    If called within the TTL and the same arguments, the cached value is returned,
    If called outside the TTL or a different value, a fresh value is returned.
    """
    def __init__(self, ttl):
        self.cache = {}
        self.ttl = ttl
    def __call__(self, f):
        def wrapped_f(*args):
            now = time.time()
            try:
                value, last_update = self.cache[args]
                if self.ttl > 0 and now - last_update > self.ttl:
                    raise AttributeError
                #print 'DEBUG: cached value'
                return value
            except (KeyError, AttributeError):
                value = f(*args)
                self.cache[args] = (value, now)
                #print 'DEBUG: fresh value'
                return value
            except TypeError:
                # uncachable -- for instance, passing a list as an argument.
                # Better to not cache than to blow up entirely.
                return f(*args)
        return wrapped_f

Now let’s put the decorator into use and test it. (My testing is uncommenting the “print DEBUG” lines from the memoized_ttl decorator.) To use the decorator, you use the standard decorator calling syntax and specify your desired ttl. I am restricting the /proc/<pid>/fd/ scans to a minute interval which, based on my experience, will drop the load of the script down to below the radar.

@memoized_ttl(60)
def get_pid_socket_nodes(pid):
    """Return a list of socket device numbers for the given process ID (pid)
    pid may be an integer or a string
    Akin to: ls -l /proc/
/fd/ | sed -n 's/^.*socket:\[\([0-9]*\)\]$/\1/p'
    """
    nodes = []
    for fd in os.listdir('/proc/%s/fd' % pid):
        link = os.readlink('/proc/%s/fd/%s' % (pid, fd))
        if link.startswith('socket:['):
            nodes.append(link[8:-1])
    return nodes

And finally, here is what it looks like looking at a couple of processes of mine with open sockets:

>>> from buffered_sockets import *
>>> pid = 23283
>>> pid2 = 23279
>>> get_socket_devices(pid)
DEBUG: fresh value
['46287188']
>>> get_socket_devices(pid)
DEBUG: cached value
['46287188']
>>> get_socket_devices(pid2)
DEBUG: fresh value
['46287165']
>>> get_socket_devices(pid2)
DEBUG: cached value
['46287165']
>>> time.sleep(60)
>>> get_socket_devices(pid)
DEBUG: fresh value
['46287188']
>>> get_socket_devices(pid)
DEBUG: cached value
['46287188']
>>>

A handy decorator to keep around which I suspect will get a lot of mileage from other scripts I end up authoring. Aside from the helpful blog posts listed above, I also found Will McGugan’s Timed Caching Decorator page after writing this entire blog post. Apparently I need to improve my google searching skills because I certainly didn’t want to re-invent what others have already completed. On the other hand, if you spend this much time creating something for the first time, you tend to remember the lessons better. I’ll take that.

December 29, 2011

Installing emacs v24 on Fedora

Filed under: adminstration,blogging,emacs,linux,usability — jonEbird @ 10:05 pm

I’ve been reading about other people giving the yet to be release version 24 of emacs for some time now. When I decided to upgrade my systems to use v24, I was a bit surprised to not find anything about configuring a Fedora system to use v24 of emacs. Guess I gotta do it myself…

This tutorial is part editorial and part instructional. I thought it would be helpful to include some of the techniques I used to get emacs up and running quickly without needing to pull my hair for other’s edification.

After realizing I wasn’t going to be able to just grab a pre-built binary, I went looking for the official sources. I ended up finding the pretest download location. First step first, let’s pull down the latest emacs-24 tarball and extract it.

PRETEST_URL="http://alpha.gnu.org/gnu/emacs/pretest/"
FILENAME=$(curl -s ${PRETEST_URL} | sed -n 's/^.*a href="\(emacs-24.[0-9\.]*tar.gz\)".*$/\1/p' )
curl -o ${FILENAME} ${PRETEST_URL}${FILENAME}
tar -xzof $FILENAME
cd ${FILENAME%.tar.gz}

If that worked, you are now sitting in the extracted directory of the latest emacs-24 pretest source. Now for some instructional information. Any significantly large project will need a decent amount of development packages installed for a successful compile and that can be a pain to identify. Earlier I claimed that I didn’t pull my hair out which means I cheated. I grabbed the latest Fedora source rpm. I didn’t actually want to install the src.rpm but rather extract the emacs.spec file which will act like a blueprint for my build. I’m going to give you the answer later but if you’d like to know how to extract the specfile, try this:
Note: You do not need to do this step. Instructional only.

SRCRPM=~/Download/emacs-23.3-7.fc16.src.rpm
# Your SRCRPM may differ depending on what you end up downloading.
mkdir tmp && cd tmp
rpm2cpio $SRCRPM | cpio -ivd
sed -n -e 's/,/ /g' -e 's/^BuildRequires: //p' emacs.spec | xargs sudo yum -y install

Note the last command in that section was a command to install the necessary development packages for our build. Since I”m not requiring you to do that above, here is the command for you:

sudo yum -y install atk-devel cairo-devel freetype-devel \
  fontconfig-devel dbus-devel giflib-devel glibc-devel gtk2-devel \
  libpng-devel libjpeg-devel libtiff-devel libX11-devel libXau-devel \
  libXdmcp-devel libXrender-devel libXt-devel libXpm-devel \
  ncurses-devel xorg-x11-proto-devel zlib-devel librsvg2-devel \
  m17n-lib-devel libotf-devel autoconf automake bzip2 cairo texinfo \
  gzip GConf2-devel alsa-lib-devel desktop-file-utils python2-devel \
  python3-devel util-linux

The other part of the specfile you’ll typically want to look at, if you’re cheating like me, is the %build section. That is where you’ll find the actual commands used to configure and build the binaries. There I found the configure switches used so I don’t have to pick out which ones I’ll need. Again, just like figuring out the development packages, figuring out configure options can also be a chore. Let’s get to configuring, building and installing it now.

./configure --prefix=/usr/local/emacs24 --with-dbus --with-gif --with-jpeg --with-png \
  --with-rsvg --with-tiff --with-xft --with-xpm --with-x-toolkit=gtk
make
./src/emacs --version # Look good? The INSTALL doc suggests testing: ./src/emacs -Q
sudo make install

Well, that worked for me and hopefully it worked for you too. If you noticed, I used the --prefix=/usr/local/emac24 option above on my configure line which means everything got cleanly installed down it’s own separate base directory of /usr/local/emacs24. Since you won’t want to use that path explicitly each time you launch emacs, we’ll have to inform Fedora of our new altenative.

sudo alternatives --install /usr/bin/emacs emacs /usr/local/emacs24/bin/emacs 20000
sudo alternatives --install /usr/bin/emacsclient emacsclient /usr/local/emacs24/bin/emacsclient 20000

And there, we’re done. Congratulations. You have installed emacs version 24 on your Fedora system. Let me know if you’ve had any problems or have a better recommendation.

December 23, 2011

Installing Pithos on Fedora within a Virtualenv

Filed under: adminstration,blogging,linux,python,usability — jonEbird @ 12:40 pm

I listen to a lot of music while at home. I am a Pandora user and have been very happy with my Pandora One subscription now for over two years. The machine used for playing my music is what I call my “media PC”. It is called that because this machine sits in my entertainment stand and is connected to my Sony receiver via HDMI making the multimedia experience as good as I can get. If you put those two facts together, you can see that I am staring at my desktop a lot and I thought it would be nice to integrate my TV into rest of the decor of the house. I primarily do that by being very selective in finding desktop pictures and generally clearing off the desktop of any clutter. Think of the large 47″ LCD television as one big painting for the living room.

Which leads me to my one, sole problem with Pandora: I like to look up and read the Artist and Title of the track being played but I don’t want the browser to also consume my visual space. (I also don’t want to mess around with Adobe Air for the desktop version of Pandora) Enter Pithos. By this point, I should point out that my media PC is running Fedora Core 15 and I’m a Gnome user (let’s not talk about Gnome3). That is important because Pithos was written for gnome users.

Pithos is great. It’s a simple UI design, still allows for normal Pandora song control, easy drop-down for my stations, can still star (thumb’s up) songs all the while being small and unobtrusive. And now we are to the subject of this blog post: Installing Pithos on a Fedora Core machine.

This installation guide will follow my other guides in the same “copy & paste” format. That is, below you should be able to simply open a shell, copy the block of shell code and paste it into your terminal and be ready to launch Pithos. The one configurable item I left in there is whether or not you’d like to install Pithos within a virtualenv or not. I won’t go into detail about what virtualenv is for this discussion, but suffice to say that you’d choose it if you want to install Pithos in a alternative path that you own instead of /usr/local/bin/. Below, when you copy & paste the instructions to install Pithos, you can simply leave out the variable "I_LOVE_VIRTUALENV" or change the value from anything but “yes” to install the “normal” way. I choose to install via virtualenv to 1. keep my system site-packages clean and 2. also keep /usr/local uncluttered. When I do this, I mostly only have to worry about backing up my home directory between rebuilds.

Again: If you’d like to use virtualenv, keep the "I_LOVE_VIRTUALENV" variable set to “yes”.
Furthermore, using virtualenv you can control the env path via setting the VIRTUALENV variable. Some people have a separate directory for their virtualenv’s. E.g. VIRTUALENV=virtualenvs/pithos
(Copy and paste away!)

# Keep this variable to install within a virtualenv.
#   otherwise, skip this line or change from "yes" to anything else.
I_LIKE_VIRTUALENV="yes"
VIRTUALENV="" # Set this to control where your virtualenv is created
# --- Rest is pure copy & paste gold ---
sudo yum -y install python pyxdg pygobject2 \
  gstreamer-python notify-python pygtk2 dbus-python \
  gstreamer-plugins-good gstreamer-plugins-bad \
  bzr python-virtualenv
# FYI, those last two are not direct requirements but tools to complete this
cd; bzr branch lp:pithos pithos
if [ "${I_LIKE_VIRTUALENV}" == "yes" ]; then
  virtualenv ${VIRTUALENV:-pithos_venv}
  source ${VIRTUALENV:-pithos_venv}/bin/activate
  # The money shot... finger's crossed
  cd pithos; python setup.py install
else
  cd pithos; sudo python setup.py install --prefix=/usr/local
fi

And there you have it. A clean, aesthetically pleasing music experience. Enjoy.
Desktop Shot with Pithos

October 3, 2011

Customize Gnome Terminal in Gnome 3

Filed under: linux,usability — jonEbird @ 7:56 pm

I recently upgraded to Fedora 15 on both the laptop and the home server and I’m still getting used to Gnome 3. Through each frustrating change, I have decided to stick with it and try to learn the new ways to do things but I have been tempted more than a few times to just switch to using XFCE. Today’s challenge: Customizing gnome-terminal.

In the old days, the customizations I would do all involved edits that could be done by updating the default profile (Menu->Edit->Profiles). The main things I would update is setting my background to a certain percentage of transparency, disable scroll bar (because we’re all using GNU screen, right?), and disable the menubar by toggling “show menubar by default in new terminals” (alt-f is reserved for emacs forward-word and not File menu). The remaining item to update is the default geometry or size of the terminal when it’s launched.

Perhaps I should blog more because I found a much easier solution than I expected. When I decided to figure this out tonight I started by finding a nice tutorial for updating gnome configurations and I thought I was going to end up recommending the following command:
gsettings set org.gnome.desktop.default-applications.terminal exec-arg "'--geometry=120x35'"
Instead, while writing the previous paragraph I realized that within the profile settings there is an option now to “Use custom default terminal size”. I feel stupid but I think I’m just stubborn in my ways after using Gnome for over a decade.

July 10, 2011

64bit Google Chrome with Flash on Fedora – Take2

Filed under: adminstration,linux,usability — jonEbird @ 10:08 am

Early last year I wrote up my procedure on getting Chrome setup with Flash on a 64bit Fedora Core 13 build. It is now dated and I hope people are not still using it. However, if you like the copy & paste style of my direct installation guidelines, I thought I’d give an updated post after a recent Fedora Core 15 install.

First things first, we still need to create new Yum repositories like we did last time. The biggest difference between last time is I would no longer recommend installing the beta build of Chrome and Adobe has moved the 64bit flash player out of a lab project to an official release. The fine folks working on the Fedora project have lend their help in instructing people on how to setup Flash as well and you may want to check that page for a more detailed explanation on what we’re doing. In particular, I’ll be using a repository which is being hosted by Fedora member leigh123linux.

Here is the final copy & paste version: (Like last time, if you are already root, take out the “sudo”)

# Creating the Google repo
cat <<\EOF | sudo tee /etc/yum.repos.d/google.repo
[google64]
name=Google - x86_64
baseurl=http://dl.google.com/linux/rpm/stable/x86_64
enabled=1
gpgcheck=1
gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pub
EOF
# Slip this guy in here... who doesn't want the gchat plugin?
cat <<\EOF | sudo tee /etc/yum.repos.d/google-talkplugin.repo
[google-talkplugin]
name=google-talkplugin
baseurl=http://dl.google.com/linux/talkplugin/rpm/stable/x86_64
enabled=1
gpgcheck=1
EOF
# Now to install the latest flash release yum repo from "leigh123linux"
leighURL="http://www.linux-ati-drivers.homecall.co.uk/flashplayer.x86_64/"
latest_release=$(wget -qO- $leighURL |\
sed -n '/flash-release/s/^<LI><A HREF="\([^"]*\)".*$/\1/p' | sort -n | tail -1)
sudo yum -y localinstall --nogpgcheck ${leighURL}${latest_release}
# Actually installing Chrome and the Flash player
sudo yum -y install google-chrome-stable google-talkplugin flash-plugin
# Creating a plugins directory
[ ! -d /opt/google/chrome/plugins ] && sudo mkdir /opt/google/chrome/plugins
sudo ln -s /usr/lib64/flash-plugin/libflashplayer.so /opt/google/chrome/plugins/

If you noticed, I slipped in the Google chat plugin. That is the necessary RPM for enabling video chat. If you really didn't want it installed, simply run "sudo yum erase flash-plugin". Enjoy your surfing.

May 28, 2011

Custom Dropbox Directory

Filed under: blogging,linux,usability — jonEbird @ 10:38 am

Ever since I rebuilt my laptop I’ve keep a file called “rebuild.txt” where I detail every little customization I’ve done. The requirement is that everything that I do has to be completely command line oriented where a block copy-and-paste would redo the same customization. So far I have things documented like setting up my local email routing, autologin to the laptop, additional RPMs w/ extra repositories, iptables rules, GTK configs, etc. The latest customization I’ve performed is when I finally decided to setup Dropbox.

I like Dropbox for it’s simplicity and the fact it integrates well with Linux. My only problem was that the directory I really wanted to have synchronized was my ~/projects/ directory. I actually didn’t do any google searches on how to change it before diving into figuring it out. I figured Dropbox would create a dot-file directory to stuff configuration and found that at ~/.dropbox/ and after that it didn’t take too much work to reverse engineer the setup.

So, here is my addition to my “rebuild.txt” file for how I would redo my Dropbox setup:
(Note: I’m running Fedora 13 but that fact really only applies to how I was installing Dropbox)


cat <<\EOF | sudo tee /etc/yum.repos.d/dropbox.repo
[Dropbox]
name=Dropbox Repository
baseurl=http://linux.dropbox.com/fedora/$releasever/
gpgkey=http://linux.dropbox.com/fedora/rpm-public-key.asc
EOF

# Now install
sudo yum -y install nautilus-dropbox
# After installing, I did launch dropbox and setup an account.

# Stop dropbox
pkill dropbox

# Modify the default location?
cp -p ~/.dropbox/config.db{,.orig}
echo "update config set value = '/home/jon/projects' where key = 'dropbox_path';" |\
sqlite3 ~/.dropbox/config.db
rsync -av ~/Dropbox/ ~/projects/

# now start dropbox again
~/.dropbox-dist/dropboxd

After setting it up, I finally realized I could have google'd this and I did for curiosity sake. A common technique I saw was around using symlinks, but I found it fun to reverse engineer their configuration and was pleasantly amused that they didn't try to obfuscate the config. At this point I should probably stop poking around because I can see that the client is written in Python and I'm now finding other interesting things about the client.

November 15, 2010

Socket Option Defaults

Filed under: linux,python — jonEbird @ 10:02 pm

Working closely with the operating system, as an engineer or administrator, you often get odd questions about what particular OS settings were used. Often times, the oddest questions come from application owners which don’t have a solid handle on their app and are looking for excuses for why their application is misbehaving. Naturally their knowledge of the operating system is equally lacking if not more so.

Today’s question: What is the default OS setting for the SO_LINGER socket option?

I started off by explaining that there were no operating system configuration files where you go and adjust default socket option values and that if they were concerned with how the specific SO_LINGER option was being used that they need to keep their focus on the application. Should they be concerned with particular values being set, it’s going to be the application applying that setting via the setsockopt() system call. Their application is Java based running on top of a application server, so there are several layers of abstraction involved here. I do not mean to piss off any Java developers here, but more times than not they are not intimate with the lower level interactions of their JVMs within the OS.

Having adequately quelled that line of questioning, in terms of troubleshooting their application, I started to think why not go ahead and produce the values for all of the socket options? How about a python script for the answer?

#!/usr/bin/env python

import socket

s = socket.socket()
socket_options = [ (getattr(socket, opt), opt) for opt in dir(socket) if opt.startswith('SO_') ]
socket_options.sort()
for num, opt in socket_options:
    try:
        val = s.getsockopt(socket.SOL_SOCKET, num)
        print '%s(%d) defaults to %d' % (opt, num, val)
    except (socket.error), e:
        print '%s(%d) can\'t help you out there: %s' % (opt, num, str(e))

Running that on my Fedora Core 13 build, I get:

$ ./getsockopt.py
SO_DEBUG(1) defaults to 0
SO_REUSEADDR(2) defaults to 0
SO_TYPE(3) defaults to 1
SO_ERROR(4) defaults to 0
SO_DONTROUTE(5) defaults to 0
SO_BROADCAST(6) defaults to 0
SO_SNDBUF(7) defaults to 16384
SO_RCVBUF(8) defaults to 87380
SO_KEEPALIVE(9) defaults to 0
SO_OOBINLINE(10) defaults to 0
SO_NO_CHECK(11) defaults to 0
SO_PRIORITY(12) defaults to 0
SO_LINGER(13) defaults to 0
SO_BSDCOMPAT(14) defaults to 0
SO_PASSCRED(16) defaults to 0
SO_PEERCRED(17) defaults to 0
SO_RCVLOWAT(18) defaults to 1
SO_SNDLOWAT(19) defaults to 1
SO_RCVTIMEO(20) defaults to 0
SO_SNDTIMEO(21) defaults to 0
SO_SECURITY_AUTHENTICATION(22) can't help you out there: [Errno 92] Protocol not available
SO_SECURITY_ENCRYPTION_TRANSPORT(23) can't help you out there: [Errno 92] Protocol not available
SO_SECURITY_ENCRYPTION_NETWORK(24) can't help you out there: [Errno 92] Protocol not available
SO_BINDTODEVICE(25) can't help you out there: [Errno 92] Protocol not available
SO_ATTACH_FILTER(26) can't help you out there: [Errno 92] Protocol not available
SO_DETACH_FILTER(27) can't help you out there: [Errno 92] Protocol not available
SO_PEERNAME(28) can't help you out there: [Errno 107] Transport endpoint is not connected
SO_TIMESTAMP(29) defaults to 0
SO_ACCEPTCONN(30) defaults to 0
SO_PEERSEC(31) can't help you out there: [Errno 34] Numerical result out of range
SO_SNDBUFFORCE(32) can't help you out there: [Errno 92] Protocol not available
SO_RCVBUFFORCE(33) can't help you out there: [Errno 92] Protocol not available
SO_PASSSEC(34) defaults to 0
SO_TIMESTAMPNS(35) defaults to 0

And I like to show the actual system calls being performed since I didn’t write the program in C.

$ strace -vall -f ./getsockopt.py 2>&1 | egrep '^(socket|getsock|setsock)'
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
getsockopt(3, SOL_SOCKET, SO_DEBUG, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_REUSEADDR, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_TYPE, [1], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_DONTROUTE, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_BROADCAST, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_SNDBUF, [16384], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_RCVBUF, [87380], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_OOBINLINE, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_NO_CHECK, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_PRIORITY, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_LINGER, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_BSDCOMPAT, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_PASSCRED, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_PEERCRED, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_RCVLOWAT, [1], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_SNDLOWAT, [1], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_RCVTIMEO, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_SNDTIMEO, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_SECURITY_AUTHENTICATION, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_SECURITY_ENCRYPTION_TRANSPORT, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_SECURITY_ENCRYPTION_NETWORK, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_ATTACH_FILTER, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_DETACH_FILTER, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_PEERNAME, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOTCONN (Transport endpoint is not connected)
getsockopt(3, SOL_SOCKET, SO_TIMESTAMP, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_ACCEPTCONN, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_PEERSEC, 0xbfd0fb0c, 0xbfd0fb08) = -1 ERANGE (Numerical result out of range)
getsockopt(3, SOL_SOCKET, 0x20 /* SO_??? */, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, 0x21 /* SO_??? */, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, 0x22 /* SO_??? */, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, 0x23 /* SO_??? */, [0], [4]) = 0

That’s a whole lot of answers when I just needed to say, the operating system doesn’t automatically apply a SO_LINGER value by default on your newly created sockets but it was fun.

February 24, 2010

64bit Google Chrome with Flash on Fedora

Filed under: blogging,linux — jonEbird @ 9:22 pm

[ UPDATE as of 2011-07-10 This post is no longer advisable. Please see my updated post on setting up 64bit Chrome with Flash player for recent Fedora releases. ]

This is a quick howto on getting a 64bit Flash working with your 64bit Google Chrome browser on Fedora. The unfortunate part is that I feel obligated in writing this down for people but it’s really not that complicated after you figure out a few details.

First things first, you need to get Chrome installed. I find it funny that the top hit on google for "chrome yum repo" suggests a yum repo which points to a web server containing only a Readme that states it’s not serving chrome RPMs due to “legal concerns”? Google’s top hit should be it’s own page for Google Yum Repository. There you will find a block of text for your Yum repository which I personally put in /etc/yum.repos.d/google.repo.

Currently, the rpm does not create a plugins directory so we have to create one at /opt/google/chrome/plugins/. Once you have done that, you can visit Adobe’s 64bit Flash page where you can download the compressed tarball. Inside that tarball will be a single libflashplayer.so library which you will now want to either sym link to in the plugins directory or just copy it there.

With all that in place, you are ready to fire up Chrome and tell it about your manually installed plugin. Do that via "google-chrome --enable-plugins". All should be well and instead of testing it on youtube.com, let’s go to pandora.com instead and listen to “M.I.A.” channel. That funky channel seems appropriate for this procedure.

Here is the copy & paste version: (remove the "sudo" if you are root)

# Creating the repo
cat <<EOF | sudo tee /etc/yum.repos.d/google.repo
[google64]
name=Google - x86_64
baseurl=http://dl.google.com/linux/rpm/stable/x86_64
enabled=1
gpgcheck=1
gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pub
EOF
# Actually installing Chrome
sudo yum install google-chrome-beta.x86_64
# Creating a plugins directory
[ ! -d /opt/google/chrome/plugins ] && sudo mkdir /opt/google/chrome/plugins
# Grabbing Adobe's 64 bit Flash player
wget -qO /tmp/flash.html http://labs.adobe.com/downloads/flashplayer10_64bit.html
DLURL=$(sed -n '/^.*a href.*libflashplayer.*tar.gz/s/^.*<a href="\([^"]*\)".*/\1/p' /tmp/flash.html)
wget -qO- $DLURL | sudo tar -C /opt/google/chrome/plugins/ -xzvof -
# fireup chrome with new plugin
google-chrome --enable-plugins

And because I’ve been playing around with the combination of desktop background, Chrome theme and a Pandora skin in a nice, aesthetic color scheme, I’ll share a desktop screenshot of my 64bit Chrome playing some tunes.

February 9, 2010

Deciphering Caught Signals

Filed under: adminstration,linux,python — jonEbird @ 6:49 pm

Have you ever wondered which signal handlers a particular process has registered? A friend of mine was observing different behavior when spawning a new process from his Python script vs. invoking the command in the shell. Actually, he was consulting me about finding the best way to shutdown the process after spawning it from his Python script. You see, the program is actually just a shell wrapper which then kicks off the real program. His program would learn the process id (pid) of the wrapper and trying to send a kill signal to that was effectively terminating the wrapper and leaving the actual program running. By comparison, I asked him what happens in the shell when he tries to kill the program. Unlike being spawned in the Python script, this time the program and wrapper together would shutdown cleanly. My initial question was, “Are there different signal handlers being caught between the two scenarios?” He wasn’t sure and our dialog afterwards is what I’d like to explain to you now.

A pretty straight forward way to query what signal handlers a process has is to use “ps”. Let’s use my shell as an example:

$ ps -o pid,user,comm,caught -p $$
  PID USER     COMMAND                   CAUGHT
 3508 jon      bash            000000004b813efb

My shell is currently catching the signals being represented by the signal mask of 0x000000004b813efb. Pretty straight forward, right? Yeah, unless you havn’t done much C programming like my friend. He was not used to seeing hexadecimal numbers where each bit represents a on/off flag for each available signal. To follow along, make sure you understand binary representation of numbers first and learn that our number 0x000000004b813efb is represented in binary as 01001011100000010011111011111011. Now viewing that number and reading from right (least significant bit) to left, note which nth bit has a one or not. You can see that it is the 1st, 2nd, 4th, 5th, etc. Now all we have to do is associate those place holders with the signals they represent. Easiest way to see which numeric values are assigned to which signals is to use the “kill” command:

$ kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL
 5) SIGTRAP      6) SIGABRT      7) SIGBUS       8) SIGFPE
 9) SIGKILL     10) SIGUSR1     11) SIGSEGV     12) SIGUSR2
13) SIGPIPE     14) SIGALRM     15) SIGTERM     16) SIGSTKFLT
17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU
25) SIGXFSZ     26) SIGVTALRM   27) SIGPROF     28) SIGWINCH
29) SIGIO       30) SIGPWR      31) SIGSYS      34) SIGRTMIN
35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3  38) SIGRTMIN+4
39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12
47) SIGRTMIN+13 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14
51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10
55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7  58) SIGRTMAX-6
59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX

Armed with this knowledge, you can now provide a human readable report for which signals my shell is capturing: It has signal handlers setup for SIGHUP(1), SIGINT(2), SIGILL(4), SIGTRAP(5), etc.

A quick note about signal handlers. A signal handler is basically a jump location for your program to goto after receiving a particular signal. Think of it as an asynchronous function call, or more succinctly as a callback. That is, your program’s execution will jump to the function you’ve registered for your signal handler immediately upon receiving said signal and it does not matter where in your program’s execution you are currently at. Since the call is asynchronous, a lot of people will have a signal handler merely toggle a global flag and let their program resume it’s processing and check on that flag at a more convenient time.

Now that we know how to see which signals are being caught by a program, and what signal handlers are, let’s create a new signal handler for my shell and note the changed signal mask. Again, reviewing my currently caught signals, I notice I’m not doing anything for the 3rd signal of SIGQUIT. I want to assign a signal handler on this signal so we can see the changed signal mask. I’m going to have the shell execute a simple function upon receipt of the SIGQUIT signal.

$ function sayhi { echo "hi there"; }
$ trap sayhi 3
$ trap sayhi SIGQUIT # same thing as the number 3
$ kill -QUIT $$
hi there

Now, how about our signal mask. Has it changed?

$ ps -o pid,user,comm,caught -p $$
  PID USER     COMMAND                   CAUGHT
 3508 jon      bash            000000004b813eff

The signal mask has changed from 0x000000004b813efb to 0x000000004b813eff. The new signal mask, converting from hexadecimal to binary, is 1001011100000010011111011111111. Notice how our 3rd bit from the right is now a “1″ and before it was “0″.

Understanding how the signal masks are represented is good, but it’s still a pain if you want to quickly compare the signals being caught between two different processes. Per that point, I created a little Python script to do the work for me:

#!/bin/env python

import sys, signal

def dec2bin(N):
    binary = ''
    while N:
        N, r = divmod(N,2)
        binary = str(r) + binary
    return binary

def sigmask(binary):
    """Take a string representation of a binary number and return the signals associated with each bit.
       E.g. '10101' => ['SIGHUP','SIGQUIT','SIGTRAP']
            This is because SIGHUP is 1, SIGQUIT is 3 and SIGTRAP is 5
    """
    sigmap = dict([ (getattr(signal, sig), sig) for sig in dir(signal) if (sig.startswith('SIG') and '_' not in sig) ])
    signals = [ sigmap.get(n+1,str(n+1)) for n, bit in enumerate(reversed(binary)) if bit == '1' ]
    return signals

if __name__ == '__main__':

    if sys.argv[1].startswith('0x'):
        N = int(sys.argv[1], 16)
    else:
        N = int(sys.argv[1])

    binstr = dec2bin(N)
    print '"%s" (0x%x,%d) => %s; %s' % (sys.argv[1], N, N, binstr, ','.join(sigmask(binstr)) )

To use the my signals.py program, copy it to a file, make it executable and run it passing the signal mask of your program.

$ wget -O ~/bin/signals.py http://jonebird.com/signals.py
$ chmod 755 ~/bin/signals.py # assuming ~/bin is in your PATH
$ signals.py "0x$(ps --no-headers -o caught -p $$)"
"0x000000004b813eff" (0x4b813eff,1266761471) => 1001011100000010011111011111111;
 SIGHUP,SIGINT,SIGQUIT,SIGILL,SIGTRAP,SIGIOT,SIGBUS,SIGFPE,SIGUSR1,SIGSEGV,SIGUSR2,
 SIGPIPE,SIGALRM,SIGCLD,SIGXCPU,SIGXFSZ,SIGVTALRM,SIGWINCH,SIGSYS

Now back to my friend and his program problem. I asked him to fire off the program both from his Python script and then again directly from the shell. Each time I asked him to check on the caught signal mask of both the wrapper program and the actual binary and report the signal masks to me. As for the wrapper, it was consistently catching only SIGINT and SIGCLD, but the story was not as clear for the binary.
When kicked off via Python, the binary was catching the following signals:

  SIGQUIT,SIGBUS,SIGFPE,SIGSEGV,SIGTERM

whereas when invoked directly from the shell, the binary was catching:

  SIGINT,SIGQUIT,SIGBUS,SIGFPE,SIGSEGV,SIGTERM

Initially, I thought, “Ah ha, see it’s catching SIGINT in addition to the other signals when invoked from the shell!”, but quelled my excitement as I realized it didn’t help to explain why both wrapper and binary were both shutting down in the shell. If you sent a SIGINT to the wrapper via “kill -INT <wrapperpid>” nothing happens. Any other signal that the wrapper was not catching, such as SIGTERM (which is the default send via “kill” when you do not specifiy a signal), would cause the wrapper to terminate and orphan the binary to remain running.

The explanation lies within the shell code. We went through the various cases and when it wasn’t explained by the wrapper handling some signal and shutting down the binary, I was left with presuming the interactive shell was doing something unique. I initially observed this by running a strace against the binary and seeing the SIGINT interrupt and then later confirmed the behavior by consulting the bash source code. When you hit control-c in the shell, the shell will send a SIGINT to both processes because they are in the same process group (pgrp). I literally downloaded the bash source code to confirm this and quoting from a comment in the source code, “keyboard signals are sent to process groups”* That means a SIGINT is sent to both the wrapper and the binary. When that happens, the wrapper does nothing, as seen from prior experiments, but the binary catches it and does a clean shutdown which then allows the wrapper to complete and exit as well.

– Jon Miller

* How to efficiently root through source code is a subject for another blog. Within the bash-3.2.48.tar.gz source bundle, look at line 3230 in jobs.c.

Next Page »