jonEbird

November 15, 2010

Socket Option Defaults

Filed under: linux,python — jonEbird @ 10:02 pm

Working closely with the operating system, as an engineer or administrator, you often get odd questions about what particular OS settings were used. Often times, the oddest questions come from application owners which don’t have a solid handle on their app and are looking for excuses for why their application is misbehaving. Naturally their knowledge of the operating system is equally lacking if not more so.

Today’s question: What is the default OS setting for the SO_LINGER socket option?

I started off by explaining that there were no operating system configuration files where you go and adjust default socket option values and that if they were concerned with how the specific SO_LINGER option was being used that they need to keep their focus on the application. Should they be concerned with particular values being set, it’s going to be the application applying that setting via the setsockopt() system call. Their application is Java based running on top of a application server, so there are several layers of abstraction involved here. I do not mean to piss off any Java developers here, but more times than not they are not intimate with the lower level interactions of their JVMs within the OS.

Having adequately quelled that line of questioning, in terms of troubleshooting their application, I started to think why not go ahead and produce the values for all of the socket options? How about a python script for the answer?

#!/usr/bin/env python

import socket

s = socket.socket()
socket_options = [ (getattr(socket, opt), opt) for opt in dir(socket) if opt.startswith('SO_') ]
socket_options.sort()
for num, opt in socket_options:
    try:
        val = s.getsockopt(socket.SOL_SOCKET, num)
        print '%s(%d) defaults to %d' % (opt, num, val)
    except (socket.error), e:
        print '%s(%d) can\'t help you out there: %s' % (opt, num, str(e))

Running that on my Fedora Core 13 build, I get:

$ ./getsockopt.py
SO_DEBUG(1) defaults to 0
SO_REUSEADDR(2) defaults to 0
SO_TYPE(3) defaults to 1
SO_ERROR(4) defaults to 0
SO_DONTROUTE(5) defaults to 0
SO_BROADCAST(6) defaults to 0
SO_SNDBUF(7) defaults to 16384
SO_RCVBUF(8) defaults to 87380
SO_KEEPALIVE(9) defaults to 0
SO_OOBINLINE(10) defaults to 0
SO_NO_CHECK(11) defaults to 0
SO_PRIORITY(12) defaults to 0
SO_LINGER(13) defaults to 0
SO_BSDCOMPAT(14) defaults to 0
SO_PASSCRED(16) defaults to 0
SO_PEERCRED(17) defaults to 0
SO_RCVLOWAT(18) defaults to 1
SO_SNDLOWAT(19) defaults to 1
SO_RCVTIMEO(20) defaults to 0
SO_SNDTIMEO(21) defaults to 0
SO_SECURITY_AUTHENTICATION(22) can't help you out there: [Errno 92] Protocol not available
SO_SECURITY_ENCRYPTION_TRANSPORT(23) can't help you out there: [Errno 92] Protocol not available
SO_SECURITY_ENCRYPTION_NETWORK(24) can't help you out there: [Errno 92] Protocol not available
SO_BINDTODEVICE(25) can't help you out there: [Errno 92] Protocol not available
SO_ATTACH_FILTER(26) can't help you out there: [Errno 92] Protocol not available
SO_DETACH_FILTER(27) can't help you out there: [Errno 92] Protocol not available
SO_PEERNAME(28) can't help you out there: [Errno 107] Transport endpoint is not connected
SO_TIMESTAMP(29) defaults to 0
SO_ACCEPTCONN(30) defaults to 0
SO_PEERSEC(31) can't help you out there: [Errno 34] Numerical result out of range
SO_SNDBUFFORCE(32) can't help you out there: [Errno 92] Protocol not available
SO_RCVBUFFORCE(33) can't help you out there: [Errno 92] Protocol not available
SO_PASSSEC(34) defaults to 0
SO_TIMESTAMPNS(35) defaults to 0

And I like to show the actual system calls being performed since I didn’t write the program in C.

$ strace -vall -f ./getsockopt.py 2>&1 | egrep '^(socket|getsock|setsock)'
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
getsockopt(3, SOL_SOCKET, SO_DEBUG, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_REUSEADDR, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_TYPE, [1], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_DONTROUTE, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_BROADCAST, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_SNDBUF, [16384], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_RCVBUF, [87380], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_OOBINLINE, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_NO_CHECK, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_PRIORITY, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_LINGER, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_BSDCOMPAT, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_PASSCRED, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_PEERCRED, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_RCVLOWAT, [1], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_SNDLOWAT, [1], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_RCVTIMEO, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_SNDTIMEO, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_SECURITY_AUTHENTICATION, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_SECURITY_ENCRYPTION_TRANSPORT, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_SECURITY_ENCRYPTION_NETWORK, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_ATTACH_FILTER, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_DETACH_FILTER, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, SO_PEERNAME, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOTCONN (Transport endpoint is not connected)
getsockopt(3, SOL_SOCKET, SO_TIMESTAMP, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_ACCEPTCONN, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, SO_PEERSEC, 0xbfd0fb0c, 0xbfd0fb08) = -1 ERANGE (Numerical result out of range)
getsockopt(3, SOL_SOCKET, 0x20 /* SO_??? */, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, 0x21 /* SO_??? */, 0xbfd0fb0c, 0xbfd0fb08) = -1 ENOPROTOOPT (Protocol not available)
getsockopt(3, SOL_SOCKET, 0x22 /* SO_??? */, [0], [4]) = 0
getsockopt(3, SOL_SOCKET, 0x23 /* SO_??? */, [0], [4]) = 0

That’s a whole lot of answers when I just needed to say, the operating system doesn’t automatically apply a SO_LINGER value by default on your newly created sockets but it was fun.

November 8, 2010

Book Review: Influence: The Psychology of Persuasion

Filed under: blogging,opinion — jonEbird @ 8:36 pm

I just finished “Influence: The Psychology of Persuasion” by Robert b. Cialdini. I thought it was a very entertaining and educational read. It has a decidedly business / salesman focus in respect to the psychology of sales techniques but I would say the lessons are applicable to other areas of life. In fact, in each chapter Robert provides suggestions in how to resist the very potent sales techniques people often employ.

Here is quote from the epilogue which serves as a nice encapsulation of the book,

We have been exploring several of the most popular of the single pieces of information that we use to prompt our compliance decisions. They are the most popular prompts precisely because they are the most reliable ones, those that normally point us toward the correct choice. That is why we employ the factors of reciprocation, consistency, social proof, liking, authority, and scarcity so often and so automatically in making our compliance decisions. Each, by itself, provides a highly reliable cue as to when we will be better off saying yes than no..

I now plan to skim back through the chapters to review the content and internalize the content. I wish to use the knowledge of psychological influence to help improve my effectiveness in “getting work done through others”. If you work in a substantially large enough company, chances are your performance evaluations have touched on this subject before.

Robert Cialdini urges a strong rebuttal against people using the techniques, covered in the book, in a deceitful manner. I definitely do not wish to use deceitful techniques. I think a good example of using a technique covered in the book for getting work done through others is the principle of reciprocity. In fact, I’ve used this technique myself before having read the book. It’s quite simple. You go and do a favor for someone else, perhaps show them something new to better their work condition, and then they are more obligated to reciprocate the favor in return. If properly administered you can even get them to sidestep change control. (Just kidding, change controls are good.)

May 19, 2010

Happy Belated Birthday to Me

Filed under: adminstration,architecture,blogging — jonEbird @ 9:34 pm

A little play on blog titles going on here. Today I enter a new chapter in my career as I accept the Linux and Unix Architect role at Nationwide. I originally applied for the position back in February and I had hoped to land the position on my birthday in April, but the process was delayed for uncontrollable reasons. I like to be able work towards a work promotion as a birthday present for myself. In recognition for all the hard work just like four years ago when I congratulated myself in achieving Senior Systems Administrator.

I have been preparing for this position for the last few years and yet in some respects it feels like I don’t know what I’m entering. That might terrify some people but it is this very fact that most energizes me. The last time I felt this way I was transitioning from a purely development role into Administration. At the time, my high school friend Cory Sanders encouraged me in saying, “You’ll do fine.” I’ve received the same kind of encouragement recently and I appreciate the support. I’m energized about this opportunity because I will be learning so much. When I switched over to System Administration I spend hours studying admin books while my wife (then girlfriend) was working at Tim Horton’s. Fueled by coffee and donuts I climbed the learning curve as fast as I could.

My best description of a Architect is someone who leverages their broad technical experience in helping the business make decisions in what to pursue, where to invest and ultimately where to focus further development. I have always had a great deal of success in influencing people outside of my control but I plan to study that art in the book Influence: The Psychology of Persuasion. I am currently learning about personalties via Personality Plus. Somewhere in the middle of those books, I’ve already borrowed The New Rational Manager where I hope to gain a better systematic approach to problem resolution. I already consider my troubleshooting skills to be very good but I was impressed in how, now fellow, Architects were able to keep a room full of people technicians focused on the resolution in such a organized manor.

Beyond the academic focus, this new position is largely about building relationships. I need to devote time with teams and individuals at lunch, over coffee and in the hallways. They need to be comfortable coming to me with their problems and I need them to be receptive to directions I set for them. This will be the “easy” part of the job. As a person of a Peaceful Phlegmatic personality disposition building relationships is naturally easy.

Finally, I hope to largely increase my business acumen as the next Linux and Unix Architect. I am looking forward to having strategic conversations with our business partners such as RedHat, HP, Sun Oracle, Novell, IBM, Veritas and many more. I expect to be working closer with existing management on budgetary decisions. Frankly, I am struggling to enumerate further business categories for which I should be focusing on which underscores my ignorance in the field. I have had a subscription to Entrepreneur Magazine for nearly a year, watch business videos online from Stanford University and try to participate in local TechColumbus networking and business related activities. I will look for any further opportunities that present themselves to me and take it from there.

When I started mentoring with our previous Architect, I told him my goal was to take his job, “…but don’t worry, I want you to move on to bigger and better work first.” I need those kind of goals to keep me motivated. This next goal will be quite lofty and I have no idea how or when I might achieve it, but the next position I am setting my sights on is CTO. I have a feeling it’s going to be longer than four years from now that I’ll be able to write that blog post.

May 15, 2010

Sun’s Future under Oracle

Filed under: blogging,opinion — jonEbird @ 8:34 am

Nice insight into the transformation of Sun under Oracle’s helm. They talk about Exadata 2, the appliance which is using their newly acquired sparc hardware.

The machine costs more than $1 million, stands over 6 feet tall, is two feet wide and weighs a full ton. It is capable of storing vast quantities of data, allowing businesses to analyze information at lightening fast speeds or instantly process commercial transactions.

The part I don’t like about that statement is the fact that they’re trying to build both a data warehousing solution as well as a OLTP in the same machine. Sounds horribly inefficient. What I’ve taken from the article is that Larry seems to be a very savvy businessman but it’s interesting that they’ve failed in developing new software for a decade and their revenue increases came from acquisitions.

He plans on continuing to buy up more companies, in the hardware space, but once he’s done he’ll have to produce products and that is where I question their ability to deliver. It’s nice to see that he’s stopped the bleeding within Sun but now you’ve got a bruised and battered, over the hill player recovering from fractured ribs and a couple concussions available to you on the bench. It just seems like they’re taking Brett Favre and using him to coach rugby. I guess that could work?

February 24, 2010

64bit Google Chrome with Flash on Fedora

Filed under: blogging,linux — jonEbird @ 9:22 pm

[ UPDATE as of 2011-07-10 This post is no longer advisable. Please see my updated post on setting up 64bit Chrome with Flash player for recent Fedora releases. ]

This is a quick howto on getting a 64bit Flash working with your 64bit Google Chrome browser on Fedora. The unfortunate part is that I feel obligated in writing this down for people but it’s really not that complicated after you figure out a few details.

First things first, you need to get Chrome installed. I find it funny that the top hit on google for "chrome yum repo" suggests a yum repo which points to a web server containing only a Readme that states it’s not serving chrome RPMs due to “legal concerns”? Google’s top hit should be it’s own page for Google Yum Repository. There you will find a block of text for your Yum repository which I personally put in /etc/yum.repos.d/google.repo.

Currently, the rpm does not create a plugins directory so we have to create one at /opt/google/chrome/plugins/. Once you have done that, you can visit Adobe’s 64bit Flash page where you can download the compressed tarball. Inside that tarball will be a single libflashplayer.so library which you will now want to either sym link to in the plugins directory or just copy it there.

With all that in place, you are ready to fire up Chrome and tell it about your manually installed plugin. Do that via "google-chrome --enable-plugins". All should be well and instead of testing it on youtube.com, let’s go to pandora.com instead and listen to “M.I.A.” channel. That funky channel seems appropriate for this procedure.

Here is the copy & paste version: (remove the "sudo" if you are root)

# Creating the repo
cat <<EOF | sudo tee /etc/yum.repos.d/google.repo
[google64]
name=Google - x86_64
baseurl=http://dl.google.com/linux/rpm/stable/x86_64
enabled=1
gpgcheck=1
gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pub
EOF
# Actually installing Chrome
sudo yum install google-chrome-beta.x86_64
# Creating a plugins directory
[ ! -d /opt/google/chrome/plugins ] && sudo mkdir /opt/google/chrome/plugins
# Grabbing Adobe's 64 bit Flash player
wget -qO /tmp/flash.html http://labs.adobe.com/downloads/flashplayer10_64bit.html
DLURL=$(sed -n '/^.*a href.*libflashplayer.*tar.gz/s/^.*<a href="\([^"]*\)".*/\1/p' /tmp/flash.html)
wget -qO- $DLURL | sudo tar -C /opt/google/chrome/plugins/ -xzvof -
# fireup chrome with new plugin
google-chrome --enable-plugins

And because I’ve been playing around with the combination of desktop background, Chrome theme and a Pandora skin in a nice, aesthetic color scheme, I’ll share a desktop screenshot of my 64bit Chrome playing some tunes.

February 9, 2010

Deciphering Caught Signals

Filed under: adminstration,linux,python — jonEbird @ 6:49 pm

Have you ever wondered which signal handlers a particular process has registered? A friend of mine was observing different behavior when spawning a new process from his Python script vs. invoking the command in the shell. Actually, he was consulting me about finding the best way to shutdown the process after spawning it from his Python script. You see, the program is actually just a shell wrapper which then kicks off the real program. His program would learn the process id (pid) of the wrapper and trying to send a kill signal to that was effectively terminating the wrapper and leaving the actual program running. By comparison, I asked him what happens in the shell when he tries to kill the program. Unlike being spawned in the Python script, this time the program and wrapper together would shutdown cleanly. My initial question was, “Are there different signal handlers being caught between the two scenarios?” He wasn’t sure and our dialog afterwards is what I’d like to explain to you now.

A pretty straight forward way to query what signal handlers a process has is to use “ps”. Let’s use my shell as an example:

$ ps -o pid,user,comm,caught -p $$
  PID USER     COMMAND                   CAUGHT
 3508 jon      bash            000000004b813efb

My shell is currently catching the signals being represented by the signal mask of 0x000000004b813efb. Pretty straight forward, right? Yeah, unless you havn’t done much C programming like my friend. He was not used to seeing hexadecimal numbers where each bit represents a on/off flag for each available signal. To follow along, make sure you understand binary representation of numbers first and learn that our number 0x000000004b813efb is represented in binary as 01001011100000010011111011111011. Now viewing that number and reading from right (least significant bit) to left, note which nth bit has a one or not. You can see that it is the 1st, 2nd, 4th, 5th, etc. Now all we have to do is associate those place holders with the signals they represent. Easiest way to see which numeric values are assigned to which signals is to use the “kill” command:

$ kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL
 5) SIGTRAP      6) SIGABRT      7) SIGBUS       8) SIGFPE
 9) SIGKILL     10) SIGUSR1     11) SIGSEGV     12) SIGUSR2
13) SIGPIPE     14) SIGALRM     15) SIGTERM     16) SIGSTKFLT
17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU
25) SIGXFSZ     26) SIGVTALRM   27) SIGPROF     28) SIGWINCH
29) SIGIO       30) SIGPWR      31) SIGSYS      34) SIGRTMIN
35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3  38) SIGRTMIN+4
39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12
47) SIGRTMIN+13 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14
51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10
55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7  58) SIGRTMAX-6
59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX

Armed with this knowledge, you can now provide a human readable report for which signals my shell is capturing: It has signal handlers setup for SIGHUP(1), SIGINT(2), SIGILL(4), SIGTRAP(5), etc.

A quick note about signal handlers. A signal handler is basically a jump location for your program to goto after receiving a particular signal. Think of it as an asynchronous function call, or more succinctly as a callback. That is, your program’s execution will jump to the function you’ve registered for your signal handler immediately upon receiving said signal and it does not matter where in your program’s execution you are currently at. Since the call is asynchronous, a lot of people will have a signal handler merely toggle a global flag and let their program resume it’s processing and check on that flag at a more convenient time.

Now that we know how to see which signals are being caught by a program, and what signal handlers are, let’s create a new signal handler for my shell and note the changed signal mask. Again, reviewing my currently caught signals, I notice I’m not doing anything for the 3rd signal of SIGQUIT. I want to assign a signal handler on this signal so we can see the changed signal mask. I’m going to have the shell execute a simple function upon receipt of the SIGQUIT signal.

$ function sayhi { echo "hi there"; }
$ trap sayhi 3
$ trap sayhi SIGQUIT # same thing as the number 3
$ kill -QUIT $$
hi there

Now, how about our signal mask. Has it changed?

$ ps -o pid,user,comm,caught -p $$
  PID USER     COMMAND                   CAUGHT
 3508 jon      bash            000000004b813eff

The signal mask has changed from 0x000000004b813efb to 0x000000004b813eff. The new signal mask, converting from hexadecimal to binary, is 1001011100000010011111011111111. Notice how our 3rd bit from the right is now a “1″ and before it was “0″.

Understanding how the signal masks are represented is good, but it’s still a pain if you want to quickly compare the signals being caught between two different processes. Per that point, I created a little Python script to do the work for me:

#!/bin/env python

import sys, signal

def dec2bin(N):
    binary = ''
    while N:
        N, r = divmod(N,2)
        binary = str(r) + binary
    return binary

def sigmask(binary):
    """Take a string representation of a binary number and return the signals associated with each bit.
       E.g. '10101' => ['SIGHUP','SIGQUIT','SIGTRAP']
            This is because SIGHUP is 1, SIGQUIT is 3 and SIGTRAP is 5
    """
    sigmap = dict([ (getattr(signal, sig), sig) for sig in dir(signal) if (sig.startswith('SIG') and '_' not in sig) ])
    signals = [ sigmap.get(n+1,str(n+1)) for n, bit in enumerate(reversed(binary)) if bit == '1' ]
    return signals

if __name__ == '__main__':

    if sys.argv[1].startswith('0x'):
        N = int(sys.argv[1], 16)
    else:
        N = int(sys.argv[1])

    binstr = dec2bin(N)
    print '"%s" (0x%x,%d) => %s; %s' % (sys.argv[1], N, N, binstr, ','.join(sigmask(binstr)) )

To use the my signals.py program, copy it to a file, make it executable and run it passing the signal mask of your program.

$ wget -O ~/bin/signals.py http://jonebird.com/signals.py
$ chmod 755 ~/bin/signals.py # assuming ~/bin is in your PATH
$ signals.py "0x$(ps --no-headers -o caught -p $$)"
"0x000000004b813eff" (0x4b813eff,1266761471) => 1001011100000010011111011111111;
 SIGHUP,SIGINT,SIGQUIT,SIGILL,SIGTRAP,SIGIOT,SIGBUS,SIGFPE,SIGUSR1,SIGSEGV,SIGUSR2,
 SIGPIPE,SIGALRM,SIGCLD,SIGXCPU,SIGXFSZ,SIGVTALRM,SIGWINCH,SIGSYS

Now back to my friend and his program problem. I asked him to fire off the program both from his Python script and then again directly from the shell. Each time I asked him to check on the caught signal mask of both the wrapper program and the actual binary and report the signal masks to me. As for the wrapper, it was consistently catching only SIGINT and SIGCLD, but the story was not as clear for the binary.
When kicked off via Python, the binary was catching the following signals:

  SIGQUIT,SIGBUS,SIGFPE,SIGSEGV,SIGTERM

whereas when invoked directly from the shell, the binary was catching:

  SIGINT,SIGQUIT,SIGBUS,SIGFPE,SIGSEGV,SIGTERM

Initially, I thought, “Ah ha, see it’s catching SIGINT in addition to the other signals when invoked from the shell!”, but quelled my excitement as I realized it didn’t help to explain why both wrapper and binary were both shutting down in the shell. If you sent a SIGINT to the wrapper via “kill -INT <wrapperpid>” nothing happens. Any other signal that the wrapper was not catching, such as SIGTERM (which is the default send via “kill” when you do not specifiy a signal), would cause the wrapper to terminate and orphan the binary to remain running.

The explanation lies within the shell code. We went through the various cases and when it wasn’t explained by the wrapper handling some signal and shutting down the binary, I was left with presuming the interactive shell was doing something unique. I initially observed this by running a strace against the binary and seeing the SIGINT interrupt and then later confirmed the behavior by consulting the bash source code. When you hit control-c in the shell, the shell will send a SIGINT to both processes because they are in the same process group (pgrp). I literally downloaded the bash source code to confirm this and quoting from a comment in the source code, “keyboard signals are sent to process groups”* That means a SIGINT is sent to both the wrapper and the binary. When that happens, the wrapper does nothing, as seen from prior experiments, but the binary catches it and does a clean shutdown which then allows the wrapper to complete and exit as well.

– Jon Miller

* How to efficiently root through source code is a subject for another blog. Within the bash-3.2.48.tar.gz source bundle, look at line 3230 in jobs.c.

September 27, 2009

Presenting at Inaugural CoPUG

Filed under: hadoop,python — jonEbird @ 8:34 pm

Tomorrow I will be presenting an Introduction to Hadoop: Driven by Python for the inaugural Central Ohio Python Users Group or just CoPUG for short.

I have high hopes for CoPUG. The organizer, Eric Floehr, appears to be well organized, competent individual although I have only exchanged emails and have yet to meet in person. While in Atlanta, last year for PyWorks, I learned of the very strong PyAtl group lead by none other than the current editor of the Python Magazine, Brandon Rhodes. Although I am not sure, I wonder if their Python group has something to do with PyCon coming to Atlanta in 2010. Can I dream of PyCon someday coming to Columbus?

My Introduction to Hadoop: Driven by Python slides provided under the Creative Commons Attribution 3.0 United States License.

September 16, 2009

Server Death

Filed under: adminstration,blogging,linux — jonEbird @ 7:41 pm

I often joke that the only people that read my weblog are bots, so it shouldn’t bother me if my site is down but it does. Last week the server, which was also doubling as a workstation for the wife, died. “The computer is not working”, the Wife explained. I didn’t check it out immediately as I just assumed that X had crashed or something else preventing her from using firefox. Like I said, I’m not too overly concerned with my site’s uptime.

But when I finally did check it out, sure enough, it was not looking good. Absolutely no display on the monitor. Considering I had replaced my video card not too long ago and I could no longer ssh into the machine, I am thinking that either the CPU and/or the motherboard are dead.

DeadPC
Hercules taking a look

After Hercules and I surveyed the situation, we decided to pull the sheet over it’s head. It’s had a nice long life (pc years) since 2004.

I headed to microcenter today to checkout what kind of motherboards, CPUs and even memory that they had on sale. If you consider my last machine was running with only 756M of memory, an ageing AMD 2Ghz processor on a abit kv8 motherboard while happily serving my website and handling the Wife’s facebook usage, then you can understand I was looking for the smallest, cheapest solution I could find. That solution was looking to be somewhere around $225.

Not willing to rush into a $200+ investment, I instead bought a IDE enclosure which is capable of serving my data via USB for a mere $21 bucks.

Now for the Restoration of my Website

I really shouldn’t even be talking about this. I should have had regular MySQL dumps along with full web content backed off to another machine. Aside from a laptop, the other “real” pc in the house is a Acer I bought as a media machine which sits in my entertainment center. It was never intended to be running 24×7, so I only did on-demand backups of my important files which were actually outside of my website. Another justification for not having regular backups was that I had two internal Seagate drives configured in a software mirror. I always figured if I had some sort of hardware problem, I’d be able to replace it and in worse case never really lose my data.

So I have my hard drive and am now looking to get my WordPress site back online with the pc in the living room. After plugging in the harddrive, I need to activate the MD device and mount up my filesystem:

[jon@pc ~]$ sudo mdadm --assemble --scan
mdadm: /dev/md/0_0 has been started with 1 drive (out of 2).
[jon@pc ~]$ cat /proc/mdstat
Personalities : [raid1]
md127 : active raid1 sdd1[1]
      241665664 blocks [2/1] [_U]

unused devices: <none>
[jon@pc ~]$ sudo mount /dev/md127 /mnt

My two machines were off from each other by two Fedora releases. I wondered if I could do a chroot, startup MySQL and get a fresh, clean dump of the database…

[jon@pc ~]$ sudo su -
[root@pc ~]# chroot /mnt
[root@pc /]# ls
bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt
opt  proc  root  sbin  selinux  srv  sys  tmp  usr  var

[root@pc ~]# mount -t proc none /proc
[root@pc ~]# /etc/init.d/mysqld status
mysqld dead but subsys locked
[root@pc ~]# /etc/init.d/mysqld restart
Stopping MySQL:                                            [  OK  ]
Starting MySQL:                                            [  OK  ]
[root@pc ~]# /etc/init.d/mysqld status
mysqld (pid 9394) is running...
[root@pc ~]# mysqldump -u root -p wordpress > wordpress.mysqldump
Enter password:
[root@pc ~]# wc -l wordpress.mysqldump
354 wordpress.mysqldump

Cool!

The rest of the migration involved an rsync of /var/www/html/ content, adjustments of the default Apache config, granting access for my WordPress user to use the database and finally updating my router to now direct requests for port 80 to my media pc.

At this point, I guess I’ll be running this site from the living room until I decide what to do about my server / workstation. I’ve always wanted to build a slimmed down, efficient virtual server to host my website and then migrate it between server and laptop during maintenance / patching of my machines, but my AMD processor didn’t support the Virtualization assistance, so it was painfully slow. I think I’ll keep an eye out for a used, server-class machine. Let me know if you find any, bots. Thanks. ;-)

September 4, 2009

Finding My Strengths

Filed under: blogging — jonEbird @ 3:17 pm

Being a #1 best selling book, there is a decent chance you’ve heard of Strengths Finder 2.0. At my work, our whole team got a copy of it. I’m busy finishing up another book, but meanwhile everyone else has completed the online assessment of their strengths and have started sharing amongst the team. So, I have decided to at least take the online assessment and share my top five strengths as well.

I must admit, I was skeptical of the effectiveness of the poll but am now pleasantly surprised to reveal such an accurate description of my strengths. Enough so to encourage me to hurry up and start reading the book. Once you complete the assessment, it will then provide you a guideline or an action plan to further take advantage of your strengths. I particularly liked the fact that it provided a nice html, printable version of that action plan which I can then easily share with others. So, without further ado, I give you my top 5 strengths:

Jon Miller’s, Strengths Finder 2.0, top five strengths

August 10, 2009

Hadoop Elephant Makes a Big Splash

Filed under: blogging,hadoop,python — jonEbird @ 5:27 pm

Big news in the world of Hadoop today. My Running Large Python Tasks With Hadoop is published in the July Edition of Python Magazine. This marks my second article with the magazine and I had a lot of fun doing it. My interest in the anti-rdbms will continue as I continue to find interesting ways to organize data in the enterprise.

While providing a gentle introduction to Hadoop, my article also introduces readers to my HadoopCalculator which you can install a couple of different ways. First way is done via git where you can pull my HadoopUtils repo from github via:

git clone git://github.com/jonEbird/Hadoop-Utils.git


That will bring a few more scripts than just my HadoopCalculator. The second way to install is to use the Python setuptools utility easy_install or pull down the source package from the Cheese Shop.

Thank you for reading this far. I lied. The big news today in the Hadoop world is Doug Cutting joining Cloudera. Had you going, didn’t I? Recently, while Doug was still with Yahoo!, the Microsoft and Yahoo Partnership had people wondering what impact that would have on the Hadoop ecosystem. Today, Yahoo! is the largest Hadoop user and for obvious reasons contributed a lot to the community. Cloudera was already a well known player in the Hadoop community but their stock has risen immensely with the addition of Doug Cutting. If they were selling stock, I’d buy.

« Previous PageNext Page »