jonEbird

Every once and a while I like to re-read manuals about pieces of software which I already feel quite comfortable in using in hopes to learn a new trick or two. Today, I was browsing the freely available GNU Emacs Manual and was breezing through it until I hit the registers section.

Have you every copied a region of text in plans to yank it back in multiple locations within your current buffer but in between this work, you realize you need to do some intermediate killing and yanking and therefore have overwritten your original region of text you had copied? Well, using registers is one way in solving that problem.

Bookmarks have a similar function as with registers within emacs. I am actually grouping them together in this weblog because their key sequences are so very similar. Bookmarks are, like the name implies, a way to keep track of position within a buffer. They can be saved and later referenced to not only re-open the particular file you were editing but take you back to the position within that file as well.

I can not seem to remember anything, from emacs commands or keystrokes to basic shell commands, without coming up with some mnemonic for memorizing it. I have just came up with one such mnemonic for using Registers and Bookmarks and I thought I’d share it with my bots (machines which read this weblog).

Register and Bookmark Mnemonics

Each key sequence starts with:
C-x r - Think “r” for register.
The general pattern is:
C-x r <key> <register> - “register” is any single digit or letter.

“key” is what I’m calling each category of register usage. Let’s explore them:

• “<space>” - mark - Just like C-spc to set the mark, this asks that the mark be set in our register.
• “s” - save - save the region’s text into the register.
• “n” - number - save a particular number into the register.
• “m” - bookmark - We’ll explore bookmarks further, but notice how you set bookmarks with what I consider the register key sequence.

Those are some of the basic storing actions, but with the same C-x r prefix you can perform other actions with the contents of the registers:

• “+” - increment - When the register is a number, this increments it’s value. Convenient to use with macros.
• “i” - insert - This action can be used for numbers, regions of text and even marks!
• “j” - jump - jump to a mark. This assumes you have already stored a mark in the particular register.

We’ll shift a bit into bookmarks but stay succinct within the mnemonics section here.

• “m” - bookmark - Repeated from above. Note that this is a interactive function which allows you to name your bookmark with an intelligent name. The default will be the basename of your current buffer’s filename. So, if you’re editing /path/to/emacs_notes.txt it will default to store the bookmark under “emacs_notes.txt” but maybe you want it to be called “emacs notes”. If so, go ahead and type that out and hit RET.
• “l” - list bookmarks - This opens a new pseudo buffer with the list of all of your bookmarks.
• “b” - bookmark jump - Jump to the named bookmark. This is a interactive function as well.

Bringing it Together - Examples

Save current mark to register “l”:
C-x r <space> l
Move to mark saved in register “l”:
C-x r j l
Save number in register “n”:
C-x r n n
Now, increment that number and then insert it:
C-x r + n C-x r i n

Love the emacs notes you’re editing, bookmark it:
C-x r m emacsnotes RET
Buried into multiple install READMEs for a particular product and want to return later:
C-x r m installreadme RET

Finally, the prettiest of the commands, let’s review our bookmarks:
C-x r l

Final Note on Registers and Bookmarks

There is a variable named bookmark-save-flag which when set to the value of “1″ will have the action of automatically updating your ~/.emacs.bmk file with any updates to your bookmarks. I recommend setting this in your ~/.emacs file so you don’t have to “M-x bookmark-save” periodically. Add the following to your ~/.emacs file:
(set-variable (quote bookmark-save-flag) 1 nil)

Finally, I’d be remiss if I didn’t mention how I generated that one-liner emacs-lisp line, even if it’s off topic for this weblog entry. I like to use various interactive functions and then capture their effective execution, in emacs-lisp form, via the “repeat-complex-command” function which is bound to “C-x M-:“. In this situation, I used the “set-variable” interactive function to set “bookmark-save-flag” to “1″ and then punched in “C-x M-:” and copied the one-liner for my ~/.emacs file. So, there you go, a bonus tip for those who’ve read this far.

For years I thought the best way to enhance my Windows experience, with the common Unix/Linux tools I’m most comfortable with, was to do so with cygwin. That was until now. At work, where we are forced to use Windows, I recently had my laptop rebuild and afterwards my re-install of cygwin wasn’t going too well. Finally fed up, I then recalled seeing reviews about colinux and how it was advertised as being tightly integrated into the Windows experience. Before looking for the install media, I then saw that there are a couple of distros which are then built on top of colinux. One of which, andLinux, is a full Ubuntu release and I presumed that their layering on top of colinux was naturally providing additional support and/or features and decided to go with andLinux for the install.

After completing the install, I must say, I am very pleased and impressed with the work they have done. I’m not sure how much of the credit goes to colinux and how much goes to andLinux, but they both get a A+ in my book. Here are my top reasons for choosing andLinux over cygwin:

1. Full Linux operating system running on top of Windows.
   Not actually being virtualized and is therefore quick. It is a special patch to the Linux kernel which allows this tight integration with winblows.
2. Each window / app launched takes the same look&feel decorations as each other windows app.
   Translation: Doesn’t look like crap.
   Also, each app’s icon is properly displayed in the task bar instead of the same, repeated icon used in cygwin.
3. Transition from wired to wireless is seamless.
   This was a piece a co-worker asked me to test out and I’m writing this up while on my wireless network at home. After suspending my laptop at work, I then un-suspended it at home and I didn’t have to touch a thing. My existing terminal window could still query hosts and I even tested an install of a quick package.
4. It’s running Ubuntu.
   Means to get additional apps, which you might be missing, you get to do "apt-get install <missingapp>" instead of re-launching setup.exe.
5. Clean terminal.
   What is this bullet point doing here you ask? Well, it is what motivated me to move away from cygwin in the first place today. I was previously trying to use mrxvt and after multiple issues, decided to punt.
   The default terminal appears to be gnome-terminal and yet I choose the XFCE version over the KDE version.
6. Xming X11 Server included.
   No need for hummingbird’s crappy X server. This is discrete and works very well.
7. Automatic TAP (bridged) networking configured.
   There is about 4 screens used during the install and none were about networking. Just works.
8. “cofs” filesystem
   My C:\ drive is mounted at /mnt/win via their ‘cofs’ filesystem. Sweet.

If you are like me and are stuck using Windows for whatever reason, I would highly suggest checking out andLinux. They are currently in beta but I’m okay with dealing with any minor hiccups. I have used it for only two days and already feel light-years away from my previous cygwin days.

In anticipation of heading down to PyWorks 2008, I have been thinking about creating a business card for the sake of keeping in contact with people I meet. One of my main goals, while attending and speaking at PyWorks, is to network with people and mark 2008 as the year which I start participating and contributing within the OSS community. While exercising my creativity in designing a nice business card, I have also been reading about Google’s android mobile platform, and I came across a interesting intersection between the two when I saw a demonstration video.

A Google developer, working on the zxing project (pronounced “zebra crossing”), has printed a 2D barcode encoding of his personal information on the back of his business card. With the builtin camera, on his android phone, he can scan in a barcode and immediate use the encoded data. It is an impressive demonstration of integrating technology with our mobile devices. Check out the video which has inspired me to not only do the same but also write this small informational note about 2D barcodes.

If you didn’t catch it, the format of the 2D barcode on his business card is QR Code. Among the other 2D barcode formats, QR Code barcodes are most popular in Japan where it was invented by Denso-Wave. The popularity of QR Codes in Japan has grown to the level of being supported by nearly every mobile device there and that also means finding QR Codes available on a increasing amount of printed media from fliers to magazines and coupons.

There are other competing 2D barcode formats that I could choose from but after doing researching and not seeing any distinctive advantages, I have concluded to follow suit with the google developer and hope that the android phone’s application and popularity will help propel QR Code’s popularity over the other 2D barcode formats.

Since 2D barcodes are nothing more than encoding and decoding data, the first thing to decide is what data we would like to encode. Since I actually do not have a business of my own and furthermore use a work issued phone, the data I encode will probably be a URL of my website. There are other interesting encodings, though, which include email address, sms, geographic locations, etc. See zxing’s wiki about BarcodeContents for a better discussion for suggested format, including their primary suggestion of using the MECARD format which is typically a composite of Name, Address, Phone number and Email address.

Once you know what you would like to encode in your 2D barcode, I’m guessing you will need software to help with that. With the assumption that you are not going to be encoding/decoding barcodes with a large frequency, my suggestion is that you use online utilities to help you. Interestingly, it turns out that the google chart api can now generate QR Code online. That is both convenient for repeated generation of QR Code but also in dynamic generation of barcodes. But alas, Jason Delport has created a google app engine application to record your text and generate the QR Code for you by generating the google chart api link for you. At that point, you can either use the supplied URL or simply download the png image. Finally, for performing online decoding of the barcode I have found the zxing online decoder to be the best and least intrusive one available.

The main reason 2D barcodes have not really taken off here in the States is because people have not yet came up with a really good idea to propel it into the mainstream. That, my friends, is going to be up to you and me to accomplish. Or, wait, we could just let google usher it in for us? But seriously I think support for mobile devices to read 2D barcodes is great step forward. Afterwards, I can envision graphic designers coming up with clever barcode prints and ways of intriguing people to scan the code for more details, but then it would be people like us who come up with new categories of data to be encoded in the barcodes for new, innovative ways to use them.

To learn more, try the collection of interesting links provided again by the zxing folks.

For the 2008 PyWorks convention, I will be presenting about LDAP and Python. The presentation is really about demystifying LDAP and encouraging people to use and extend LDAP for their config file needs. In efforts to make my point, the last half of my presentation will be a time for a demo. This entry is your basic landing point where you can download the scripts, presuming you are looking for a copy of the scripts and/or slides after seeing my presentation? (oh! nevermind, your google search landed you here)

For the demo, I will be leveraging the fail2ban project. It is a python based application which scans typical application logs for security failures and bans IPs from being able to connect again. It also uses the builtin ConfigParser module for reading it’s 30+ config files, which is why I have chosen to use it. For the demo, I have created two scripts:

The first one, configparser2ldap.py is used to process a set of config files and automatically generate LDAP schema as well as LDIF data.

Next, I have my ldapconfig.py module where I extended the ConfigParser module to support making queries to LDAP. I am basically overriding the read() method only and leaving the rest of the module alone. This way the only modifications to the fail2ban application are how it is instantiating the ConfigParser and I won’t have to become a full time fail2ban developer if I want to centralize the configuration data in LDAP.

And that is really the main point of my presentation: The power of centralizing your configuration data and how it can drastically change how you administer your large scale server farm.

Downloads

LDAP + Python Slides.

configparser2ldap.py script to auto-generate LDAP schema and LDIF from ConfigParser compatible config files.

ldapconfig.py python module which inherits the ConfigParser and supports optionally pulling config data from LDAP.

An Idea for a helpful Admin Tool

What if you got a page and/or ticket for an obscure server’s particular service? The unique problem is that your environment is huge, you’re still relatively new to the company, co-workers are not there to help you and you have never heard of this server. When logging in, you’re hoping that the person has a nice RC script under /etc/init.d/, that you can find the app via a “lsof -i:<port>”, find the application’s home and locate some log files. But what if the application install was not that nice and did not conform to the norms that you are used to?

To either a small or very large degree, you will be reverse engineering this application. If you’re really unlucky, the application who supports it also has no idea about it nor knows anything about Unix-like machines. So, what if there was an application which is polling upon logging into the server, told you, “In case you are looking for the application binX, which typically listens on port XX, it was most likely started last time by issuing the script /path/to/funky/path/binX.sh”. I’m guessing it would freak you out and immediately flood your emotions with confusion, gratitude and curiosity.

So, would such an application be difficult to write?

• Poll any events for read/write/access under key dirs, such as /etc/init.d/, /etc/*conf ? (use the inotify syscall introduced in Linux kernel 2.6.16)
• Track users logging into the system (could correlate later)
• Watch for any new ports being listened on, then record the binary name.
• Reverse engineer this application to automatically collect interesting data on it.
• Intelligently parse an strace (note to self, checkout: http://subterfugue.org/)
• Utilize systemtap for Linux and DTrace for Solaris. pseudo code { observe new socket being opened, so show me the last 10 files opened and executed. correlate application with startup script }

Now, if your data was collected in a easily usable format, you can collect similar data from other machines and start to make broader correlations.

The whole process is really about automating the process of reverse engineering an application. I do that alot. I believe others would like an application which aided or performed the entire reverse engineering for them.

What is the best programming language for a system administrator? Queue the language war, please. The typical arguments are “your language can’t do this”, “this library doesn’t have a consistent naming convention”, well “my language is faster”, yeah and “your syntax is hideous to read much less use”, blah blah blah. No, I’m not a professional developer but I do spend a significant time doing development as a systems administrator. My programs are not huge year long projects, will probably never reach million lines of code and usually never need superb speed. For administrators, the most important aspect of the language of choice is productivity and maintainability.

When choosing your language, I recommend picking one that has a decent user community, is available on numerous platforms, has had significant time to mature in proving itself and has an extensive modules/library support. Meeting these requirements will leave you using a language that should keep you efficiently producing solutions to your administrative tasks.

First let’s eliminate some languages based on maintainability. Goodbye Haskell, lisp, scheme, Erlang and any other purely functional languages you have used or know of. I’d venture to say that less than 2% of system administrators are comfortable using any one of those languages. And you can obviously not choose a language which only yourself are going to be able to maintain. Aside from staying away from the obscure, the program should be intuitive to read. People can argue on the virtues of their favorite language and why it lends itself to writing maintainable code, but writing maintainable code is truly a skill. You can write obfusticated code in any language. It takes practice and a conscience effort of keeping your code clean and organized well. Here, practice makes perfect, is the key.

Secondly, and in my opinion the most important aspect of the language of choice is staying efficient. Ideally, each program should be succinct and to the point. I no longer use C/C++ regularly, even though that’s the language I started with, because you simply have to write much more code which another language can do in half or less of work. Try looking at one of the ‘P’s of the LAMP stack and see which fits you better and you can see yourself being productive in. That is, evaluate Python, Perl, PHP and Ruby (okay, not a ‘P’ but whatever). Don’t use a language that doesn’t make sense to you. Don’t waste your time.

And finally, time to explain this title and tell a little story where some customer data was delayed during one day’s production incident. One day, we had a production issue where messages were accidentally dequeued from a IBM Webphere MQSeries queue. A tool which was used to grab just one message dequeued all of the messages. To top it all off, the same tool kept seg faulting while trying to requeue the same messages. The solution left to us was to manually parse out each of the discrete messages into separate files. Once in that state, we had another known tool which could upload the messages separately. There were three developers and myself on the phone and we were all racing to the solution. My language of choice was Python and the rest of the developers used the language that they use professionally, Java. So who reached the solution first? Well I wouldn’t be writing this if I hadn’t won, would I? For me, Python makes sense and I can efficiently write code which I like to think other people will be able to understand and update. That is what is most important for your language of choice.

[ As un-entertaining as it is, you can view the Python solution. ]

Nothing too fancy here. Just a list of the most common things I find desirable while writing shell scripts.

1. Use meaningful variable names

This point is strictly for the sake of readability. Too often when trying to read somebodies script I’ll actually do various search & replaces of their variables because they used variables like “w”, “w2″, “w3″. It was quick and dirty for the author, but the inheritor of that script would appreciate if you had used more meaningful variable names.

2. Comment your code.

This goes without saying, really…

3. Visually separate any optional settings sections.

Don’t know about you, but sometimes I get lazy and don’t feel like using getopts. Instead, I’ll throw my what would be optional arguments as hard coded variables at the top of my script. I think this is fine, but you’ll want to visually segregate these optional variables from the rest of the script.
I like to use a — dashed line of about 50-70 characters and even put the words “do not modify beyond this point” to further emphasize what you’re encouraged to change and what shouldn’t normally be touched.

4. Use relative pathing for accessing files to the script.

Never assume the user’s cwd is the same as the script and use “./” to run or source another file. I like to set a variable REL_DIR=`dirname $0` and use it to reference the directory where the very script is running from.
E.g. You have a functions script you’d like to source, then with that REL_DIR variable you would “. ${REL_DIR}/<some-file>“.
I’m actually surprised on how often this happens.

5. Always print a usage statement for improper usage and/or when -h option used.

My code excerpt typically looks like:

USAGE="Usage: `basename $0` <my options here>"
if [ -z "$SOME_ARG" ]; then
    print $USAGE 1>&2
    exit 1
fi

6. Conscientiously use STDOUT vs. STDERR in different scenarios.

Not a script faux pau really, but it can help during the development process. Use STDOUT only for informational messages and/or optional debugging info. Then STDERR would only be used for errors. That way, when running the script you can optionally turn off stdout (1>&-) and easily check that nothing was printed to STDERR. When the output is mixed you’ll have a greater chance of missing the error.
One example of this technique in action is when using the tar command. Try leaving out the verbose (’v') option when creating or extracting your archive, then you can easily see when you might have had a permissions issue or something else related.

7. Keep all required variables defined in the script.

Define the required variables at the top of the script. Even mention that they are REQUIRED. A good example of this is scripts that use Sybase’s isql utility. Anytime I run isql, I like to set something like:

# required variables for isql
SYBASE=/some/path/to/sybase
LD_LIBRARY_PATH=$SYBASE/lib

What you want to avoid is a situation where the script works because you’ve got the required variable set in your env, but only because it’s set in one of your dot files.

8. Cron’ed Scripts.

Two common principles I like to emphasize here:
1. Keep all required variable/env settings in the script! cron does NOT source your dot files.
2. Redirect stdout, but leave stderr unmanaged. This is a cheap technique, but whenever I don’t have time to test for all possible errors I simply setup my .forward file and let cron email me the output produced from the cron script. Though, to be complete, you should really manage your stderr in other fashions.

9. Keep your exit/return codes categorized.

Not always important for small scripts, but a good practice.
For any sort of error checking your script might perform, use a unique error code for each situation that you decide to exit the shell script. That will make invocations of your script more manageable.

10. Avoid “Magic” Numbers

Anywhere you are comparing a value to some, seemingly, arbitrary number, go ahead and set that value to a meaningful variable name. Then your comparison reads alot better.
Using “$CURRENT_VALUE -gt $THRESHOLD” is much better than finding “$CURRENT_VALUE -gt 83” buried in some script and not having any clue what the number 83 signifies aside from the surrounding code.

11. Use unique temporary files.

Never do this: /path/to/some/command -option > command.out.
You are assuming that you are sitting in a directory where you have permissions to create a temporary file and secondly that no one will ever be running the same script at the same time you are.
Some shells make creating temporary files easy with commands such as mktemp. I typically employ a convention where I define my temporary file space as “TEMP=/tmp/.myshellname$$_“. Then lets say I need a temp file to capture the output from ps. I might redirect it to ${TEMP}raw_ps.
And finally, at the end of the script, or defined in a shell function, you can cleanup each temporary file with one line: rm -f ${TEMP}*.

In general, well written code/scripts should read well and be organized well. Every principle discussed above has one purpose: maintainability.

Lately I’ve found myself trying to further learn web technology and have been
evaluating various web frameworks. The motivation is to become much more
proficient with the most widely accepted user interface: The web browser.

Being a tradition Unix/Linux user, when I think of widely understood,
standardized interfaces, I’m going to be thinking of the good ‘ole shell. Just
run your unknown utility with a ‘-h’ or a ‘–help’ to get the impatience
user’s guide. That is really nice. A key piece of that consistency is how the options
are typically processed via the get-opts library.
Most people are familiar with the standard get-opts routines that are readily
available in your language of choice… from C to python. For those of you
unfamiliar with the get-opts routine, it is the library which makes all
command line options to a program standardized. That is why passing ‘-fp’ to
ps is the same as ‘-f -p’. What I admire about this library is
how successful it has been in unifying all sorts of utilities on Unix/Linux
for years. Heck, even tar came around after those early years!
So why not we take get-opts to the next level of convenience, usability and
power? I’ve already seen this trend start with python’s replacement of the
stock get-opts with href="http://www.python.org/doc/2.4/lib/module-optparse.html">optparse.
Can you guess where I’m going with this?

What if we supplied a bit more information than merely whether or not an
option takes an argument or not, and then use that extra information for
maximum power and usability? Imagine developing your latest utility and testing via command line, but then turning around and invoking it via a web browser, GTK window, TK window, etc? And why not?

In the simplest of implementations, a web page could provide radio buttons for
toggling various options as well as text form for any additional options. I
envision a metamorphosis of information and interface options while being ease to
use. After we get the basics working, we can start implementing more advanced
features, such as: grouping like options together in sections, allowing the
user to toggle layout to alphabetical, search options, keep advanced options
hidden by default, remember the last options used and much more! Think of what that’ll do for the Linux newbies?

I can not currently
call myself a web developer and honestly I am not anxious to become one
either, yet I want to make some of my utilities available via the web. I’ll spend an equal amount of time exposing my utility via the web as I spent developing it in the first place! Are you
the same? Care to develop the next generation of get-opts?