Using awk to work on data files

In order to average numbers in a file we can use awk as so:

$ cat meyer-heavy.txt | awk '{ sum+=$1;count++ } END {print sum/count}'

To find the largest value:

$ cat meyer-heavy.txt | awk 'BEGIN{max=-9999} $1 > max {print $1; max=$1}' 

Improved version (only prints largest value, not intermediate ones): 

$ cat meyer-heavy.txt | awk 'BEGIN{max=-9999} $1 > max {max=$1} END{print max}'

To find the smallest value:

$ cat meyer-heavy.txt | awk 'BEGIN{min=9999} $1 < min { print $1; min=$1 }' 

Fixed version (doesn't assign blank lines as min value): 

$ cat meyer-heavy.txt | awk 'BEGIN{min=9999} /\-[0-9]+/&&$1 < min { min=$1 } END{print min}'

Where $1 is for column 1, use $2 for column 2 etc...

Grep and recursion

Grep has for a long time had a recursive option "-r" but I have never been able to get it to do what I want because if you write

$ grep -r test *.txt

The *.txt part specifies the files to look in, and since the folders I want to recurse don't end with .txt it doesn't recurse into them.
The correct way to cause grep to recurse is to specify file patterns separately.

$ grep -r test --include "*.txt" ./

Getting the correct default paper size in LibreOffice Writer

For years OpenOffice/LibreOffice has ticked me off by endeavoring to give me documents that after I walk down the hall to the printer, are asking for A4 sized paper, though I live in the US where we use "Letter" sized paper.

Even though I set the defaults for LibreOffice, and even though the printer and any printer settings are ALL set to Letter.

Well, I found this documentation of what is a really stupid way to have to make this change actually happen. Create a template, and set it as "default" and I guess this works.

From: https://help.libreoffice.org/Writer/Changing_the_Default_Template

1. Create a document and the content and formatting styles that you want.
2. Choose File - Templates - Save.
3. In the New Template box, type a name for the new template.
4. In the Categories list, select "My Templates", and then click OK.
5. Choose File - Templates - Organize.
6. In the Templates list, double-click the "My Templates" folder.
7. Right-click the template that you created, and choose Set as Default Template.
8. Click Close.

Now if I can only get Impress to stop setting my default language as Finnish...

Git tutorial

While on the subject of danielmiessler.com, I noticed his GIT tutorial which is really easy to read:
http://danielmiessler.com/study/git/

Using tcpdump

One thing I can never seem to remember is how to get tcpdump to show entire packets.

The option is -s (for snaplength). So either -s 1514 or -s 0 will cause tcpdump to capture entire packets.

More good info on using tcpdump is at danielmiessler.com/study/tcpdump/.

Apache reverse proxy to redmine

I had all kinds of issues trying to get this to work. I'm trying to proxy from my servers /redmine folder to a container VM running redmine. The problem has been that the container was running redmine in its root folder.

The solution was to tell the container to run redmine in a /redmine folder, from http://www.redmine.org/projects/redmine/wiki/HowTo_Install_Redmine_in_a_sub-URI

Simply had to add:
Redmine::Utils::relative_url_root = "/redmine"
to the end of Redmine's environment.rb and restart Apache

Recording test audio

Linux distros I have used have no built in record software which I have wanted in order to test microphone levels before creating screencasts.
In Arch, install the "alsa-utils" package, then you can record/play with the following utilities:

$ # Record 5 seconds of audio, then play it back.
$ arecord -d 5 test-mic.wav
$ aplay test-mic.wav

VMWare player in Arch Linux

I don't use vmware too often, I prefer VirtualBox, but when I went to start it, it couldn't find kernel headers, I found what looked to be a solution in the Arch wiki (https://wiki.archlinux.org/index.php/Vmware#VMware_module_patches_and_installation) but still had issues. I finally installed the package mentioned in the wiki "wmare-patch" from the AUR repositories and everything was fine.

I will simply need to remember to re run this program each time I have the issue.

# vmware-patch

Using netstat

I use

# netstat -pant4

all the time to see what is listening/connected to my computer.
I found some other useful commands I didn't know about at http://openmaniak.com/netstat.php

To display all the opened network sockets (extended informations):

#netstat -aute
-a: All
-u: UDP
-t: TCP
-e: Extended

To display the summary statistics for each protocol

#netstat -s
-s: Summary statistics for each protocol.

To display the extended interfaces statistics:

#netstat -ie
-i: Interface
-e: Extended information

Parsing columns in BASH

I sometimes need "cut" functionality that works for fields with more than a single space. For instance, I sadly have a script just to kill nepomukindexer because it constantly goes berzerk and uses all my 8gb of ram. 

So I need to parse the process IDs of all the processes that contain the word nepomukindexer (there are hundreds at times).

First try

I figured cut would do the trick.

ps -ef | grep nepomukindexer | cut -d " " -f 5 

This only works if there are exactly 4 spaces between the two fields, and that isn't always the case.

Second try

I still don't understand why sed doesn't seem to work with the "+" regular expression symbol the way I expect. The following seems like it should replace all instances of one or more spaces with a single space, but it does nothing.

ps -ef | grep nepomukindexer | sed 's/ +/ /g'

Edit: I just found a workaround for this. I changed this to

ps -ef | grep nepomukindexer | sed 's/  */ /g'
                                      ^^ two spaces

and it worked, the only difference being the matching part which says match a single space followed by 0 or more additional spaces, and replace the whole mess with a single space.

Third try

The solution I found is to use awk.

ps -ef | grep nepomuk | awk -F" " '{print $2}'

Using this command, I pipe the results through kill and all is well.

Blogilo take 2

I've had no luck at all with Blogilo over the past month or two. But I just noticed an item in the Arch AUR called "python-gdata-svn". After installing it, I got a test post to work, so this is my second try. If you see this, then this may be something that needs to be installed in Arch (64 bit anyway) in order for Blogilo to work.

Edit: Seems to work, even allowed me to update my original post. Though truthfully I can't say for sure that its because I installed python-gdata, but it did work immediately afterwards...

Non-blogilo-edit:
Apparently Blogilo is trying to upload some HTML elements that Blogger doesn't care for.
The problem returned so started Blogilo from command line and saw the following output:

blogilo(7501): Could not regexp the id out of the result: "The element type &quot;br&quot; must be terminated by the matching end-tag &quot;&lt;/br&gt;&quot;." 

So I switched to HTML view and removed the <div><br></div> but still had issues:

blogilo(7501): Could not regexp the id out of the result: "The element type &quot;br&quot; must be terminated by the matching end-tag &quot;&lt;/br&gt;&quot;."

But I didn't see any "&nbsp" so I quit and restarted and saw a hitherto invisible "&nbsp;" that I removed at which point the post worked. 

Blogging clients: Blogtk, Drivel

After the issues with Blogilo, I installed BloGTK and Drivel. I'm making this post from BloGTK. Setup was similar to that of Blogilo. The interface is pretty similar. There is a place for "tags" but it won't allow me to enter anything.

I have not been able to get "drivel" to successfully log in as of yet.

Disable wildcard expansion when calling BASH scripts

This is a rarely needed, but very useful item. I wanted to create a script that I could pass wildcards into. But bash will automatically expand wildcards before the script can get them, so running

$ myscript.sh *

will call myscript with a bunch of filenames instead of the "*" character.

The solution is to create an alias to disable globbing:

noglob_helper() {
    "$@"
    case "$shopts" in
        *noglob*) ;;
        *) set +f ;;
    esac
    unset shopts
}

alias noglob='shopts="$SHELLOPTS"; set -f; noglob_helper'

Now you can preface your script with the alias and the wildcards will not be expanded:

$ noglob myscript.sh *

This code is from Simon Tatham's home page (the author of PuTTY among other things) http://www.chiark.greenend.org.uk/~sgtatham/aliases.html