Category Archives: Linux

Linux

Adding Infiniband to my Bitcoin Mining Cluster for HPC Tasks (Part 1 – Overview)

I now have a cluster with InfiniBand network hardware in my garage. This is my bitcoin mining cluster that I’ve had running for a few years. Last summer I upgraded the CPUs from the cheapest available (Sempron 140s) to something faster but compatible with the same motherboards (Phenom II X4 975 BEs) so that I could run simulations for work, but I ran into scaling issues using 100 Mb/s Ethernet. At that point I got a cheap TrendNet gigabit switch and switched the cluster over to that, and I was able to scale across 2-3 machines (8-12 cores), but after that things really started to slow down.

I’m happy to report that with the InfiniBand hooked up as of yesterday I can now scale across all 6 IB-equipped compute nodes with approximately linear scaling. Unfortunately the older single-data-rate (SDR) hardware I got for ‘cheap’ from eBay didn’t produce the kind of dramatic reductions in latency I expected compared to gigabit Ethernet. I expect for that I’ll really need to upgrade to QDR IB at some point, but I’ll also probably want faster machines at that point. Total cost for the IB upgrade (24-port Cisco Topspin switch $350, 7 Mellanox HCAs for $39.90/ea – $279.30, 7 CX4 cables with latch connectors at $16.00/ea – $112.00) was $741.30. I intend to add some posts later with more details of the setup, as I know there some interest out there in setting up cheap IB hardware at home.

Installing Debian Wheezy (7.0) Linux on the Chromebook Pixel

UPDATE 2013-04-29

I have created a github for this here. If you have patches please submit pull requests!

UPDATE: I continue to update the kernel as more fixes make it into git. You can check all of the Pixel-related files I’m posting in this Drive folder.

UPDATE 2013-03-27: New kernel with fix for the audio pops, see my G+ post from today.

The Chromebook Pixel is a very nice (if expensive) piece of hardware, designed to run Chrome OS, which is a variant of Linux. Since being noted as favored by Linus Torvalds, inventor and lead maintainer of Linux, support for the various Pixel hardware components has rapidly been added to the kernel git repository.

Not everything is working great just yet, but all of the essential features are working. Here’s a walkthrough that I hope will be sufficient, based almost entirely on other people’s work and howtos. I’ll link to those where I can. Several Google software engineers have been helpful on Google+, and a bunch of work has been done by Linux kernel maintainers.

This was my starting point: DaveM’s howto in his Linux git repo

  1. Get a chromebook pixel
  2. Enable developer mode
  3. Download the Debian Wheezy netinstall image (yes, it supports the wifi in the installer) (here)
  4. Boot from the installer (Ctrl-L at boot screen, escape when it says to press escape to choose a boot device, choose your USB drive with the Debian installer)
  5. Install as normal to the internal SSD. I used LVM/encryption and it worked just fine. When you reboot, pull the USB drive and it should boot from the internal SSD.
  6. Once booted, the trackpad will not work. A USB mouse will work just great. Download and install my build of the 3.9-rc1 kernel (.deb files and config, full source) built from Linus’ merging of patches and configured with help from Benson Leung
  7. To your /etc/modules add: 
    ath9k
atmel_mxt_ts
chromeos_laptop
tpm_tis force=1 interrupts=0

    Those are the modules for the wifi, the touchpad (both the atmel and chromeos_laptop), and the tpm chip, that to keep it from rebooting when you try to suspend (thanks Duncan Laurie!).

  8. This is no longer necessary with the updated downloads. To your /etc/rc.local, above exit 0, add echo TSCR > /proc/acpi/wakeup. This is a hack to keep it from waking right up after going to sleep (thanks, Benson Leung!)
  9. This appears not to be necessary, actually Create a file called 01i8042 in /etc/pm/sleep.d to properly sleep and wake the keyboard on suspend. It should have this as its contents. 
    #!/bin/sh

###############################################################################
# Pm-utils script to unbind i8042 on hibernate/suspend and
# bind it on thaw/resume.
#
# Copyright: Copyright (c) 2009 Nicolay Doytchev
# License:   GPL-3
###############################################################################

###############################################################################
# INSTALL:
#   1. Copy this script to /etc/pm/sleep.d/
#   2. Make it executable:
#       sudo chmod +x /etc/pm/sleep.d/01i8042
#
# UNINSTALL:
#   1. Delete the script from /etc/pm/sleep.d/
#       sudo rm /etc/pm/sleep.d/01i8042
###############################################################################

case "$1" in
    hibernate|suspend)
        echo -n "i8042" > /sys/bus/platform/drivers/i8042/unbind
    ;;
    thaw|resume)
        echo -n "i8042" > /sys/bus/platform/drivers/i8042/bind
    ;;
esac

    Make it executable. Found this here

  10. You may need to install the firmware-atheros package for the bluetooth to work, which may require adding non-free to the end of the deb lines in /etc/apt/sources.list

I think that’s it, but it’s always hard to correctly recreate these things after the fact without redoing it (and I’ve spent enough time on this already). Let me know if you have any problems or improvements, or go comment on Linus’ G+ post about it.

I want to add a special thank-you to Linus Torvalds for posting about his updates on Google+.

EDIT: More stuff
First, if you hate tap-to-click like me, here’s an xorg.conf with the correct dpi setup and multi-finger click.

Here is a folder containing all the relevant files and info (plus my other configs I decide to upload) discussed in this post and the comments.

Also, FYI the keyboard backlight does not come on after suspend right now.

EDIT 2013-03-11 20:33 MDT: By the way, if you run i3 or some other nerdy window manager like me, and you forgot how to make Debian sleep when the lid is closed, you just need to uncomment the appropriate line in /etc/default/acpi-support.

Finally I can use my ScanSnap S1500 in Linux!

I have been using Fujitsu ScanSnap double-sided auto-feeding scanner for years. I started with a Mac model (S300M I think? no Windows drivers?) and then moved to a Windows model, the S1500.

For the last couple of years I’ve been running a Windows virtual machine in VMWare Workstation primarily so that I can use my scanner and the included OCR features (turn scans into text).

Recently, I’ve been having a lot of trouble with VMWare Workstation, and I run libvirt/qemu-kvm on my web server, so I decided to try it on my workstation as well. It works pretty well for Windows 7, not a fast as VMWare when it worked properly, but the scanner will not function correctly using it.

Last time I looked into using the S1500 on Linux I found almost nothing. I could scan stuff but it wasn’t very useful for a paperless office workflow, the whole reason I have such a scanner.

I am happy to say that, on Debian Wheezy/Sid, gscan2pdf with the libsane-perl backend and tesseract for OCR seem to work nicely. Some caveats:

  • I downloaded the newest version of gscan2pdf and installed it. At the moment that is 1.0.6.
  • The ‘Page Options’ tab of the ‘Scan Document’ window only works properly when I choose options that don’t make much sense. I selected ‘ADF Duplex’ in the ‘Standard’ tab, and then in the ‘Source document’ section of the ‘Page Options’ tab, I have selected ‘Single-sided’ and ‘Side to scan: Facing’. Otherwise, the page numbers come out strange. I do still get double-sided scanning, though.

I couldn’t find a lot about this on the web, so I hope it helps someone else in my position. Happy scanning!

EDIT 2012-10-28 Strangely enough, I was able to get the scanner working just fine in my Windows VM using Spice USB redirection. It’s good to have the fallback, but I’ll try to stick with Linux, it’s much more convenient and removes a dependency.

Firefox 13.0.1 for Linux Not Working With Flash Plugin?

I’ve been a little annoyed with how Ubuntu has been going lately (even though I think it’s a great distribution for new Linux users), and so I’ve been moving my systems back to Debian. My workstation, whose filesystem originated I think on Ubuntu 8.04 and was running 10.04, was the last one to go.

I had some issues because I run RAID1 for my system disks, and then some issues with crypto and booting, but finally I got it up and running, only to find that the brand-new Firefox 13.0.1. did not work with Flash. At all. I couldn’t even see the flash plugin in about:plugins. My searches were all confounded by apparent problems between the latest Flash and Firefox 13 on Windows. However, I’m partly writing this post to help change that.

This morning, my laptop (already running the same version of Debian) came due for its firefox upgrade, so I decided to see whether the same thing happened on upgrade. Sure enough, it did. I’m not sure why I didn’t think of this on Thursday with the workstation, but I checked the filetype of the firefox 13.0.1 binary, and found that it was 32-bit. This is a little funny if you’re an old hand at running 64-bit firefox on Linux, because it used to be the case that none of the plugins were 64-bit and required using nspluginwrapper to run them at all. In this case I had the opposite problem — most of my plugins were 64-bit, but I had a 32-bit browser that couldn’t link to them.

I’m not sure why the Firefox site always gave me the 64-bit download automatically before, and this time gave me the 32-bit download, but I’ll spare you the trouble of trying to find it yourself. Go here (the Mozilla ftp server in the Firefox releases directory), select your release (in this case 13.0.1), and then linux-x86_64. Then select your language (for me, en-US), and finally, the 64-bit Firefox download.

Once I loaded the 64-bit Firefox, sure enough, my plugins came back. Here’s some stuff to help others find this via search engines:

Debian firefox 13 libflashplayer.so linux 32-bit 64-bit flash plugin about:plugins

Extracting certain lines from a file using Perl

As often happens to me when processing data, I needed to extract a specific set of lines from many files. If the files are small or few, or the list of lines is short, this can be done manually. If the list of lines or number of files are large, well, that’s what computers are for.

As Google will reveal, there are a number of ways to hack this up with bash, grep, sed, and other command-line tools, but none (that I know of) are really designed for this.

Here’s my script. It’s designed around some heart model stuff, and we use base-0 (first line is line 0, second line is line 1, etc), while many other tasks require base-1 (first line is line 1, second line is line 2, etc), and I’m sure there are other numbering schemes out there. Therefore, I included an optional base parameter. Without that it assumes base 1.

Note: I have no idea if this works on non-UNIX-like systems (i.e. Windows).

Use it like:

extract_lines.pl <file with list of lines> <file from which to extract lines> [optional base number]

And now, the code:

#!/usr/bin/env perl

use strict;

unless(@ARGV == 2 || @ARGV == 3){
    die "Usage: extract_lines.pl <line number file> <source file> [base]\n";
}

open(NUMBERS, "<$ARGV[0]") || die "Failed to open line number file $ARGV[0] for read: $!\n";
chomp(my @numbers = <NUMBERS>);
close(NUMBERS);

open(SOURCE, "<$ARGV[1]") || die "Failed to open source file $ARGV[1] for read: $!\n";
chomp(my @source = <SOURCE>);
close(SOURCE);

my $base = 1;

if(@ARGV == 3){
    $base = $ARGV[2];
}

# sort inputs just in case
my @sorted = sort { $a <=> $b } @numbers;

# save mem
undef @numbers;

my $nextline;

$nextline = shift(@sorted);
my $currline = $base;

foreach my $line (@source){
    if($nextline == $currline){
        print $line . "\n";
        if(@sorted > 0){
            $nextline = shift(@sorted);
        }
    }
    $currline++;
}

Bug reports welcome, just leave a comment.