Category Archives: Tech

Tech

Adding Infiniband to my Bitcoin Mining Cluster for HPC Tasks (Part 1 – Overview)

I now have a cluster with InfiniBand network hardware in my garage. This is my bitcoin mining cluster that I’ve had running for a few years. Last summer I upgraded the CPUs from the cheapest available (Sempron 140s) to something faster but compatible with the same motherboards (Phenom II X4 975 BEs) so that I could run simulations for work, but I ran into scaling issues using 100 Mb/s Ethernet. At that point I got a cheap TrendNet gigabit switch and switched the cluster over to that, and I was able to scale across 2-3 machines (8-12 cores), but after that things really started to slow down.

I’m happy to report that with the InfiniBand hooked up as of yesterday I can now scale across all 6 IB-equipped compute nodes with approximately linear scaling. Unfortunately the older single-data-rate (SDR) hardware I got for ‘cheap’ from eBay didn’t produce the kind of dramatic reductions in latency I expected compared to gigabit Ethernet. I expect for that I’ll really need to upgrade to QDR IB at some point, but I’ll also probably want faster machines at that point. Total cost for the IB upgrade (24-port Cisco Topspin switch $350, 7 Mellanox HCAs for $39.90/ea – $279.30, 7 CX4 cables with latch connectors at $16.00/ea – $112.00) was $741.30. I intend to add some posts later with more details of the setup, as I know there some interest out there in setting up cheap IB hardware at home.

Installing Debian Wheezy (7.0) Linux on the Chromebook Pixel

UPDATE 2013-04-29

I have created a github for this here. If you have patches please submit pull requests!

UPDATE: I continue to update the kernel as more fixes make it into git. You can check all of the Pixel-related files I’m posting in this Drive folder.

UPDATE 2013-03-27: New kernel with fix for the audio pops, see my G+ post from today.

The Chromebook Pixel is a very nice (if expensive) piece of hardware, designed to run Chrome OS, which is a variant of Linux. Since being noted as favored by Linus Torvalds, inventor and lead maintainer of Linux, support for the various Pixel hardware components has rapidly been added to the kernel git repository.

Not everything is working great just yet, but all of the essential features are working. Here’s a walkthrough that I hope will be sufficient, based almost entirely on other people’s work and howtos. I’ll link to those where I can. Several Google software engineers have been helpful on Google+, and a bunch of work has been done by Linux kernel maintainers.

This was my starting point: DaveM’s howto in his Linux git repo

  1. Get a chromebook pixel
  2. Enable developer mode
  3. Download the Debian Wheezy netinstall image (yes, it supports the wifi in the installer) (here)
  4. Boot from the installer (Ctrl-L at boot screen, escape when it says to press escape to choose a boot device, choose your USB drive with the Debian installer)
  5. Install as normal to the internal SSD. I used LVM/encryption and it worked just fine. When you reboot, pull the USB drive and it should boot from the internal SSD.
  6. Once booted, the trackpad will not work. A USB mouse will work just great. Download and install my build of the 3.9-rc1 kernel (.deb files and config, full source) built from Linus’ merging of patches and configured with help from Benson Leung
  7. To your /etc/modules add: 
    ath9k
atmel_mxt_ts
chromeos_laptop
tpm_tis force=1 interrupts=0

    Those are the modules for the wifi, the touchpad (both the atmel and chromeos_laptop), and the tpm chip, that to keep it from rebooting when you try to suspend (thanks Duncan Laurie!).

  8. This is no longer necessary with the updated downloads. To your /etc/rc.local, above exit 0, add echo TSCR > /proc/acpi/wakeup. This is a hack to keep it from waking right up after going to sleep (thanks, Benson Leung!)
  9. This appears not to be necessary, actually Create a file called 01i8042 in /etc/pm/sleep.d to properly sleep and wake the keyboard on suspend. It should have this as its contents. 
    #!/bin/sh

###############################################################################
# Pm-utils script to unbind i8042 on hibernate/suspend and
# bind it on thaw/resume.
#
# Copyright: Copyright (c) 2009 Nicolay Doytchev
# License:   GPL-3
###############################################################################

###############################################################################
# INSTALL:
#   1. Copy this script to /etc/pm/sleep.d/
#   2. Make it executable:
#       sudo chmod +x /etc/pm/sleep.d/01i8042
#
# UNINSTALL:
#   1. Delete the script from /etc/pm/sleep.d/
#       sudo rm /etc/pm/sleep.d/01i8042
###############################################################################

case "$1" in
    hibernate|suspend)
        echo -n "i8042" > /sys/bus/platform/drivers/i8042/unbind
    ;;
    thaw|resume)
        echo -n "i8042" > /sys/bus/platform/drivers/i8042/bind
    ;;
esac

    Make it executable. Found this here

  10. You may need to install the firmware-atheros package for the bluetooth to work, which may require adding non-free to the end of the deb lines in /etc/apt/sources.list

I think that’s it, but it’s always hard to correctly recreate these things after the fact without redoing it (and I’ve spent enough time on this already). Let me know if you have any problems or improvements, or go comment on Linus’ G+ post about it.

I want to add a special thank-you to Linus Torvalds for posting about his updates on Google+.

EDIT: More stuff
First, if you hate tap-to-click like me, here’s an xorg.conf with the correct dpi setup and multi-finger click.

Here is a folder containing all the relevant files and info (plus my other configs I decide to upload) discussed in this post and the comments.

Also, FYI the keyboard backlight does not come on after suspend right now.

EDIT 2013-03-11 20:33 MDT: By the way, if you run i3 or some other nerdy window manager like me, and you forgot how to make Debian sleep when the lid is closed, you just need to uncomment the appropriate line in /etc/default/acpi-support.

Finally I can use my ScanSnap S1500 in Linux!

I have been using Fujitsu ScanSnap double-sided auto-feeding scanner for years. I started with a Mac model (S300M I think? no Windows drivers?) and then moved to a Windows model, the S1500.

For the last couple of years I’ve been running a Windows virtual machine in VMWare Workstation primarily so that I can use my scanner and the included OCR features (turn scans into text).

Recently, I’ve been having a lot of trouble with VMWare Workstation, and I run libvirt/qemu-kvm on my web server, so I decided to try it on my workstation as well. It works pretty well for Windows 7, not a fast as VMWare when it worked properly, but the scanner will not function correctly using it.

Last time I looked into using the S1500 on Linux I found almost nothing. I could scan stuff but it wasn’t very useful for a paperless office workflow, the whole reason I have such a scanner.

I am happy to say that, on Debian Wheezy/Sid, gscan2pdf with the libsane-perl backend and tesseract for OCR seem to work nicely. Some caveats:

  • I downloaded the newest version of gscan2pdf and installed it. At the moment that is 1.0.6.
  • The ‘Page Options’ tab of the ‘Scan Document’ window only works properly when I choose options that don’t make much sense. I selected ‘ADF Duplex’ in the ‘Standard’ tab, and then in the ‘Source document’ section of the ‘Page Options’ tab, I have selected ‘Single-sided’ and ‘Side to scan: Facing’. Otherwise, the page numbers come out strange. I do still get double-sided scanning, though.

I couldn’t find a lot about this on the web, so I hope it helps someone else in my position. Happy scanning!

EDIT 2012-10-28 Strangely enough, I was able to get the scanner working just fine in my Windows VM using Spice USB redirection. It’s good to have the fallback, but I’ll try to stick with Linux, it’s much more convenient and removes a dependency.

Retiring Virtually Shocking

I started Virtually Shocking when I had dreams of becoming the web’s foremost cardiac electrophysiology blogger, especially as it relates to simulation. As it turns out, I didn’t really have the interest, drive, or time required to achieve such lofty status. I would much rather blog about whatever interests me. Some of that relates to cardiac simulation, some does not.

I’ve therefore moved the blog to http://blog.brocktice.com. I’ve set up redirects and all of that good stuff, so old links should all still work.

I’ve taken down my fancy Virtually Shocking theme and replaced it with an oldish-looking default wordpress theme. This blog is about content, not presentation. At some point I’ll spiff it up a little.

I’m sad to see Virtually Shocking go, but if I’d been honest with myself, this should have become blog.brocktice.com a long time ago. I may still post Cardiac Electrophysiology stuff over at http://cardiosolv.com/blog/.

Which Internet Weather Site is Most Accurate?

I have been a long-time user of the Weather Underground, while my wife uses some Weather Channel app or something on her phone. We frequently have disagreements about what the high temperature for the day is/was supposed to be because we use these different services.

I really like the Weather Underground, but I must confess her numbers/rain probabilities seemed to jive better with what was happening, so I did a search to see if someone had analyzed the various weather sites for consistency and accuracy. Someone did, and how.

I’ll be honest. I mostly just skimmed through the site, because I was just interested in the conclusions (at first). An important caveat is that the analysis was done for only one location, by one guy, but I have to admire the effort and the statistical analysis.

In conclusion: I’m going to see if I can find an IntelliCast app for my phone.