Category Archives: Tech

Tech

Resetting the web server on the Morningstar Tri-Star MPPT 60 Solar Charge Controller

I have a solar-powered communications tower site for my WISP running on the MPPT60 as listed in the title. That controller is way more charge controller than I needed for the site, and pricey too, but I bought that one because it has a built-in network-connected monitoring and management system.

Usually I just check the web interface to make sure everything is good. It can be a little flaky and today it stopped responding. I didn’t want to have to cut power to the device, maybe possible without taking down my tower equipment’s power for a few minutes, maybe not, and wanted to make sure it was still functioning correctly.

The solution for me, at least this time, was to connect to the device in MS View (it was still pinging), and right-click on the controller, then select ‘Properties’, then the ‘Control’ tab, and then select ‘Reset Communications Server’ and click ‘Force Coil On’.

A few minutes later I could access the web status page again. I hope this helps someone else because searching for this information turned up next to nothing.

gpg-agent not working for ssh

I recently did a linux reinstallation on my workstation after more than 4 years of cruft accumulation, but couldn’t get ssh to work with my cryptostick. I did all of the normal stuff required to disable ssh-agent and enable ssh-agent support in gpg-agent.

Turns out I needed to install the ‘gpgsm’ package (on Debian/Ubuntu), which replaces the normal gpg-agent with one that supports a smart card.

I’m posting this mostly for my own reference, but maybe it will help you too.

Sywtbals? – Assignment 2

This is the second in my “So you want to be a Linux sysadmin?” series. See the first post here and the category here.

Now that you have completed Assignment 1 and have a working sandbox Linux system installed, it’s time to cover the basics.

First, when I started using Linux, I got a boxed set of RedHat that included a nice cheat sheet sticker to go on a keyboard wrist rest. In lieu of that, I just saw this posted today. I recommend you keep it bookmarked and perhaps printed out for reference: Linux Command Shelf Cheat-Sheet. There’s a PDF for download on that site. I also recommend you somehow (RSS or Facebook) subscribe to NixCraft. I still regularly learn useful things from that blog.

Now, when taking care of a Linux (or other *NIX) system, you’re often going to need to download and extract file archives. Typically these are either singly-zipped files (.gz or .bz2 from gzip and bzip2, respectively), or so-called “tarballs” (.tar, .tar.gz, .tgz, .tar.bz2, etc). Tarballs are created with the tar program, whose name comes from Tape ARchive. The additional suffixes indicate that the tarball has been compressed in some way. In other words, a simple .tar file contains a bunch of files all rolled up into a single archive, but not compressed, while .tar.gz (.tgz for short) files have additionally been zipped up in some fashion. Incidentally, tar is also still used to write files to tape, if that’s your thing.

The best way to get a file for download from the command-line (which I will focus on almost entirely in this series) is wget. So, open up a terminal window. You should have a simple prompt waiting for your input. Check to see whether you have wget by typing


which wget

The which command searches through all of the directories on your PATH, an environment variable that tells the shell where to find programs, for the command you list, in this case wget. It’s a good way to check for the presence of a program on a system. If that doesn’t work, but you think it’s around somewhere, you can sometimes use locate, but we’ll not cover that today.

Now, if you have wget, you should get in return the full path to the program, something, like /usr/bin/wget. If so, you can proceed to the next step where it says ‘Downloading a file with wget’.

If not, the utility will print nothing. I tried a made-up program called ‘snarfblat’:


brock@gamont:~$ which snarfblat
brock@gamont:~$

Here’s a successful which call for wget:


brock@gamont:~$ which wget
/usr/bin/wget
brock@gamont:~$

If you don’t have it, try sudo apt-get install wget on a Debian-based distribution or sudo yum install wget on a RedHat-based distribution. If you’re not set up with sudo, we will cover that later. For now become root with su - followed by the root password, and then run the above commands without the sudo at the beginning. Type exit or press Ctrl-D to exit the root shell once you have installed it. Double-check that it’s now installed by calling which wget again. If you get stuck, please email me or comment here so we can take care of it and you can continue.

Downloading a file with wget

For this tutorial, I’m going to have you download a sample tarball that I have created. It contains a bunch of directories and files with various names and contents. We will use it in this assignment and later assignments to practice command-line basics.

For now, go to a terminal (I recommend you make a directory for this like mkdir ~/sywtbals, where ~/ is a shortcut for specifying your home directory. In my case that is /home/brock. Then change to that directory: cd ~/sywtbals.

Now, run wget to pull down the file.


wget http://blog.brocktice.com/sywtbals/1.tgz

It should look like this:


brock@water:~/sywtbals$ wget http://blog.brocktice.com/sywtbals/1.tgz
--2013-05-05 17:30:33--  http://blog.brocktice.com/sywtbals/1.tgz
Resolving blog.brocktice.com... 209.188.113.105
Connecting to blog.brocktice.com|209.188.113.105|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 215040 (210K) [application/x-gzip]
Saving to: “1.tgz”

100%[======================================>] 215,040      259K/s   in 0.8s    

2013-05-05 17:30:34 (259 KB/s) - “1.tgz” saved [215040/215040]

brock@water:~/sywtbals$

Give yourself a self-high-five if you noticed I changed machines in the middle of this post from gamont to water.

Now that you’ve got the file, let’s take a look at it with ls, the file-listing tool:


brock@water:~/sywtbals$ ls -l 
total 212
-rw-r--r-- 1 brock brock 215040 May  5 13:40 1.tgz
brock@water:~/sywtbals

I used the -l for the ‘long’ listing that shows details about each file. This shows us a few interesting things. One is the permissions:


-rw-r--r-- brock brock

This tells us the permissions for the user, group, and everyone else. A file may only have one group and one user assigned to it. There’s SELinux (Security-Enhanced Linux), the NSA’s enhancements to make Linux permissions more fine-grained, but honestly I don’t understand SELinux well enough to do more than turn it off, so I’m not going to discuss it here.

So, in this case the user has read and write permissions, the group has read permissions, and everyone else also has read permissions. The user is brock and the group is brock. Note that even though the user and group names are the same they are totally separate. One represents a userid, and the other represents a groupid. Observe the use of the id command to elaborate:

brock@water:~/sywtbals$ id
uid=1000(brock) gid=1000(brock) groups=1000(brock),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),100(users),108(netdev),109(bluetooth),112(fuse),115(scanner),120(libvirt),1001(family)
brock@water:~/sywtbals$

Here you can see that (and again, this is just a coincidence because I was the first user added to the system), the user id brock is 1000, and the group id brock is also 1000. However, the group id brock could just as easily be 2005 or something.

Let’s say this file is somewhat secret and I don’t want anyone else (except root of course) on the system to be able to read it. We’ll change it like this:

brock@water:~/sywtbals$ chmod go= 1.tgz
brock@water:~/sywtbals$ ls -l
total 212
-rw------- 1 brock brock 215040 May  5 13:40 1.tgz

What I did is to tell chmod to set the group and other permissions to be equal to nothing. As a result, only the user has any permissions at all.

On the other hand, what if we want to allow anyone on the system to read and write the file?


brock@water:~/sywtbals$ chmod go+rw 1.tgz 
brock@water:~/sywtbals$ ls -l
total 212
-rw-rw-rw- 1 brock brock 215040 May  5 13:40 1.tgz
brock@water:~/sywtbals$

You can also use numerical codes to set permissions. All I ever remember about that offhand is that 0 means nothing and 7 is everything. I prefer the other notation, but you can look it up. A common 4-letter admonition you’ll see as a Linux sysadmin is RTFM — read the fucking manual.

Usually you can find the so-called manfile for a program or configuration file like this:


man chmod

This will show the output in the system pager, a program that pages through a text file. Historically this was normally a program called, sensibly, more. Later, an improved program called less replaced it. UNIX humor — less is more. Anyway, there are only four things you need to know to get around less reasonably well. The q key quits. The spacebar moves to the next page. And the / key starts a search. The Esc key gets you back to the main mode. There’s a lot more you can do with less, but those four things will make up 99% of your less usage.

The most important thing you can do with the / search in many manpages as a beginner is to find the Examples section, if there is one. It will usually contain examples of what you’re trying to do. Of course you can also RTFM. Here’s me searching for the examples in the chmod manpage. Note the /examples at the end.


CHMOD(1)                         User Commands                        CHMOD(1)

NAME
       chmod - change file mode bits

SYNOPSIS
       chmod [OPTION]... MODE[,MODE]... FILE...
       chmod [OPTION]... OCTAL-MODE FILE...
       chmod [OPTION]... --reference=RFILE FILE...

DESCRIPTION
       This manual page documents the GNU version of chmod.  chmod changes the
       file mode bits of each given file  according  to  mode,  which  can  be
       either a symbolic representation of changes to make, or an octal number
       representing the bit pattern for the new mode bits.

       The format of a symbolic mode is  [ugoa...][[+-=][perms...]...],  where
       perms  is  either zero or more letters from the set rwxXst, or a single
       letter from the set ugo.  Multiple symbolic modes can be  given,  sepa‐
       rated by commas.

       A  combination  of the letters ugoa controls which users' access to the
       file will be changed: the user who owns it  (u),  other  users  in  the
/examples

Unfortunately, when I hit the Enter key after typing /examples I was greeted with Pattern not found (press RETURN). So I’ll have to properly RTFM. I press the spacebar to scroll down. (Arrow keys and page up/down can also be used to navigate.) Eventually I find this documentation on the octal numerical codes:


       granted  to  users  that are in neither of the two preceding categories
       (o).

       A numeric mode is from one to  four  octal  digits  (0-7),  derived  by
       adding up the bits with values 4, 2, and 1.  Omitted digits are assumed
       to be leading zeros.  The first digit selects the set user ID  (4)  and
       set group ID (2) and restricted deletion or sticky (1) attributes.  The
       second digit selects permissions for the user who owns the  file:  read
       (4),  write  (2),  and  execute  (1); the third selects permissions for
       other users in the file's group, with the same values; and  the  fourth
       for other users not in the file's group, with the same values.

       chmod never changes the permissions of symbolic links; the chmod system
       call cannot change their permissions.  This is not a problem since  the
       permissions  of  symbolic links are never used.  However, for each sym‐
       bolic link listed on the command line, chmod changes the permissions of
       the pointed-to file.  In contrast, chmod ignores symbolic links encoun‐
       tered during recursive directory traversals.

SETUID AND SETGID BITS
       chmod clears the set-group-ID bit of a regular file if the file's group
       ID  does  not  match the user's effective group ID or one of the user's
       supplementary group IDs, unless the user  has  appropriate  privileges.
 Manual page chmod(1) line 44

There you go, the numerical codes are read = 4, write = 2, and execute = 1. Execute means that if it's a script or a program, it can be run, or if it's a directory, you can navigate (change directory or cd) into it. Anyway, to set read/write for user and group, and nothing for everyone else, you'd use (4+2=6)(4+2=6)(0) or chmod 660.

So, now that we've beaten the permissions thing to death. Just a few other notes. In this output:

-rw------- 1 brock brock 215040 May 5 13:40 1.tgz

The last few bits are the file size (215040 bytes) and file modification time, ending with the filename. You can get a handy "human-readable" size using the -h for human flag like so:


brock@water:~/sywtbals$ ls -lh 1.tgz 
-rw-rw-rw- 1 brock brock 210K May  5 13:40 1.tgz
brock@water:~/sywtbals$

I usually use the -h when doing a detailed listing. Because it rounds, however, sometimes you may want to omit the -h to see if a file size matches, has changed, etc.

OK! Let's extract this sucker. We'll use a tar command to extract. First, let's get a preview of the file contents. Nice people generally put everything to tar in a directory first, so that the directory extracts nicely and doesn't make a mess. Sometimes, (and occasionally even with good reason), people will put all the contents of the tar right at the top level, and extracting without knowing that will make a big mess. (I'll cover a quick tip to fix that when we get to pipes. Don't let me forget.).

To get a preview of the file contents, we use the t flag to tar:


brock@water:~/sywtbals$ tar tvf 1.tgz
drwxrwxr-x brock/brock       0 2013-05-05 13:31 1/
drwxrwxr-x brock/brock       0 2013-05-05 13:30 1/sequential/
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential/sequential_file_85
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential/sequential_file_34
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential/sequential_file_60
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential/sequential_file_43
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential/sequential_file_30
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential/sequential_file_64
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential/sequential_file_11
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential/sequential_file_77
...
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential_padded/sequential_file_00093
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential_padded/sequential_file_00049
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential_padded/sequential_file_00013
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential_padded/sequential_file_00068
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential_padded/sequential_file_00018
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential_padded/sequential_file_00055
-rw-rw-r-- brock/brock      12 2013-05-05 13:31 1/sequential_padded/sequential_file_00005
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential_padded/sequential_file_00098
-rw-rw-r-- brock/brock      12 2013-05-05 13:31 1/sequential_padded/sequential_file_00001
-rw-rw-r-- brock/brock      13 2013-05-05 13:31 1/sequential_padded/sequential_file_00026
brock@water:~/sywtbals$

The command tar tfv 1.tgz can be broken down as follows: tar is the program, t tells it to test the file, f tells it it's getting a filename for input (1.tgz in this case), and v means verbose, in that it prints out all the names of the files being extracted. You'll always need f to extract a file. The philosophies on v vary. Some people say using it causes errors to get lost in the shuffle and go unnoticed. Others like to see which files are being extracted. If you do a run like this with t first, then you can go ahead and do the actual extraction later without v and see any errors.

I'm a nice guy, so I put the contents in a directory called 1. (I'm also an idiot, this is Assignment #2, but I digress). So let's go ahead and extract the file.


brock@water:~/sywtbals$ tar xf 1.tgz 
brock@water:~/sywtbals$

In this case I omitted the v option, and changed from test mode to extract mode. tar followed the UNIX convention of outputting no message at all if everything worked correctly.

Do a quick listing on the created directory just to double-check:


brock@water:~/sywtbals$ ls -l 1
total 8
drwxr-xr-x 2 brock brock 4096 May  5 13:30 sequential
drwxr-xr-x 2 brock brock 4096 May  5 13:31 sequential_padded
brock@water:~/sywtbals$

Looks good! Questions?

I have a bonus question. tar figured out something about our 1.tgz file and took care of it without us asking. Do you know what that is?

So you want to be a Linux sysadmin?

Introduction

There is a shortage of good Linux system administrators. Some friends of mine have an interest in helping to fill that shortage, so I have finally decided to embark upon a series of blog posts based on my experience. First, let’s get some things out of the way.

Qualifications

There are plenty of linux sysadmins out there, with varying degrees of experience, and varying methods and opinions. I do not claim to be the leading expert on the subject, I’m just trying to write up what I do know. Constructive comments and suggestions are welcome. People telling me what an idiot I am will have their comments deleted. That said, here are my qualifications regarding Linux system administration. I offer them both to show you I’m not just making this stuff up, and also to give you one example of how people learn Linux sysadmin skills without a formal education or certificate program.

  • I started using Linux in 1998 with a Linux for Dummies book and the included copy of Red Hat 4.something.
  • I’ve run Linux servers in some form and with varying degrees of success since college in 2000.
  • I took about half of a computer science major in college as a double-major until I dropped that to focus on research, next point.
  • When I joined a computational science research lab as an undergrad in fall of 2002, I took over the Linux/Irix system administration for the lab, and started porting old Irix applications to run on Linux.
  • When our lab moved to Johns Hopkins University after Hurricane Katrina, I set up our compute nodes (formerly run by a Tulane sysadmin) as a high-performance computing (HPC) cluster. This was the real beginning of my HPC experience.
  • Shortly thereafter, we ordered a new cluster for the lab. I handled the process of deciding specifications, working with vendors to get quotes and negotiate the deal, and then running the cluster (from Penugin Computing) once it was delivered.
  • After graduating, I did some consulting work for Penguin Computing on a variety of HPC tasks. I think this went pretty well based on the feedback I got from Penguin and end customers.
  • At this point I’m running a high-performance cluster in my garage for my work at CardioSolv Ablation Technologies, and also running my own server hosting this blog and a bunch of other stuff.

Disclaimers

  • Just to reiterate: I do not claim to be the leading expert on the subject, I’m just trying to write up what I do know.
  • With Linux, there are usually a minimum of three different ways to do a certain thing or solve a different problems. Every time I work next to another sysadmin I learn stuff from them, and at least a few times they’ve learned things from me. Nothing anyone writes about how to do stuff is gospel.
  • I will not be responsible if you destroy systems or data from following my instructions without understanding them.

With that out of the way, let’s get started.

Welcome to Linux

There are many books you can buy and sites you can read that will give you a history of Linux and an overall philosophy for how it works, etc etc. I’m going to assume if you’re reading this that you already know all that stuff and jump right in. In my experience, the best way to learn how to be a Linux sysadmin is to try to make things work. You’ll learn a lot in the process, so I’m going to do this as a series of assignments.

Assignment 1

If you haven’t already, set up a Linux computer or a virtual machine (using VirtualBox, VMware, or the like), strictly for practicing on. I recommend you start with Debian or CentOS, or possibly Ubuntu Server. I prefer Debian, but most popular Linux distributions are based on either Debian or RedHat, so learning either of those will get you off to a good start. To be good at Linux system administration, you’ll eventually need to know your way around both. We’ll get to that (and some other interesting distributions like Gentoo) in later assignments.

Set up this machine with three partitions: /, /boot, and /home. Leave a minimum of 200MB for /boot, 20GB for /, and the rest assign to /home.

When done, you should be able to log in as your primary user (i.e. brock) as well as root, the god-like default system administration account.

Please reply or email me with any questions.

Adding Infiniband to my Bitcoin Mining Cluster for HPC Tasks (Part 1 – Overview)

I now have a cluster with InfiniBand network hardware in my garage. This is my bitcoin mining cluster that I’ve had running for a few years. Last summer I upgraded the CPUs from the cheapest available (Sempron 140s) to something faster but compatible with the same motherboards (Phenom II X4 975 BEs) so that I could run simulations for work, but I ran into scaling issues using 100 Mb/s Ethernet. At that point I got a cheap TrendNet gigabit switch and switched the cluster over to that, and I was able to scale across 2-3 machines (8-12 cores), but after that things really started to slow down.

I’m happy to report that with the InfiniBand hooked up as of yesterday I can now scale across all 6 IB-equipped compute nodes with approximately linear scaling. Unfortunately the older single-data-rate (SDR) hardware I got for ‘cheap’ from eBay didn’t produce the kind of dramatic reductions in latency I expected compared to gigabit Ethernet. I expect for that I’ll really need to upgrade to QDR IB at some point, but I’ll also probably want faster machines at that point. Total cost for the IB upgrade (24-port Cisco Topspin switch $350, 7 Mellanox HCAs for $39.90/ea – $279.30, 7 CX4 cables with latch connectors at $16.00/ea – $112.00) was $741.30. I intend to add some posts later with more details of the setup, as I know there some interest out there in setting up cheap IB hardware at home.