Monthly Archives: March 2008

Killing Zombies

Occasionally, on our cluster, a node will crash. If a job was running on it that spanned multiple nodes, sometimes the other nodes won’t get the message that their fellow has crashed, and they will just keep running whatever processes are on them.

I call these “Zombie” processes, because they just lumber along eating CPU time and rotting, keeping other jobs from using the node. Today, after noticing a particularly bad zombie infestation, I finally created a script that checks for zombified machines and then restarts them. This script is compatible with Torque and relies on Scyld Beowulf “b-commands” and IPMI, but you could easily replace them with similar utilities like rsh or ssh.


#!/bin/bash
# For all of the nodes in the main cluster...
for NODE in `seq 0 119`; do
     # Calculate the load and convert it to an integer
     LOAD=`bpsh $NODE uptime | awk '{ print $11 }' | sed "s/\,//"`
     LOAD=`printf %1.0f $LOAD`
     # Figure out whether the node should be running anything
     ASSIGNED=`qstat -f | grep $NODE | wc -l`
     if [ $ASSIGNED -gt 0 ]; then
          ASSIGNED=1
     fi
     # If the node is running something but shouldn't be, reboot it.
     if [ $LOAD -gt 1 ] && [ $ASSIGNED -eq 0 ]; then
          echo Node $NODE is a zombie! Kicking. >> /root/logs/zombies.log
          # This relies on IPMI
          ipmitool -H 10.54.2.$(( 100 + $NODE )) -U (some user) -P (some password) power reset
     fi
done

You can download the file directly: Zombie Checker

(Here’s another post on Zombies).

To Miss New Orleans

I have just returned from my second-to-last visit to New Orleans to see Amanda. Soon, we’ll be living in the same place and those trips will no longer be necessary. Despite all of the hassles of flying, despite missing Amanda, the one benefit of living apart has been an excuse to go see New Orleans regularly even after I’d moved away.

In two months, I’ll visit for one last week, and then the trips will largely stop.

We hope to move back there when Amanda is done with residency, if we can swing it. There will probably be some visits for holidays, as my family’s home is still just outside the city. Nonetheless, it’s slowly dawning on me that my monthly, re-charging dose of New Orleans is coming to an end.

At this point, the legacy of Katrina in my life is diminishing. Assuming that all continues to go well, I’ll be graduating from Johns Hopkins rather than Tulane. All sorts of other things both in my head and in my life have changed as a result of the flood. However, I will once again be living with my wife, we’ll go on with our post-school and post-New-Orleans lives. I’ll finally start to get some closure on something that can never really be fixed or undone.

Cognitive Distortions and the Five Hindrances (Post 1)

(This is the first in a series of unknown length on my thoughts on Buddhism’s Five Hindrances and their relationship to the concept in psychotherapy of Cognitive Distortions.)

I was first introduced to the psychological concept of cognitive distortions by Maria. Per the linked Wikipedia article, cognitive distortions are thought processes that “maintain negative thinking and … emotions”. Learning to refute them is termed “cognitive restructuring”.

On reading Maria’s old post, I was immediately reminded of Buddhism, and had in fact just listened to a series of podcasts on the “Five Hindrances”. However, there is no simple mapping between cognitive distortions and the hindrances. I’m tempted to try to make a Venn diagram on the relationship between the two sets.

My readings so far in Mind Over Mood have reaffirmed my suspicion that the two concepts are inter-related. Take, for example, this statement from the book’s introduction:

Mind over Mood teaches you to identify your thoughts, moods, behaviors, and physical reactions in small situations as well as during major events in your life … you learn how to make changes in your life when your thoughts are alerting you to problems that need to be solved.

The major function for me of meditation has been to find a baseline. Our thoughts build up during the day like layers as we work on something, are interrupted, hear interesting news, spontaneously think of things that were bugging us days ago. Sometimes we follow a train of thought for a while, based mostly on assumptions. Gil Fronsdal calls these bits of imagination ‘stories’. We create stories, often about the intentions of others, build up long conversations in our heads imagining confrontations and how other people in our life will react to what we say. Aside from getting lost daydreaming these stories, we tend to accumulate feelings about our imagined conversations and conversation partners, even though the conversations have never occurred. Gil usually calls the state we arrive in after this as being ‘caught’.

Here are some examples from intueri.org of people building stories in their minds:

She’s tired of working at the coffee shop. She expected so much more for herself.

Her mother is a real estate agent, hawking million-dollar homes to the contemporary gentry. Her father is a entertainment lawyer. Her older sister manages her own ballet studio. Her younger brother designs computer games for a major software corporation.

And then there’s her. A barista at a small coffee shop. Between a tall Americano and double-shot espresso, her mind wanders—

I’m the failure of the family. They were embarrassed to talk about me at the wedding; everyone else is so successful and then there’s me, the coffee girl. I hate spending time with my family. I’m trying the best that I can to get back on my feet, but they will never understand. They’re too busy with the expensive details of their lives—

She calls out, “Tall Americano!” and offers a warm smile to the elderly woman who approaches the counter to pick up the steaming beverage.

(From The Things That We Hide)

A man walked past her on the sidewalk. His dark eyes darted to her face before snapping back to an invisible point in the distant horizon.

She saw his surreptitious glance.

He thinks I’m hot, she thought to herself. He wants to ask me out, I know it.

Her stiletto heels clicked loudly against the concrete catwalk and, after he passed her, she swiveled her hips with more panache to offer him a teasing view of her backside.

They all look back—what’s there not to like?

With a gentle toss of her head, her long hair floated over her shoulder and landed softly on her back, revealing more of her voluptuous figure. Throwing her shoulders back and pushing her chest out, she continued to sashay along the sidewalk while the crowd parted around her.

I’m so gorgeous that everyone wants to look at my delicious body.

(From Perception Spectrum.)

The interesting thing about the Perception Spectrum post is that it contains several other examples of stories that could be built from the same situation. I suggest you read the whole post and think about how the stories are being built based on assumptions.

In order to extricate ourselves from the state of being ‘caught’ in our stories, it’s helpful to understand how we feel when we are not caught. Meditation gives us practice at identifying our ‘stories’ (complexes of distortions?) in a controlled environment. Having practiced in a controlled environment, we find it easier to identify story-building that occurs during our daily lives.

Can you think of an instance in which you built up a story and later found it to be far from reality?