Gödel, Escher, Bach: An Eternal Golden Braid. It has been described as, “The book for which there is no appropriate description.” It incorporates symbolic logic, cognitive science, Zen Buddhism, mathematics, consciousness, art, music, and much more. I first heard of it my freshman year of college while browsing E2, where it was (mostly) discussed with reverence and awe. Intrigued, I ventured over to the Tulane book store, and was happy to find that they carried it.
I bought it. The first time.
About this time five years ago, I was a nervous junior undergraduate studying Biomedical Engineering at Tulane University. I had just been accepted as an undergraduate member of Dr. Natalia Trayanova’s computational cardiac electrophysiology lab. The goal at that time was to complete a research project for my undergraduate thesis.
So very many things have happened since then. Here are the highlights:
That brings us to the present day. Now, looking forward a little:
In the next two weeks, the cluster set-up will be completed. We will have free rein on 140 compute nodes (20 old, 120 new), all managed from one head node. The new nodes will be connected by the fastest Infiniband interconnects available on the market, and each node will have 8 GB of RAM available, with the potential to hold 64 GB each. There are four 3.0 GHz Opteron cores per node, yielding a total of 480 processors and 960 GB of RAM on the new nodes alone.
To give you some perspective on what that means, let me give you some details about the kinds of models we run. When I joined the lab, our two largest models consisted of a 4mm thick slice of the canine heart, and a very smooth, idealized model of the rabbit heart. These models are composed of 1.6 million and 0.82 million tetrahedral elements, respectively. It took something like an hour of wall clock time per millisecond of simulation time to run these models. (In other words, to get one millisecond worth of simulation data it was necessary to wait about an hour.) We could run one or two simulations at a time, at that speed.
My newest model, and currently the largest model in use in the lab, is composed of 28 million tetrahedral elements. On a cluster similar to our new one (Lonestar on TeraGrid), using 32 processors, it takes about 22 minutes of wall-clock time to simulate one millisecond in the model. Using a crude estimated unit of speed of (minutes real time / millisecond simulation time / tetrahedral element), and focusing only on the number of simulations we can run at once, not the number of CPUs required:
We have increased our simulation speed by almost 100 fold. We can run two to four simulations of that size at a time, vs one or two the old way. But that’s not all. We can now run bigger models. Much bigger models. We are now capable of running something the size of a dog heart (we have verified this). More importantly, we now have the technical capacity to run a model the size of the human heart, with a resolution near that of the size of a cardiac cell, and to model contraction in addition to electrical activity. It remains only to develop such models. We are prepared to store the results: the new cluster has a storage capacity of 28 TB online, with the ability to add something like 40 or 50 TB more simply by expanding the existing storage device.
In my time in the lab, I have watched our abilities expand from serial jobs with relatively small models to massively parallel jobs with the capacity to model electrical and mechanical activity in the human heart. We are just beginning a very exciting time in the lab and in the field, and what’s really killing me is that fact that there’s so much more to tell you.
But I can’t just yet.
(This post was partly inspired by a conversation with Maria and Amanda)
So, the cluster is here.
The good news: I think we’ll be able to use it (albeit without Infiniband for now) by the end of the day.
The bad news: FedEx clearly dropped one entire rack. Yes. Unbelievable. About $110,000 worth of equipment. When it got here, there were plastic parts coming out from under the door of the crate. From the looks of it, it faceplanted off of a loading dock or a forklift. The tilt sensor was definitely tripped. Some of it may still be good, but it all has to be thoroughly inspected. Penguin is just building us a whole new rack. We lucked out in a way — it was the only rack with no ‘special’ components. All of the other racks have something special. One has the head node (not a big deal). One has the Infiniband switch (very expensive with long lead time to replace), and one has a fileserver with a Xyratex storage unit (also very expensive with long lead time). This rack just had compute nodes and a regular gigabit switch.
I’ll have more news and pictures once we get further along.
Amazon.com is really the leader right now in so-called “cloud computing”, where there’s some anonymous cluster of servers that you somehow use for various reliable services.
They started with “S3“, meaning “Simple Storage Service”. S3 is a back-end web service that allows you to place, retrieve, and share files via some sort of web back-end interface. As I’m not really a web developer, that interface is beyond me for the moment. However, various programs have cropped up that act as a front-end to S3, my favorite being JungleDisk. Once JungleDisk is fired up, you have what appears to be a network disk with unlimited storage. It is, in fact, effectively unlimited, though it of course comes at a cost. There’s a bandwidth cost whenever you move things to and from, and an ongoing storage cost. The ongoing storage cost is something like a charge based on your “average daily balance” of storage used, and is currently charged at a mere $0.15/GB/month. So, if you want to store 100 GB, that’s $15/mo. Not bad. Remember, too, that you’re never paying for storage that you don’t need, unlike some other services.
After they got S3 stabilized, Amazon.com came out with another beta “cloud” service, the “EC2“, for “Elastic Compute Cloud”. They let you run virtual machine images, basically, charged at an hourly rate. This was kind of interesting, and had some usefulness for a lot of people, but the machines were somewhat small and weak.
No more.
Within the last few days, they launched larger machine images with an x86_64 architecture, up to 4 virtual cores, and nearly 16GB of available RAM. Now we’re talking something interesting for people like me, and our lab. As a test, I created my own custom image, containing our simulation software, and fired it up. For $0.80/hour, I can get the equivalent of 2 of our 2-core Opteron cluster nodes. It’s a little slower than that, but only just. It also probably doesn’t scale as well as our current system, and I know it won’t scale as well as the system that will be arriving on Monday, but something tells me that this will change. I’m not the only one out there that wants to run cluster applications on EC2, as a quick google search will reveal.
Furthermore, this is a relatively cheap and ubiquitous platform that would allow someone to run high-performance applications without the overhead of purchasing a complete cluster. It would be good for, say, starting a business that required a large amount of computing power without having to purchase, store, feed, cool, and house all of that hardware up front. Once things got rolling it would be possible to use revenue to purchase and maintain such dedicated hardware.
The one major down-side of EC2, as I see it, is that it doesn’t save any of the data on the machine once the machine is shut down. One has to ship the data off to S3 (no charge) or another machine (bandwidth charges apply) before shutting it down. Nonetheless, as the tools for interacting with S3 improve, I expect that this limitation will disappear as well.
I should note that I’m not affiliated with either Amazon.com or JungleDisk in any way, except as a happy user.
When doing a literature search, it’s a good idea to start from a few articles and then (if they are along the lines of what you are looking for) use their references and articles that reference them to expand the search.
One handy way of doing that is with the HubMed Graph Browser. You get to it by finding an article (like mine here) and then selecting the “Graph” link next to “Related” in the line of options at the bottom.
Once you load the TouchGraph, you can see the related articles, change the depth of relationships graphed, zoom in and out, and so on. It can be a nice alternative to the normal related articles list, graphically showing distance and relation.