I recently did a linux reinstallation on my workstation after more than 4 years of cruft accumulation, but couldn’t get ssh to work with my cryptostick. I did all of the normal stuff required to disable ssh-agent and enable ssh-agent support in gpg-agent.
Turns out I needed to install the ‘gpgsm’ package (on Debian/Ubuntu), which replaces the normal gpg-agent with one that supports a smart card.
I’m posting this mostly for my own reference, but maybe it will help you too.