1. Google's PAM module does not scale securely
Google Authenticator is just an implementation of OATH's TOTP, which stands for "Time-based One-Time Password." It really just boils down to a way to convert a timestamp into a 6-digit number using a preshared secret known only to your phone and to the authentication server. The important bit here are the words "One-Time" -- an attacker should not be able to reuse the password after it's been used once. But this is where the PAM module falls short.
Google's PAM module works by installing a file containing the secret, scratch tokens, plus a few other parameters (more on that later) into either the user's home directory or some other location on the system you are trying to secure via 2-factor authentication. Default configuration expects to find that file in $HOME/.google_authenticator, but it can also be installed centrally in /etc or /var and made work via a few PAM tricks.
However, if you have more than one server you are trying to secure (and most admins do), you cannot install the same secret file on multiple servers, as this would break the "One-Time" part of "TOTP." Since the servers don't communicate PAM state data with each-other, an attacker can take the token you used to authenticate on serverA, and reuse that token to authenticate on serverB. Since serverB has no way of knowing that the token has already been used on serverA, the authentication succeeds, thus largely defeating the purpose of 2-factor authentication.
The only way to remain secure is to use different secrets on serverA and serverB, but clearly this does not scale past 2-3 nodes and 2-3 users. If you have 10 users and 10 servers, you have to provision and keep track of a hundred secrets.
The only other way to remain secure when using the Google Authenticator PAM module is to store the secret and state data in some central location, such as on a shared filesystem. But here we run into another problem.
2. Google Authenticator stores the secret and state data in the same file
I'm rather surprised at this architectural decision -- usually you want to separate your configuration data from your state data and keep them in separate locations -- if only because this will minimize the risk that a failed attempt at saving your state will overwrite your Google Authenticator secret and thus lock you out.
This is an architecture choice that comes from the era before SELinux or other mandatory access control systems. On a system with "classic" unix permissions based on users and groups, a root process reading /etc/passwd would be able to also write to /etc/passwd, but this is no longer true on a SELinux-enabled system. These days we can tightly limit which files the processes are able to write to, and which only read -- even if the process is running as root.
So, if we were trying to distribute token files to multiple systems in order to be able to (securely) use the same preshared secrets on multiple servers, the fact that both the secret and state data are in the same file would rule out NFS and many similar networked filesystems -- we don't want the secrets travelling in cleartext over the wires. For the central storage to be secure, you'd have to find a stable and reliable networked filesystem that supports encryption in transit, and that suddenly makes the problem really complicated as such filesystems are few in number. It also doesn't solve the last problem.
3. Your preshared secrets are only as good as your weakest system
Even if you found a way to securely distribute Google Authenticator state data across multiple servers, you'd still have the problem that a root-level security incident on any one of your servers would require that all Google Authenticator tokens are immediately reissued. Not much of a concern if you only have a few users, but if you have dozens or hundreds of them, this becomes a nightmare scenario. You can't have your entire infrastructure grind to a halt while you feverishly reissue tokens to users only because someone did something dumb on "wwwtest02" and an attacker managed to gain root-level access and could read all preshared secrets.
So, all of the above was what prompted us to come up with totpcgi, which securely centralizes Google Authenticator-based 2-factor authentication infrastructure, plus adds a few features we liked, such as a way to accept "password+token" in one field, plus optionally encrypt the preshared secret with the user's password -- so that even if something goes wrong and the attacker is able to read all preshared secrets, the information will not be of any use without also knowing the users' passwords.