And yet the NSA is moving to prime numbers.
A large public key isn't a very good reason to not adopt quantum-safe crypto, it just means that it requires having the Tor project to be able to scale to a larger degree. I suggest hash tables, a percentage of which are pseudorandomly downloaded. Otherwise the Tor project won't scale to 10x the relays ... even ignoring quantum cryptography.