To start off with, both the new and the old syntax work, although the errors they throw will be different. Correct syntax below, respectively:
tor --service install -option -f "C:\blah\torrc" tor -install -option -f "C:\blah\torrc"
The new/correct syntax installs the service correctly. However, Tor does not quit after invoking the installation and continues to run as if you executed tor.exe with a bunch of garbage options. You will most likely get errors related to a nonexistent torrc, or bad syntax. Ignore them. Fire up services.msc and look at the properties for "Tor Win32 Service". Under "Path to executable:" You should see something like the below. It's not strictly syntactically correct (there should be no quotes around the -f), but it works without any issues.
"C:\Program Files\Vidalia Bundle\Tor\tor.exe" --nt-service "-f" "C:\blah\torrc"
The important thing to remember here is to *declare your DataDirectory in your torrc!* Without this, Tor is going to create duplicates. I would personally recommend creating a Tor DataDirectory in the root of a drive, because it makes handling permissions easy, which is the next issue we have to tackle. Odds are at some point you'll run into an error like this when trying to start your newly-installed service:
Service failed to start : An exception occurred in the service when handling the control request.
Which is Windows' way of saying it's crapping out because of a permissions issue. There is a special user named SERVICE under whom all of the Windows system services are run. This user needs "Full Control" access to the Tor DataDirectory and the torrc file. Be careful about this. Accidentally giving control of %appdata% or %homepath% to every single system service is more than a small security risk, which is why I don't recommend using the default Tor and Vidalia folders in the %appdata% directory. Doing so intelligently involves using advanced NTFS permissions to give SERVICE "Read & Execute" permissions that apply to "This folder only" from %appdata% all the way back to the root of the drive, and "Full Control" permissions to Tor and Vidalia that apply to "This folder, subfolders, and files". This is all assuming that your torrc is in Vidalia and Tor is your DataDirectory.
Which is why it's infinitely easier to simply make a C:\Torservice directory and grant SERVICE full control over it, and use it to store torrc and as your DataDirectory. No need to stay up all night and bug people on IRC about it...
Long story short? Plenty of bug reports to file and wikis to improve. I know almost zero Windows users use Tor as an NT-Service, but I hope this saves some adventurous soul a few hours of frustration. Thanks to Sebastian and Goldstein for pitching in.
~Justin Aplin