[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: [tor-bugs] #29399 [Internal Services/Tor Sysadmin Team]: Retire host and services for tordnsel and check (chiwui)



#29399: Retire host and services for tordnsel and check (chiwui)
-------------------------------------------------+-------------------------
 Reporter:  ln5                                  |          Owner:  anarcat
     Type:  task                                 |         Status:  closed
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:  fixed
 Keywords:                                       |  Actual Points:
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------
Changes (by anarcat):

 * status:  accepted => closed
 * resolution:   => fixed


Comment:

 == step 4 done

 data removal scheduled everywhere:

 {{{
 anarcat@curie:tsa-misc(master)$ ./retire -v -H chiwui.torproject.org
 retire-all --parent-host=fsn-node-01.torproject.org
 starting tasks at 2020-04-09 16:39:51.630866
 checking for ganeti master on host fsn-node-01.torproject.org
 ganeti node detected with master fsn-node-01.torproject.org
 checking on fsn-node-01.torproject.org if instance chiwui.torproject.org
 is running
 instance chiwui.torproject.org not running, no stop required
 scheduling chiwui.torproject.org instance removal on host fsn-
 node-01.torproject.org
 scheduling gnt-instance remove chiwui.torproject.org to run on fsn-
 node-01.torproject.org in 7 days
 warning: commands will be executed using /bin/sh
 job 10 at Thu Apr 16 20:39:00 2020
 scheduling chiwui.torproject.org backup disks removal on host
 bungei.torproject.org
 checking for path "/srv/backups/bacula/chiwui.torproject.org/" on
 bungei.torproject.org
 scheduling rm -rf "/srv/backups/bacula/chiwui.torproject.org/" to run on
 bungei.torproject.org in 30 days
 warning: commands will be executed using /bin/sh
 job 24 at Sat May  9 20:40:00 2020
 Error: The certificate retrieved from the master does not match the
 agent's private key. Did you forget to run as root?
 Certificate fingerprint:
 59:C4:A7:B7:3C:DD:A2:04:61:92:5B:35:97:03:66:64:1D:3C:55:85:DF:2E:40:BA:2B:3D:E2:A1:D2:11:2F:F5
 To fix this, remove the certificate from both the master and the agent and
 then start a puppet run, which will automatically regenerate a
 certificate.
 On the master:
   puppet cert clean pauli.torproject.org
 On the agent:
   1a. On most platforms: find /home/anarcat/.puppet/etc/ssl -name
 pauli.torproject.org.pem -delete
   1b. On Windows: del
 "\home\anarcat\.puppet\etc\ssl\certs\pauli.torproject.org.pem" /f
   2. puppet agent -t

 Error: Try 'puppet help node clean' for usage
 failed to revoke instance pauli.torproject.org on host
 chiwui.torproject.org: Encountered a bad command exit code!

 Command: 'puppet node clean chiwui.torproject.org'

 Exit code: 1

 Stdout: already printed

 Stderr: already printed


 completed tasks, elasped: 0:00:12.384885 (user 2.74 system 0.05 chlduser
 0.0 chldsystem 0.0 RSS 34.9 MB)
 anarcat@curie:tsa-misc(master)$ ./retire -v -H chiwui.torproject.org
 retire-all --backup_host=''
 starting tasks at 2020-04-09 16:41:08.346772
 No idea what '--backup_host' is!
 completed tasks, elasped: 0:00:00.002826 (user 0.21 system 0.03 chlduser
 0.0 chldsystem 0.0 RSS 30.8 MB)
 [1]anarcat@curie:tsa-misc(master)$ ./retire -v -H chiwui.torproject.org
 retire-all --backup-host=''
 starting tasks at 2020-04-09 16:41:13.611470
 not wiping instance chiwui.torproject.org data: no parent host
 Notice: Revoked certificate with serial 23
 Notice: Removing file Puppet::SSL::Certificate chiwui.torproject.org at
 '/var/lib/puppet/ssl/ca/signed/chiwui.torproject.org.pem'
 chiwui.torproject.org
 Submitted 'deactivate node' for chiwui.torproject.org with UUID
 84ccf106-f275-4f7e-8571-d414a47a4a3d
 completed tasks, elasped: 0:00:08.504086 (user 3.09 system 0.05 chlduser
 0.0 chldsystem 0.0 RSS 34.2 MB)
 }}}

 note that in the above the puppet run failed because it tried to connect
 using a normal user. this was worked around in 4d025f3 and reran
 correctly.

 == step 5

 removed this block from LDAP:

 {{{
 269 host=chiwui,ou=hosts,dc=torproject,dc=org
 host: chiwui
 hostname: chiwui.torproject.org
 objectClass: top
 objectClass: debianServer
 architecture: amd64
 access: restricted
 admin: torproject-admin@xxxxxxxxxxxxxx
 sshRSAHostKey: ssh-rsa
 AAAAB3NzaC1yc2EAAAADAQABAAABAQDUKfP+b2Isj3UlWmVRAeXpOcyZslJypugDdunLUWXsx2IjzKzhExqkgiDigsv0Fr7SbFKuJSBmZM/q0X6iLXUAuTPDREhubMcQ9iGONvh26H/ocniXpgtbBzzZ8d6sDK/NLupOXHjfBXN/IWhCdwN/JC6lm1qjLAf5BQ7ukVeVKt7gBXXW4rGUkCw+eWLFS1IjKWASm9ubE9t+uVaoYeUP0PSwSrgIrb9hjCsMHBFTOXvSgrX2Nr85ZUetUPvHyo/GPUIdteK8ouMrRe4yJi6rIyMeze2a7ohtEJ2q1IDaE3Jr5BlzIyXeEK+LN1VykiiChde0pGbInzHWzgk8wi3R
 root@chiwui
 sshRSAHostKey: ssh-ed25519
 AAAAC3NzaC1lZDI1NTE5AAAAILDW4yvM1jKFwZpSMHl/+HqPsLA2H58w028TmHQ5Zmqu
 root@chiwui
 distribution: Debian
 allowedGroups: check
 allowedGroups: tordnsel
 purpose: [[check.torproject.org]]
 purpose: tordnsel
 l: Falkenstein, Saxony, Germany
 dnsTTL: 300
 ipHostNumber: 116.202.120.176
 ipHostNumber: 2a01:4f8:fff0:4f:266:37ff:fe69:3bda
 physicalHost: gnt-fsn
 }}}

 == step 6

 removed the following DNS records:

 {{{
 exitlist                IN      NS      chiwui4
 chiwui2                 IN      A       116.202.120.177
 chiwui4                 IN      A       116.202.120.176
 }}}

 or, in other words, this commit in dns/domains.git:

 {{{
 commit f61867cdd2832444c1b3abe0e74a21f6e5e74f05 (HEAD -> master)
 Author: Antoine Beaupré <anarcat@xxxxxxxxxx>
 Date:   Thu Apr 9 16:49:40 2020 -0400

     retire chiwui (#29399)

 diff --git a/torproject.org b/torproject.org
 index 8ab0832..241a9ca 100644
 --- a/torproject.org
 +++ b/torproject.org
 @@ -83,7 +83,6 @@ dip                   IN      CNAME   gitlab-02
  donate                 IN      CNAME   crm-ext-01
  staging.donate         IN      CNAME   crm-ext-01
  test.donate            IN      CNAME   crm-ext-01
 -exitlist               IN      NS      chiwui4
  exonerator             IN      CNAME   materculae
  gitlab    IN CNAME gitlab-02
  gettor                 IN      CNAME   static
 @@ -202,8 +201,6 @@ $INCLUDE
 "/srv/letsencrypt.torproject.org/var/hook/snippet"
  macppc                 IN      A       50.195.45.81 ;old ip 74.95.122.145
  macx86                 IN      A       50.195.45.82 ;old ip 74.95.122.149
  watsoni                        IN      A       50.195.45.86
 -chiwui2                        IN      A       116.202.120.177
 -chiwui4                        IN      A       116.202.120.176

  ; internal networks
  macrum-priv            IN      A       172.30.133.1
 }}}

 remove the following sudo entries:

 {{{
 %check                  chiwui=(check)                          ALL
 %tordnsel               chiwui=(tordnsel)                       ALL
 %check          chiwui=(root)   /usr/local/sbin/apache2-vhost-update
 }}}

 or, in other words, this commit in puppet:

 {{{
 commit 66a02f3b4361167bfe45bd85361826a0b5076efd (HEAD -> master)
 Author: Antoine Beaupré <anarcat@xxxxxxxxxx>
 Date:   Thu Apr 9 16:48:41 2020 -0400

     retire chiwui (#29399)

 diff --git a/modules/nagios/templates/obsolete-packages-
 ignore.d-hostspecific.erb b/modules/nagios/templates/obsolete-packages-
 ignore.d-hostspecific.erb
 index 3b727533..60801d2a 100644
 --- a/modules/nagios/templates/obsolete-packages-ignore.d-hostspecific.erb
 +++ b/modules/nagios/templates/obsolete-packages-ignore.d-hostspecific.erb
 @@ -7,7 +7,6 @@ ignore = []
  case @fqdn
  when "alberti.torproject.org" then           ignore << %w{userdir-ldap
 userdir-ldap-cgi}
  when "moly.torproject.org" then              ignore << %w{megacli}
 -when "chiwui.torproject.org" then            ignore << %w{tor prometheus-
 node-exporter}
  end

  ignore.flatten.join("\n")
 diff --git a/modules/roles/manifests/check.pp
 b/modules/roles/manifests/check.pp
 deleted file mode 100644
 index 51e0fc4c..00000000
 --- a/modules/roles/manifests/check.pp
 +++ /dev/null
 @@ -1,35 +0,0 @@
 -# deprecated, to be replaced by roles::check_rewrite
 -class roles::check {
 -       include apache2
 -       include apache2::ssl
 -       ssl::service { 'check.torproject.org': notify  => Exec['service
 apache2 reload'], key => true, }
 -
 -       ferm::rule{
 -               "tordnsel-exit":
 -                       description     => "Allow tordnsel exit queries",
 -                       rule            => "&SERVICE(tcp, (8000 10080
 10443 10110 5190 6667 6697 9030))",
 -                       ;
 -               "tordnsel-dns":
 -                       description     => "Allow tordnsel dns queries",
 -                       rule            => "&TCP_UDP_SERVICE(10053)",
 -                       ;
 -               # XXX MAGIC-IP-ADDRESS
 -               "do-track":
 -                       domain      => '(ip)',
 -                       description => 'do TRACK for tordnsel traffic',
 -                       table       => 'raw',
 -                       chain       => 'PREROUTING',
 -                       rule        => 'daddr 116.202.120.177 proto tcp
 dport (http https) jump RETURN',
 -                       ;
 -               "tor-nat":
 -                       description     => "redirect some incoming to high
 ports",
 -                       table           => 'nat',
 -                       chain           => 'PREROUTING',
 -                       rule            => 'daddr 116.202.120.177 proto
 tcp dport  80 DNAT to :10080;
 -                                           daddr 116.202.120.177 proto
 tcp dport 443 DNAT to :10443;
 -                                           daddr 116.202.120.177 proto
 tcp dport 110 DNAT to :10110;
 -                                           daddr 116.202.120.176 proto
 udp dport  53 DNAT to :10053;
 -                                           daddr 116.202.120.176 proto
 tcp dport  53 DNAT to :10053 ',
 -                       ;
 -       }
 -}
 diff --git a/modules/sudo/files/sudoers b/modules/sudo/files/sudoers
 index 7052c067..a1b7c52f 100644
 --- a/modules/sudo/files/sudoers
 +++ b/modules/sudo/files/sudoers
 @@ -44,7 +44,6 @@ letsencrypt           nevii=(dnsadm)
 NOPASSWD: /srv/dns.torproject.org/bin/update
  %atlas                 STATICMASTER=(atlas)                    ALL
  %bridgedb              polyanthum=(bridgedb,bridgescan)
 ALL
  %buildmasters          rouyi=(jenkins)                         ALL
 -%check                 chiwui=(check)                          ALL
  %collector             COLLECTORHOSTS=(collector)              ALL
  %consensus-health      henryi=(consensus-health)               ALL
  %dip                   gitlab-01=(git)                         ALL
 @@ -63,7 +62,6 @@ letsencrypt           nevii=(dnsadm)
 NOPASSWD: /srv/dns.torproject.org/bin/update
  %rtfolks               rude=(rtmailarchive)                    ALL
  %torarchive            archive-01=(torarchive)                 ALL
  %tordebadm             palmeri=(tordeb)                        ALL
 -%tordnsel              chiwui=(tordnsel)                       ALL
  %torhelp               STATICMASTER=(torhelp)                  ALL
  %tormedia              listera=(tormedia)                      ALL
  %torperf               ferrinii=(torperf)                      ALL
 @@ -122,7 +120,6 @@ noc         peninsulare=(root)      ALL

  # various roles can do other interesting things
  %bridgedb      polyanthum=(root)               /usr/local/sbin/apache2
 -vhost-update
 -%check         chiwui=(root)   /usr/local/sbin/apache2-vhost-update
  %rtfolks       rude=(root)             /usr/local/sbin/apache2-vhost-
 update

  %buildmasters          rouyi=(root)
 /usr/sbin/service jenkins *
 }}}

 == step 7

 removed from tor-passwords

 == step 8

 DNSWL N/A

 == step 9

 removed from spreadsheet

 == step 10

 N/A

 == step 11

 remove from reverse DNS in hetzner.

 we're all done here, good bye chiwui, you served us well!

 thanks to the metrics team and special thanks for irl for finally bringing
 us to this point, you rock! :)

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29399#comment:25>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs