Re: [tor-bugs] #30020 [Internal Services/Tor Sysadmin Team]: switch from our custom YAML implementation to Hiera

["Tor Bug Tracker & Wiki" <blackhole@xxxxxxxxxxxxxx>]

#30020: switch from our custom YAML implementation to Hiera
-------------------------------------------------+-------------------------
 Reporter:  anarcat                              |          Owner:  anarcat
     Type:  project                              |         Status:
                                                 |  assigned
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:  #29387                               |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Old description:

> We currently use a custom-made YAML database for assigning roles to
> servers and other metadata. I started using Hiera for some hosts and it
> seems to be working well.
>
> Hiera is officially supported in Puppet and shipped by default in Puppet
> 5 and later. It's the standard way of specifying metadata and class
> parameters for hosts. I suspect it covers most of our needs in terms of
> metadata and should cover most if not all of what we're currently doing
> with the YAML stuff in Puppet.
>
> We should therefore switch to using Hiera instead of our homegrown
> solution.

New description:

 We currently use a custom-made YAML database for assigning roles to
 servers and other metadata. I started using Hiera for some hosts and it
 seems to be working well.

 Hiera is officially supported in Puppet and shipped by default in Puppet 5
 and later. It's the standard way of specifying metadata and class
 parameters for hosts. I suspect it covers most of our needs in terms of
 metadata and should cover most if not all of what we're currently doing
 with the YAML stuff in Puppet.

 We should therefore switch to using Hiera instead of our homegrown
 solution.

 This involves converting:

  * `if has_role('foo') { include foo }` into `classes: [ 'foo' ]` in hiera
  * hardcoded macros in the ferm module's `me.conf.erb` into exported
 resources
  * templates looping over allnodeinfo into exported resources
  * the `$roles` array into Hiera
  * the `$localinfo` into Hiera (assuming all the data is there)
  * the `$nodeinfo` and `$allnodeinfo` arrays into Hiera (assuming we can
 switch from LDAP for host inventory)
  * basically any other stuff of the kind

 Ideally, all YAML data should end up in the hiera/ directory somehow. This
 is the first step in making our repository public (#29387) but also using
 Hiera as a more elaborate inventory system (#30273).

 The idea of switching from LDAP to Hiera for host inventory will
 definitely need to be evaluated more thoroughly before going ahead with
 that part of the conversion, but YAML stuff in Puppet should definitely be
 converted.

 The general goal of this is both to allow for a better inventory system
 but also make it easier for people to get onboarded with Puppet. By using
 community standards like Hiera, we make it easier for new people to get
 familiar with the puppet infrastructures and do things meaningfully.

--

Comment (by anarcat):

 i did more work here. the following macros have now been safely removed:

 {{{
 HOST_STATIC
 HOST_ROLE_PEOPLE
 HOST_ROLE_METRICSBOT
 HOST_ROLE_JABBER_SERVER
 HOST_ROLE_WEBLOG_SOURCE
 HOST_ROLE_WEBLOG_SINK
 }}}

 This also led to the removal of a custom SSH keys generation template
 (`modules/roles/templates/weblog_sink/webstats-authorized_keys.erb`),
 although it hasn't been converted to the native `ssh_authorized_keys`
 because of the format difference between the custom fact we use to export
 the ssh keys and the one expected by the type. This could be fixed in
 another refactoring at some other time.

 Now, I'm working on the `static_*` stuff, which is like `weblog_*` but a
 little more complicated because the config files are not (yet) built with
 config::fragment. The SSH firewall configuration was a little more
 complicated but it's been migrated already. Next up is the authorized_keys
 which should follow the same pattern as the weblog stuff and then the
 config::fragment conversion. There are also corner cases with more sub-
 roles for that one that will need to be taken into account, but those can
 hopefully be converted into class parameters.

 There are now 36 roles left in the `roles` class. There were about 50
 roles, split between `site.pp` and the `roles` class, when I started this,
 about a week ago, so i think it would be fair to assume this first part of
 the conversion will be done in a week or two.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/30020#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online

_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs