Tech:Puppet

From Meta
Jump to navigation Jump to search

Miraheze uses a puppet master-slave configuration for deploying and managing configuration across all of the servers.

Puppet Agents[edit | edit source]

Puppet agents are all servers in the cluster and are able to access the puppet master in order to collect resources and manifests that have been pre-compiled on the master. Puppet agents aren't really overly special except that they have puppet installed on them and have a signed upstream cert on the master.

The manifest is ran every 30 minutes on all agents (which differs from the previous masterless set up where it was every hour unless a change was made). To manually run puppet on an agent you need to run the following as root:

puppet agent -tv

In the past puppet runs could be disabled by disabling the cron tab and was limited only to an SRE member - now this process is strongly discouraged. To disable puppet runs then you need to run the following as root:

puppet agent --disable="<reason>"

To then re-enable puppet runs:

puppet agent --enable

Puppet Master[edit | edit source]

The puppetmaster is the central server that hosts the private git repo and the public git repo (from GitHub) and compiles the manifests for agents to run.

Certificates

When reinstalling a server you need to clean all certificate information about the particular server. This can be done by running:

puppetserver ca clean --certname <node>

When installing a new server, it may be necessary to check what hostname it is using to make a new request for a certificate or maybe just to generally check if any new certificate requests exists. This can be done by running:

puppetserver ca list

When decommissioning a server, it is necessary to revoke the certificate of the server in order to prevent it being used to access the contents of the puppetmaster. This should also be done in the event any server is compromised.

puppetserver ca revoke --certname <node>

When adding a server to the puppetmaster, it is necessary to sign the certificate request. The following command will verify the certificate is legitimate and then authorise it to use the puppetmaster's contents.

puppetserver ca sign --certname <node>
master

If in the process of debugging you are unsure what the puppermaster is telling an agent to run or is passing on to an agent, it is possible to get a full JSON output of what is being to the server by running:

puppet master --compile <node>
node

When reinstalling or decommissioning a host, it is necessary to tell the puppetmaster to forget everything it currently knows about the host. This can be done by running:

puppet node clean <node>

When working with facts, you can get a JSON output of all facts the puppetmaster is aware of that each node knows. This can be done by running:

puppet node find <node>

Adding a new puppet agent (server) to the Puppetserver[edit | edit source]

This section is only a part of the installation process, see Tech:Server_lifecycle for all steps.

Here are the steps you should follow when adding a new puppet agent (server) to the Puppetserver:

  • Step 1: Run https://phabricator.miraheze.org/P220 (you will have to do it a few times as at the apt-install step, it forgets the commands to run after). If you cannot just copy-paste, use a shorter URL to download the script: wget -O puppet.sh https://meta.miraheze.org/m/5KV
  • Step 2: (On the puppetserver) cd /etc/puppetlabs/puppet/git && git pull
  • Step 3: (On the agent) execute puppet agent -tv --server puppet3.miraheze.org --waitforcert 60
  • Step 4: (On the puppetserver) Check puppetserver ca list, and make sure that the fingerprints match
  • Step 5: (On the puppetserver) After you have made sure that the fingerprints match, execute puppetserver ca sign --certname [servername].miraheze.org
  • Step 6: (On the agent) execute puppet agent -tv --server puppet3.miraheze.org
  • Note: The agent will automatically detect the signed certificate and proceed from there.
  • Step 7: (On the agent) verify that puppet agent -tv works with out --server puppet3.miraheze.org.

Removing puppet agent (server) on the Puppetserver[edit | edit source]

This section is only a part of the decommission and reimage processes, see Tech:Server_lifecycle for all steps.

Here are the steps you should follow when removing a puppet agent (server) from the Puppetserver:

  • Step 1: (On the puppetserver) execute sudo puppet node deactivate <host>
  • Step 2: (On the puppetserver) execute puppetserver ca clean --certname <host>
  • Step 3: (On mon2) execute puppet agent -tv