Tech:Puppet

From Meta
Jump to navigation Jump to search

Miraheze uses a puppet master-slave configuration for deploying and managing configuration across all of the servers.

Puppet Agents[edit source]

Puppet agents are all servers in the cluster and are able to access the puppet master in order to collect resources and manifests that have been pre-compiled on the master. Puppet agents aren't really overly special except that they have puppet installed on them and have a signed upstream cert on the master.

The manifest is ran every 10 minutes on all agents (which differs from the previous masterless set up where it was every hour unless a change was made). To manually run puppet on an agent you need to run the following as root:

puppet agent -t

In the past puppet runs could be disabled by disabling the cron tab and was limited only to operations member - now this process is strongly discouraged. To disable puppet runs then you need to run the following as root:

puppet agent --disable="<reason>"

To then re-enable puppet runs:

puppet agent --enable

Puppet Master[edit source]

The puppetmaster is the central server that hosts the private git repo and the public git repo (from GitHub) and compiles the manifests for agents to run.

Certificates

When reinstalling a server you need to clean all certificate information about the particular server. This can be done by running:

puppetserver ca clean --certname <node>

When installing a new server, it may be necessary to check what hostname it is using to make a new request for a certificate or maybe just to generally check if any new certificate requests exists. This can be done by running:

puppetserver ca list

When decommissioning a server, it is necessary to revoke the certificate of the server in order to prevent it being used to access the contents of the puppetmaster. This should also be done in the event any server is compromised.

puppetserver ca revoke --certname <node>

When adding a server to the puppetmaster, it is necessary to sign the certificate request. The following command will verify the certificate is legitimate and then authorise it to use the puppetmaster's contents.

puppetserver ca sign --certname <node>
master

If in the process of debugging you are unsure what the puppermaster is telling an agent to run or is passing on to an agent, it is possible to get a full JSON output of what is being to the server by running:

puppet master --compile <node>
node

When reinstalling or decommissioning a host, it is necessary to tell the puppetmaster to forget everything it currently knows about the host. This can be done by running:

puppet node clean <node>

When working with facts, you can get a JSON output of all facts the puppetmaster is aware of that each node knows. This can be done by running:

puppet node find <node>

Adding a new puppet agent (server) to the Puppetmaster[edit source]

Here are the steps you should follow when adding a new puppet agent (server) to the Puppetmaster (puppet1):

  • Step 1: Set hiera value puppet_major_version to 4 for the host in the puppet repo.
  • Step 2: (On the puppetmaster) cd /etc/puppet/puppetserver/git && git pull
  • Step 3: (On the agent) execute puppet agent -tv --server puppet1.miraheze.org --waitforcert 60
  • Step 4: (On the puppetmaster) Check puppetserver ca list, and make sure that the fingerprints match
  • Step 5: (On the puppetmaster) After you have made sure that the fingerprints match, execute puppetserver ca sign --certname [servername].miraheze.org
  • Step 6: (On the agent) execute puppet agent -tv --server puppet1.miraheze.org
  • Step 7: Set hiera value puppet_major_version to 6 for the host in the puppet repo.
  • Step 8: (On the puppetmaster) cd /etc/puppet/puppetserver/git && git pull
  • Step 9: (On the agent) execute puppet agent -tv --server puppet1.miraheze.org
  • Step 10: (On the agent) execute apt-get --purge remove *puppet* -y
  • Step 11: (On the agent) execute apt-get update
  • Step 12: (On the agent) execute apt-get install puppet-agent -y
  • Step 13: (On the agent) execute /opt/puppetlabs/bin/puppet agent -tv --server=puppet1.miraheze.org
  • Step 14: (On the puppetmaster) execute puppetserver ca clean --certname [servername]
  • Step 15: (On the agent) execute /opt/puppetlabs/bin/puppet agent -tv --server puppet1.miraheze.org
  • Step 16: Follow step 4-5 again.

The agent will automatically detect the signed certificate and proceed from there.

Removing puppet agent (server) on the Puppetmaster[edit source]

Here are the steps you should follow when removing a puppet agent (server) from the Puppetmaster (puppet1):

  • Step 1: (On the puppetmaster) execute puppet cert clean <host>
  • Step 2: (On the puppetmaster) execute puppet node clean <host>
  • Step 3: (On the puppetmaster) execute puppet node deactivate <host>
  • Step 4: (On db4) execute su - postgres -s /bin/bash -c "psql -d puppetdb -f /srv/puppetdbRemoveHost.sql"
  • Step 5: (On the misc1) execute puppet agent -tv