Tech:Graylog

From Meta

Graylog is a log management solution for logs stored on the servers. As part of of a central logging project, Miraheze is exploring Graylog for the central log storage. The web interface is available at https://graylog.miraheze.org/. Access is restricted to Site Reliability Engineering department personnel. Said people can use their LDAP credentials for authentication.

Architecture[edit | edit source]

Graylog runs on graylog121.miraheze.org as of now. There are three daemons running there: graylog-server for the actual log management, elasticsearch for storing the logs and mongod for storing Graylog's configuration.

                                                                                                                  
                     +----------------------------------+                                        +------------------------------------------+
                     | test131.miraheze.org               |                                        | graylog121.miraheze.org                    |
                     | +------------+                   |                                        |                                          |
                     | |            |                   |                                        | +---------------+      +---------------+ |
+----------------+   | | MediaWiki  |-\                 |                                        | |               |      |               | |
|                |   | |            |  ---\             |                                    ------|graylog-server -------- elasticsearch | |
| Miraheze User  |   | +------------+      --\          |             12210/tcp   ----------/    | |               |\     |               | |
|                |   |                   +------------+ |              ----------/               | +-------|-------+ \    +---------------+ |
+----------\-----+   | +-------------+   |            | |   ----------/                          |         |          |                     |
            ------\  | |             |   | syslog-ng  -----/          TLS encrypted              |          \         \                     |
                   ----|   NGINX     -----            | |                                        |  +-------|-------+  \  +---------------+ |
                     | |             |   +------------+ |                                        |  |               |   \ |               | |
                     | +-------------+    /             |                                        |  |     NGINX     |    ||    mongod     | |
                     |                   /              |                                        |  |               |     |               | |
                     | +-------------+  /               |                                        |  +------|--------+     +---------------+ |
                     | | /dev/log    | /                |                                        +---------|--------------------------------+
                     | | (kernel logs|/                 |                                                  |                                 
                     | | , etc.)     |                  |                                                  |                                 
                     | +-------------+                  |                                                  |                                 
                     |                                  |                                                  |                                 
                     +----------------------------------+                                        +---------|---------+                       
                                                                                                 |                   |                       
                                                                                                 |  Tech Team member |                       
                                                                                                 |                   |                       
                                                                                                 +-------------------+                       

In the example above, test131 runs syslog-ng, which is responsible for receiving the logs locally and sending them to graylog-server. By setting base::syslog::syslog_daemon to 'syslog_ng' in puppet, base::syslog will install syslog-ng and configure it to listen on 127.0.0.1:10514 (for anything on the server sending its logs to that destination, such as MediaWiki and NGINX) and 'system' for services such as ssh and kernel logs.

Streams[edit | edit source]

Streams are Graylog's categories of data. By default, the All messages stream is the stream for every message sent to Graylog. Streams are useful to limit access for certain members. For example, MediaWiki Administrators can only access the streams for MediaWiki and NGINX logs.

Quering the data[edit | edit source]

Graylog has a search syntax that's close to Lucene's syntax. For MediaWiki and NGINX, custom fields have been defined: go to https://graylog.miraheze.org/search and click on 'Fields' on your left. Using these fields, you can query the data. For example:

  • View NGINX logs for your IP address: nginx_remote_addr:"1.2.3.4"
  • View all SSH logs: application_name:"sshd"
  • View all MediaWiki errors and warnings: application_name:"mediawiki" AND (mediawiki_level:"ERROR" OR mediawiki_level:"WARNING")

Access[edit | edit source]

For security reasons, the Graylog interface is not accessible without a SOCKS5 proxy, just like Proxmox' interface. In order to make the process of using tunnels as easy as possible, please install SmartProxy: Chrome or Firefox. We'll be using port 8089 (although other ports will work too) on your desktop or laptop, which will be used for a SOCKS5 proxy over SSH. If you have access to graylog121, you can use graylog121.miraheze.org. If you don't have access to graylog121, use either of the Bastion servers (bast*.miraheze.org).

In SmartProxy, create a proxy server: Proxy Server > Add server > Name = "Miraheze Proxy", Address = "127.0.0.1", Port = "8089", Protocol = "SOCKS5" > Save. Afterwards, create a proxy rule: Proxy Rules > Add rule > Rule type = "Search Domain and SubDomain", Domain = "graylog.miraheze.org", then "Apply Proxy" to "Miraheze Proxy" > Save and then click "Save" on the bottom of the page as well.

You can also see this quick video on what the configuration looks like for SmartProxy

OpenSSH[edit | edit source]

If using OpenSSH, you can use ssh -D 8089 <server>.miraheze.org.

PuTTY[edit | edit source]

It is recommended to save this config to a session. Choose a server you would like to connect to. Go to Connection > SSH > Tunnels, enter 8089 in Source port and select the radio buttons Dynamic and Auto. If you are planning to use Graylog for an extended period of time, without using PuTTY for executing commands on servers (idle state), you may hit a timeout: see this for a fix.

Administration[edit | edit source]

Configuring Graylog is a combination of Puppet usage and using the web interface for configuration (where configuration will eventually be stored in MongoDB on graylog121.miraheze.org). role::graylog is used for graylog121's configuration. base::syslog contains the configuration for every server logging to Graylog.