Observability

List of platforms in use by PUL IT and/or DLS

HoneyBadger
- see also pulibrary/pul-it-handbook/services/honeybadger.md
- Used for error tracking per-project.
- Also used for some uptime monitors, though TODO not sure if those are working / on?
DataDog
- see also pulibrary/pul-it-handbook/services/datadog.md
- Used for apm and log aggregation
- VM-level metrics with dashboards (especially network volume, latency)
- Service health metrics and dashboard, powered by data from our health endpoints
CheckMK
- there’s also a /staging instance
- see also pulibrary/pul-it-handbook/services/checkmk.md
- Monitoring and alerts, especially lower-level metrics
SigNoz
- Currently in trial implementation
- use “sign in with SSO”
- provides log aggregation and APM
Grafana
- sign in with github
- Set up by DLS to quickly implement project-specific metrics
- Temporary home for some stats and dashboards that may move to signoz
- Runs on nomad

You can look at the little status page honeybadger builds for us.

Long load times or “bad gateway” errors

Check APMs for large latency times for the page that’s reported as slow
Dashboards for CPU and Memory use (or run htop on boxes)
Curl from localhost and from load balancer to check for network latency from various places
Tigerdata mount point (for figgy): ls /mnt/tigerdata
Passenger queue sizes: sudo passenger-status
- if the queue is full, look at the requests sudo passenger-status --show=requests > requests.txt
Search logs for patterns in requests
- Is everything erroring or just some things?
- There are logs from the load balancer, logs from the local web server, application logs, and traefik logs
- note the status code isn’t always correctly parsed from the log line
- search for any unusual error messages you find - stackoverflow, the project’s ticketing system, reddit
Traefik
- This traefik Network diagram is a helpful reference
- Dashboard: Traefik’s view of server health