Observability

List of platforms in use by PUL IT and/or DLS

• HoneyBadger
  • see also pulibrary/pul-it-handbook/services/honeybadger.md
  • Used for error tracking per-project.
  • Also used for some uptime monitors, though TODO not sure if those are working / on?
• DataDog
  • see also pulibrary/pul-it-handbook/services/datadog.md
  • Used for apm and log aggregation
  • VM-level metrics with dashboards (especially network volume, latency)
  • Service health metrics and dashboard, powered by data from our health endpoints
• CheckMK
  • there’s also a /staging instance
  • see also pulibrary/pul-it-handbook/services/checkmk.md
  • Monitoring and alerts, especially lower-level metrics
• SigNoz
  • Currently in trial implementation
  • use “sign in with SSO”
  • provides log aggregation and APM
• Grafana
  • sign in with github
  • Set up by DLS to quickly implement project-specific metrics
  • Temporary home for some stats and dashboards that may move to signoz
  • Runs on nomad

Scenarios

Are our prod sites up?

You can look at the little status page honeybadger builds for us.

Slow or intermittent site

Long load times or “bad gateway” errors

• Check APMs for large latency times for the page that’s reported as slow
• Dashboards for CPU and Memory use (or run htop on boxes)
• Curl from localhost and from load balancer to check for network latency from various places
• Tigerdata mount point (for figgy): ls /mnt/tigerdata
• Passenger queue sizes: sudo passenger-status
  • if the queue is full, look at the requests sudo passenger-status --show=requests > requests.txt
• Search logs for patterns in requests
  • Is everything erroring or just some things?
  • There are logs from the load balancer, logs from the local web server, application logs, and traefik logs
  • note the status code isn’t always correctly parsed from the log line
  • search for any unusual error messages you find - stackoverflow, the project’s ticketing system, reddit
• Traefik
  • This traefik Network diagram is a helpful reference
  • Dashboard: Traefik’s view of server health
  • check whether it’s on by going to the /challenge path of an application
  • You can also see current logs in the nomad UI
  • You should be able to see it in the load balancer UI as the first upstream defined for any application that’s using it
  • the signoz logs for traefik show the request’s protected path, and the series of nginx config stanzas that request passed through as well as the VM that ultimately served the request.