We had an interesting problem yesterday where we saw one of our github post-recieve hooks had “randomly” stopped working. This hook was responsible for positing commits containing issue numbers to JIRA (our bug tracker). While not at all vital, it was extremely convenient and there were several developers lamenting its loss.
The first course of action was to identify the last successful update. Having narrowed down the time in which a change would have to have occurred on the ops side of things we ping ops and asked if there had been any changes during that window. Nope. Next we checked any recent code changes. Nope. Had any automated tasks run on the box? Had something changed in the load balancer? Had something changed on JIRA? Nothing.
So now… we were thoroughly confused. If you hit the githook manually with curl:
curl -X POST blah.example.com/blah/githook
you could get reasonable responses and it showed up in our logs. When we used github’s convenient little test hook button:
We saw nothing in the logs and no output to the hook.
Obviously something was wrong on github’s side. But what could have changed? We added another hook to ping on a server that wasn’t in front of our load balancer. Now the test hook button worked just fine.
Finally someone had the clever idea to check the host header in the requests.
When we set the host header without a port e.g.:
mongodb.org
we could get a response to show up in the logs.
If however we specified the port e.g.:
mongodb.org:80
the request was getting caught by our load balancer and forwarded elsewhere. We had been making the wrong assumptions the entire time.
In summary, github started adding the port to the Host header of their post-receive hook requests and our load balancer was misconfigured. I’m just sharing this info incase anyone else runs into a similar issue.
