Aggressive NeuVector Stopped SSO

One weekend in early September, I got a call: SimpleSAMLphp (SSP), part of our SSO infrastructure, was not starting. People needed to log in, so this was not a problem I could leave for Monday.

Nothing obvious had changed. We had not upgraded SSP recently. Somewhere in the back of my mind was the uncomfortable question: Are we in the middle of a security incident? I started with the SSP logs and found a more specific failure: PHP lint could not start inside the container.

PHP lint validates the metadata files SSP needs. It runs when SSP starts, and it can run again when a change to our metadata repository triggers a sync through a webhook. I checked whether NeuVector was stopping it. To my surprise, it was.

That made little sense. SSP had been running in this cluster since before we introduced NeuVector, which had already learned its processes. But users were waiting, and I could investigate the history once they could log in again. I approved PHP lint in NeuVector. SSP came up, and users could log in. Quick victory. Now I wanted to know why I had needed to approve an old process at all.

I checked NeuVector's access log for a policy change and found no logins around the failure. The event log held a stranger clue. About 25 days before the outage, NeuVector had recorded this message (timestamp omitted):

Group.Auto.Remove Auto remove unused group: [SSP group] and related network/response rules.

Unused? SSP runs around the clock. Its StatefulSet still had two pods, and they had been running when NeuVector removed the group. I found another automatic removal about five days earlier. NeuVector was treating a live application as if it had left the cluster.

I kept going backward through the Kubernetes events. In the days leading up to the second removal, the NeuVector controller had been failing and coming back up repeatedly. We had missed the pattern while dealing with other fires. One event showed how little memory was left on its node:

The node was low on resource: memory.
Threshold quantity: 100Mi, available: 65236Ki.
Container [NeuVector controller] was using 1395400Ki, request is 0.

The node had about 64 MiB available. NeuVector was using roughly 1.3 GiB in that event, and its previous runs used a little over 2 GiB. The controller had been OOM-killed multiple times. In our installation, NeuVector had no memory requests or limits set, so the scheduler had no stated minimum memory requirement to use when assigning it a node. It could land on a node without enough room to keep running.

This cluster had started small. We created it to move SSP to the cloud, independent of our organization's network. More workloads and resource use accumulated over time, and NeuVector's own memory use grew. That week, the available memory and the controller's needs finally collided.

The one-day unused-group cleanup setting made the instability matter more. If NeuVector failed to recognize a group it had seen before, that group could be removed after only a day. My reading of the events is that the restarting controller lost track of SSP long enough for cleanup to remove its group and learned rules. I also found similar memory issues around the first removal. That gave me a plausible explanation for both removals, although I could not see the exact moment NeuVector lost track of SSP.

Once NeuVector landed on a node where it could stay up, it discovered SSP again. The old process rules were gone, so SSP went through learning as if it were new. Looking back from the outage weekend, the approximate timeline was:

During Discover mode, NeuVector watched SSP and learned the process calls it saw. PHP lint never ran in that window: the existing pods stayed up, and no metadata change triggered a sync. NeuVector moved through Monitor to Protect without learning that process.

Then scheduled AKS node maintenance cycled the nodes, and SSP's pods had to restart on the new nodes. This time PHP lint ran. NeuVector was in Protect mode, saw a process missing from its newly learned profile, and blocked it. SSP could not finish starting. That was the failure behind the weekend call.

Approving PHP lint restored service, but it only fixed the immediate block. As part of the root cause analysis, I submitted a remediation plan to the rest of the team for approval: increase the cluster's available memory by scaling vertically (using larger nodes) or horizontally (adding another node), set NeuVector's memory request based on observed usage and its limit with room for peaks, and extend unused-group retention from one day to one week. The request would give the scheduler a useful minimum when placing NeuVector; the longer cleanup window would give it more time to recover from a missed observation.

What stayed with me was how ordinary the final blocked process was. PHP lint had not changed. SSP had not suddenly become a new application. NeuVector had forgotten what it knew about a service that had been there the whole time.