Can Frontier Models Really Solve Project Security Problems? Lessons from a Real Pull Request

Can today’s frontier models solve project security problems? They increasingly appear capable of finding vulnerabilities and proposing fixes. But model involvement must be documented, and every security patch—human- or AI-produced—still needs technical review at the actual runtime boundary.

FeatBit security advisory GHSA-vgp4-8v92-44m6 and pull request #938 form the evidence chain for this case. External security contributor tonghuaroot reported the vulnerability through GitHub’s private security-advisory workflow and prepared an initial fix in the advisory’s temporary private fork. The maintainers later moved the implementation to public PR #938, where the patch, review comments, follow-up commits, and merged solution are visible.

The advisory is currently accessible only to authorized collaborators; public readers can verify the implementation history through PR #938. Neither record attributes the original report or implementation to Fable 5, Mythos 5, or another frontier model. The visible AI contribution is limited to GitHub Copilot’s review comments and one co-authored autofix commit.

That evidence boundary makes the case useful: it gives us a concrete standard for judging any security patch without inventing model attribution. Did the proposed behavior match the product contract? Was the runtime boundary protected? Were edge cases covered? Were existing libraries evaluated? Did maintainers with sufficient project and technical knowledge verify the result?

The Case: An Authenticated User Could Influence a Webhook Destination

FeatBit sends outbound webhooks when configured events occur. If an authenticated organization member can supply a webhook URL without adequate destination controls, the backend may be induced to request addresses that external users should not reach: loopback services, private networks, link-local endpoints, or cloud metadata services.

That is the essential SSRF pattern. OWASP describes SSRF as abuse of server-side functionality to read or update internal resources through a URL influenced by an attacker.

The original security advisory classifies the issue as High severity and CWE-918. It credits tonghuaroot as the reporter and records FeatBit self-hosted back-end API versions <= 5.4.2 as affected at the time of the report. This article describes the code and review history without inferring release status beyond the advisory record.

The advisory provides several facts that the shorter PR description does not:

• An authenticated organization member—including a low-privilege member—could reach the vulnerable send path.
• The issue was non-blind: the downstream status, headers, and response body were returned to the caller.
• The caller could control outbound headers and the JSON body.
• Both the immediate POST /api/v{version}/webhooks/send route and stored webhooks used the same unvalidated sender.
• Redirect following and alternative IP encodings expanded the bypass surface beyond obvious private IPv4 strings.

These details are important because they establish exploitability, privilege requirements, affected data flow, and impact. The contribution did more than label a suspicious line of code: it documented a reproducible project-level vulnerability and prepared a fix for maintainer review.

The advisory included end-to-end evidence

The report reproduced the issue against a deployed FeatBit API server with a controlled sentinel service on the backend network. A unique canary response was returned through the webhook delivery; a decimal-encoded IPv4 address reached the same service; and the backend attempted to connect to the link-local cloud-metadata address. Its negative control then showed the proposed validator rejecting internal, encoded, metadata, and IPv6 loopback targets before a socket was opened while preserving access to a public destination.

This evidence established the vulnerability and validated the initial direction. It did not establish that the first validator was complete—the later review found the DNS-rebinding window, omitted address ranges, and information-disclosure issue discussed below.

But identifying the vulnerability class, defining the intended product behavior, and implementing a robust defense are three different tasks.

The Deeper Gap: The Fix Did Not Fully Define the Business Contract

The advisory reveals a more important limitation than any omitted IP range: the work moved from a generic SSRF diagnosis to a network-level fix without first making FeatBit's intended webhook contract explicit.

The vulnerable flow combined two related but different product behaviors:

• Live Debug: an interactive test route accepted a destination URL, headers, and JSON body, sent the request immediately, and returned downstream request and response details.
• Persisted delivery: saved webhooks used the same sender when feature-flag or segment events occurred later.

FeatBit's webhook documentation confirms that Live Debug is intentional product behavior. While creating or editing a webhook, a user can send a test payload to the configured URL and inspect request and response details. The security problem was therefore not simply that "the endpoint returns a response" or that "private addresses exist." The problem sat at the intersection of permissions, resource ownership, destination policy, diagnostic output, and self-hosted deployment requirements.

The PR did not change who could invoke Live Debug

The advisory records that POST /api/v{version}/webhooks/send inherited only the controller's authenticated-user requirement. A low-privilege organization member could call it directly. PR #938 changed the outbound HTTP handler, but it did not add a webhook-management permission or otherwise narrow access to the test operation.

After the network fix, that member could no longer reach blocked internal ranges, but could still ask the backend to send caller-defined requests to allowed public destinations and receive the response. The highest-impact SSRF path was mitigated; the business authorization question remained.

The API did not bind the test to an authorized webhook resource

The dashboard builds a Live Debug request from the webhook currently being created or edited. The backend, however, accepts the URL, headers, payload, and related fields directly in SendWebhook; it does not load an organization-owned webhook and derive the sensitive request parameters from a resource the caller is authorized to manage.

Those are different contracts:

• "Test this webhook configuration that I am allowed to manage."
• "Make the FeatBit backend send this request that I supplied."

A public-destination filter makes the second contract safer, but it does not enforce the first. A business-aligned design would decide whether draft testing is required, which permission controls it, and which fields the server should accept or derive.

A blanket public-only policy may conflict with self-hosted integrations

The merged PR configured the shared webhook HttpClient with an external-only policy. That directly addresses the reported internal-network attack, but it also establishes a product rule: Live Debug and persisted delivery can no longer target private destinations.

FeatBit is an open-source, self-hosted platform. It is reasonable to infer that some operators may intentionally connect FeatBit to an internal automation, audit, or integration service that has no public address. The public webhook documentation asks users to enter the URL FeatBit should call and does not currently state that only public Internet endpoints are supported.

This does not mean private destinations should remain unrestricted. It means the team needed to decide and document whether internal webhooks are unsupported, administrator-allowlisted, governed by deployment-level configuration, or handled differently in hosted and self-hosted environments. The PR discussion and tests do not show that product decision being made.

Live Debug response details need a deliberate disclosure policy

The advisory correctly identifies the complete downstream response as an impact multiplier: it made the vulnerability non-blind. Yet showing request and response details is also part of the documented Live Debug experience.

The right design might combine permission checks, response-size limits, sensitive-header redaction, content-type restrictions, or a deliberately reduced diagnostic view. The key is that "return everything" and "return nothing" are both product decisions. PR #938 sanitized the validator's own error after review, but it did not revisit the broader response contract.

This is the core alignment lesson: a technically valid vulnerability class does not uniquely determine the correct product behavior. A model or contributor can produce a stronger network boundary and still leave authorization too broad, bind the operation to the wrong resource model, or break legitimate deployment patterns. Security review must ask not only "Can this attack still work?" but also "Does the resulting behavior match what this product promises and what its users legitimately need?"

What the Initial Implementation Did Well

The advisory states that a self-contained validator and regression test were prepared in its temporary private fork. The FeatBit maintainer later recorded that the work was moved to PR #938 because Copilot could not review the private-fork PR. The public initial commit added that custom WebhookTargetValidator. It attempted to:

• Accept only absolute HTTP and HTTPS URLs.
• Resolve hostnames before a request was sent.
• Reject loopback, private, link-local, cloud metadata, CGNAT, and selected reserved address ranges.
• Unwrap IPv4 addresses embedded in IPv6 representations.
• Add tests for alternative IP encodings and malformed URLs.
• Stop delivery and record an error when a target failed validation.

This was not superficial code. The implementation accounted for several non-obvious SSRF bypass techniques, including decimal and hexadecimal IPv4 forms, IPv4-mapped IPv6, NAT64, and 6to4 addresses.

For reviewers, the first implementation made the threat concrete, exposed its assumptions, and supplied test cases. The subsequent review then strengthened both the implementation and the architectural approach.

Review Finding 1: Close the DNS Rebinding Window

The most important review finding was a time-of-check-to-time-of-use gap.

The validator resolved the hostname before sending the request. Later, HttpClient performed its own DNS resolution when establishing the connection. An attacker controlling DNS could make the hostname resolve to a public address during validation and an internal address during the actual connection. The preflight check would pass while the request still reached the protected network.

The GitHub Copilot review comment on the PR identified this DNS rebinding risk directly.

This is the distinction between checking a value and controlling the operation that uses it. For SSRF defense, a separate IsValidTarget(url) call is not a strong boundary if the networking stack later resolves and connects independently.

The safer design is to enforce the policy in the HTTP handler or connection path, where DNS resolution and connection establishment can be governed together. That is why this issue could not be solved by adding more IP ranges to the same preflight method.

Review Finding 2: Avoid a Handwritten Address Policy

The initial IsPublic method manually classified IP ranges. It covered many important cases, but the first version still treated several non-public or non-routable IPv4 ranges as acceptable.

The review identified multicast 224.0.0.0/4, reserved 240.0.0.0/4, limited broadcast 255.255.255.255, and common test or benchmark ranges as examples. A follow-up commit added more conditions.

That patch improved the list, but it also demonstrated the maintenance trade-off: every newly discovered omission required another branch in a security-critical method. A dedicated library could provide more focused review and reuse for this evolving network policy.

Security code often fails at the edges. A contributor—human or AI—can remember many reserved ranges and still omit one. The engineering response should not be “make the checklist longer” when an appropriate maintained abstraction already exists.

Review Finding 3: Separate Internal and User-Facing Errors

The first implementation included the validation failure reason in the persisted delivery error. That reason could contain a resolved IP address or DNS exception detail, and the delivery object was returned through the API.

The first review comment recommended keeping detailed information in server logs while returning a generic message to the caller. A Copilot autofix was then co-authored into the PR to sanitize the user-facing response.

This was a smaller issue than the DNS rebinding gap, but it is exactly the kind of business-context detail a project-level security fix must get right. Diagnostic information useful to an operator is not automatically appropriate for an API consumer.

The Architectural Improvement: Research Before Reimplementation

The decisive improvement came from stepping back from the custom validator and asking a broader engineering question:

Is there an existing, security-focused library that already solves this at the correct layer?

The maintainers considered two .NET options in the PR discussion:

• microsoft/AntiSSRF, which provides an HttpMessageHandler and configurable policies for outbound HTTP requests.
• idunno.Security.Ssrf, which also provides SSRF-aware handlers for HttpClient and ClientWebSocket and supports preflight checks when that behavior is required.

The final PR used Microsoft.Security.AntiSSRF. The replacement commit removed the custom validator and its tests, added the library package, and configured its handler for the webhook HttpClient.

That commit changed 299 lines while adding only 18 and deleting 281. The number is not proof of security by itself, but it captures the architectural improvement: less bespoke policy code, enforcement closer to the connection, and a smaller FeatBit-owned surface to review and maintain.

Using a library does not remove the need for review. The team still had to choose the right policy, allow plaintext HTTP because FeatBit webhooks support it, and disable the library’s default X-Forwarded-For behavior because a dummy header could cause third-party receivers to reject webhook requests. Reuse reduced the amount of security machinery FeatBit owned; it did not eliminate product-specific decisions.

What This PR Tells Us About Evaluating Frontier Models

This PR is not a Fable 5 or Mythos 5 evaluation. It is a real-world rubric for what a credible frontier-model security evaluation would need to measure.

Evaluation area	Evidence a frontier model would need to produce
Project-level detection	Trace the attacker-controlled webhook URL to the privileged outbound request and explain the SSRF impact.
Threat enumeration	Cover private networks, metadata endpoints, redirects, alternate IP encodings, IPv4-in-IPv6 forms, and DNS rebinding.
Boundary placement	Protect connection-time resolution rather than relying only on a detached preflight check.
Ecosystem research	Compare maintained SSRF libraries before proposing custom network-policy code.
Product contract	Define who may test a webhook, which authorized resource supplies the destination, whether internal integrations are supported, and what Live Debug may disclose.
Project fit	Preserve required webhook behavior across hosted and self-hosted deployments while controlling disclosure, headers, observability, and compatibility.
Verification	Supply adversarial tests and clearly state residual uncertainty instead of declaring the system secure.

The four review findings already described—boundary placement, address-policy completeness, API disclosure, and library selection—are the practical tests behind this rubric. A future model evaluation should be judged on that complete engineering outcome, not only on whether it identifies the vulnerability or produces a plausible diff.

A Better Workflow for Frontier-Model Security Changes

Teams can get substantial value from frontier models without treating their output as the security decision. The workflow begins with provenance: record the model, agent, task specification, tools, and human edits so later performance claims are verifiable.

1. Define the product contract before asking for code

Write down the intended actor, permission, owned resource, accepted inputs, supported destination classes, deployment differences, and response semantics. For this case, "test a webhook configuration I may manage" is a much more useful starting contract than "accept a URL and send a request."

Without that contract, a model can produce code that is technically stricter while still being wrong for the product.

2. Ask for attack surfaces before asking for code

Have the model map trust boundaries, attacker-controlled inputs, privileged operations, and external integrations. Require file and data-flow evidence for every finding.

3. Convert the finding into a threat model

Document the attacker, required access, controlled input, sensitive operation, full path to impact, bypass conditions, and enforcement boundary. This makes the difference between preflight URL validation and connection-layer enforcement explicit before implementation.

4. Search the ecosystem before writing security primitives

Ask the model to inspect the project’s dependencies and research maintained solutions. Compare candidates on enforcement layer, redirect behavior, DNS handling, policy configuration, release activity, license, tests, and integration cost.

The default rule should be: do not hand-roll parsers, cryptography, authorization frameworks, URL allowlists, network-range policy, or SSRF defenses unless existing options fail a documented requirement.

5. Require an adversarial second pass

The reviewer—human or model—should try to defeat the proposed fix rather than summarize it. Test resolution changes, redirects, special-purpose IPv4 and IPv6 ranges, proxies, error disclosure, and legitimate customer configurations. Also test whether a low-privilege caller can bypass the intended UI workflow, whether sensitive parameters can be derived from an authorized resource, and whether self-hosted internal integrations have an explicit policy.

6. Keep final risk acceptance human-owned

Security approval includes business decisions that are not in a generic vulnerability description: which destinations customers need, whether HTTP must remain supported, how errors should appear, what telemetry operators need, and what compatibility risk a new handler introduces.

The model can prepare evidence. A maintainer must accept the residual risk. That maintainer needs more than permission to approve: they need deep knowledge of the project, the runtime, the threat model, and the relevant security ecosystem. Without that practical depth, human review can become a ceremonial click rather than an independent control.

7. Make the release reversible

Code review can reduce risk before merge; it cannot prove production behavior. When a security hardening change may reject previously accepted customer configurations, ship it with observable, reversible rollout controls where the architecture permits.

FeatBit’s AI release engineering approach treats AI-assisted changes as release decisions, not merely code-generation outputs. Progressive rollout, telemetry, and rollback reduce the blast radius when an implementation is correct in principle but wrong for an unexpected production case. Our guide to AI code review tools versus human reviewers covers the complementary review boundary in more detail.

The Answer Is Yes—But Not Alone

Can current frontier models solve project security problems?

This PR does not measure a specific frontier model. It shows that finding a valid security issue and producing an initial patch are only part of the job; production readiness also depends on an explicit product contract, architecture, ecosystem research, adversarial review, and product context.

FeatBit’s response demonstrates the value of that process: the community report was reproduced, reviewed in public, iterated through concrete findings, and ultimately moved from bespoke validation code to a focused security library at the connection layer. It also shows where the process can go further: authorization, resource binding, internal-integration policy, and Live Debug disclosure should be reviewed as first-class parts of the fix rather than treated as consequences of a network preset.

The right operating model is not “AI finds and fixes security vulnerabilities.” It is:

AI expands detection and implementation capacity. Maintained security components reduce bespoke risk. Adversarial review tests the boundary. Humans decide whether the result fits the product. Controlled release limits what happens if everyone is still wrong.

That is a less dramatic promise than autonomous security engineering. It is also factual, testable, and useful today.