Disabling Pickle Did Not Remove Pickle

SGLang lets operators disable direct Pickle transport for its internal messages. I expected that choice to remove Pickle from the receive path. It changed the envelope and left the decoder where it was.

I ran a stock SGLang v0.5.18 service with a real model across two GPUs. I configured one of its internal receivers to be reachable from a second machine and sent it a message from a peer that had not authenticated. Code chosen by that peer ran inside the receiver and wrote a small marker file containing its own process ID. The ID matched the receiver the service had launched. The result was the same with the setting on and off, on three fresh starts each.

The receive path explains why both modes executed code. The deployment determines who can reach it. In the default single-node layout, the receiver stays local to the host. My test used a supported configuration that made it reachable from another machine.

The switch changed the envelope

SGLang runs a model as several cooperating processes. They pass work and results to each other over ZeroMQ, a messaging library that can carry messages over local sockets or over TCP. This is inter-process communication, or IPC, but it is not necessarily confined to one host.

The receiver I followed was the detokenizer, the process that turns the model’s output token IDs back into text. In normal operation it receives messages from the scheduler. Before it can act on one, it has to turn the incoming bytes into a Python value.

Python Pickle does more than read fields from a data container. Its reconstruction instructions can identify a function and call it with supplied arguments. Those calls happen while loading the object. The Python documentation therefore warns against unpickling untrusted data.

In the tested release, SGLANG_USE_PICKLE_IPC defaults to true. The receive function calls PyZMQ’s recv_pyobj, which reconstructs the received Python object using Pickle. The PyZMQ documentation makes the same trust requirement explicit.

Setting the flag to false selects MessagePack instead. MessagePack represents data such as numbers, byte strings and structured records. Choosing it can remove the need to reconstruct arbitrary Python objects. Here, though, the accepted message types included PickleWrapper, a record carrying Pickle bytes.

The wrapper was accepted as a top-level message. Once MessagePack had decoded that record, SGLang immediately unwrapped it with Python’s Pickle decoder. The outer format had changed, but the sender could still reach the operation that invokes functions.

The wrapper existed for compatibility: it let the typed transport carry Python objects that its ordinary message types could not represent. A peer could select that same path.

The check came after the effect

The v0.5.18 receive and unwrap functions show the order. This condensed pseudocode omits logging and other message types.

# Condensed pseudocode, not an exploit client.
def receive(socket):
    if use_pickle_ipc:
        return socket.recv_pyobj()

    message = typed_msgpack_decode(socket.recv())
    if isinstance(message, PickleWrapper):
        return pickle.loads(message.data)
    return message

# The receiver checks the reconstructed value afterward.
message = receive(socket)
dispatch(message)

The MessagePack decoder checks whether the outer record belongs to an accepted type. It accepts the wrapper. Later, the application dispatcher checks whether the reconstructed result is a message the detokenizer knows how to handle.

Neither check prevents a callable from running during the intervening Pickle reconstruction. The receiver’s event loop does not call its dispatcher until sock_recv has returned.

In the experiment, reconstruction ran the marker-writing action. Dispatch then received a value it did not recognise, rejected it, and the detokenizer exited. The error came after the code had already run.

A check on the finished object cannot undo work done while building it. A regression test that only expects an invalid-message error would miss this. It also has to show that the rejection happened without the supplied action running.

When an internal receiver becomes reachable

Who can send to the decoder depends on where the receiver binds.

The tested source has three branches. Without data-parallel attention, the component receivers use Unix-domain sockets, which never leave the host. Data-parallel attention is a serving mode that spreads attention work across parallel workers, and turning it on moves those receivers to TCP. Even then they bind to loopback, unless the launch also supplies a distribution address. Give it an address that is not loopback and the receivers bind there instead.

A route from another machine needs all three: data-parallel attention, a non-loopback distribution address, and a network path to the receiver’s port. I tested this supported, non-default configuration.

I kept the public HTTP endpoint on loopback and exposed only the detokenizer’s receiver to a second authorized machine. The crafted message went to that internal receiver; delivery through the inference API was not demonstrated.

A loopback HTTP endpoint does not keep a separately bound component receiver local. If the service runs with data-parallel attention and a distribution address, its internal receivers listen on that address. Their network rules determine who can reach them; the serialization setting does not make untrusted messages safe.

From a code path to a running service

Source review and a reconstruction of the receive logic showed that the decoder was unsafe. I still needed to establish whether the process created by an ordinary launch would accept the same bytes.

A local campaign reached the launcher-created receiver over Unix-domain sockets. I then tested a real service from a second physical machine, exercising the deployment condition and the decoder together.

Several smaller model fixtures never produced a working inference baseline under the two-GPU layout, so I excluded them. A marker from an unhealthy launch would not demonstrate the effect in a working inference service. The final campaign used the Qwen3-30B-A3B model that SGLang’s own two-GPU test exercises, and every trial had to answer an ordinary request before the crafted message was sent.

The service was stock v0.5.18 at commit 71de97b264b04dcd514cf904003028aefe9775c8, on one host with two NVIDIA A100-SXM4-80GB GPUs, tensor and data parallelism both set to two, data-parallel attention on. The second machine only sent messages. It was not another model worker.

The witness was a marker write. Reconstruction selected a callable and its arguments, and that callable wrote the receiver’s process ID to a file, tying the effect to the listening process.

A crash would show that the input disrupted the service. Here the selected action finished before the crash, inside the process that owned the receiver. That is code execution.

What six fresh starts established

I started the service three times with the setting on and three times with it off. Each trial began with a successful ordinary request. Then I connected from the second machine without sending anything. Connecting alone produced no marker.

Sending the crafted message produced exactly one marker in every trial, and every marker’s process ID matched the receiver the launcher had created. Three out of three with raw Pickle. Three out of three with MessagePack carrying a PickleWrapper.

All six connect-only outcomes are recorded in one hash-bound operator transcript. Five of the six trials also have separate state, inference, marker and service records. The first default-mode trial’s baseline and listener identity are recorded only in that transcript. The connect-only control rules out the handshake as the cause of the marker; it does not compare a crafted message against a well-formed one.

The result is remote code execution in the reachable receiver under the tested configuration. An unauthenticated peer selected an action that ran with the receiver’s own permissions, in both receive modes.

What a repair has to change

Disabling direct Pickle transport did not mitigate this, because an accepted message type brought Pickle back after the outer decode. A repair has to follow the bytes through the whole receive path, not just the first function that touches them.

For any transport a peer can write to, I would remove both the raw recv_pyobj path and PickleWrapper from the accepted message types. Every field needs a data-only representation the decoder can validate without calling a function the sender picked. Once that validation has passed, peer-controlled bytes must not reach pickle.loads through an adapter further down.

Peers also need to authenticate, and to be authorized, before the decoder accepts work from them. That narrows who can send. A non-executing format narrows what an accepted sender can make reconstruction do. Both belong in the design, and neither replaces the other.

Where cross-host traffic is unnecessary, keep the receivers local. Where TCP is required, restrict it at the host and network layers to the components that need it. That limits which peers can reach the decoder while the path is being repaired; the serialization setting did not prevent execution in these tests.

I would test a repair against both raw and wrapped inputs, checking for zero side effects as well as rejection. Valid typed messages should still reach their handlers. An exception on its own does not show that a repair works.

Coordination and source status

CERT/CC received the report on July 14, 2026. On September 17 the coordinator confirmed it as arbitrary code execution under the non-loopback, data-parallel-attention condition. The case is VU#765030, and the identifier assigned to it is CVE-2026-93034.

The executed results above are for v0.5.18. A separate commit-pinned source inspection on October 1 found the same receive and unwrap primitives in v0.5.20 and in the main revision inspected that day. That is source evidence. It is not a runtime test of those versions, and it is not a claim about a continuous affected range.

A public release check on October 7 found v0.5.21, released after that source inspection. This post has not assessed that release’s receive path or runtime behavior, so the October 1 inspection does not establish its fix status.

There is an earlier public SGLang report involving Pickle in shm_broadcast.MessageQueue. It concerns a different receive path. This post follows the manager-message decoder and the launcher-created detokenizer.

A typed outer format can still carry instructions that execute during object reconstruction. To know what a serialization setting protects, follow every accepted message through to the operation that turns its bytes into a value.

This post documents CRUCIBLE-2026-131, coordinated through CERT/CC as VU#765030 and assigned CVE-2026-93034. The full-service experiments used stock SGLang v0.5.18.