Pitfalls of Connecting LangExtract to OpenAI-Compatible APIs: Model Routing, Strict Mode, and Silently Dropped Chunks

Google's LangExtract is a Python library that extracts structured entities from long texts, with each extraction mapped back to its character position in the source text. You just describe the task and provide one or two examples, and it automatically chunks the text, calls the model chunk by chunk, parses the JSON, and aligns the results. The default backend is Gemini, but it includes a built-in provider based on the official OpenAI SDK, which in theory can connect to vLLM, LiteLLM, or any hosted gateway.

However, problems arise when you point model_id at a model that doesn't follow OpenAI naming conventions (for example, a locally deployed qwen3-coder-plus). This article is based on hands-on testing with langextract 1.7.1 (openai 3.35.1, Python 3.13), using a local mock server that logs all requests and can return 400/429/503, fenced JSON, or truncated output on demand to reconstruct what actually happens on the wire. The author also discloses that they developed the gateway used in the examples, APIClaw, so gateway-related opinions should be taken with some reservation.

The Model Name Determines Which Provider Is Used

The most intuitive way to call it looks like this:

import langextract as lx

result = lx.extract(
    text_or_documents=text,
    prompt_description=prompt,
    examples=examples,
    model_id="qwen3-coder-plus",
    api_key="your-key",
    model_url="https://apiclaw.biz/v1",
)

Looks fine, but LangExtract's routing is implemented via regex matching on model_id. Only names starting with gpt-3.5, gpt-4, gpt-5, or o1 through o9 go to the OpenAI provider. Names starting with qwen, deepseek, llama, mistral, gemma, phi, gpt-oss, etc., get routed to the Ollama provider—even if you passed model_url.

The test results are telling:

• qwen3-coder-plus and gpt-oss-120b were sent to the Ollama client, POSTing to <base_url>/v1/api/generate, returning Can't find Ollama qwen3-coder-plus. Try: ollama run qwen3-coder-plus.
• plain-model, which matches no pattern at all, errors out directly: No provider registered for model_id='plain-model'.
• gpt-4o-mini with model_url correctly hits /v1/chat/completions.

In other words, the "obvious" code above only works if your gateway model happens to look like an OpenAI model name.

Force the OpenAI Provider with ModelConfig

The way around the routing regex is to explicitly construct a ModelConfig:

import os
import langextract as lx

config = lx.factory.ModelConfig(
    model_id="YOUR_MODEL_ID",
    provider="OpenAILanguageModel",   # "openai" also works
    provider_kwargs={
        "api_key": os.environ["APICLAW_API_KEY"],
        "base_url": "https://apiclaw.biz/v1",
        "temperature": 0.0,
        "max_output_tokens": 2000,
        "max_workers": 4,
    },
)

result = lx.extract(
    text_or_documents=text,
    prompt_description="Extract every person mentioned, with their role. Use exact text.",
    examples=examples,
    config=config,
    show_progress=False,
)
for e in result.extractions:
    print(e.extraction_class, e.extraction_text, e.attributes, e.char_interval)

This way, regardless of what model_id is called, requests reliably go to POST /v1/chat/completions. Two details require attention:

• api_key must be passed explicitly. LangExtract only reads the OPENAI_API_KEY environment variable when the model ID contains gpt. When the provider is explicitly specified but the name is qwen3-coder-plus, omitting api_key results in API key not provided. even if the environment variable is set.
• base_url can come from an environment variable. When base_url is not passed, the SDK reads OPENAI_BASE_URL. Either way, keep the trailing /v1.

Enabling config also produces a warning: With 'config', schema constraints are still applied via examples.—this directly relates to the next section.

What's Actually in Each Request

Each text chunk becomes one request, containing:

• A system message: You are a helpful assistant that responds in JSON format.
• A user message: containing the prompt description, examples arranged as Q:/A: with their expected JSON, and the current chunk text;
• n: 1;
• response_format of type json_schema, named langextract_extractions, with strict: true, and a schema generated from your examples.

By default, requests contain neither temperature nor max_tokens, leaving everything to server defaults. Setting them explicitly in provider_kwargs adds temperature: 0.0 and max_tokens: 2000 to the request. For extraction tasks, it's recommended to set both: low temperature for consistency, and an output limit large enough to accommodate a long list of entities.

What to Do When the Server Doesn't Support json_schema

Strict json_schema output is not supported by all models and compatible services. Some models only support json_object, and some support neither. When the mock server returns 400 for json_schema, the entire extract() throws InferenceRuntimeError: OpenAI API error: Error code: 400.

The solution is to disable schema constraints:

result = lx.extract(..., config=config, use_schema_constraints=False)

With this disabled, the request switches to response_format: {"type": "json_object"} and succeeds immediately. The prompt examples still show the model the expected structure, just without server-side hard constraints.

Parse Failures Are Silently Dropped—This Is the Default Most Worth Changing

This is the most dangerous part found in testing. By default, LangExtract swallows parse errors for each chunk. In the following two cases, the run only prints a single log line and returns zero extractions with no exception:

• The model wraps its answer in explanation text and fences (first outputs Here is the result:, followed by a fenced json block). In json_object mode, LangExtract expects bare JSON, so the chunk is dropped, with the log Chunk omitted from results (parse error): Failed to parse JSON content.
• The response is truncated due to finish_reason: "length", the incomplete JSON fails to parse, and it's likewise silently dropped.

In a long document of 40 chunks, a few dropped chunks look like "the model found nothing in that section"—which leads you to completely wrong extraction conclusions. The fix is:

result = lx.extract(
    ...,
    config=config,
    use_schema_constraints=False,
    fence_output=True,                                  # accept fenced JSON responses
    resolver_params={"suppress_parse_errors": False},   # fail loudly instead
)

fence_output=True correctly parses fenced responses; suppress_parse_errors: False turns silent drops into a catchable ResolverParsingError, letting you retry. Testing also found that responses starting with a <think>...</think> block parse fine, so reasoning models are not inherently a problem.

How Chunking, Concurrency, and Multiple Passes Amplify Request Volume

On a 5,320-character document, the measured request counts were as follows:

Setting	Request count	Max concurrency
Default (max_char_buffer=1000)	6	6
max_workers: 3 (in provider_kwargs)	6	3
extraction_passes=3	18	6
max_char_buffer=4000	2	2

Each request repeats the prompt description and examples, so the smaller the chunk, the higher the proportion of fixed overhead in each request. max_workers defaults to 10, meaning long documents may fire 10 requests simultaneously; if your plan has per-minute or concurrency limits, be sure to lower it. extraction_passes reruns the entire document serially—three passes triples both request count and elapsed time.

Increasing max_char_buffer reduces request count, but longer chunks also mean the model has to track more content; it's best to test recall on your own documents before raising it significantly.

Retries Come Only from the SDK—LangExtract Does Not Retry Itself

With the mock server continuously returning 503, the SDK retries twice by default (three requests total), after which extract() throws InferenceRuntimeError. When returning 429 with Retry-After: 2, the SDK waits correctly and the third request succeeds after about 4.6 seconds. LangExtract itself has no retry loop, so large-scale batch jobs require you to wrap a retry layer at the call level; also remember that a single failed chunk fails the entire document.

Practical Checklist

• Unless model_id starts with gpt- or o1~o9, explicitly specify the provider with ModelConfig(provider="OpenAILanguageModel").
• Pass api_key explicitly in provider_kwargs; environment variable keys only work for model names starting with gpt.
• Set temperature and max_output_tokens yourself.
• When the server rejects json_schema, switch to use_schema_constraints=False.
• Set suppress_parse_errors: False, and add fence_output=True for chatty models, to avoid chunks being silently dropped.
• Lower max_workers according to rate limits, and treat extraction_passes as a request-volume multiplier when estimating cost.

While these findings were verified on APIClaw, they apply equally to any OpenAI-compatible endpoint—the routing logic, schema constraints, and silent chunk dropping are all behaviors of the LangExtract client itself, independent of the specific gateway.