Layer
An inference framework for AI routing and model triage that supports local model loading, editable installation, and automatic routing through Router, avoiding the overhead of HTTP calls.
Timeline (last 90 days)
Layer is an inference framework for AI routing and model dispatch, supporting local installation in Editable mode and direct model loading, eliminating HTTP call overhead.
When running Layer examples, you must use the Python interpreter in the Venv under the Layer directory rather than the system global Python, because Layer depends on third-party packages such as Torch, Flash, and Tension, and reusing it directly avoids version mismatches or environment conflicts.
Router is Layer's routing layer, essentially a lightweight classification/decision model that reads the input text (and optionally State), determines the problem domain and forwards it to a downstream model, and the Routing field in the returned result records the target model and reason.
In Layer's Router example, setting MaxLoaded=1 means the maximum number of models resident in GPU memory at the same time is 1, and switching models requires unloading the old model before loading the new one, trading time for GPU memory.
In Layer's Router example, the input 'billing and refund question' was routed to the English model with reason 'English Latent Text', and the final Department selection was Billing with probability 0.98, Technical with probability 0.0132, and confidence 0.8984; the input consumed 80 Tokens, and the output does not count toward Tokens.
Layer provides an interface compatible with System1; the local service starts on port 127.0.0.1:8000, and the request parameters are exactly identical to the official TypeSafe interface; when the Model field is set to JV1, Layer cannot recognize the model and will automatically fall back to Routing mode, whereas manually specifying English, MartinLingo or Type Decision is equivalent to specifying the model in Layer.load.
Layer is compatible with the official TypeSafe interface in input and output; when using the TypeSafe SDK, you only need to replace BaseURL, API Key and Model to point to Layer; the example installs the SDK via uv add typeSafe-sdk, creates a Client pointing to localhost:8000, and filling Model with jf-1 automatically goes through Routing.
All three invocation methods (SDK, HTTP and third-party TypeSafe SDK) have been tested, and basically all of the official test files pass, with only two test cases related to "酷炸导单" failing; the presenter believes the problem lies in the test case design.
Layer's request response speed is at the millisecond level; the number of tests is not large but the magnitude is stable.
All Facts (9)
Layer is an inference framework for AI routing and model dispatch, supporting local installation in Editable mode and direct model loading, eliminating HTTP call overhead.
50%UnverifiedWhen running Layer examples, you must use the Python interpreter in the Venv under the Layer directory rather than the system global Python, because Layer depends on third-party packages such as Torch, Flash, and Tension, and reusing it directly avoids version mismatches or environment conflicts.
50%UnverifiedRouter is Layer's routing layer, essentially a lightweight classification/decision model that reads the input text (and optionally State), determines the problem domain and forwards it to a downstream model, and the Routing field in the returned result records the target model and reason.
50%UnverifiedIn Layer's Router example, setting MaxLoaded=1 means the maximum number of models resident in GPU memory at the same time is 1, and switching models requires unloading the old model before loading the new one, trading time for GPU memory.
50%UnverifiedLayer is compatible with the official TypeSafe interface in input and output; when using the TypeSafe SDK, you only need to replace BaseURL, API Key and Model to point to Layer; the example installs the SDK via uv add typeSafe-sdk, creates a Client pointing to localhost:8000, and filling Model with jf-1 automatically goes through Routing.
50%UnverifiedLayer's request response speed is at the millisecond level; the number of tests is not large but the magnitude is stable.
50%UnverifiedLayer provides an interface compatible with System1; the local service starts on port 127.0.0.1:8000, and the request parameters are exactly identical to the official TypeSafe interface; when the Model field is set to JV1, Layer cannot recognize the model and will automatically fall back to Routing mode, whereas manually specifying English, MartinLingo or Type Decision is equivalent to specifying the model in Layer.load.
50%UnverifiedIn Layer's Router example, the input 'billing and refund question' was routed to the English model with reason 'English Latent Text', and the final Department selection was Billing with probability 0.98, Technical with probability 0.0132, and confidence 0.8984; the input consumed 80 Tokens, and the output does not count toward Tokens.
50%UnverifiedAll three invocation methods (SDK, HTTP and third-party TypeSafe SDK) have been tested, and basically all of the official test files pass, with only two test cases related to "酷炸导单" failing; the presenter believes the problem lies in the test case design.
50%