Solver plugins¶

A solver plugin connects Scaly models to an optimization solver. It is a separate Python package that generates a C wrapper around the solver's native interface. Read this page when implementing a backend. The solver guide covers using an existing solver.

Scaly generates the problem's objective, constraints, and derivatives as C functions, called oracles. The plugin's wrapper calls these functions and passes their results to the native solver. The wrapper and oracles compile together, so a numerical solve does not call back into Python.

Anatomy of a plugin¶

A plugin is a Python package that:

1. depends on scaly;
2. exposes a backend object through the scaly.solvers entry-point group;
3. when it wraps its own native solver, bundles the vendored shared library under <pkg>/lib/ and its C headers under <pkg>/include/, built by a hatch build hook (plugins/scaly-piqp and plugins/scaly-ipopt are the two reference implementations). plugins/scaly-sqp has no library of its own and generates C against scaly-piqp's.

# pyproject.toml
[project.entry-points."scaly.solvers"]
mysolver = "scaly_mysolver:BACKEND"

The entry-point name is the backend string users pass as the second argument to sc.solver(problem, backend).

The backend protocol¶

BACKEND must satisfy scaly.solvers.registry.SolverBackend, which defines the common backend metadata and the code generation hook. A backend accepts either quadratic programs, abbreviated QP, or nonlinear programs, abbreviated NLP. Scaly's just-in-time compiler, abbreviated JIT, uses the library and header paths when it builds the generated code.

NLP backends also satisfy scaly.solvers.registry.NlpSolverBackend:

require_backend(..., "nlp") validates these declarations at runtime. QP backends do not declare these members.

Discovery, protocol-version validation, and kind checking live in src/scaly/solvers/registry.py. Library and header discovery (src/scaly/solvers/paths.py) derives everything else from this metadata: solver_loadable("mysolver"), compile/link flags, diagnostics, and the per-plugin exact-path override env var SCALY_MYSOLVER_LIB.

The codegen contract: render_wrapper¶

def render_wrapper(fun: Function, ctx: SolverWrapperCtx) -> list[str]: ...

fun is the concrete graph instance being rendered. fun.descriptor (a SolverDescriptor, src/scaly/solvers/model.py) carries the problem dimensions, input/output signatures, oracle Functions and sparsity patterns. Its compile_options records structural choices. Tuning options are Python runtime data. ctx supplies the C names and helpers needed by the wrapper through scaly.codegen.solver.SolverWrapperCtx:

• ctx.symbol is the solver's mangled C identifier. Prefix every static the template declares with it. Each solver wrapper compiles in its own translation unit.
• ctx.raw_symbol is the name of the function the template must define.
• ctx.stats_symbol is the scaly_solver_stats static the template must fill on every call. Core declares it and exports the <symbol>_stats(...) accessor, and the plugin only writes the fields.
• ctx.raw_symbol_of(fn) gives the C symbol of an oracle/derivative Function or ExternalOracle from the descriptor. Scaly renders ordinary functions into the same translation unit. An external oracle contributes its declared source and raw symbol directly before the wrapper. Declare supplied functions static when sharing their source between solvers, since each bridge is a separate translation unit.

The returned lines are C source, emitted verbatim into the translation unit between the oracle kernels and the external solve function. The entry point follows Scaly's application binary interface, or ABI.

Required signature¶

With I = len(desc.input_signature) and O = len(desc.output_signature):

static void <ctx.raw_symbol>(const double* in0, ..., const double* in{I-1},
                             double* out0, ..., double* out{O-1},
                             double* w,
                             const scaly_solver_option* options) { ... }

w is the caller's scratch workspace. Pass it after the output pointers of every oracle _raw call. An oracle that needs workspace cannot accept NULL. Do not use it for the wrapper's own storage. Solver workspaces and O(n²) buffers belong in static locals, since the wrapper is non-reentrant by contract (see the generated interface).

Oracle calling convention¶

Each descriptor Function has a bridge entry named by ctx.raw_symbol_of(fn), with signature static void <symbol>(const double* <in0>, ..., double* <out0>, ..., double* w), with inputs and outputs in the Function's declared order. External oracles retain their declared flat-buffer interface ending in double* w.

Statistics¶

Fill every field of ctx.stats_symbol on every call, including early returns after an error. The fields have the following meanings:

Set diagnostics that the backend cannot provide to zero. Time operations with scaly_clock_s(), which Scaly emits into every generated unit that contains a solver. Maintain t_total ≈ t_fe + t_solver + t_qp + t_globalization + t_glue. A direct QP backend reports its solve in t_qp. An NLP backend that cannot separate its internal work reports it in t_solver.

Map native statuses through the solver header's enum constants rather than integer literals. An upstream rename or renumbering then causes a compile error instead of an incorrect status.

Fatrop's C header supplies no return-status enum. Its wrapper maps the integer FatropOcpCStats.return_flag using the pinned Fatrop 1.2.0 IpSolverReturnFlag order in include/fatrop/ip_algorithm/ip_algorithm.hpp, preserving the flag in native_status. A package test checks that enum's names and order against the wrapper's mapping. An exception from the native solve maps to ERROR with -1.

Options¶

prepare_options runs during Python solver construction, before any C generation. It checks names, types and combinations and supplies the backend's defaults. It returns two dicts. Core stores structural choices in desc.compile_options. It packs runtime tuning into a null-terminated scaly_solver_option array and passes it through the _with_options entry on every numerical call. The host entry sets each bridge's option pointer before executing the graph. The wrapper receives that pointer as its final options argument.

Option kind 0 stores an integer, kind 1 a number and kind 2 a string. The C option interface describes the layout. Apply options on every call, including when reusing a native workspace. Runtime option names and values from the user's solver must never enter rendered source. Compile choices select one fixed interface, such as PIQP's dense or sparse matrix layout. Include only parameters with a useful call-time purpose in the runtime dict, such as convergence tolerances, iteration limits and diagnostics.

Keep defaults in prepare_options. Core calls it with an empty dict when rendering the exported default arrays and the standard five-argument entry. This entry uses backend defaults. Configured Python and ahead-of-time calls use the same _with_options path. Configurations with the same compilation choices share one source and cache key.

Descriptor families¶

Core normalizes every Problem into one SolverDescriptor. Plugins consume flat buffers and static metadata. They do not inspect Expr nodes or reconstruct the user's trees.

Every typed solver uses the same flattened leaf order:

inputs  = [variable leaves], [box-multiplier leaves], lam_eq, lam_ineq, [parameter leaves]
outputs = [variable leaves], [box-multiplier leaves], lam_eq, lam_ineq

desc.n_var_blocks is the number of variable leaves and determines all four fixed-group offsets. desc.param_names names the leaves after 2 * n_var_blocks + 2 fixed inputs. The wrapper must scatter its flat native solution and box multipliers back into the declared variable blocks. Equality and inequality arrays are always present, including when their sizes are zero.

Bound convention¶

Core oracles use IEEE negative infinity for an absent lower bound and IEEE positive infinity for an absent upper bound, for variable and inequality bounds in both descriptor families. A plugin must translate those values after evaluating the oracle and before calling a solver that uses a finite sentinel. The PIQP adapter maps them to -PIQP_INF and PIQP_INF, the macro from the vendored header, and the IPOPT adapter maps them to -2e19 and 2e19. Apply the same translation on initial setup and on every update path. Do not make a wrapper depend on the identity of sc.NO_LB or sc.NO_UB. Substitution and code generation preserve their values, not Python object identity.

Quadratic programs¶

kind == "qp", selected by sc.solver(problem, "piqp").

• The problem shape is min 0.5 x' P x + c' x subject to A x = b, l <= G x <= u, and box bounds.
• desc.oracle takes parameter leaves and emits P, c, [A_eq, b_eq], [G_ineq, l_ineq, u_ineq], x_lb, x_ub. Empty constraint blocks are omitted from the oracle but remain size-zero multiplier groups in the solver signature.
• With compile_options["sparse"] selected, the oracle emits compact compressed sparse column values in the baked order of P_sparsity, A_sparsity, and G_sparsity. P_sparsity contains the upper triangle. Otherwise the patterns are absent and the oracle emits dense row-major matrices.
• The oracle emits IEEE infinities for absent bounds, and the wrapper converts them to the QP solver's native convention.
• A QP plugin is a standalone solver only. scaly-sqp does not consume this contract for its subproblems. Its wrapper is written against PIQP's C API and links scaly-piqp's library. See the user guide.

Nonlinear programs¶

kind == "nlp", selected by sc.solver(problem, "ipopt"), "sqp" or "fatrop".

• The problem shape is min f(x,p) subject to h_eq(x,p) = 0, two-sided inequalities, and box bounds.
• Core concatenates the variable leaves into one internal x for the oracles. desc.base maps (x, *params) to f and, when constraints exist, stacked g = [h_eq; g_ineq].
• desc.grad has the same inputs and returns the dense objective gradient. desc.jac returns the compact sparse Jacobian of g in desc.jac_sparsity order.
• desc.hess takes (x, *params, lam:f[, lam:g]) and returns the compact Lagrangian Hessian in desc.hess_sparsity order.
• desc.bounds takes only parameter leaves and returns x_lb, x_ub[, l_ineq, u_ineq].
• Core asks the backend for hess_triangle and hands the wrapper an oracle and pattern already cut to that layout. IPOPT selects lower, and scaly-sqp selects upper.
• Any NLP oracle may instead be an ExternalOracle with the same signature. Its source defines raw_symbol using the flat-buffer convention, and workspace_size contributes to root and nested workspace packing.
• The bounds oracle emits IEEE infinities for absent bounds, and the wrapper converts them to the NLP solver's native convention.

Stage layouts¶

desc.stage_layout returns a StageLayout from scaly.solvers.stages. Its nx and nu tuples describe variables ordered [x_0, u_0, ..., x_N, u_N]. dynamics[k] holds the original equality rows into stage k+1, in next-state order. eq[k] and ineq[k] hold stage-local rows in the original stacked [equalities; inequalities] vector. Variable bounds are separate.

The optional descriptor constructor field _stage_layout supplies a layout. Otherwise the property detects and checks one from the Jacobian and Lagrangian Hessian patterns on first access. Backends without a stage requirement need not read it. This is an additive descriptor field under protocol v8.

The layout guarantees structurally diagonal next-state Jacobian blocks, not their numerical coefficients. The Fatrop wrapper normalizes each dynamics row to F(x, u) - xnext using its coefficient at the initial point, checks that it stays constant during the solve, and converts multipliers with the same scale.

A backend with stage_oracles = True receives desc.stage_oracles when the problem supplies stage Functions, instead of desc.jac, desc.hess and their sparse patterns. Core constructs these dense derivatives for one stage before mapping over the horizon. desc.grad references the same Function as desc.stage_oracles.grad. Fatrop declares this opt-in. Other backends retain whole-problem oracles by default. The problem checks that its layout matches the Functions' uniform state/control sizes, with no controls at the terminal stage. base and bounds retain the ordinary NLP contract. StageOracles.jac takes (x, *params) and returns dynamics, initial, path and terminal Jacobians in local [x, u] order, with a leading stage axis on dynamics and path blocks. StageOracles.hess takes (x, *params, scale, lam) and returns the stage and terminal Lagrangian Hessians, each followed by a gradient row. These rows exclude the incoming transition's linear -xnext term and box multipliers. The wrapper adds box multipliers and permutes local [x, u] order to Fatrop's [u, x] order. Dynamics already have coefficient -1 on xnext. This optional descriptor field is additive under protocol v8. General problems continue to use the sparse-oracle contract above.

Versioning¶

SOLVER_PLUGIN_PROTOCOL_VERSION (src/scaly/solvers/registry.py) covers the whole contract above: descriptor semantics and oracle output orderings, the SolverWrapperCtx fields, the _raw calling convention, and the scaly_solver_stats layout (SCALY_SOLVER_STATS_VERSION tracks the struct ABI itself, and a stats change bumps both). Any breaking change to any of these bumps the protocol version, and get_backend refuses plugins declaring a different version.

History:

• v1: packaging metadata only (hand-written templates in core).
• v2: plugin-owned codegen via render_wrapper.
• v3: external NLP oracles and the stats-v2 QP/globalization timing fields.
• v4: the stats-v3 per-solve diagnostics tail (primal_viol, step_inf, alpha, merit_penalty, backtracks, qp_iter), appended after _pad0. The struct grows from 96 to 136 bytes.
• v5: backend-selected NLP Hessian triangles and the compact oracle output convention that the descriptor pattern is the handed layout.
• v6: typed Problem/Function solver signatures, variable-block metadata, and the fixed warm-start and result order.

• v7: IEEE-infinity semantics for absent bounds in core QP and NLP oracles. Plugins normalize them to native solver sentinels.

• v8: call-time option arrays, prepare_options, the _with_options entry and its context argument on generated internal calls. The preparation hook separates fixed compilation choices from runtime tuning.

• v9: core renders solver bridges with their oracles as external libraries using the module build recipe. Wrapper options use one pointer, and oracle calls end at w.

What core owns¶

Plugins must not duplicate any of this:

• Problem normalization and oracle assembly (sc.problem / sc.solver), including derivative factories and sparsity detection.
• The universal C ABI entry point, workspace packing, and the _raw kernel rendering (Program IR).
• The scaly_solver_stats struct, the SCALY_SOLVE_* status enum, and scaly_clock_s. Plugins fill and use them, never redefine them.
• JIT compilation, caching (keyed on source and flags), and library/header discovery.
• The typed Solver and Function call interfaces and Function.solver_stats().

Checklist for a new plugin¶

1. Create the package. If the solver has its own C API, add a hatch build hook that bundles the library and headers. Use plugins/scaly-piqp as a reference.
2. Define BACKEND with the metadata fields, prepare_options and render_wrapper. Expose it through the scaly.solvers entry point.
3. Implement the wrapper. Call the oracles, normalize infinite bounds, and drive the solver's C API. Use native enum constants for statuses and fill the statistics on every return path. Prefix static state with ctx.symbol.
4. Add tests under plugins/scaly-<name>/tests/. Compare with analytic or reference solutions, compile a nested solver call, and check the statistics. The PIQP and IPOPT suites provide examples.
5. Nothing in src/scaly/ should need to change. If it does, the protocol is missing something. Raise it as a core issue instead of forking core.