Open-source ML systems research

Custom-DL-Optimizer

A benchmark-driven safety layer for PyTorch inference.

Give it your model and representative traffic. It races built-in compiler paths and optional runtimes, rejects candidates that are wrong or impractical, and returns the fastest valid callable with the evidence attached.

See it choose a plan Read the guide

pip install custom-dl-optimizer

LIVE DECISION Profiling serving-mix Native fallback armed

Weighted input signatures 8+ eligible plan types Parity before selection Validated decision cache

The complete decision in one loop

It does not optimize by assumption. It measures.

Here a provider posts the lowest raw latency, then fails numerical parity. FX + Inductor becomes the deployable winner, and the decision is cached for guarded reuse.

Input: A model plus weighted serving signatures
Policy: Correctness, startup, memory, and gain thresholds
Output: A callable module plus an auditable JSON report

Animated Custom-DL-Optimizer run showing weighted workload profiling, candidate construction, latency measurement, parity rejection, valid-plan selection, and cache persistence. — 01 Profile 02 Build 03 Measure 04 Validate 05 Select 06 Cache

Different models want different backends.

A fixed optimization path can help a dense CNN and regress a depthwise network. Selection stays model-specific.

Cold start changes the winner.

Compilation and lazy first calls are included when the deployment has a finite request horizon.

The fastest output can still be wrong.

Every candidate must preserve output structure and clear configured numerical tolerances before it can win.

Inside the optimizer

One policy across PyTorch, compiler, runtime, and quantized plans.

Backends keep doing what they do best. Custom-DL-Optimizer owns the comparison boundary: representative traffic, isolation, correctness, lifecycle cost, constraints, and fallback.

01
Profile traffic
Normalize real batches, shapes, dtypes, keyword forms, and weights.
02
Build candidates
Clone the module and construct eligible built-in and provider plans.
03
Measure cost
Record setup, first calls, serial distributions, memory, and bootstrap bounds.
04
Gate validity
Reject parity failures, runtime errors, and policy violations.
05
Select and cache
Replace the baseline only after the confidence gate, then revalidate later.

Useful beyond a benchmark chart

Choose the workflow that matches the job.

DEPLOY / 01

Pick a plan for the traffic you actually serve.

Model batch-size and shape frequency, cap setup and memory, then select by steady-state or projected lifecycle cost. A validated cache avoids repeating the full race on every process start.

Open deployment guide

SELECTED PLAN

fx_inductor

Mean gain: 1.18x
95% gate: passed
Break-even: 214 calls
Baseline: native

Code in. Callable result out.

See exactly what the package gives you.

Use the optimized result like the original module, inspect the evidence that selected it, and persist a portable decision record for review or experiment tracking.

from custom_dl_optimizer import (
    ExecutionTarget, InferenceOptimizer,
    OptimizationPolicy,
)

optimizer = InferenceOptimizer(
    target=ExecutionTarget("cuda"),
    policy=OptimizationPolicy(
        enable_compile=True,
        plan_cache_dir=".custom-dl-cache",
    ),
)

decision = optimizer.select_signature(model.eval(), sample)
output = decision(sample)

candidate = decision.report.selected_candidate
speedup = candidate.speedup_vs_native
bundle = decision.save_bundle("artifacts/run")

print(f"selected      {decision.selected_plan}")
print(f"mean_cost_ms  {candidate.selection_cost_ms:.2f}")
print(f"cost_ci_high  {candidate.selection_cost_ci_high_ms:.2f}")
print(
    "speedup       baseline"
    if speedup is None
    else f"speedup       {speedup:.2f}x vs native"
)
print(f"parity        {candidate.parity}")
print(f"gate_passed   {decision.report.confidence_gate_passed}")
print(f"output_shape  {tuple(output.shape)}")
print(f"saved         {bundle}")

$ python optimize.py

selected      fx_inductor
mean_cost_ms  8.43
cost_ci_high  8.49
speedup       1.18x vs native
parity        True
gate_passed   True
output_shape  (8, 10)
saved         artifacts/run

Callabledecision(sample) runs the selected plan.

Evidencedecision.report records costs, bounds, parity, and failures.

Bundlemanifest.json and report.json preserve the decision.

This output is an illustrative CUDA run, not a universal benchmark. Plan names and measurements change with the model, hardware, PyTorch build, workload profile, and enabled providers.

Bring the strongest alternatives

Compare maintained runtimes under the same policy.

Custom-DL-Optimizer complements backend compilers. It gives their outputs a shared correctness, measurement, constraint, and selection boundary.

Read provider contract

BUILT INPyTorch eagerAMP / layoutFX rewritesFX + Inductor

OPTIONAL PROVIDERSTorch-TensorRTONNX RuntimeTorchAOPrivate backend

One comparable reportPer-case latency, lifecycle bounds, parity, constraints, cache state, and failures

Evidence, including regressions

The system is useful because no backend wins everywhere.

The historical T4 study below motivated adaptive selection. Most eager gains came from established precision, layout, and compiler techniques, while the fixed custom path regressed on MobileNet-V2.

Model	Eager FP32	Inductor	Experimental	vs eager	vs Inductor
ResNet-50	366.997 ms	99.254 ms	89.307 ms	4.11x	1.11x
MobileNet-V2	112.703 ms	54.312 ms	64.372 ms	1.75x	0.84x
VGG-16	639.399 ms	273.326 ms	273.402 ms	2.34x	1.00x
EfficientNet-B0	146.788 ms	70.784 ms	66.409 ms	2.21x	1.07x
DenseNet-121	360.750 ms	194.997 ms	193.169 ms	1.87x	1.01x

From first call to paper artifact

Everything needed to inspect, extend, and reproduce the decision.

Custom-DL-Optimizer v3

Replace the baseline only when the evidence is clear.

Get started View repository

Custom-DL-Optimizer

It does not optimize by assumption. It measures.

Different models want different backends.

Cold start changes the winner.

The fastest output can still be wrong.

One policy across PyTorch, compiler, runtime, and quantized plans.

Profile traffic

Build candidates

Measure cost

Gate validity

Select and cache