Since models have gotten so much better at math, proofs and coding, I started thinking about other productive uses of the ever cheaper and more intelligent tokens, other than proving theorems and solving open math problems.

Towards harnessing models’ capabilities for accelerating real-world problems, I think it’ll be incredibly useful to build a comprehensive dataset of optimization problem instances, and agents have gotten smart and cheap enough to make this happen. I started with semidefinite programs (SDPs) as they constitute a relatively small class that I’m quite familiar with.

Dataset design

The problem instances should be stored in a common format that can encompass multiple types of optimization problems and is easy to convert to input into multiple optimization solvers. I chose JuMP’s MathOptFormat, which can describe a wide variety of optimization problems, has comprehensive solver support, and is language agnostic. The metadata for each problem is stored in a separate .yaml file, see for instance the example below.

schema_version: 1
id: sdp/parrilo_cop5_k2_membership_sdp
title: Parrilo COP5 level-2 membership Gram SDP
category: sdp
description: "Fixed feasibility problem for Theorem 3.1 of Shen and Zhong. The fixed rational separator is M*=DHD+J/300; variables are its degree-4 SOS Gram entries for (sum_i x_i^2)^2 (x^circ2)^T M* (x^circ2)."
tags: [continuous, imported, sdp, copositive, polynomial, semidefinite]
provenance:
  kind: imported
  source_id: parrilo-copositive-separators
  citation: Shen and Zhong, Explicit Separators for Consecutive Levels of Parrilo's Sum-of-Squares Hierarchy over the Copositive Cone, arXiv:2608.27743, Theorem 3.1 and certificate archive k5_certificate_bundle.json.
  source_url: https://arxiv.org/abs/2608.27743
  license: CC-BY-4.0
expected:
  status: optimal
solution:
  solver: Mosek 11.2.4
  status: optimal
  solve_time: 0.042208881000988185
  date: '2026-09-23'

Data pipeline

To collect this dataset, I set up a roughly 3-stage pipeline, making heavy use of parallel GPT Sol agents and GPT Luna subagents.

1. Source discovery: This stage consists of launching dozens of parallel agents to comb the entire internet for SDP instances. It searched github code, papers, books, lecture notes, existing datasets and publicly available collections. These sources are deduplicated, cataloged, and ranked by quality. The discovery stage stopped when most sources discovered each round turned out to be duplicates.

2. Instance extraction/conversion: This is the most expensive stage regarding time and computational resources, where sources are downloaded, extracted and converted to the common MathOptFormat. In the end, 500+ individual converter scripts totaling 150k+ lines of code are written. These scripts range from simple ones converting from one format to another, to implementing solutions of textbook exercises, and more complex ones that reproduce SDP formulations taken from research papers.

3. Quality control: To ensure that every instance is high quality and the dataset is as free of errors as possible, I ran multiple rounds of cross-checks and quality control. These included deduplication, metadata checks, various sanity checks and finally solving each problem with an SDP solver and comparing the solution to a documented optimum where available. After this stage, I ended up with more than 5k individual instances in the final dataset.

Here are some statistics of the instances in the dataset, detailed comparisons can be found in this dashboard.

Solver comparisons

Next I benchmarked a wide variety of solvers on these instances. To ensure the run finishes in a reasonable amount of time, I used a fairly strict 20 second timeout. Below I show some key visualizations, the full suite is available here.

Accuracy

First I compared the accuracy of the solvers, how many instances they can solve and whether their solution value agrees with the consensus.

Speed

Next, I compared the performance of the solvers, by their measured solve time on each instance. Note that since there is a timeout of 20 seconds, this comparison might be biased towards solvers that perform well on small to medium scale problems, as large scale problems that take more than 20 seconds to solve are not reflected here.

What’s next?

The goal of this dataset is to collect all the SDP problems that have been formulated and are publicly available. Having a comprehensive dataset of SDP instances lets us benchmark existing solvers against real-world problems, as well as providing another test suite for solver interfaces (for example, it has already found a couple of bugs in JuMP). It also paves the way for teaching LLMs to write custom solvers that finds and exploits instance-specific structure, something that I will be looking into in the future.