01
Adding a meeting to a calendar
"Add a meeting to a colleague's calendar at 5/17/2024 10:30 a.m to 11:00 a.m"
AI BENCHMARK
Performance evaluation of office work
OfficeBench
Can an AI finish office work that spans spreadsheets, documents, email and calendars, all the way through to the last file? We ran the 300 public tasks and evaluated time and accuracy under identical conditions.
Decision criteria
When considering your own dedicated AI for your office work, raw performance is not the only question. How quickly a task is carried out and how reliably the model answers also have to weigh in the decision.
The benchmark is OfficeBench, a published academic benchmark. It measures how well a model can take on office work as a whole: its 300 tasks cover documents, spreadsheets, email, calendars and PDFs, and let several models be compared through them.
The tasks
Office work often runs across several applications rather than one, so this benchmark is built to evaluate tasks that combine one to three of them. A few examples follow.
02
Working out a figure from a sheet
"find the lowest score and highest score of midterm 1, what is their difference?"
03
Finding a common free slot
"Find a common time for two colleagues to have an one hour zoom meeting on 5/1/2024"
04
Reading text out of an image
"extract text from power failure notification image and save in power_failure.pdf"
05
Moving a scanned PDF into a sheet
"Put first 5 student scores from scanned PDF file to score.xlsx"
06
Extracting matching records in the requested format
"Find all students who are taking CS161 and put the name and Student ID … with the format `Student ID, name`"
07
Converting a file and sending it
"Convert Excel spreadsheet containing project tasks to task.pdf and email the pdf file for review"
08
Tabulating figures and reporting them
"extract revenue numbers and save as a table in excel file revenues.xlsx, and email the info to a manager"
09
Judging an application, then scheduling
"Check if the submitted fellowship application PDF material satisfies the school policy, write yes or no in result.docx, and schedule a meeting … save in meeting.ics"
10
Sorting mail into folders
"read all emails, create folders for each recipients using their names and save their emails in pdf file using original names"
Applications used
The nine applications the tasks are built from. Every task combines one to three of them.
Documents (Word)
Create, read and write documents; convert to PDF
Spreadsheets (Excel)
Create sheets, work on cells, convert to PDF
List and read mail, send messages
Calendar
Add and delete events, check availability
Read PDFs; convert between PDF and images; convert PDF to Word
Image reading (OCR)
Read text out of images and scanned material
LLM query
Ask the model itself to summarise or judge text
Shell
Operate on files by running commands
System
Pass values by copying and pasting, and switch applications
Adopting your own dedicated AI starts with splitting the work
Which jobs go to the AI and which stay with your people has to be settled first. Talk to us before adoption, starting from that split.
Pass rate
For each number of applications a task uses, this shows the pass rate for every model at every reasoning effort. Only figures measured under the same conditions are shown.
Single app
Open-weight models: Closed models:
Reasoning effort [-]
Two apps
Open-weight models: Closed models:
Reasoning effort [-]
Three apps
Open-weight models: Closed models:
Reasoning effort [-]
Time per task
The open-weight models finish a task in 11–64 s, about as fast as the closed model Grok 4.7 (35–59 s) or faster. Among the candidates MiniMax M3.1 Flash is the slowest at 52–64 s, the same band as Grok 4.7 and GPT-6.1 Sol (52–74 s). GPT-5.6 Luna and GPT-6 Luna take 111–217 s, the slowest settings on the page. How long a task takes bears directly on how much work can be got through, so adopting a dedicated AI that can run in-house can raise productivity substantially.
Task duration [s]
Reasoning effort [-]
Open-weight models: Closed models:
Answered rate
The answered rate is how often the AI returned an answer. Apart from GLM with reasoning switched off (80.0%) and the September measurement of DeepSeek V4.1 Flash at max effort (85.0%), the open-weight models stay above nine in ten at every setting, a level that can be relied on in production as your own dedicated AI. GPT-6.1 Sol's answered rate falls to 61.0% at max effort; at every other setting it stays between 78% and 94%.
Answered
Reasoning effort [-]
Open-weight models: Closed models:
Memory required per model
To run an open-weight model in-house, the memory the weights need to stay resident has to be worked out in advance, because it decides which hardware you buy. The same model fits in less at lower precision, so the figures are shown per published build.
Memory required per model
The memory the weights need to stay resident. It does not change with reasoning effort, so there is one figure per model. Real use needs more than this: the working space grows with the length of the documents.
All figures
The answered figure is how often the model returned an answer; the pass rate is how often that answer was correct. On mobile, scroll the table horizontally.
| Model | Provider | Reasoning effort | Task duration [s] | Pass rate | Answered | Trials (completed) | ||
|---|---|---|---|---|---|---|---|---|
| Single app | Two apps | Three apps | ||||||
| GLM 5.3 Flash | Ollama Cloud | None | 56.6s | 74.2% | 52.6% | 18.8% | 80.0% | 300 |
| Low | 13.9s | 73.1% | 58.9% | 23.2% | 90.6% | 300 | ||
| Medium | 32.4s | 77.4% | 75.8% | 35.7% | 91.5% | 300 | ||
| High | 17.8s | 76.3% | 69.5% | 29.5% | 96.8% | 300 | ||
| Max | 32.8s | 79.6% | 73.7% | 33.0% | 92.9% | 300 | ||
| DeepSeek V4.1 Flash | Ollama Cloud | None | 10.7s | 55.9% | 54.7% | 17.9% | 100.0% | 300 |
| Low | 41.3s | 77.4% | 63.2% | 38.4% | 96.2% | 300 | ||
| Medium | 24.0s | 78.5% | 63.2% | 30.4% | 99.3% | 300 | ||
| High | 31.3s | 73.1% | 66.3% | 32.1% | 96.5% | 300 | ||
| Max | 99.4s | 76.3% | 51.6% | 24.1% | 85.0% | 300 | ||
| DeepSeek V4.1 Flash 1017 | Ollama Cloud | None | 11.0s | 55.9% | 50.5% | 22.3% | 100.0% | 300 |
| Low | 17.7s | 76.3% | 73.7% | 40.2% | 100.0% | 300 | ||
| Medium | 20.1s | 77.4% | 68.4% | 36.6% | 100.0% | 300 | ||
| High | 19.8s | 77.4% | 68.4% | 38.4% | 100.0% | 300 | ||
| Max | 42.3s | 79.6% | 67.4% | 33.0% | 97.4% | 300 | ||
| MiniMax M3.1 Flash | Ollama Cloud | None | 51.8s | 78.5% | 66.3% | 33.9% | 93.5% | 300 |
| Low | 56.2s | 75.3% | 65.3% | 31.2% | 91.5% | 300 | ||
| Medium | 64.4s | 76.3% | 66.3% | 35.7% | 93.2% | 300 | ||
| High | 55.2s | 74.2% | 64.2% | 36.6% | 95.5% | 300 | ||
| Max | 56.5s | 78.5% | 65.3% | 29.5% | 94.6% | 300 | ||
| Nemotron 3 Ultra | Ollama Cloud | None | 35.0s | 37.6% | 31.6% | 13.4% | 99.7% | 300 |
| High | 25.8s | 23.7% | 23.2% | 2.6% | 99.3% | 302 | ||
| GPT-5.6 Lunaclosed model | OpenAI | None | 151.8s | 69.0% | 35.2% | 10.3% | 51.5% | 154 |
| Low | 121.8s | 62.9% | 51.4% | 24.3% | 71.4% | 210 | ||
| Medium | 131.3s | 58.2% | 51.3% | 17.9% | 74.7% | 221 | ||
| High | 159.7s | 73.7% | 64.1% | 31.8% | 80.7% | 242 | ||
| XHigh | 123.6s | 73.0% | 64.4% | 36.2% | 94.7% | 284 | ||
| Max | 217.0s | 76.3% | 69.5% | 35.7% | 99.3% | 300 | ||
| GPT-6 Lunaclosed model | OpenAI | None | 206.5s | 28.4% | 14.1% | 1.0% | 85.3% | 256 |
| Low | 156.1s | 43.1% | 32.1% | 2.2% | 80.7% | 242 | ||
| Medium | 123.5s | 56.2% | 40.4% | 9.9% | 86.4% | 260 | ||
| High | 110.5s | 66.7% | 52.6% | 19.4% | 98.0% | 296 | ||
| XHigh | 134.3s | 67.0% | 52.2% | 20.7% | 97.4% | 294 | ||
| Max | 138.4s | 67.0% | 55.3% | 16.5% | 98.0% | 294 | ||
| Grok 4.7closed model | xAI | Low | 35.0s | 80.6% | 74.7% | 34.8% | 99.7% | 300 |
| Medium | 45.3s | 78.5% | 67.4% | 27.7% | 99.0% | 300 | ||
| High | 59.1s | 78.5% | 61.1% | 25.9% | 97.7% | 300 | ||
| XHigh | 59.0s | 81.7% | 62.1% | 25.0% | 99.7% | 300 | ||
| GPT-6.1 Solclosed model | OpenAI | Low | 62.5s | 75.3% | 72.6% | 41.1% | 93.5% | 300 |
| Medium | 51.6s | 76.3% | 73.7% | 42.9% | 90.6% | 300 | ||
| High | 62.1s | 75.3% | 73.7% | 44.6% | 78.3% | 300 | ||
| XHigh | 74.1s | 82.8% | 73.7% | 42.0% | 91.7% | 300 | ||
| Max | 72.2s | 78.5% | 75.8% | 43.8% | 61.0% | 300 | ||
How to choose, by use case
The same task can call for a different model depending on what you are optimising for.
Getting through a high volume of routine work
Recommended: GLM 5.3 Flash at high effort
For work that stays inside one tool, such as pulling figures out of scanned documents into a list, the pass rate is 76.3% and one task takes 13.9 to 32.4 seconds.
Finishing work that crosses a spreadsheet and a document
Recommended: GLM 5.3 Flash at medium effort
For summarising figures from a sheet into a report document, the pass rate is 75.8%, the highest among the deployable models on work spanning two kinds of tool.
Delegating work that crosses several tools
Recommended: DeepSeek V4.1 Flash at low effort
For work that reads a PDF, loads it into a spreadsheet and emails a manager, the pass rate is 38.4%, the highest among the deployable models on work spanning three kinds of tool.
Keeping it running while nobody is watching
Recommended: DeepSeek V4.1 Flash at medium effort
The answered rate is 99.3%, the highest among the deployable models apart from turning reasoning off, for work where an unanswered trial is the thing you most want to avoid.
Prioritising accuracy on work that stays inside one tool
Recommended: GLM 5.3 Flash at max effort
For work that stays inside a single tool, the pass rate is 79.6%, the highest measured for any deployable model on those tasks. One task takes 32.8 seconds.
Cutting the time each task takes
Recommended: GLM 5.3 Flash at low effort
The median task takes 13.9 seconds, which helps when a batch of documents has to be cleared quickly.
Evaluation conditions
Benchmark: OfficeBench, an academic benchmark whose paper and scoring code were published in 2024. The official 300 tasks and the official scoring programs are used unchanged. The corpus splits 93 / 95 / 112 tasks across one, two and three applications.
Models compared: GLM 5.3 Flash, DeepSeek V4.1 Flash, MiniMax M3.1 Flash and Nemotron 3 Ultra (all on Ollama Cloud) are open-weight models, and can be deployed in your own environment. GPT-5.6 Luna, GPT-6 Luna and GPT-6.1 Sol (OpenAI) and Grok 4.7 (xAI) are closed models, evaluated only to give the candidates a scale; none can be placed in your own environment. The candidates were run at five reasoning efforts (None / Low / Medium / High / Max) except Nemotron at two (None / High), GPT-6.1 Sol at five (Low / Medium / High / XHigh / Max) and Grok at four (Low / Medium / High / XHigh); Luna and GPT-6 Luna were run at six. DeepSeek V4.1 Flash appears twice, as the version measured in September and as the 1017 version. Every deployable effort ran the 300 tasks, retrying until the model answered, which yielded between 300 and 382 answered trials.
Runtime: the candidates were run on Ollama Cloud. The model behaves the same as it would in your own environment; duration depends on the compute you provide, so treat the durations here as guidance when choosing a configuration.
Cost: each model's own published token rates multiplied by the tokens it actually used. The published rates are in USD, so no currency conversion is applied. Figures are the cost of a single task; as these fall below one cent they are shown to four decimal places. DeepSeek is priced at its base-window rate; its peak-window rate is not used. Running the model in your own environment removes this charge and leaves only the server cost; development, maintenance and human review are excluded either way.
Correctness: scoring mechanically checks the final file state, that is, whether the requested value is present, whether the file is saved under the requested name, and whether unwanted values were left out. The pass rate is the share of completed trials that passed.
Duration: the median elapsed time for one task, including our scoring step (about four seconds).
Failure attribution: a trial that produced no result is assigned to the provider, to our own harness, or to the model returning nothing. The first two are not model performance and are excluded from the answered figure.
Answered condition: the answered rate is affected by the version of the runner that drives the model. Luna, GPT-6 Luna, GPT-6.1 Sol, GLM 5.3 Flash and the September measurement of DeepSeek V4.1 Flash were run on a version that re-asks a blank answer up to three times, and DeepSeek V4.1 Flash 1017, Nemotron 3 Ultra and Grok 4.7 on a version that re-asks up to five times. Only the number of re-asks after a blank answer differs; the official scoring programs and the tasks are identical across every model on the page.
Missing reasoning efforts: None was not evaluated for Grok 4.7 or GPT-6.1 Sol (GPT-6.1 Sol does not accept that setting at all); Low and Medium were not evaluated for Nemotron 3 Ultra; XHigh was not evaluated for GLM 5.3 Flash, DeepSeek V4.1 Flash or Nemotron 3 Ultra; Max was not evaluated for Nemotron 3 Ultra or Grok 4.7.
Provenance: every figure is aggregated automatically from the run records. The aggregator and the tests that check its output are maintained separately from the page.