How Reliable Is Automated Plane Spotting?
Introduction
📘 About the post
This project was completed for Assignment 2 in CSCI 6510: Trustworthy AI, a course exploring the opportunities, limitations, and risks of modern AI systems.
Learn more: Trustworthy AI – CSCI 6510
An AI agent labels an aircraft with 90% confidence. How much should we trust that number?
In the previous blog post, we saw how agents can be leveraged to extract plane clips from a video. That was back on September 7, 2026, about a month before the writing of this post. Back then, we discovered Claude Opus 5 performed reasonably well in this task, but still made a couple mistakes: A landing DHL aircraft was labeled as taking off, and an AirOne aircraft became OneAir.
Since then, Anthropic has released Claude Opus 5.5, reporting improvements over Opus 5 on all nine benchmarks in its release comparison. That sounds promising, but better benchmark scores do not necessarily translate into a more reliable plane spotting workflow. We still need to check whether the agent gets the labels right—and whether its confidence scores help us decide when to trust them.
To investigate reliability, we split this post into two parts. First, we revisit the original problem with the newer model to see whether it still makes the same mistakes. Second, we ask the agent to assign a confidence score to each label and test whether those scores can be trusted—or, in other words, whether the agent is well calibrated. For example, when a well-calibrated agent reports 90% confidence, it should be correct about 90% of the time.
1.1) Better Model ↔ Better Results?
Picking up where we left off last time, we will first assess how well the agent performs with the new model by asking it to identify the aircraft in the same video. We will reuse the long-form clips generated by the previous agent, since generating clips from the full-length video again could produce a different set. This keeps the inputs consistent and lets us focus on the accuracy of the identifications.
We also ask the agent to report confidence percentages for the aircraft and flight-stage labels. This gives us a way to assess both its accuracy and its calibration. The prompt is provided below.
You are an aircraft-identification assistant for a plane spotter.
Analyze the clips provided in
/dataand identify each distinct aircraft and its flight stage.For each aircraft appearance, report: […]
Interestingly, simply switching to the newer model seemed to resolve the problems from the original post. The DHL aircraft was correctly identified as landing, and AirOne was correctly identified as AirOne. The agent also appeared to find more aircraft and identify them more precisely.
Boeing 767-300 freighter (probable; 767-300ER BDSF/BCF) — Kalitta Air (for DHL; yellow DHL tail)
- Appearance
- #10
- Registration
- Unknown
- Source clip
- clip_06
- In the clip
- 0.0–31.0 s
- Duration
- 31 s
Night. '…LITTA AIR' titles after enhancement → KALITTA AIR. Conical tail cone and compact bogie suggest 767; Kalitta also flies 777Fs for DHL, so 777F not excluded. 28–31 s view may be a cross-faded continuation.
Footage © EMAplanespotting
Boeing 747-400 converted freighter (BCF/BDSF) — Aerotranscargo (brand 'AIR ONE')
- Appearance
- #3
- Registration
- Unknown (ER- prefix area unreadable)
- Source clip
- clip_02
- In the clip
- 0.8–40.0 s
- Duration
- 39.2 s
Red 'atc' tail, 'AIRONE' titles; 4 engines, upper-deck hump. Long upper deck with window row → converted pax-to-freighter -400, not factory -400F. Gear down from start; touchdown partly hidden by fence.
Footage © EMAplanespotting
Opus 5, now identified to be a landing aircraft (left), and an AirOne (right)Correcting these examples was encouraging. However, success on two known errors is only limited evidence of improvement. It does not establish how reliably the model will handle other aircraft or viewing conditions.
And while reviewing the remaining results, I encountered a different problem.
1.2) A Correct Answer Can Still Be Difficult to Check
The agent identified a TUI aircraft. The plane I was looking at was OneAir.
At first, this seemed like an obvious mistake. But when I watched the clip again, I noticed a TUI aircraft in the background, behind the OneAir plane. That was the aircraft the agent meant. It had, in fact, identified the OneAir plane separately in another clip.
I had nearly dismissed a correct answer because I could not tell what it referred to. This ambiguity would make manual evaluation difficult, particularly in scenes containing several aircraft.
To make the identifications easier to verify, I added one instruction:
Throughout the video segment, highlight the aircraft of interest you are referring to.
The highlight made the label easier to verify and reduced ambiguity during manual review. For a meaningful evaluation, the reviewer and the agent must be referring to the same aircraft; otherwise, a misunderstanding can be mistaken for an identification error.
2.1) How Well Calibrated Is the Agent?
So far, the newer, fancier model seems to have improved accuracy on the examples we revisited. But how well does it know when it is right?
To find out, I ran the prompt with the highlighting instruction three times. I then manually checked the airline, aircraft type, and flight-stage labels from each run. When the agent returned Unknown but I could determine the correct label, I counted that prediction as incorrect. The results are shown below.
| Stated confidence | Judged on | Rangeof what it said | Calls1 / 2 / 3 | Correctof those judged | Actually right | Verdict |
|---|---|---|---|---|---|---|
| High80–100 % | Airline | 80–97 %mean 88 % | 4313 / 16 / 14 | 43 / 43 | 100 % | under-confident by 12 pts |
| Aircraft type | 80–97 %mean 88 % | 4313 / 16 / 14 | 43 / 43 | 100 % | under-confident by 12 pts | |
| Flight stage | 80–97 %mean 92 % | 3512 / 11 / 12 | 35 / 35 | 100 % | under-confident by 8 pts | |
| Medium50–79 % | Airline | 55–75 %mean 65 % | 177 / 4 / 6 | 17 / 17 | 100 % | under-confident by 35 pts |
| Aircraft type | 55–75 %mean 65 % | 177 / 4 / 6 | 17 / 17 | 100 % | under-confident by 35 pts | |
| Flight stage | 50–75 %mean 63 % | 208 / 4 / 8 | 18 / 20 | 90 % | under-confident by 27 pts | |
| Lowunder 50 % | Airline | 5–45 %mean 21 % | 3010 / 2 / 18 | 14 / 30 | 47 % | under-confident by 26 pts |
| Aircraft type | 5–45 %mean 21 % | 3010 / 2 / 18 | 5 / 30 | 17 % | well calibrated | |
| Flight stage | 5–45 %mean 24 % | 3510 / 7 / 18 | 6 / 35 | 17 % | over-confident by 7 pts | |
| All calls | Airline | 5–97 %mean 61 % | 9030 / 22 / 38 | 74 / 90 | 82 % | under-confident by 21 pts |
| Aircraft type | 5–97 %mean 61 % | 9030 / 22 / 38 | 65 / 90 | 72 % | under-confident by 11 pts | |
| Flight stage | 5–97 %mean 59 % | 9030 / 22 / 38 | 59 / 90 | 66 % | under-confident by 6 pts |
| Stated confidence | Judged on | Rangeof what it said | Callsclip 1 | Correctof those judged | Actually right | Verdict |
|---|---|---|---|---|---|---|
| High80–100 % | Airline | 80–95 %mean 88 % | 13 | 13 / 13 | 100 % | under-confident by 12 pts |
| Aircraft type | 80–95 %mean 88 % | 13 | 13 / 13 | 100 % | under-confident by 12 pts | |
| Flight stage | 80–97 %mean 92 % | 12 | 12 / 12 | 100 % | under-confident by 8 pts | |
| Medium50–79 % | Airline | 60–75 %mean 68 % | 7 | 7 / 7 | 100 % | under-confident by 32 pts |
| Aircraft type | 60–75 %mean 68 % | 7 | 7 / 7 | 100 % | under-confident by 32 pts | |
| Flight stage | 55–75 %mean 63 % | 8 | 7 / 8 | 88 % | under-confident by 24 pts | |
| Lowunder 50 % | Airline | 15–45 %mean 24 % | 10 | 5 / 10 | 50 % | under-confident by 26 pts |
| Aircraft type | 15–45 %mean 24 % | 10 | 2 / 10 | 20 % | well calibrated | |
| Flight stage | 15–45 %mean 22 % | 10 | 1 / 10 | 10 % | over-confident by 12 pts | |
| All calls | Airline | 15–95 %mean 62 % | 30 | 25 / 30 | 83 % | under-confident by 22 pts |
| Aircraft type | 15–95 %mean 62 % | 30 | 22 / 30 | 73 % | under-confident by 12 pts | |
| Flight stage | 15–97 %mean 61 % | 30 | 20 / 30 | 67 % | under-confident by 6 pts |
| Stated confidence | Judged on | Rangeof what it said | Callsclip 2 | Correctof those judged | Actually right | Verdict |
|---|---|---|---|---|---|---|
| High80–100 % | Airline | 80–97 %mean 88 % | 16 | 16 / 16 | 100 % | under-confident by 12 pts |
| Aircraft type | 80–97 %mean 88 % | 16 | 16 / 16 | 100 % | under-confident by 12 pts | |
| Flight stage | 80–97 %mean 93 % | 11 | 11 / 11 | 100 % | under-confident by 7 pts | |
| Medium50–79 % | Airline | 55–75 %mean 64 % | 4 | 4 / 4 | 100 % | under-confident by 36 pts |
| Aircraft type | 55–75 %mean 64 % | 4 | 4 / 4 | 100 % | under-confident by 36 pts | |
| Flight stage | 55–75 %mean 66 % | 4 | 4 / 4 | 100 % | under-confident by 34 pts | |
| Lowunder 50 % | Airline | 10–15 %mean 13 % | 2 | 0 / 2 | 0 % | over-confident by 13 pts |
| Aircraft type | 10–15 %mean 13 % | 2 | 0 / 2 | 0 % | over-confident by 13 pts | |
| Flight stage | 5–45 %mean 30 % | 7 | 3 / 7 | 43 % | under-confident by 13 pts | |
| All calls | Airline | 10–97 %mean 77 % | 22 | 20 / 22 | 91 % | under-confident by 14 pts |
| Aircraft type | 10–97 %mean 77 % | 22 | 20 / 22 | 91 % | under-confident by 14 pts | |
| Flight stage | 5–97 %mean 68 % | 22 | 18 / 22 | 82 % | under-confident by 14 pts |
| Stated confidence | Judged on | Rangeof what it said | Callsclip 3 | Correctof those judged | Actually right | Verdict |
|---|---|---|---|---|---|---|
| High80–100 % | Airline | 80–95 %mean 87 % | 14 | 14 / 14 | 100 % | under-confident by 13 pts |
| Aircraft type | 80–95 %mean 87 % | 14 | 14 / 14 | 100 % | under-confident by 13 pts | |
| Flight stage | 85–95 %mean 92 % | 12 | 12 / 12 | 100 % | under-confident by 8 pts | |
| Medium50–79 % | Airline | 55–75 %mean 63 % | 6 | 6 / 6 | 100 % | under-confident by 37 pts |
| Aircraft type | 55–75 %mean 63 % | 6 | 6 / 6 | 100 % | under-confident by 37 pts | |
| Flight stage | 50–70 %mean 61 % | 8 | 7 / 8 | 88 % | under-confident by 26 pts | |
| Lowunder 50 % | Airline | 5–45 %mean 20 % | 18 | 9 / 18 | 50 % | under-confident by 30 pts |
| Aircraft type | 5–45 %mean 20 % | 18 | 3 / 18 | 17 % | well calibrated | |
| Flight stage | 10–45 %mean 23 % | 18 | 2 / 18 | 11 % | over-confident by 12 pts | |
| All calls | Airline | 5–95 %mean 51 % | 38 | 29 / 38 | 76 % | under-confident by 25 pts |
| Aircraft type | 5–95 %mean 51 % | 38 | 23 / 38 | 61 % | under-confident by 9 pts | |
| Flight stage | 10–95 %mean 53 % | 38 | 21 / 38 | 55 % | well calibrated |
Overall, the agent was underconfident: across the 90 predictions in each category, its mean stated confidence of about 60% sat well below its actual accuracy of 66–82%. Breaking this down by confidence band:
- High (80–100%): All 121 predictions were correct, compared with mean stated confidence of 88–92% across the three categories.
- Medium (50–79%): 52 of 54 predictions were correct—100% for airline and aircraft type, and 90% for flight stage. This was well above the mean stated confidence of roughly 65%.
- Low (below 50%): Correct predictions decreased significantly. Aircraft type and flight stage were correct only 17% of the time, in line with or slightly below the stated confidence. Airline predictions were correct 47% of the time.
The confidence scores were therefore useful for separating stronger predictions from weaker ones, even though they did not consistently match the actual success rates. Based on this experiment, I would trust high-confidence results, inspect medium-confidence results, and reject low-confidence labels until I could check them manually.
Conclusion
We set out to see whether better benchmark scores would translate into more reliable aircraft identifications. In this experiment, Opus 5.5 corrected the mistakes from the previous post. Bounding boxes made it easier to see which aircraft each label described, and confidence scores helped flag predictions that needed closer review. Although the agent was underconfident overall, every high-confidence prediction was correct in our sample, while low-confidence predictions were much less reliable.
Arguably, this small experiment provides limited evidence of the agent’s reliability, and we still need to test whether these findings hold in other settings. Nevertheless, for me, the key takeaway is that clear instructions and careful evaluation matter alongside choosing a better model. Before relying on an agent’s answers, we need to resolve ambiguity about what it means and check whether its confidence reflects how often it is right.