TL;DR
The researchers tested 6 popular LLMs (DeepSeek-R1, GPT-4.1, Claude-3.7, DeepSeek-V3, Qwen2.5-Coder, Llama-3.3) on 944 LeetCode problems across 5 programming languages. They measured Pass@1 (solved on the first try), syntax errors, runtime errors, logic errors and suboptimal algorithms (exceeding the time limit). The result: DeepSeek-R1 and GPT-4.1 are consistently the best on every metric, Llama-3.3 is the worst. Python and JavaScript produce the fewest errors for every model, while Go has the most syntax problems.
The main finding: models often generate logically correct but slow code. Llama-3.3 produces algorithms that exceed the time limit on 35-50% of problems across languages. Even top models suffer: GPT-4.1 and Claude-3.7 hit Time Limit Exceeded in 15-20% of cases. The reason is that an LLM thinks in patterns, not in complexity math. The model generates a "working solution" (a nested loop, naive brute force) without weighing O(n²) against O(n log n).
The fix is simple: add "Optimize the algorithm's time complexity" to the prompt. That single phrase cuts Time Limit Exceeded by 25-35% for every model. DeepSeek-R1 responds especially well — its optimality errors drop from 15% to 5% in Python. It works as an explicit reminder: the model stops producing the first working solution and starts analyzing options and picking an efficient algorithm. The study also showed that a structured prompt (role + task + examples + constraints + test cases + instructions) gives more stable results than a bare "write code for...".
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Approach Blueprint
STEP 1: Set the role and language
"You are a developer. Solve it in Python."
STEP 2: Describe the task
A clear statement + input/output examples
STEP 3: State the constraints
Value ranges, edge conditions
STEP 4: Provide a code stub
A class/function with its signature
STEP 5: Show test cases
Sample data for checking
STEP 6: Add instructions
"Code only, no comments" +
"Optimize the algorithm's time complexity"
Everything runs in one prompt, one request.
Example in Practice
Task: You need a script for sales analytics. There's a CSV with 100,000 rows (product, revenue). You need the top 10 products by revenue. The first attempt with ChatGPT produced code that takes 2 minutes to run — the client won't wait.
Prompt:
You are a Python developer. Solve the task.
Description:
Given a list of products with revenue, find the top 10 by revenue.
Example:
Input: [("Laptop", 1500), ("Mouse", 25), ("Keyboard", 80)]
Output: [("Laptop", 1500), ("Keyboard", 80), ("Mouse", 25)]
Constraints:
- up to 100,000 products
- revenue from $0 to $10,000,000
Stub:
def top_sales(products: list) -> list:
# your code
Test cases:
products = [("Product1", 1000), ("Product2", 5000), ("Product3", 3000)]
Expected: [("Product2", 5000), ("Product3", 3000), ("Product1", 1000)]
Instructions:
Executable code only, no explanations.
Optimize the algorithm's time complexity — there may be a lot of data.
Result:
The model will return code with an efficient sorting algorithm or a heap (O(n log k) instead of O(n²)). Without the word "optimize", it may hand back a naive nested loop. With that phrase, DeepSeek-R1 and GPT-4.1 generate a fast solution in 85-90% of cases instead of 60-70%.
Why It Works
The LLM's weakness in coding: The model generates from patterns in its training data without computing mathematical complexity. It "sees" a similar problem → produces a similar solution. If the data is full of simple examples with loops, it will give you a loop, even if that's O(n²) on a large array. An LLM doesn't ask itself "will this finish in a second on 100,000 items" — it has no concept of runtime, only text patterns.
The LLM's strength: A model is excellent at following explicit instructions and can compare solution options if asked. When the prompt says "optimize complexity", that's a trigger for a different generation pattern. The model starts not just writing code but reasoning: "this problem has a naive O(n²) approach and an optimal O(n log n) one via sorting, I'll pick the second".
How it works mechanically: The phrase "optimize the algorithm's complexity" acts as a context anchor. Without it, the model samples from the space of "all working solutions" → often lands on inefficient ones. With it, it samples from the subspace of "efficient solutions" → the odds of optimal code go up. It works especially well for reasoning models (DeepSeek-R1), which are trained to take explicit reasoning steps before writing code.
Additional levers:
Programming language: Python and JavaScript produce 40-50% fewer syntax errors than Go or C++. If the task doesn't require a specific language, pick Python for maximum reliability.
Model choice: DeepSeek-R1 and GPT-4.1 solve 20-30% more problems on the first try than Llama-3.3. If you work with code all the time, these models are worth the money/resources.
Prompt structure: Adding a role ("you are a developer"), examples (input/output) and constraints (data sizes) reduces logic errors. LLMs work better when the context is structured rather than just "write a function for...".
Generation parameters: If you have API access, top-p=0.95, temperature=0.1 give more stable results for code than the defaults. A low temperature removes "creativity", which in code means unpredictability.
Prompt Template
You are a {role} developer. Solve the task in {language}.
Task description:
{detailed description of what needs to be done}
Example:
Input: {sample input}
Output: {sample expected result}
Explanation: {how the result was obtained}
Constraints:
- {value range}
- {edge conditions}
- {specific requirements}
Code stub:
{class/function structure with signature}
Test cases:
{example 1}
{example 2}
{example 3}
Instructions:
Executable code only, no explanations or comments.
Optimize the algorithm's time complexity.
{additional code requirements}
How to fill it in:
{role}— Python / JavaScript / Java / C++ / Go{language}— the same language (Python is preferable for reliability){description}— what the code should do, without implementation details{examples}— 1-2 simple cases that show the logic{constraints}— data sizes, number ranges, special conditions{stub}— an empty function/class with the correct signature{test cases}— 2-3 examples for checking, including edge cases{additional}— code style, error handling, output format
Key phrase: "Optimize the algorithm's time complexity" — a must if you're working with large data (>1000 items) or heavy computations.
Limitations
⚠️ Not a silver bullet: A structured prompt + "optimize" improve the result but don't guarantee 100% correctness. Even top models (DeepSeek-R1, GPT-4.1) solve only 70-80% of problems on the first try. The remaining 20-30% need iterations, checking or human fixes.
⚠️ Depends on difficulty: On simple problems (sorting a list, finding an element) the word "optimize" barely changes anything — the model will produce decent code anyway. The effect shows up on medium and hard problems (dynamic programming, graphs, combinatorics), where there are several solutions with different complexity.
⚠️ Go and C++ are trickier: If the task requires Go or C++, expect 2-3 times more syntax errors than in Python. Models know strict languages less well. You may need several attempts or manual syntax fixes.
⚠️ Llama-3.3 is weaker at code: Despite strong general abilities, Llama-3.3 produces suboptimal algorithms on 35-50% of problems even with the word "optimize". If coding is your main use case, go with DeepSeek-R1, GPT-4.1 or Claude-3.7.
⚠️ Testing required: Any generated code needs to be tested on edge cases (empty array, maximum values, duplicates). LLMs handle typical cases well but miss edge cases in 15-20% of solutions.
Resources
Holistic Evaluation of State-of-the-Art LLMs for Code Generation — Le Zhang, Suresh Kothari (Department of Computer Science, Iowa State University, 2025)
The study is based on the LeetCode platform and evaluates these models: DeepSeek-R1, DeepSeek-V3, Qwen2.5-Coder-32B, Llama-3.3-70B, GPT-4.1, Claude-3.7 Sonnet.
Data repository: https://figshare.com/s/26448e92798aab34e407