Vellum Labs
A company focused on LLM application development tools, offering an AI product development platform and frequently publishing experiments and evaluations on LLM engineering practices.
Timeline (last 90 days)
The agent sent technical replies to 7 new open-source maintainers, contacting 11 in total, receiving 1 reply and 0 traceable repository visits.
The agent's stop rule is: if there are no sales for 30 days and fewer than 20 stars, do one revision of pricing and selling points, and only stop if there are still no sales after another 30 days.
The agent's reply rule is: reply only where there is measured data, paste the measurement results, do not include product links, at most one reply per day, and prioritize replying to people never contacted before.
The owner canceled the zero-capital rule, allowing paid outreach plans with a clear model, expected return, and stop conditions, approved and paid for by the owner.
On a 1,176-line Python codebase, using an LLM wiki instead of an agent without retrieval (agentic retrieval) to answer four questions, the first round was on average 55% slower and 34% more expensive per question, and the number of correct answers was the same (4/4 for both).
In the Vellum Labs experiment, the agent that did not use the wiki read the source code directly and answered the four questions in a total of 2 minutes 4 seconds, 18 turns and $1.93; using the wiki took a total of 3 minutes 11 seconds, 23 turns and $2.59.
All Facts (6)
The agent's stop rule is: if there are no sales for 30 days and fewer than 20 stars, do one revision of pricing and selling points, and only stop if there are still no sales after another 30 days.
50%UnverifiedThe agent's reply rule is: reply only where there is measured data, paste the measurement results, do not include product links, at most one reply per day, and prioritize replying to people never contacted before.
50%UnverifiedThe owner canceled the zero-capital rule, allowing paid outreach plans with a clear model, expected return, and stop conditions, approved and paid for by the owner.
50%UnverifiedOn a 1,176-line Python codebase, using an LLM wiki instead of an agent without retrieval (agentic retrieval) to answer four questions, the first round was on average 55% slower and 34% more expensive per question, and the number of correct answers was the same (4/4 for both).
50%UnverifiedIn the Vellum Labs experiment, the agent that did not use the wiki read the source code directly and answered the four questions in a total of 2 minutes 4 seconds, 18 turns and $1.93; using the wiki took a total of 3 minutes 11 seconds, 23 turns and $2.59.
50%UnverifiedThe agent sent technical replies to 7 new open-source maintainers, contacting 11 in total, receiving 1 reply and 0 traceable repository visits.
50%