Published · Reviewed by Horacio Poblete, CEO and founder
When an AI proposal disappoints, it’s rarely out of bad faith. What weighs is the asymmetry: the seller knows which parts are easy and which are expensive, and the buyer often doesn’t. A well-rehearsed demo looks like a finished product, and a third-party model with a good interface looks like proprietary technology. To balance that conversation, a few concepts and a list of questions you bring to the table are enough.
The concepts you need
Seven ideas cover almost every conversation with an AI vendor:
- Language model: a system trained on enormous volumes of text to predict the next word. It generates plausible text; it doesn’t consult a database of truths.
- Using a model versus training one: in the vast majority of cases, the company uses ready-made models from large vendors, through an app or an integration. Training a model from scratch is rare and expensive.
- Why it’s wrong with confidence: the model produces the most likely answer, not the verified one. A confident tone doesn’t signal accuracy, which is why every deliverable needs review and sources.
- Context: the model answers better when it receives the right documents at the moment of the question. Connecting the company’s documents makes the answer specific, as long as they’re organized, up to date and have the right permissions.
- Agents: systems in which the model carries out steps, uses tools and acts in other systems. They’re worth more and risk more, and the central question is what they do without human approval.
- Custom training: it’s less necessary than proposals suggest. Most needs are met with good context and good instructions, without training anything.
- Cost per use: every query costs something, and the cost grows with volume, with the size of the documents and with agents that run many steps. The cost of the pilot doesn’t predict the cost of operations.
Promise, product and hidden cost
Every proposal mixes three things, and the CEO’s job is to separate them before comparing prices.
- Promise: what the solution will do someday. Phrases like "it learns over time" or "it gets smarter with use" call for an explanation: learns how, with what data, validated by whom.
- Product: what works today and what you can see running with your data, more than once, without the vendor adjusting it beside you.
- Hidden cost: what doesn’t show up in the price and does show up in operations, such as preparing data, integrating systems, paying for usage, reviewing what AI produces and keeping everything working when the underlying model changes.
An example from a CEO’s week: a proposal arrives for an "AI legal assistant" that promises to analyze all the company’s contracts. Looking closely, the product is an assistant that answers questions about the documents loaded into it. The hidden cost is that someone needs to organize those contracts first, and a lawyer still reviews every analysis used in a negotiation. None of that makes the proposal bad. It just makes the comparison honest.
Warning signs, as questions
Warning signs appear when you ask a simple question and the answer gets vague. Treat each one as a question, not an accusation:
- What’s yours and what comes from third parties? Almost every solution uses models from large vendors, and that’s normal. What matters is what the vendor adds on top, and whether "our own model" means a model they trained or a third-party model they configured.
- Whose data did the demo use? A demo with the vendor’s data shows the best case. Ask to see the solution on a case of yours, with your documents.
- Where does that return come from, if you don’t know our processes yet? A return figure presented before any diagnosis is a hypothesis. Ask which assumptions support it and how they’ll be verified.
- What’s the maintenance and exit plan? Who adjusts the solution when it gets worse, how much it costs to maintain and what the company takes with it if it ends the contract.
- How does the solution avoid exposing sensitive information? Sensitive information disclosure is among the risks in the OWASP Top 10 for LLM Applications, and it’s worth asking what has been done about it.
Questions to ask any AI vendor
Take this list to your next meeting and ask for the answers in writing:
- What exactly is the scope of the deliverable, and what’s left out?
- Which parts of the solution are yours and which come from third parties, including the model?
- Can we see the solution working with our data, on a case of ours, before signing?
- Which company data does the solution access, where is it stored and who else has access to it?
- Does the contract guarantee that our data won’t be used to train models?
- How will results be measured, and what’s today’s starting point?
- How much does it cost to operate at real volume, and what makes that cost grow?
- When AI gets it wrong, who notices, who fixes it and how is it recorded?
- What happens when the underlying model changes version or price?
- What can the agents do without human approval?
- What will our team be able to maintain and adapt on its own at the end of the project?
- If we end the contract, what do we take with us: data, instructions, workflows and documentation?
On the data question, remember that the major vendors’ enterprise plans usually state in the contract that data isn’t used to train models. A solution built on top of them should keep that guarantee in writing, and the guide on data security when using AI details what else to check.
A tool, a project or internal capability
Before comparing vendors, decide what the company is buying: a tool, a project delivered by a third party or the capability to do it on its own. Each choice calls for different criteria, and many proposals mix all three without saying so. The question of consulting or training, which comes first usually resolves itself once that decision is explicit.
In Almai’s view, every AI project should end with the company knowing how to maintain, adapt and evolve what was built. Permanent dependence on a vendor is a cost, even when it doesn’t show up in the proposal.
How a CEO builds that repertoire
Judgment about AI is formed through frequent use on work that matters. A CEO who writes, analyzes reports and prepares meetings with AI every week learns in practice how much context it needs and how much review work is left over. The guide on AI in the executive’s routine shows where to start, and Almai’s Exclusive 1:1 Classes do that work with each executive, starting from their own analyses, documents and decisions.
Two exercises help before the next meeting with a vendor:
- Test the case yourself: take a document similar to the one in the use case, without sensitive data, and try a simple version in the AI assistant the company already uses. If an afternoon of testing comes close to the demo, the proposal’s value lies elsewhere, such as integration, scale or support.
- Take the proposal to AI: ask it to list the unstated assumptions and the questions a skeptical CFO would ask. You arrive at the meeting with the agenda ready.
AI doesn’t start with the tool. It starts with leadership.
