At AI Summit NYC, Definity CEO Freddy Castro joined the panel “The Custom Code Conundrum: Fine-Tuning vs. Starting from Scratch” to unpack a challenge a lot of teams are facing right now:

When does AI customization create real business value, and when does it become an expensive experiment?

Throughout the conversation, Freddy brought the panel back to a practical perspective: AI decisions shouldn’t start with models. They should start with outcomes, architecture, and the reality that what you build today will need to change quickly.

Below are the key ideas he shared, and how teams can apply them when deciding whether to fine-tune, build custom, or keep it simple.

1) The first “red flag”: using AI when you don’t need AI

One of the most common ways organizations overspend, Freddy noted, is surprisingly simple:

Sometimes the problem doesn’t call for AI at all.

In the current wave of excitement, teams often assume AI is the default solution, even when traditional software automation or straightforward engineering would solve the issue faster, cheaper, and more reliably.

Practical takeaway, before debating fine-tuning vs. retrieval vs. custom builds, ask:

• Is the current pain caused by a lack of intelligence, or just a lack of workflow automation?

• Would a rules-based workflow or integration fix this without adding model risk?

AI can be transformative, but it’s not automatically the best tool for every job.

2) “Training exercises” aren’t bad… but they shouldn’t be funded like full initiatives

Freddy also highlighted a pattern he sees frequently: teams spend significant money on customization with little to show for it because the effort was never tied to a clear outcome.

Experimentation is useful. The problem is treating it like a major transformation initiative without the foundation to justify it.

When there’s no measurable objective, even a technically “successful” build can produce only a slightly improved version of an existing process, and the ROI never materializes.

Practical takeaway:

• If your initiative is exploration, treat it like exploration: small scope, fast cycle, controlled investment.

• If your initiative is value delivery, treat it like value delivery: measurable KPIs, clear ownership, and architectural discipline.

3) Start with a business KPI, not a model decision

Across the panel, Freddy repeatedly came back to one principle:

If you want AI to deliver value, you need a business metric first.

That could be revenue growth, cost savings, cycle-time reduction, productivity, quality, or customer satisfaction… it must be something that can be measured and used to drive decisions.

Without that KPI anchor, teams can end up building for novelty rather than impact, and customization becomes a high-cost initiative disconnected from the business.

Practical takeaway, before choosing the “AI approach,” define:

• What KPI are we moving?

• What baseline are we starting from?

• What does success look like in 30, 60, and 90 days?

The model is a means, the KPI is the destination.

4) Build modular, because change is guaranteed

Freddy’s architectural point was direct: AI systems should be built in small, interchangeable pieces, not monolithic commitments.

Why? Because the pace of AI improvement is relentless. What looks “best” today may not be best in a few months, or even a few weeks. That means teams must design for iteration, replacement, and integration from the beginning.

Freddy emphasized modularity across everything, prompts, data, retrieval layers, model choices, and even fine-tuning when it’s appropriate.

Practical takeaway, instead of betting on one big solution, structure AI work like building blocks:

• a retriever module

• a prompt module

• an evaluation module

• a UI/workflow module

• a model module

Each should be replaceable without collapsing the whole system.

5) Plan for “disposability”, build expecting you’ll replace it

A word Freddy used that stood out: disposability.

In traditional software, teams aim to build systems that last. In AI, your advantage often comes from being able to evolve fast, which means accepting that parts of what you build today will need to be replaced.

That doesn’t mean you build carelessly. It means you build intentionally with the expectation of change:

• new model capabilities

• new constraints

• improved retrievers

• better evaluation methods

• shifting governance requirements

Practical takeaway:
Design with an upgrade path. The goal isn’t to “finish” AI, it’s to create a system that can adapt without expensive rewrites.

6) “Bet the house” on value, not on spending

One of the strongest points Freddy made was about organizational commitment.

A major warning sign is when teams are emotionally committed (“let’s do it!”) but not structurally committed, meaning leadership isn’t truly backing the initiative as a business bet.

When an AI initiative is treated as strategically important, it creates clarity:

• The problem gets properly defined

• The pain gets measured

• The costs become visible

• The decision criteria get sharper

And importantly, Freddy clarified: committing to value doesn’t mean spending wildly. In fact, real commitment can prevent overspending by forcing teams to choose the right solution rather than the most expensive one.

Practical takeaway:
Treat AI as a business initiative first. The tighter the value discipline, the more likely you’ll avoid unnecessary customization.

If you’d like to revisit the discussion, you can watch the full recording of “The Custom Code Conundrum: Fine-Tuning vs. Starting from Scratch” featuring Freddy Castro and the full speaker lineup.