AI Coding Assistants - Is the investment worth it?

Written By Charles Wright

Its an exciting time right now, not just for software developers but for everyone that writes any type of code. For CTOs in particular there is a major drive towards the use of AI, not just for software development, but across organisations as a whole. For companies that produce their own applications, data platforms and other technology products AI supported coding represents a massive opportunity for both productivity gains and cost reduction.

How good are AI coding assistants?

Two years ago we saw an explosion in generative AI capabilities with ChatGPT and its more recent competitors. Now the market is highly competitive with Google, Anthropic and multiple other companies constantly one-upping each others models. Benchmark scores across across coding, mathematics, language and more are being constantly improved.

Early on, chat interfaces provided a way to quickly generate code, troubleshoot errors and many other core programming tasks. However, early models were highly flawed; the code often contained many errors and it was hard to integrate due to the limited context you could provide the model.

By comparison we’re now living in a utopia, 1000+ line code bases can be generated in minutes from a single prompt, and the code runs the first time its executed. This has been enabled by better models, models that can “reason” and that can ingest and output more information. However, a major or turning point has been the use of tools. Previously, LLMs could only take in and spit out text, the introduction of agentic AI introduced the ability to provide tools to LLMs. These tools can largely be described as the ability to execute specific pieces of code, such as calling an API. Now LLMs can navigate a file system, research online, execute code and perform many other tasks. This has allowed for a leap forward in the practicality and automation AI coding tools can provide. On top of this the introduction of Model Context Protocol (MCP) has introduced online catalogues of these tools simplifying integration and reducing the need to develop your own.

All these benefits mean that not only can non-coders now build applications, but your existing engineers can code up to 55% faster and will be up to 20% more productive across their entire role.

Therefore, for developers its not just exciting, its also frightening, much of these tools comprise a significant portion of their job responsibilities. What happens when the AI tools can truly do everything better than you? Software developers are expensive to hire, command high compensation packages and take a long time to train.

Well we’re not far off, AI coding tools are already much faster than humans and they’re cheaper to run. We’re already in a position where its more practical to buy a subscription to an AI coding tool than to hire a graduate software engineer.

What are the drawbacks?

AI coding tools currently fall short when looking at the big picture. They struggle to:

  • Design - The AI tools are likely to default to generic tech stacks or ones their system prompts (the rules behind the scenes) tell them to prefer.

  • Plan - It typically requires a direct prompt or workflow in order to plan out tasks to complete.

  • Work across a code base - When developing new features, AI tools often only edit a limited number of files, without performing a checks to identify dependencies and references.

  • Code duplication - When attempting to correct itself or add a new feature legacy code is often left in place causing duplications, broken UI and other errors.

  • Documentation - Code can often be hard to follow and the AI often needs to be explicitly to add comments and explanations.

  • Overwriting - Generated code will overwrite old code that is still in use, changing entire web pages and other large scale changes.

  • Assumptions - Often extra code will be added that was not specifically requested. This can be great for UI enhancement adding navigation, delete, edit and other functionality automatically, but if you have a specific vision for those features you might have to rewrite what’s produced.

  • Ignore common security practices - You’ll have to directly request security features or manually implement them yourself. As a result the current tools excel at building internal applications, rather than applications that are customer facing or online.

  • Error cycles - If the AI can’t fix an error first time it will often cycle one or two different solutions over and over, regardless of the error messages fed to it.

  • Test code - You’ll be the one verifying that code actually works in most cases, especially when it comes to UI design.

  • Model changes - Both a benefit and a curse, as models are constantly updated, one week you’ll be able to predict an outcome of a prompt, the next it will behave entirely differently.

How should you use these tools?

You cant rely blindly on AI, you still need senior developers to guide and operate AI coding tools. There’s a skill to using these tools in their current state, here are some tips:

Write specifications

The temptation with strong tools is to write a short prompt and let the AI get on with it. However, this makes the previously described challenges more prevalent and leads to the tools defaulting to their preferred technology stack and approach. Instead write a detailed specification and attach it as part of the prompt. System design is especially important to ensure the application you build is not only functional in the first place but also can cope with future feature development.

Intervene as needed

Vibe coding is a recent approach which solely leverages AI to do the coding, when errors are created the code is overwritten or created with another prompt. While this allows for rapid, hands off development, it causes compounding errors, technical debt and cycles of ineffective error resolution. There is still a need to make manual code adjustments, correct references across multiple files and suggest alternate techniques for the AI to use.

Test, test, test

A good majority of errors are not recognised by the AI and the output will suggest the code works correctly. Testing not just the code that has been written but all the points it interacts with the rest of the code base is essential. I’d also suggest requesting the AI add debugging into the code from the start, so when errors do occur the output from the terminal or browser can be passed to the AI for easier resolution.

My recommendations

Invest now, compared to the massive productivity benefits the costs are tiny. The models powering the tools and the tools themselves get more powerful every week and they require experience to utilise effectively.

  • Start small and test these tools now.

  • Pick a single tool to start, tools like Cursor provide their own IDE and fast editing, tools like Cline allow for a lot of flexibility, GitHub Copilot has a cost effective pricing model.

  • Don’t make any major changes to process and personnel yet. Keep hiring junior developers and graduates, as without them there will be no seniors! Get at least 6 months under your belt, learn and use the tools, set up governance and ethics frameworks.

  • Set quantitively measurable KPIs and outcomes. Being able to make a business case supported by quantitative improvements is the best way to secure future investment.

If you’re interested in seeing how AI coding assistants can support your teams, or where you should apply AI across your organisation, reach out and book a meeting with Obsidian Shift.

References

  • https://arxiv.org/abs/2302.06590

  • https://www.reuters.com/technology/artificial-intelligence/jpmorgan-engineers-efficiency-jumps-much-20-using-coding-assistant-2025-03-13/

Charles Wright