The good, the bad and the ugly: What Australia’s CIOs are learning about real-world AI

The atmosphere among Australia’s top technology leaders has notably shifted in recent months. The era of pure AI experimentation and wide-eyed fascination has ended, replaced by the harsh reality of execution capability gaps and demonstrating tangible return on investment.

At Factor’s CIO Day in Melbourne, we swapped high-level theory for genuine conversations about on-the-ground experiences with AI, sharing our own insights while listening to some of the country’s most influential technology executives.

At our closed-door roundtable, “The good, the bad and the ugly: What real-world AI initiatives are actually teaching us”, we explored what is actually delivering results on the front lines, the hard-fought wins and the lessons learned from mistakes.

The good: Unexciting initiatives delivering value

The strongest ROI stories in the room weren’t the flashy, consumer-facing AI initiatives. Instead, they were often unglamorous internal use cases focused on eliminating tech debt or improving operational efficiency.

• Streamlining admin bottlenecks: From automating after-hours client intake processes straight into booking systems to deploying agents that triage and redirect contact centre emails, AI is successfully freeing up staff for higher-value work.
• Reducing tech debt: Teams are experimenting with using AI to identify and respond to production errors within minutes. This allows lean engineering teams to focus on continuous deployment rather than constant firefighting.
• Building ROI into the process: One leader shared how they saved 200 days of manual labour by using AI to compare policy documents, costing less than A$20 in tokens. They also ensured the AI solution demonstrates its own ROI by logging its actions and volume metrics, ensuring that when the board asks for the impact, the data already exists.
• Solving the impossible: The real wins are novel problems that were never feasible for humans to solve previously, rather than existing processes with AI sprinkled on. For example, while you cannot expect a contact centre operator to speak a dozen languages fluently, an AI agent can. Similarly, machine learning can be employed to spot patterns and identify data quality issues where there is no trusted source to compare it to.

The bad: How AI exposes cracks in the foundation

AI can be an incredible accelerator. But speed cuts both ways. If you plug AI into messy data or automate a broken process, you don’t solve the problem, you just accelerate mistakes. Building the wrong thing ten times faster is still building the wrong thing.

AI isn’t necessarily creating new problems, but it is acting as a magnifying glass on existing flaws.

• Risk and governance friction: Multiple CIOs flagged that delivery teams want to move far faster than their risk and compliance functions can assess. Without embedding foundational AI literacy across risk teams,  AI threats can be exaggerated and stall momentum. Governance must be approached collaboratively from the outset, not treated as a final check at deployment.
• Access to quality data: AI outputs are only as good as the data informing them. Leaders shared examples where gaps in their data, human input errors or siloed data stores caused fabrications, forcing pilots to be shut down.
• Cybersecurity and sprawl: Without a tight, value-driven scope, AI use cases can quickly sprawl dangerously. Furthermore, tools like Copilot have exposed heavy, pre-existing cyber costs, such as the need for comprehensive data tagging and secure access architectures, sparking internal friction over whether the remediation should be funded by AI or security budgets.

The ugly: Unexpected traps and surprise impacts

Moving past small experiments has revealed unique challenges that leaders never anticipated:

• AI is an ecosystem, not a product: AI maturity impacts literacy, governance, operating model, infrastructure. It cannot be treated as a siloed, standalone technology project and must be baked into organisation-wide strategic plans.
• Adoption challenges: Small experiments frequently see rapid adoption and excitement in week one, only to be largely abandoned by week three. Without a clear strategy for long-term cultural change, AI initiatives can quickly stall.
• Outsourced thinking: Daily AI usage is actively changing human behaviour, with multiple leaders noting that even highly capable engineers and business analysts occasionally outsource their critical thinking to AI. But without deep organisational context or wider knowledge, AI generates inadequate requirements, leading to poorer outcomes and slower delivery cycles

Shifting from proof of concept to proof of viability

The collective experiences of the leaders in the room at CIO Day demonstrated that the hard part of AI isn’t the AI itself. It’s everything around it: the data quality, architectural readiness, governance structures and cultural alignment. And it’s only by focusing on everything that you can see real value from AI projects.

At Equal Experts, we’ve been supporting our clients to move away from proof of concepts towards proof of viability. Before embarking on any project, we work alongside our partner to define the desired outcomes and assess if it is really achievable. If we can’t identify the tangible business value or prove that it can work long-term in production, we won’t start the project. It’s what makes the difference between an initiative that delivers meaningful results and one that never goes beyond a line in a report.

Bridging the execution gap

Technology leaders need partners who remove complexity, not add to it. At Equal Experts, we de-risk your AI plans by staffing projects with senior consultants who average 17+ years of industry experience. Our smaller, highly experienced teams move faster, upskill your internal capability and deliver value from day one. Large organisations working with Equal Experts have reduced consultant headcount by an average of 44%, without compromising on outcomes.

We help organisations achieve AI maturity in a phased evolution:

• Gauge: Understand your starting point with a clear picture of existing capabilities, opportunities and challenges.
• Ground: Secure the groundwork required for safe innovation across data, infrastructure, culture and governance
• Grow: Identify the specific AI use cases that solve real problems and deliver measurable value for the organisation.
• Go beyond: Treat AI as a fundamental evolution of how the organisation operates, embedding AI across people, process and culture.

A great place to start is to book a complimentary 30-minute Technology Principal-led key challenges discovery session. We’ll help you identify some of the high-impact use cases for AI in your organisation and validate them quickly, turning potential into real action.

Book a complimentary 30-minute session or reach out directly to the Equal Experts Australia team to learn more.

About the author

Matthew Waugh is Director of Sales and Business Development for Equal Experts APAC. With over 10 years of experience in sales and business development, Matthew is a lead in strategic initiatives that drive growth across the Asia-Pacific region for Equal Experts. Connect with Matthew on LinkedIn.