Generative AI powered Support Ticket Categorization
We helped a leading financial services company automate its customer support ticket classification system using AI, achieving 80%+ accuracy on 500+ daily tickets.
Addressing customer complaints promptly and effectively is crucial for minimizing dissatisfaction and fostering stronger customer loyalty. Furthermore, analyzing these complaints provides valuable insights for ongoing service improvement, ultimately attracting new customers. However, the unstructured nature of this feedback means companies often rely on support staff to manually sort and route complaints, a process that becomes increasingly inefficient and unsustainable as the customer base expands.
A leading financial services company specializing in lending faced this exact challenge. With 400–500 support tickets received daily, the manual categorization process was slow, error-prone, and lacked scalability.
The Problem with Manual Classification
For a fast-growing fintech company processing loans and payments, resolving customer tickets quickly was critical. However, manually categorizing 400–500 unstructured tickets daily into 25+ categories led to the following challenges:
- High ticket misclassification rates due to human error and subjective judgment.
- Delays in routing tickets to the right team.
- Zero visibility into recurring issues (e.g., refund delays, payment failures).
The company needed a system that could automatically classify tickets with high accuracy while adapting to evolving customer queries—without retraining models or adding human reviewers. Recognizing the inefficiencies, the company approached Akaike Technologies to automate and optimize its ticket classification system using Gen AI.
The Solution
Akaike deployed a classification model tailored to the client’s needs. The solution focused on ticket data extraction, pre-processing, prompt engineering, and LLM optimization to ensure accurate and efficient categorization.
Key Steps in the Solution:
Step 1: Simplifying the Category Framework:
- Historical ticket data was analyzed to identify patterns and fine-tune parameters for future ticket classification.
- Less frequent categories were merged into an “Other” category, reducing the total number of categories from 25 to 11.
Step 2: Prompt Engineering for Precision:
- Prompts were designed to guide the LLM in classifying tickets into the 11 refined categories.
- For complex tickets, the LLM was asked to rationalize its classification decision, ensuring viable reasoning for each categorization.
The system was built on the inference capabilities of Sentence Transformers and foundation model, eliminating the need for retraining on new tickets. Similarity models were integrated to improve accuracy, including:
- Fuzzy Matching: Identified and categorized tickets even with slight variations in wording, such as misspellings or synonyms.
- Semantic Similarity Models: Compared the meaning of customer queries rather than relying on exact keyword matches, enabling accurate grouping of tickets with similar issues but different phrasing.
Prompts were engineered to ensure the LLM could understand and classify tickets accurately without retraining, even as new tickets were introduced.
Impact & Results
Akaike’s approach combined technical expertise with a deep understanding of the client’s business needs. The logic guiding our solution-building helped improve ticket classification and increase business efficiency in several ways:
1. Domain-Specific Prompts:
- Incorporated fintech jargon into prompt examples.
- Avoided costly model retraining—saving $15k/month in cloud compute.
3. Transparency:
- LLM rationales provided auditable logs for compliance teams.
We achieved 80% accuracy on ticket classification using the methods above.
The LLM’s reasoning step reduced the misclassification of hybrid tickets (e.g., a refund request triggered by a payment failure), thus ensuring the resolution of ambiguous ticket types.
Here are some other areas that saw an improvement:
By focusing on prompt engineering, LLM optimization, and inference-based classification, we delivered a solution that not only automated ticket classification but also provided a foundation for ongoing operational improvements.
The Akaike Edge
Akaike's expertise lies in end-to-end management of AI lifecycle processes including problem identification and data collection to model deployment and ongoing monitoring, utilizing industry-leading frameworks, tools, and libraries.
%20(1).png)
