LLM-based document parsing product to extract information from financial documents
Project: Artemis
LLM-based document parsing
to extract information from
financial documents
Company
Hercules AI, builds AI products to automate document extraction, processing, and verification
Direct Team
2 Backend Eng 2 Frontend Eng,
2 AI Engineers, 1 Annotator,
1 Designer, 2 QA Specialists
Role
Lead Product Manager
Duration
6 Months
I led the end-to-end development to launch Artemis, 0 -> 1
Given the technical complexity, I planned and roadmapped our feature set beforehand so that (1) the engineering team could build each step quickly, and (2) we could put something in users' hands to test hypotheses.
Problem Statement
Our customer, a large investment firm, were drowning in a massive volume of Investment documents
Costed the investment firm 1M+ annually
Failed in-house and market solutions
80 FTEs processing 5,000 equity notices /month
Manually processing and analyzing documents takes 20 mins
Result
Artemis reduced processing times by 90% per document and secured a $1.2M contract
We developed Artemis, a human-in-the-loop learning AI system that automated the manual extraction process. This reduced processing time from 20 minutes to under 2 minutes per document, enhanced data extraction accuracy by 95%, and secured a $1.2 million annual contract
GOING DEEPER...
Constraints
Users felt the task was too complex for a technical solution
Diverse document formats
Constantly changing structures and file types
User Skepticism
Users doubted AI accuracy and feared increased workload
PROJECT Goals
We needed to build an AI system that was effective and robust
Reduce Processing Time
Goal 1
Develop an intelligent document
extraction system with near 100%
accuracy across evolving
document formats
Goal 2
Rebuild trust in AI solutions among users who were skeptical due to previous failures
Goal 3
Research insight
Existing solutions failed because they relied on template matching and needed a lot of training data
Competitive Research
Most products were doing basic template matching, which meant users had to manually annotate each file type, reducing any efficiency gains
User research
User complained that every client of theirs sent documents in different file types and formats
finding the solution
Zero-shot LLMs could address the scalability issue by requiring minimal data
A key Insight from competitive analysis was that AI models actually needed a lot of data to adapt to a new document. This data had to be hand-annotated by the analysts - this was the bottleneck
Reading AI research
I went through AI research papers and narrowed down on LLMs with zero-shot learning and one-shot learning to minimize the data needed for training models
Technical requirements
Shared these insights with our AI team and broke down the technical requirements.
Collaborating closely
with design + Eng
I sketched wireframes of the core interface functionalities based on my understanding of the system architecture
Priortization and Roadmapping
Used Impact vs. Effort Matrix to prioritize and test our ideas fast
Given the technical complexity of the solution we needed to build, I had to plan and fully roadmap our feature set beforehand so that (1) the engineering team can build each step quickly and (2) so we can put something in the hands of users to test hypothesis.
ROADBLOCK
Users still felt our product was a black box, and it didn’t spark trust
Bringing teams together to build, test, and improve fast
Facilitated cross-functional collaboration between design, engineering, and users to develop and validate a minimal functional product bi-weekly. Leveraged agile methodologies to iterate rapidly, incorporating incremental features based on user feedback and strategic priorities.
Building Trust with ai
Transparency and user agency are key to rebuilding trust in AI
Building Trust: I laid out a human-in-the-loop AI training system with two core principles
Show the model's output transparently
Make them feel that the model's confidence scores were well calibrated
Motivate the analysts to make corrections
and show their corrections having an impact on the model performance.
For each document, we had a set of extracted fields with confidence scores. Users could review and verify extracted data, correcting any inaccuracies
Surfaced data with low confidence scores at the top, allowing users to focus on verifying uncertain data first
Every 50 corrections, we'd retrain the model and show the updated performance on their data - continually improving its accuracy over time.
This allowed us to build trust in two stages:
First when users realized that they could trust the model's confidence scores
Second, when users saw the model learn from their corrections
outcome
We launched Artemis 1.0 two weeks ahead of schedule!
This project really drove home how important it is to understand what users actually need and to build trust by being transparent and involving them every step of the way. Honestly, it was exciting to see how combining AI with thoughtful design could not only tackle a really tricky technical problem but also win over skeptical users. The success of Artemis showed me firsthand that when you align innovative tech with a user-first approach, you create solutions that truly make an impact—and deliver real business value
Reduced processing time (north star metric) from 20 mins to 2 mins per document
with 95% data extraction accuracy
Secured a $1.2 million annual contract
Reflection
For AI products, building trust is just as important as the functional product experience
“Working on this project taught me that for AI products, building trust is just as important as delivering a great functional experience. I realized how critical it is to truly understand user needs and involve them every step of the way, especially when skepticism is high. By combining innovation with thoughtful design choices, we didn’t just solve a tough technical challenge—we built transparency and trust with our users. Seeing the success of Artemis firsthand reinforced for me that aligning innovation with user-centric design can create solutions that are not only impactful but also deeply valued by the people who use them.”