AI Systems & Evaluation Engineer Intern

About WalletGyde WalletGyde is building a Financial AI platform designed to help people better understand, manage, and improve their financial lives through trustworthy, explainable, and personalized financial intelligence. As we continue evolving our Financial AI architecture, one of our highest priorities is making sure the system does more than generate convincing responses. We want to understand whether every financial answer is **correct, grounded in the right data, produced through the right logic, and communicated responsibly**. Over the past year, our team has built a growing reliability infrastructure around our Financial AI, including deterministic financial calculations, Ground Truth datasets, regression testing, financial personas, Guardrails, Tone & Trust standards, QA workflows, failure classification, and what we refer to internally as the **WalletGyde Financial Harness**. We are now looking for an **AI Systems & Evaluation Engineer Intern** to help us strengthen and evolve these systems. The Role This is not a traditional machine learning internship focused primarily on model training. You will work across the systems surrounding modern AI models to help WalletGyde answer one critical question: How do we know WalletGyde produced the right financial answer, for the right reason, with the right behavior? You will help investigate where AI failures originate, improve how we evaluate system performance, strengthen our Ground Truth methodology, build regression coverage, and help convert manual evaluation processes into more repeatable engineering systems. This role sits at the intersection of **AI engineering, evaluation, financial logic, QA, data analysis, and responsible AI**. What You Will Work OnGround Truth & Financial Accuracy You will help audit, validate, version, and expand WalletGyde’s Ground Truth datasets and expected financial answers. This includes reviewing how expected answers connect to approved financial definitions, formulas, time periods, and deterministic calculation logic. AI Evaluation Help improve how WalletGyde measures AI performance using a combination of: Deterministic evaluators Model based graders Expected versus actual comparisons Financial accuracy testing Behavioral evaluation Human review You will help us move beyond simply asking whether a chatbot response passed or failed and toward understanding **why** a response succeeded or failed. Failure Analysis Investigate failures across multiple parts of the AI stack, including: Financial data Date and period logic Deterministic calculations Retrieval and RAG Orchestration and routing Guardrails Model responses Evaluation methodology The goal is to determine the actual root cause rather than assuming every incorrect response is an LLM problem. Regression Testing Help maintain and expand WalletGyde’s regression suite as new edge cases and failures are discovered. You will work with existing test cases and help ensure engineering improvements do not unintentionally break previously correct behavior. QA Engineering Help convert portions of our existing manual QA methodology into repeatable engineering workflows. This may include test runners, structured evaluation outputs, regression reports, automated checks, and eventually CI/CD evaluation gates. Guardrail Evaluation Help build executable tests around behaviors such as: Uncertainty Unsupported financial claims Clarification Privacy Financial boundaries Hallucination prevention Responsible redirection Safe handling of incomplete information Tone & Trust Evaluation WalletGyde has developed a Tone & Trust framework designed around principles such as: **Never Shame** **Never Pretend Certainty** **Answer First** **Encourage the User** **Connect Insight to Action** You will help explore how qualitative standards like these can become measurable, reproducible evaluation criteria. Financial Harness Engineering Support the broader WalletGyde Financial Harness connecting: **Financial Data → Deterministic Logic → Ground Truth → Retrieval → AI Reasoning → Guardrails → Tone → Product Response → Evaluation** You will help us understand how these components interact and how failures can be traced through the system. Evaluation Automation Work alongside WalletGyde engineers and external technical partners to help move portions of our evaluation methodology into automated workflows, regression pipelines, observability systems, and CI/CD. What You Will Inherit You will not be starting from a blank slate. WalletGyde has already developed significant evaluation infrastructure, including: **140 Ground Truth evaluation questions** **26 core regression cases** **20 structured financial personas** **Financial Metrics & Calculation Logic** **Deterministic Financial Intelligence functions** **Failure classification methodology** **Guardrail framework** **Tone & Trust framework** **Existing QA workflows** **Financial Harness evaluation methodology** Your job will be to understand what exists, identify where it can improve, and help us evolve it into a more mature Financial AI reliability system. Potential Q4 Projects Depending on experience and progress, projects may include: **Ground Truth vNext** Help create a cleaner and more versioned Ground Truth structure with validated expected answers, clearer financial definitions, and stronger traceability. **Evaluation Pipeline v1** Help develop a repeatable workflow that runs evaluation cases and captures expected versus actual behavior. **Failure Attribution Framework** Help improve how WalletGyde identifies whether failures originate from data, calculations, date resolution, retrieval, orchestration, Guardrails, model behavior, or evaluation itself. **Regression Suite Expansion** Turn newly discovered failures into permanent regression tests. **Guardrail Evaluation Suite** Help convert key Guardrail requirements into executable tests. **Tone Evaluation Suite** Explore automated and human assisted approaches for evaluating WalletGyde’s communication principles. **Financial Harness Reporting** Help create structured evaluation outputs that can eventually support dashboards, observability, and engineering decision making. What We Are Looking For Strong candidates will have experience or interest in several of the following: Python SQL REST APIs Git and GitHub Large Language Models RAG systems AI evaluation Automated testing Data analysis Prompt evaluation Model evaluation AI reliability Debugging complex systems CI/CD concepts Financial data or FinTech systems Technical documentation You do not need to be an expert in every area. We care deeply about your ability to **learn quickly, investigate complex systems, ask strong questions, and reason through ambiguous technical problems**. The Kind of Person Who Will Thrive Here We are looking for someone who is naturally curious. When an AI response is wrong, your instinct should not be: “The model made a mistake.” Your instinct should be: “Why did this fail, and at what layer did the failure actually begin?” Maybe the data was incomplete. Maybe the wrong financial period was selected. Maybe the deterministic calculation was incorrect. Maybe retrieval missed a transaction. Maybe the Ground Truth answer itself was wrong. Maybe the model received the correct information but explained it incorrectly. We want someone who enjoys following that chain until the actual cause is understood. You should also be comfortable reading and understanding existing systems before immediately rebuilding them. Why This Internship Is Different Many AI internships focus primarily on prompts, model experimentation, or building chatbot features. This role focuses on a different question: **How do you engineer trust around an AI system operating in a high stakes domain like personal finance?** You will gain hands on exposure to: Financial AI architecture LLM evaluation Ground Truth engineering Deterministic financial systems AI reliability Responsible AI Guardrail testing Regression engineering RAG systems FinTech data Early stage AI product development Evaluation automation You will also have the opportunity to see how an early stage company moves from experimental AI systems toward more measurable, observable, and production ready infrastructure. What Success Looks Like By the end of the internship, we want you to be able to look at a Financial AI failure and systematically determine: **What happened** **Where it happened** **Why it happened** **How it should be fixed** **How we prevent it from happening again** Ultimately, your work will help WalletGyde move from manually asking whether our AI is correct toward building systems that can **continuously measure, diagnose, and improve Financial AI reliability.** How to Apply Please submit your resume along with a brief response to the following: **Tell us about a technical problem you investigated where the first explanation turned out not to be the actual root cause. How did you figure out what was really happening?** GitHub, technical projects, AI evaluation work, research, or other examples of systems you have built or analyzed are encouraged but not required. **WalletGyde, Inc.** Building trustworthy financial intelligence through better data, better evaluation, and better AI systems. Job Type: Internship Pay: From $100.00 per month Work Location: Remote