Note: The job is a remote job and is open to candidates in USA. Info Way Solutions is a company seeking a Product Analyst to join their team. The role involves drafting detailed user stories, maintaining product backlogs, and analyzing AI performance metrics to provide actionable product recommendations.
Responsibilities
- 3+ years experience drafting detailed Jira user stories, maintaining product backlogs, and defining acceptance criteria based on business requirements in an Agile product team or equivalent environment
- 2+ years experience in data analysis with the ability to interpret AI performance metrics and translate findings into actionable product recommendations
- 1+ year experience working with LLM-based applications, including understanding of hallucination patterns, output quality issues, grounding techniques, bias detection, and mitigation strategies
- 1+ year hands-on experience executing structured AI agent evaluation plans, tracking iterations, maintaining golden datasets, and performing regression testing
- 1+ year experience supporting the development, evaluation, and continuous improvement of AI-powered products, AI agents, copilots, or generative AI solutions
- Experience translating AI product requirements into prompts, agent workflows, business rules, and measurable success criteria
- Experience defining AI product success metrics and measuring improvements in output quality, user satisfaction, task completion, or productivity
- Experience partnering with engineering, UX, data science, and business stakeholders to prioritize, evaluate, and refine AI product capabilities
Skills
- 3+ years experience drafting detailed Jira user stories, maintaining product backlogs, and defining acceptance criteria based on business requirements in an Agile product team or equivalent environment
- 2+ years experience in data analysis with the ability to interpret AI performance metrics and translate findings into actionable product recommendations
- 1+ year experience working with LLM-based applications, including understanding of hallucination patterns, output quality issues, grounding techniques, bias detection, and mitigation strategies
- 1+ year hands-on experience executing structured AI agent evaluation plans, tracking iterations, maintaining golden datasets, and performing regression testing
- 1+ year experience supporting the development, evaluation, and continuous improvement of AI-powered products, AI agents, copilots, or generative AI solutions
- Experience translating AI product requirements into prompts, agent workflows, business rules, and measurable success criteria
- Experience defining AI product success metrics and measuring improvements in output quality, user satisfaction, task completion, or productivity
- Experience partnering with engineering, UX, data science, and business stakeholders to prioritize, evaluate, and refine AI product capabilities
- Experience in HR technology, learning technology, content creation platforms, or knowledge management solutions as a Business Analyst, Product Analyst, Product Owner, or equivalent role
- Hands-on experience building and executing specialized AI test cases, evaluating LLM outputs, prompt variations, and agentic workflows
- Hands-on experience with prompt engineering, prompt optimization, and evaluation of AI-generated content quality, accuracy, completeness, and consistency
- Experience designing evaluation rubrics, benchmark datasets, golden datasets, and quality measurement frameworks for AI-generated outputs
- Experience conducting side-by-side model comparisons, experimentation, or A/B testing to improve AI product performance
- Familiarity with Retrieval Augmented Generation (RAG), grounding strategies, hallucination mitigation, and content sourcing concepts
- Familiarity with automated AI evaluation tools, frameworks, and testing methodologies
- Experience with human-in-the-loop workflows and designing review, validation, and approval processes for AI-generated outputs
- Experience working with AI governance, responsible AI practices, bias detection, content safety, and model quality evaluation
- Experience evaluating content quality against defined business, compliance, instructional design, or organizational standards
- Strong written and verbal communication skills, with the ability to document evaluation methodologies, findings, recommendations, and trade-offs for both technical and business audiences
Company Overview