Test Analyst - AI Assurance

Hexaware Technologies

4+ YrsNot disclosedIndia (On-site)
Posted: 6hr agoOpening: 1

Skills Required

Key skills are marked withstar
  • Key skillSoftware Testing
  • Key skillTest Case Design
  • Key skillTest Planning
  • Manual Testing
  • Functional Testing
  • Regression Testing
  • Requirements Analysis
  • Defect Management
  • Jira
  • SQL
  • User Acceptance Testing
  • API Testing
  • Test Automation
  • Agile Methodologies
  • Documentation

Job Description

Test Analyst - AI Assurance
Role Overview
Strong understanding of GenAI system architecture, including Retrieval-Augmented Generation (RAG) and Agent-based systems, with hands-on experience in implementing Python-based AI evaluation frameworks such as Ragas, DeepEval, and similar tools. Proven ability to perform Model and Agent quality evaluations using platforms like Arize, LangFuse, and Confident AI, while driving Responsible AI (RAI) validation and reporting key RAI metrics. Experienced in golden dataset curation, synthetic dataset generation, and establishing AI observability and tracing through OpenTelemetry (OTEL) instrumentation to enable effective monitoring, diagnostics, and continuous quality improvement of AI systems.
Key Responsibilities
•  Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
•  Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
•  Experience in Golden dataset curation & Synthetic dataset generation
•  Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc. 
•  Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
•  Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
•  Experience in Integrating AI evaluation into CI/CD pipeline
•  Experience in Responsible AI validation 

 Required Skills and Capabilities
•  Strong understanding of GenAI System Architecture of RAG & Agents
•  Proficiency in Python & AI evaluation SDKs
•  AI Evaluation Frameworks (Ragas, DeepEval) 
•  Model Evaluation 
•  Agent Evaluation 
•  Responsible AI (RAI) Validation 
•  RAI Metrics Reporting 
•  Golden Dataset Curation 
•  Synthetic Dataset Generation 
•  AI Observability 
•  AI Tracing 
•  OpenTelemetry (OTEL) 
•  LLM Evaluation 
•  AI Quality Engineering 
•  AI Governance 
•  AI Monitoring 
•  AI Testing & Assurance 
•  Root Cause Analysis (AI Systems)
•  Knowledge of regulatory considerations for AI systems
Role-Specific Requirements
•  Proficiency in Python & AI evaluation SDKs
•  Strong understanding of RAG & agentic system architecture
•  Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
•  Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
•  Experience in Golden dataset curation & Synthetic dataset generation
•  Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc. 
•  Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
•  Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
•  Experience in Integrating AI evaluation into CI/CD pipeline
•  Experience in Responsible AI validation
•  Knowledge of regulatory considerations for AI systems
Experience Requirements
More than 4 years of experience in AI Assurance Testing

 

 

Similar Jobs (35064)

Manual Functional Test Engineer

Persistent Systems

Not disclosedPune

Skills required:

Manual Testing, Test Case Design, Test Planning, Functional Testing, Regression Testing, Defect Tracking, Jira, Exploratory Testing, System Testing, SQL, API Testing, Test Management Tools, Agile Methodologies, Documentation

Mainframe Developer

Persistent Systems

Not disclosedBengaluru

Skills required:

Programming, Java, Python, C#, JavaScript, SQL, Object-Oriented Programming, REST APIs, Web Development, Database Design, Git, Unit Testing, Debugging, Software Development Life Cycle, Agile Methodologies

Senior Business Analyst

Persistent Systems

Not disclosedPune

Skills required:

Business Analysis, Requirements Gathering, Process Mapping, Gap Analysis, BPMN, Data Analysis, SQL, User Stories, Stakeholder Management, Solution Design, Documentation, Client Management, Presentation Skills

Senior Databricks Developer

Persistent Systems

Not disclosedPune

Skills required:

Databricks, Apache Spark, PySpark, Delta Lake, Python, SQL, Data Pipelines, Unity Catalog, Microsoft Azure, AWS, Lakehouse Architecture, Structured Streaming, MLflow, Data Modeling, Airflow

Azure Data Architect

Persistent Systems

Not disclosedPune

Skills required:

Data Architecture, Data Modeling, Data Warehousing, Data Governance, SQL, Cloud Data Platforms, Snowflake, Databricks, ETL, Master Data Management, Data Lakes, Lakehouse Architecture, Metadata Management, Data Security, Stakeholder Management

AWS Full Stack Engineer

Persistent Systems

Not disclosedPune

Skills required:

JavaScript, TypeScript, React, Node.js, Python, REST APIs, GraphQL, SQL, NoSQL, System Design, Microservices, Docker, Cloud Platforms, CI/CD, Unit Testing

Data Engineer

Persistent Systems

Not disclosedPune

Skills required:

Data Engineering, SQL, Python, ETL, Data Pipelines, Apache Spark, Airflow, Data Warehousing, Data Modeling, Cloud Platforms, Kafka, dbt, Snowflake, AWS, Git

Senior Data Engineer

Persistent Systems

Not disclosedPune

Skills required:

Data Engineering, SQL, Python, Apache Spark, Data Pipelines, Airflow, Kafka, Data Modeling, Data Warehousing, Snowflake, Databricks, Cloud Platforms, dbt, Data Quality, CI/CD

Senior Databricks Developer

Persistent Systems

Not disclosedPune

Skills required:

Databricks, Apache Spark, PySpark, Delta Lake, Python, SQL, Data Pipelines, Unity Catalog, Microsoft Azure, AWS, Lakehouse Architecture, Structured Streaming, MLflow, Data Modeling, Airflow

AWS Full Stack Engineer

Persistent Systems

Not disclosedPune

Skills required:

JavaScript, TypeScript, React, Node.js, Python, REST APIs, GraphQL, SQL, NoSQL, System Design, Microservices, Docker, Cloud Platforms, CI/CD, Unit Testing

View All 35064 Jobs