[VCK] Senior QA Automation Engineer (AI Systems)

Company Description

We are Software Mind, an awesome team of engineers who are ready to ramp up any top-notch company’s projects! Our aim? To always be one step ahead. Become part of a multicultural company in constant growth with an excellent work environment certified by Great Place To Work!

Job Description

About the Project
Software Mind is building a private, tenant-isolated AI assistant for the real estate title and settlement industry. The platform is a retrieval-first (RAG) system that ingests historical email, documents, and structured metadata into a per-tenant vector index, and serves grounded, cited, expert-weighted answers through a chat-style Q&A interface with single sign-on and full audit logging.
The platform is AWS-native with a Python/FastAPI backend, Vue.js frontend, OpenSearch/Pinecone vector store, and OpenAI/Anthropic/Bedrock as LLM provider. You will join a senior, cross-functional LATAM-based team where hands-on AI delivery experience not just familiarity is the baseline expectation.
You build the validation harness that determines whether the AI system meets accuracy standards this is your primary deliverable. This is not generic test automation. You need to understand what 'correct' means for a RAG-based retrieval system and design test frameworks that can surface retrieval failures, hallucinated citations, confidence score drift, and RBAC access violations.

Your Responsibilities
Design and implement the validation harness for RAG output quality: retrieval accuracy, citation correctness, and grounding

Build automated test suites for the AI Extraction Gateway across Simple RAG and Complex RAG implementations

Develop and execute accuracy rubric test cases in collaboration with the BA and Designated Subject Matter Expert

Automate regression testing for confidence score calibration and source weighting behaviour

Test RBAC enforcement and role-specific filtered view access controls

Validate audit logging completeness and document lifecycle traceability

Build and maintain the incremental ingestion pipeline test suite

Contribute to go/no-go decision packs: produce accuracy reports and test evidence documentation

Tech Stack: Python, pytest, AWS, REST API Testing, Jira, Confluence

Qualifications

Must-Have Skills & Experience
6+ years in QA automation engineering; senior seniority required

Strong test automation engineering skills Python preferred (pytest or equivalent framework)

Experience with API testing and contract testing

Comfortable designing test frameworks for non-deterministic or probabilistic systems

Experience in agile/scrum environments with Jira-based test management

Additional Information

Nice-to-Have
Prior experience testing RAG systems, LLM outputs, or semantic search relevance this is a strong differentiator

Familiarity with AI evaluation frameworks: RAGAS, TruLens, or custom rubric-based evaluation approaches

Background in compliance-sensitive testing: audit trail validation, access control verification, or regulated-data environments

We are accepting applications from LATAM countries