Senior Prompt Engineer - Data Science & Quality Analysis (India) at Checkmate

, , India -

Full Time

Start Date

Immediate

Expiry Date

01 Mar, 26

Salary

0.0

Posted On

01 Dec, 25

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Prompt Engineering, Data Analysis, Python, SQL, A/B Testing, Statistical Analysis, LLM Workflows, Experiment Management, Model Optimization, Quality Checks, Communication Skills, Analytical Mindset, Language Models, Adversarial Evaluation, Model Safety Testing, Continuous Improvement

Industry

IT Services and IT Consulting

Description

Checkmate is building advanced Voice AI systems for some of the largest restaurant and retail brands in the U.S., including several in the top 10. Our AI solutions are live in production with real customers, achieving over 80% accuracy. This represents a $1 billion market opportunity. Join us at this pivotal moment of growth to shape AI products used daily by thousands of staff and customers, combining cutting-edge LLM innovation with real business impact. Essential Job Functions • Design, test, and optimize LLM prompts for conversational AI, text classification, and structured data extraction tasks. • Build evaluation pipelines to analyze prompt performance using quantitative metrics, human-in-the-loop feedback, and business KPIs. • Conduct prompt experiments and regression testing to ensure stability, accuracy, and safety as models evolve. • Collaborate with Machine Learning, Product, and Operations teams to translate business objectives into scalable, data-driven prompt-engineering strategies that enhance model accuracy, efficiency, and real-world usability. • Use Python/SQL to analyze model outputs, identify anomalies, and automate quality checks. • Document best practices and contribute to internal frameworks for prompt evaluation and continuous improvement. • Communicate findings effectively to technical and non-technical stakeholders, driving measurable business impact through insight-driven decisions. B.S. or higher in a quantitative discipline (Data Science, Computer Science, Engineering, or related field) or in a field relevant to language models (Linguistics, Philosophy, Cognitive Science, etc.). 5+ years of relevant experience with a B.S. degree, or 3+ years of experience with a Master’s degree. Demonstrated proficiency in Python for automation, evaluation, and experimentation with LLM workflows. Proven experience in prompt engineering and working with LLMs (GPT-4, Claude, Gemini, and LLaMA) for text generation, reasoning, and structured data extraction. Proficiency in Python and SQL for data analysis, evaluation scripting, and workflow automation. Strong background in A/B testing, statistical analysis, and performance metric evaluation, with the ability to design experiments and interpret data-driven insights for continuous model optimization. Familiarity with prompt-evaluation tools such as LangFuse or Galileo, and Weights and Biases for experiment management and regression testing. Deep understanding of advanced prompting techniques, including few-shot prompting, reasoning-based prompting, multi-turn dialogue design, agentic orchestration, and DSPy/AdaFlow-style programmatic prompting frameworks. Experience applying CO-STAR and TIDD-EC! prompting frameworks for structured reasoning, instruction design, and context control in production-grade LLM systems. Excellent requirement-elicitation and communication skills, with the ability to translate business objectives into prompt-engineering solutions. Analytical mindset with a process-driven approach to optimizing model behavior, data quality, and operational workflows. Academic or applied research experience related to language models, prompt engineering, or LLM-based systems is a strong plus. Familiarity with LLM architectures, embeddings, and fine-tuning techniques preferred. Experience with LLM red-teaming, adversarial evaluation, or model safety testing is a plus. Candidates must be flexible and work during US hours at least till 5 pm EST, which is essential for this role & must also have their own system/work setup for remote work.

Responsibilities

The Senior Prompt Engineer will design, test, and optimize LLM prompts for various AI tasks while building evaluation pipelines to analyze prompt performance. Collaboration with cross-functional teams is essential to translate business objectives into effective prompt-engineering strategies.