Lead I - Software Engineering - AI Solutions Analyst

MUST-HAVES:

• LLM, AI, Prompt Engineering LLM Integration & Prompt Engineering
• Context & Knowledge Base Design.
• Context & Knowledge Base Design.
• Experience running LLM evals

NOTICE PERIOD: Immediate – 30 Days

SKILLS: LLM, AI, PROMPT ENGINEERING

NICE TO HAVES:

Data Literacy & Modelling Awareness Familiarity with Databricks, AWS, and ChatGPT Environments

ROLE PROFICIENCY:

Role Scope / Deliverables:

• Scope of Role Serve as the link between business intelligence, data engineering, and AI application teams, ensuring the Large Language Model (LLM) interacts effectively with the modeled dataset.
• Define and curate the context and knowledge base that enables GPT to provide accurate, relevant, and compliant business insights.
• Collaborate with Data Analysts and System SMEs to identify, structure, and tag data elements that feed the LLM environment.
• Design, test, and refine prompt strategies and context frameworks that align GPT outputs with business objectives.
• Conduct evaluation and performance testing (evals) to validate LLM responses for accuracy, completeness, and relevance.
• Partner with IT and governance stakeholders to ensure secure, ethical, and controlled AI behavior within enterprise boundaries.

KEY DELIVERABLES:

• LLM Interaction Design Framework: Documentation of how GPT connects to the modeled dataset, including context injection, prompt templates, and retrieval logic.
• Knowledge Base Configuration: Curated and structured domain knowledge to enable precise and useful GPT responses (e.g., commercial definitions, data context, business rules).
• Evaluation Scripts & Test Results: Defined eval sets, scoring criteria, and output analysis to measure GPT accuracy and quality over time.
• Prompt Library & Usage Guidelines: Standardized prompts and design patterns to ensure consistent business interactions and outcomes.
• AI Performance Dashboard / Reporting: Visualizations or reports summarizing GPT response quality, usage trends, and continuous improvement metrics.
• Governance & Compliance Documentation: Inputs to data security, bias prevention, and responsible AI practices in collaboration with IT and compliance teams.

KEY SKILLS:

Technical & Analytical Skills:

• LLM Integration & Prompt Engineering – Understanding of how GPT models interact with structured and unstructured data to generate business-relevant insights.
• Context & Knowledge Base Design – Skilled in curating, structuring, and managing contextual data to optimize GPT accuracy and reliability.
• Evaluation & Testing Methods – Experience running LLM evals, defining scoring criteria, and assessing model quality across use cases.
• Data Literacy & Modeling Awareness – Familiar with relational and analytical data models to ensure alignment between data structures and AI responses.
• Familiarity with Databricks, AWS, and ChatGPT Environments – Capable of working in cloud-based analytics and AI environments for development, testing, and deployment.
• Scripting & Query Skills (e.g., SQL, Python) – Ability to extract, transform, and validate data for model training and evaluation workflows.
• Business & Collaboration Skills Cross-Functional Collaboration – Works effectively with business, data, and IT teams to align GPT capabilities with business objectives.
• Analytical Thinking & Problem Solving – Evaluates LLM outputs critically, identifies improvement opportunities, and translates findings into actionable refinements.
• Commercial Context Awareness – Understands how sales and marketing intelligence data should be represented and leveraged by GPT.
• Governance & Responsible AI Mindset – Applies enterprise AI standards for data security, privacy, and ethical use.
• Communication & Documentation – Clearly articulates AI logic, context structures, and testing results for both technical and non-technical audiences.