Module 7: AI Engineering - The New Role

5. The Day to Day: What an AI Engineer Does at Companies

Next →

Description

Question: What does an AI Engineer do on a typical workday?

Answer: It depends on the company, but typically: integrating APIs, designing RAG systems, optimizing prompts, deploying features, collaborating with product/frontend.

In this lesson you'll understand:

  • Typical day-to-day tasks.
  • Common projects (chatbots, RAG, agents, classification).
  • How they collaborate with other roles (product, frontend, backend, data science).

Typical Tasks (Day to Day)

1. Integrating LLM APIs

What:

  • Calling the OpenAI API, the Anthropic API, the Google API.
  • Handling authentication (API keys).
  • Error handling (retry logic, fallbacks).

Example:

import openai

client = openai.OpenAI(api_key="sk-...")

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello"}]
)

print(response.choices[0].message.content)

Time: 20-30% of the day.

2. Designing and Building RAG Systems

What:

  • Converting documents into embeddings (vectors).
  • Storing them in a vector DB (Pinecone, Weaviate).
  • Searching for relevant documents (cosine similarity).
  • Generating a response with an LLM (using the documents as context).

Flow:

User query → Embed query → Search vector DB → Retrieve docs → LLM (generate answer with docs as context) → Response

Time: 30-40% of the day (on complex projects).

3. Optimizing Prompts

What:

  • Experimenting with system prompts (clear instructions).
  • Few-shot learning (adding examples).
  • Chain-of-Thought (reasoning step by step).
  • Structured outputs (JSON mode).

Goal:

  • Improving quality (accuracy, relevance).
  • Reducing cost (shorter prompts).

Time: 10-20% of the day.

4. Deploying Features

What:

  • Exposing the feature via an API (FastAPI, Express.js).
  • Dockerizing the application.
  • Deploying to the cloud (AWS, GCP, Vercel).
  • Monitoring and logging (errors, latency, cost).

Example:

from fastapi import FastAPI

app = FastAPI()

@app.post("/chat")
async def chat(message: str):
    response = openai.ChatCompletion.create(...)
    return {"response": response.choices[0].message.content}

Time: 10-20% of the day.

5. Collaborating with Other Roles

What:

  • The Product Manager: Understanding the requirements (which feature to build).
  • The Frontend Engineer: Designing the API contract (which endpoints to expose).
  • The Backend Engineer: Integrating with the existing system (databases, authentication).
  • The Data Scientist: Using the insights (e.g. "users ask about X → optimize RAG for X").

Time: 10-20% of the day.

Common Projects

1. A Support Chatbot

Goal: Answering user questions (FAQ, technical support).

Stack:

  • LLM: GPT-3.5 or Claude Haiku (low cost).
  • RAG: The product documentation → embeddings → a vector DB.
  • Frontend: React (the chat UI).

The AI Engineer's tasks:

  1. Integrating the OpenAI API.
  2. Building the RAG (documentation → embeddings → Pinecone).
  3. Optimizing the prompts (system prompt: "You are a support assistant...").
  4. Exposing the API (FastAPI).
  5. Deploying and monitoring.

Timeline: 2-4 weeks.

2. A Q&A System over Documentation

Goal: Answering questions about docs (code, manuals, procedures).

Stack:

  • LLM: GPT-4 (reasoning).
  • RAG: Docs → embeddings → a vector DB.
  • Backend: FastAPI.

The AI Engineer's tasks:

  1. Ingesting the docs (PDFs, Markdown → chunks → embeddings).
  2. Storing them in a vector DB (Chroma, Weaviate).
  3. Query: The user asks → embed the query → search for relevant chunks → the LLM generates an answer.
  4. Optimizing (chunk size, embedding model, re-ranking).

Timeline: 3-6 weeks.

3. Text Classification

Goal: Classifying text (e.g. support tickets → categories: billing, technical, account).

Stack:

  • LLM: GPT-3.5 (simple classification).
  • Prompt: Few-shot examples.

The AI Engineer's tasks:

  1. Designing the prompt (system: "Classify this ticket", few-shot examples).
  2. Integrating the API.
  3. Exposing the endpoint (POST /classify).
  4. Monitoring (accuracy, latency).

Timeline: 1-2 weeks.

4. Agents (Multi-Tool)

Goal: The LLM decides which tool to use (e.g. search the docs, call an API, run code).

Stack:

  • LLM: GPT-4 (reasoning).
  • Framework: LangChain (a ReAct agent).
  • Tools: A search API, a calculator, a database query.

The AI Engineer's tasks:

  1. Defining the tools (search, calculator, DB).
  2. Configuring the agent (LangChain ReAct).
  3. Optimizing (agent loops, tool calling).
  4. Deploying.

Timeline: 4-8 weeks (complex).

A Typical Day (Example)

9:00-10:00: Standup and Planning

  • A daily standup with the team (product, frontend, backend).
  • A review of tasks: "Today I'll optimize the RAG prompts to reduce hallucinations".

10:00-12:00: Development (RAG Optimization)

  • Experimenting with chunk size (256 vs 512 tokens).
  • Trying re-ranking (re-ranking the most relevant documents).
  • Measuring accuracy (manual testing with 20 queries).

12:00-13:00: Lunch

13:00-15:00: Code Review and Debugging

  • A code review of another AI Engineer's PR (reviewing the prompt logic).
  • Debugging: API timeouts (adding retry logic).

15:00-16:00: A Meeting with Product

  • The Product Manager asks for a new feature: "Add multimodal support (image + text)".
  • The AI Engineer explains: "We need GPT-4 Vision, a 3× higher cost, a 2-week timeline".

16:00-17:00: Deployment

  • Deploying a new version of the chatbot (with optimized prompts).
  • Monitoring: Verifying that latency <1s, cost <$0.02/request.

17:00-18:00: Learning

  • Reading Claude 3.5's documentation (new features).
  • Experimenting with structured outputs (JSON mode).

Collaboration with Other Roles

The Product Manager

What they do:

  • They define which features to build.
  • They prioritize (chatbot or RAG first?).

How you collaborate:

  • You explain the trade-offs (cost, latency, capability).
  • You propose technical solutions (GPT-4 vs GPT-3.5).

The Frontend Engineer

What they do:

  • They build the UI (the chat interface, the search bar).

How you collaborate:

  • You define the API contract (POST /chat, the response format).
  • You provide examples of requests/responses.

The Backend Engineer

What they do:

  • They handle authentication, databases, scaling.

How you collaborate:

  • You integrate with their system (e.g. user authentication).
  • You share the vector DB (Pinecone) for RAG.

The Data Scientist

What they do:

  • They analyze data (what are users asking?).
  • They identify problems (hallucinations in 20% of queries).

How you collaborate:

  • You use the insights to optimize (adding context for problematic queries).

Common Mistakes

1. Over-optimizing without measuring

Mistake: Optimizing prompts without measuring accuracy (trusting your intuition).

Solution: Create a test set (20-50 queries), measure accuracy before/after.

2. Not considering cost

Mistake: Using GPT-4 for everything (even simple tasks).

Solution: A multi-model strategy (GPT-3.5 for simple, GPT-4 for complex).

3. Not deploying fast

Mistake: Perfecting a feature locally for weeks (without user feedback).

Solution: Deploy an MVP in 1-2 weeks, iterate with feedback.

Why this matters for an AI Engineer

1. Realistic expectations

Day to day:

  • 70% coding/integration (APIs, RAG, prompts).
  • 20% collaboration (product, frontend, backend).
  • 10% learning (new LLM features, frameworks).

2. Critical skills

Most important:

  • Fast integration (APIs, RAG).
  • Optimization (prompts, cost, latency).
  • Deployment (FastAPI, Docker, cloud).

Summary

Typical tasks:

  1. Integrating LLM APIs (20-30%).
  2. Designing RAG (30-40%).
  3. Optimizing prompts (10-20%).
  4. Deploying features (10-20%).
  5. Collaborating (10-20%).

Common projects:

  • A support chatbot (2-4 weeks).
  • Q&A over docs (3-6 weeks).
  • Text classification (1-2 weeks).
  • Agents (4-8 weeks).

A typical day:

  • Development (RAG, prompts, APIs).
  • Meetings (product, frontend, backend).
  • Deployment and monitoring.
  • Learning (new features, frameworks).

Collaboration:

  • The Product Manager (requirements, prioritization).
  • The Frontend Engineer (the API contract).
  • The Backend Engineer (integration).
  • The Data Scientist (insights, optimization).

Next step: Lesson 06: Exercise: Is this for you? — An honest self-assessment.

← Previous4. AI Engineer Skills: What You Need to Know (and What You Don't)Next →6. Exercise: Is AI Engineering for You? (An Honest Self-Assessment)