DNSC 6290: LLMs: Concepts and Applications

Logo

George Washington University | Department of Decision Sciences

View the Project on GitHub jphall663/GWU_LLM

DNSC 6290: LLMs: Concepts and Applications

Materials for a technical, nuts-and-bolts course on large language models, their applications, and their evaluation.

DNSC 6290: Course outline

  1. LLM and transformer foundations
  2. Prompting and prompt development
  3. Model APIs, structured outputs, and typed decisions
  4. Context engineering, memory, and state
  5. RAG and knowledge pipelines
  6. Tool use and agentic workflows
  7. TEVV for LLM applications

Corrections or suggestions? Please open an issue.

Preliminary materials

Lecture 1: LLM and transformer foundations

Tokens, embeddings, attention, transformers, generation; reasoning models and inference-time reasoning.

An encoder--decoder transformer stacks attention and feed-forward blocks to transform token representations into vocabulary predictions.

An encoder–decoder transformer stacks attention and feed-forward layers to produce token representations. Source: Lecture 1, Slide 11.

Lecture 1 class materials

Lecture 2: Prompting and prompt development

Instructions, system prompts, examples/in-context learning, templates, task decomposition, and iterative refinement.

A prompt-development lifecycle moves from goal framing to design, examples or workflow stages, evaluation, and monitored release.

A prompt-development lifecycle moves from goal framing and design through examples, evaluation, release, monitoring, and decommissioning. Source: Lecture 2, Slide 17.

Lecture 2 class materials

Lecture 3: Model APIs, structured outputs, and typed decisions

Requests/responses, parameters, streaming, error handling; JSON schemas, Pydantic validation, and Jev-style typed decisions.

Four stacked controls for a reliable model API application: bound the request, validate data, bound the decision, and test claims.

Four API controls bound requests, validate data, constrain decisions, and test claims. Source: Lecture 3, Slide 18.

Lecture 3 class materials

Lecture 4: Context engineering, memory, and state

Context assembly, windows and budgets, conversation history, persistent memory, and state management.

Conversation history, persistent memory, and task state pass through selection, validation, ordering, and budgeting before becoming context for one model call.

Conversation history, persistent memory, and application state are selected and budgeted into context for a model call. Source: Lecture 4, Slide 6.

Lecture 4 class materials

Lecture 5: RAG and knowledge pipelines

Ingestion, chunking, retrieval embeddings, vector/hybrid search, reranking, grounded answers, and citations.

A retrieval-augmented generation diagram shows a user query retrieving relevant documents, adding them to the LLM context, and producing a grounded answer.

A query retrieves relevant documents that augment the model prompt and support a cited answer. Source: Lecture 5, Slide 14.

Lecture 5 class materials

Lecture 6: Tool use and agentic workflows

Function calling, external connectivity, MCP, bounded agent loops, orchestration, and agent identity; Text2SQL as a worked example.

An MCP host contains clients that connect over JSON-RPC transports to tool servers backed by local files, APIs, and remote systems.

An MCP host connects clients to tool servers over local or remote transports. Source: Lecture 6, Slide 9, adapted from IBM (n.d.), What is Model Context Protocol?; transport labels follow the MCP (2025) Transports specification.

Lecture 6 class materials

Lecture 7: TEVV for LLM applications

Measurement science, benchmarks and their limitations, LLM judges, embedding- and classifier-based tests, red teaming, field testing, human review, and validation of AI measurement instruments.

A model-testing workflow combines benchmark cases with embedding-based diagnostics and human-interpretable evidence.

Benchmark cases and embedding diagnostics provide complementary evidence for model testing. Source: Lecture 7, Slide 7, Sudjianto et al. (2024), Human-Calibrated Automated Testing and Validation of Generative Language Models.

Lecture 7 class materials

Software configuration

The notebooks use Python 3.13 and the repository’s locked uv environment. From the repository root, run uv sync --locked to create or update .venv, then select that project environment as the notebook kernel. Notebooks that use the OpenAI API load OPENAI_API_KEY from the ignored, repository-root .env file or prompt for it securely.

The project README covers installing uv and Python, selecting the kernel in VS Code, and lecture-specific setup instructions.

This project is maintained by jphall663.