Deploying AI safely and effectively in business and organizational contexts.
5 tips in this topic
Create a rubric before deploying. Define what "good" looks like with specific criteria: accuracy, tone, completeness, safety. Have humans rate a sample of outputs weekly. Track trends over time. Don't just measure speed—measure quality.
Three main risks: data leakage (sensitive info in prompts going to third parties), hallucination liability (AI confidently stating false information), and shadow AI (employees using unapproved tools). Address each with policies, not just technology.
Prompt injection is when user input overrides your system instructions. Prevent it by: separating user input with clear delimiters, validating inputs, using the system message for instructions (not user message), and never trusting user input to be benign.
Layer defenses: ask the model to cite sources, implement fact-checking against known databases, flag low-confidence responses for human review, and set clear expectations with users that AI can make mistakes. Never fully automate high-stakes decisions.
Use GPT-4 for prototyping and diverse tasks. Fine-tune smaller models when you have consistent, narrow use cases with thousands of examples. Fine-tuned models are cheaper to run but expensive to train and maintain. Start with GPT-4, only fine-tune if cost becomes prohibitive.