{"id":157,"date":"2026-03-08T23:07:47","date_gmt":"2026-03-08T23:07:47","guid":{"rendered":"https:\/\/blog.rebalai.com\/en\/2026\/03\/08\/building-production-ready-ai-pipelines-lessons-fro\/"},"modified":"2026-03-18T22:00:08","modified_gmt":"2026-03-18T22:00:08","slug":"building-production-ready-ai-pipelines-lessons-fro","status":"publish","type":"post","link":"https:\/\/blog.rebalai.com\/en\/2026\/03\/08\/building-production-ready-ai-pipelines-lessons-fro\/","title":{"rendered":"Building Production-Ready AI Pipelines: Lessons from Running 10K+ Generations"},"content":{"rendered":"

It was a Tuesday morning when I opened our Datadog dashboard and saw 847 silent failures from the previous night’s batch job. No alerts. No exceptions in our logs. Just a queue that had quietly eaten thousands of tokens and returned nothing useful. Our pipeline had been “succeeding” in the<\/a> sense that it wasn’t throwing errors \u2014 it was just producing garbage and writing it to the database like everything was fine.<\/p>\n

That was month two of running LLM-powered features in production<\/a>. I thought I had it figured out by then. I did not.<\/p>\n

Over the past eight months, on a three-person team, I’ve pushed somewhere north of 10,000 generations through production<\/a> pipelines \u2014 across Claude 3.5, GPT-4o, and a brief, regrettable experiment with a self-hosted Mistral instance that I will get to. Here’s what I actually learned<\/a>, as opposed to what the<\/a> documentation implied I would need to care about.<\/p>\n

Retry Logic Is a Trap If You Do It Wrong<\/h2>\n

Every guide tells you to implement retries. What they don’t tell you<\/a> is that naive exponential backoff will bankrupt you during a rate limit storm, and that retrying on the wrong error codes will just make your problems worse, faster.<\/p>\n

My first implementation looked roughly like this:<\/p>\n

import time\nimport random\n\ndef call_llm_with_retry(prompt, max_retries=3):\n    for attempt in range(max_retries):\n        try:\n            response = client.messages.create(\n                model="claude-3-5-sonnet-20241022",\n                max_tokens=1024,\n                messages=[{"role": "user", "content": prompt}]\n            )\n            return response.content[0].text\n        except Exception as e:\n            if attempt == max_retries - 1:\n                raise\n            wait = (2 ** attempt) + random.uniform(0, 1)\n            time.sleep(wait)\n<\/code><\/pre>\n
Looks fine, right? The problem is that broad except Exception<\/code>. I was retrying on context length errors (HTTP 400) \u2014 deterministic failures where no amount of waiting fixes a prompt that’s 2,000 tokens over the limit. I was also retrying on content policy rejections. And on malformed JSON responses from my own parsing layer, which weren’t even API errors.<\/p>\n
After a particularly bad Friday afternoon deploy<\/a> where this pattern caused a cascade of 400 errors that chewed through our rate limit budget retrying requests that were never going to succeed, I got specific:<\/p>\n
import anthropic\nimport time\nimport random\nimport logging\n\nRETRYABLE_STATUS_CODES = {429, 500, 502, 503, 529}\n\ndef call_llm_with_retry(prompt: str, max_retries: int = 4) -> str:\n    last_exception = None\n\n    for attempt in range(max_retries):\n        try:\n            response = client.messages.create(\n                model="claude-3-5-sonnet-20241022",\n                max_tokens=1024,\n                messages=[{"role": "user", "content": prompt}]\n            )\n            return response.content[0].text\n\n        except anthropic.RateLimitError as e:\n            # 429 \u2014 back off hard, respect Retry-After header if present\n            retry_after = getattr(e, 'retry_after', None)\n            wait = retry_after if retry_after else (2 ** attempt) * 2 + random.uniform(0, 2)\n            logging.warning(f"Rate limited. Waiting {wait:.1f}s (attempt {attempt + 1})")\n            time.sleep(wait)\n            last_exception = e\n\n        except anthropic.APIStatusError as e:\n            if e.status_code not in RETRYABLE_STATUS_CODES:\n                # 400, 401, 403, 404 \u2014 these won't get better with retries\n                logging.error(f"Non-retryable API error {e.status_code}: {e.message}")\n                raise\n            wait = (2 ** attempt) + random.uniform(0, 1)\n            time.sleep(wait)\n            last_exception = e\n\n        except anthropic.APIConnectionError as e:\n            # Network issues \u2014 retry with backoff\n            wait = (2 ** attempt) + random.uniform(0, 1)\n            time.sleep(wait)\n            last_exception = e\n\n    raise last_exception\n<\/code><\/pre>\n
The separation between retryable and non-retryable errors cut our wasted API spend by about 30% in the<\/a> first week. Not because we were hitting tons of 400s \u2014 we weren’t \u2014 but because when we did, they were expensive ones (long prompts) and we were burning budget retrying them six times.<\/p>\n
Practical takeaway: type your exceptions. If you’re using Anthropic’s SDK, anthropic.RateLimitError<\/code>, anthropic.APIStatusError<\/code>, and anthropic.APIConnectionError<\/code> are distinct and should be handled differently. Same pattern applies to OpenAI’s SDK.<\/p>\n
The Cost Math Will Surprise You<\/h2>\n
I thought I had a handle on costs. Did input\/output token estimates, built a little calculator, felt confident. Then I saw the actual bill.<\/p>\n
The issue wasn’t the per-token cost. It was everything I hadn’t accounted for: tokens burned on retries, on failed generations, on the system prompt I was including on every single request<\/em> even when most of those requests didn’t need the full context. My system prompt was 847 tokens. Across 10,000 requests, that’s 8.47 million tokens of input just for boilerplate.<\/p>\n
So I started being deliberate about prompt architecture. Short context = shorter system prompt. A simple classification task doesn’t need the five-paragraph system prompt I wrote for open-ended generation. I built a prompt registry \u2014 nothing fancy, just a dict of prompt templates keyed by task type \u2014 and matched prompt complexity to task complexity.<\/p>\n
One thing I noticed that genuinely surprised me: batch processing doesn’t just save money on some APIs, it changes your failure mode profile entirely. With synchronous requests, latency spikes cause timeouts and downstream failures. With batch, the failure shows up hours later when you check results. Both are annoying; they’re annoying in different ways, on different schedules. For our<\/a> async summarization jobs, batch made sense<\/a>. For anything user-facing, obviously not.<\/p>\n
Also: watch your output token limits. I was setting max_tokens=4096<\/code> on everything out of habit. The model doesn’t charge you for tokens it doesn’t use, but it holds a connection open while generating. For tasks that reliably produce short outputs, tighter limits improve throughput and catch runaway generations early.<\/p>\n
Observability: What I Actually<\/a> Watch<\/h2>\n
Before shipping, I imagined needing detailed traces of every reasoning step. What I actually<\/a> monitor day-to-day is much simpler and more boring.<\/p>\n
The signals that matter in my setup:<\/p>\n