What Happens When Your Only LLM Provider Goes Down?

The first time our only LLM provider had an outage mid-production, I learned this lesson the hard way. Everything just stopped.

After that, working across GPT-4, Claude, and Gemini for clients at Thunder Marketing and Sazag Infotech, the real lessons weren't about prompting. They were about reliability, cost, and not depending on any single vendor.

Why one provider is a bad bet

OpenAI has had real outages. Rate limits hit you when you least expect it. Different providers are genuinely better at different things, Claude handles long context better, GPT-4 is often stronger at reasoning.

So we built a thin abstraction that tries providers in order and falls back automatically.

1class MultiProviderLLM:
2    def __init__(self):
3        self.providers = {
4            LLMProvider.OPENAI: OpenAIClient(),
5            LLMProvider.ANTHROPIC: AnthropicClient(),
6            LLMProvider.GOOGLE: GoogleClient(),
7        }
8        self.fallback_order = [
9            LLMProvider.OPENAI,
10            LLMProvider.ANTHROPIC,
11            LLMProvider.GOOGLE,
12        ]
13
14    async def complete(self, prompt: str, preferred_provider=None, **kwargs) -> str:
15        providers = (
16            [preferred_provider] + self.fallback_order
17            if preferred_provider else self.fallback_order
18        )
19
20        for provider in providers:
21            try:
22                return await self.providers[provider].complete(prompt, **kwargs)
23            except (RateLimitError, ServiceUnavailable) as e:
24                logger.warning(f"{provider} failed: {e}")
25                continue
26
27        raise AllProvidersFailedError()

Which provider for which job

From actual production use: GPT-4 for complex reasoning, Claude when the document is long (that 200K context window matters), either one for code, a cheaper fast model like Gemini Flash for simple stuff, and GPT-4V or Claude 3 for anything with images.

1def select_provider(task: Task) -> LLMProvider:
2    if task.requires_vision:
3        return LLMProvider.OPENAI
4    if task.context_length > 100_000:
5        return LLMProvider.ANTHROPIC
6    if task.complexity == "simple":
7        return LLMProvider.GOOGLE
8    return LLMProvider.OPENAI

Keeping the bill under control

LLM costs sneak up on you fast if you're not careful. Three things helped a lot.

Caching repeated prompts, since a surprising number of prompts repeat:

1async def complete(self, prompt: str, **kwargs) -> str:
2    cache_key = hashlib.sha256(f"{prompt}:{kwargs}".encode()).hexdigest()
3    cached = await self.cache.get(cache_key)
4    if cached:
5        return cached
6
7    result = await self.llm.complete(prompt, **kwargs)
8    await self.cache.setex(cache_key, 3600, result)
9    return result

Sending simple tasks to cheaper models instead of the expensive one by default:

1async def smart_complete(prompt: str, task_type: str) -> str:
2    if task_type in ["classification", "extraction", "simple_qa"]:
3        return await gpt35_client.complete(prompt)
4    if task_type in ["summarization", "translation"]:
5        return await claude_instant_client.complete(prompt)
6    return await gpt4_client.complete(prompt)

And just writing shorter prompts. Verbose, polite prompts cost more tokens for no real benefit.

1# Wastes tokens on politeness
2prompt = """You are a helpful assistant that extracts information...
3Please be thorough and accurate. Here is the document: {document}"""
4
5# Same result, fewer tokens
6prompt = """Extract names, dates, and amounts from this document:
7{document}
8Return as JSON: {{"names": [], "dates": [], "amounts": []}}"""

That last change alone cut our token usage by 30%.

Making the output actually reliable

Asking an LLM to "just write JSON" and hoping is not a strategy. Force the structure with a parser.

1class ExtractedData(BaseModel):
2    names: list[str]
3    dates: list[str]
4    amounts: list[float]
5
6parser = PydanticOutputParser(pydantic_object=ExtractedData)
7response = await llm.complete(prompt + parser.get_format_instructions())
8data = parser.parse(response)

Wrap every call with retries and backoff, since transient failures happen more than you'd think.

1@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=60))
2async def robust_llm_call(prompt: str) -> str:
3    return await llm.complete(prompt)

And track everything, requests by provider and status, latency, and token usage, so you actually see problems before your users complain about them.

What this bought us

85% task automation accuracy in production, 99.5% availability thanks to the fallbacks, 40% lower cost from caching and tiered models, and no lock-in to a single vendor.

None of this is exciting engineering. But it's the difference between an AI feature that works reliably and one that quietly breaks the day your main provider has a bad afternoon.

FAQ

Should I use just one LLM provider in production?

I wouldn't. Outages and rate limits happen. A fallback chain across two or three providers costs little and saves you on a bad day.

How do you keep LLM costs under control?

Cache repeated prompts, send simple tasks to cheaper models, keep prompts short. That last one alone cut our tokens by 30%.

How do you get consistent output from an LLM?

Use a structured output parser like Pydantic so the response has to match a schema, instead of hoping the model formats things correctly.

Building with LLMs? Let's connect on LinkedIn or explore my projects on GitHub.