Sign in Subscribe

From Raw Data to Production LLM: An End-to-End ML Workflow on Nebula Block

Tracy Giang

• 4 min read
Share
From Raw Data to Production LLM: An End-to-End ML Workflow on Nebula Block
From Raw Data to Production LLM: An End-to-End ML Workflow on Nebula Block

Most explanations of an ML workflow stop at the model card. That's the easy part.

The real work — turning raw files into usable data, training a model on actual hardware, checking it's good enough, deploying it somewhere your product can call, and watching what happens once real traffic hits it — is where the engineering time and the budget actually go.

This post walks through that full ML workflow as it plays out in practice, end to end. Start: a folder of messy files on someone's laptop. Finish: an endpoint your production app is calling.

None of the individual steps are exotic. What's different here is that every step happens inside the same Canadian infrastructure boundary — one bill, one access model, no data crossing borders along the way.

Stage 1: Land the Raw Data

Every ML project starts as a mess: PDFs, CSVs, scraped HTML, support transcripts, whatever your source systems spit out.

Step one is getting that mess into Object Storage — Nebula Block's S3-compatible layer — organized into buckets the rest of the pipeline can predictably address. A common pattern: raw/, processed/, checkpoints/.

Because the storage layer speaks standard S3, this usually just means pointing your existing ingestion script at a new endpoint and credential pair. A boto3 upload loop, an AWS CLI sync, a Cyberduck drag-and-drop for smaller datasets — no rewrite needed.

If you're pulling from internal systems (a CRM export, a document management tool), this is also your last chance to redact or anonymize before the data leaves its source. Everything downstream treats what lands here as ground truth.

Stage 2: Process and Prepare Your Data

Raw files aren't training data yet.

A GPU Instance — an L40S or A100 is usually plenty here, saving the bigger H100/H200 cards for actual training — mounts the raw bucket and runs cleaning, deduplication, tokenization, and chunking. The processed dataset gets written back to its own bucket.

For supervised fine-tuning, this is where instruction-response pairs get assembled and checked. For retrieval-style projects, it's where documents get chunked and embedded — BGE Large EN v1.5 is a solid default, available through the Inference API without standing up your own embedding server — before landing in a vector store.

Keep this stage on its own instance, separate from training. That way you can re-run preprocessing experiments — different chunk sizes, different cleaning rules — without burning expensive GPU-hours on a card that's mostly waiting on CPU-bound text work.

Stage 3: Train or Fine-Tune the Model

This is the stage GPU instances exist for.

Depending on the project, that might mean:

  • Full fine-tuning of a smaller open model
  • Parameter-efficient fine-tuning (LoRA or QLoRA) on a larger one
  • Continued pretraining on domain-specific text

H100 and H200 instances handle the heavy end. A100 and L40S cover smaller models or tighter budgets.

The instance pulls its training set straight from Object Storage and checkpoints weights back out periodically — rather than leaving them stranded on local disk, where one instance failure could cost you hours of compute.

Since the instance is a real VM or bare-metal box with SSH access, you're not locked into one training framework. PyTorch, Hugging Face's stack, or a custom training loop all run the same way they would in a server room — you're just renting the hardware by the hour instead of owning it.

Stage 4: Evaluate Before You Trust It

A checkpoint isn't a model until someone's confirmed it actually beats what it's replacing.

This stage usually runs on the same GPU instance, or a smaller one:

  • Holdout evaluation sets
  • Task-specific benchmarks
  • An LLM-as-judge pass — using a larger hosted model like Llama 3.3 70B or DeepSeek-V3 through the Inference API to score outputs against a rubric
  • A human reviewer spot-checking the results

Anything that doesn't clear the bar goes back to Stage 3 with adjusted hyperparameters or a revised dataset. Anything that does gets tagged and promoted to a "candidate" prefix in Object Storage — separate from the working checkpoints generated mid-training.

Stage 5: Serve the Model in Production

A checkpoint sitting in a bucket isn't in production. You've got two real options here.

The lighter path: deploy behind a serverless endpoint that mounts straight from Object Storage. No separate upload step. Works well for models that fit the managed serving pattern.

The heavier path: stand up a dedicated GPU Instance running the model directly — vLLM or a similar serving framework — behind a private API with your own authentication. This is the right call when you need guaranteed dedicated throughput, or the workload has to stay fully inside a defined network perimeter instead of touching a shared endpoint.

[link: compare serverless vs. dedicated GPU deployment]

Stage 6: Watch It, Then Loop Back to Stage 1

Production traffic is itself a dataset.

Logged queries, low-confidence responses, and user corrections are exactly the raw material Stage 1 started with — just generated by the model instead of collected from a source system. They belong in the same Object Storage bucket structure, ready to feed the next fine-tuning round.

This ML workflow isn't a straight line. It's a loop. And the real value of keeping every stage on one infrastructure layer is that closing the loop doesn't mean shipping data across a border every time it comes back around.

Learn more at


Sign up for more like this.

Enter your email
Subscribe