AI Systems Lab

One URL shortener, pushed until it breaks.

One backend, 22 measured stages. Then six AI systems, from scratch.

See the stages First numbers

  • ScaleLab

    Now on S3, docker.

    S

    A URL shortener platform, pushed further at each stage and load-tested after every one.

    In progress
    3 of 22 3 of 22 stages done

  • Research Agent

    A1

    Several AI agents research a question and write a sourced report.

    Not started
    0 of 5 0 of 5 stages done

  • Advanced RAG Engine

    Now on A2.1, ingestion.

    A2

    Ask questions over your documents and get answers with citations.

    In progress
    0 of 5 0 of 5 stages done

  • LLMOps Lab

    A3

    Traces, tests and monitors the Research Agent and the RAG Engine.

    Not started
    0 of 4 0 of 4 stages done

  • Model Lab

    A4

    Fine-tune, compress and serve an open-weight model.

    Not started
    0 of 5 0 of 5 stages done

  • AI Governance Gateway

    A5

    A security layer in front of the LLM and its tools, with an attack test suite.

    Not started
    0 of 4 0 of 4 stages done

  • Multimodal Studio

    A6

    Image, voice, audio and video analysis.

    Not started
    0 of 4 0 of 4 stages done

22 stages, 3 done

Each stage adds one layer to the same URL shortener: workers, a load balancer, a cache, queues, Kubernetes.

  1. Foundation

    Done
    S0

    Python package, Postgres 17 schema for links and clicks, a migration runner, 35 tests.

    s0-foundation

  2. Links API

    Done
    S1

    FastAPI CRUD and redirect, one error format, timeouts, health checks, graceful shutdown, 82 tests.

    s1-links-api

  3. Load testing

    Done
    S2

    k6 suite of six tests, Locust users, a resource monitor. First run on a busy machine: median 14.6 ms at 200 req/s, no failures up to 600 req/s, breaks between 700 and 750.

    s2-load-testing

  4. Docker

    In progress
    S3

    Multi-stage image, compose stack and container hardening are being built.

  5. ScalingS4Planned
  6. DatabaseS5Planned
  7. CachingS6Planned
  8. Rate limiter serviceS7Planned
  9. Auth and API securityS8Planned
  10. Notification serviceS9Planned
  11. Chat serviceS10Planned
  12. File serviceS11Planned
  13. Search serviceS12Planned
  14. Analytics pipelineS13Planned
  15. Architecture splitS14Planned
  16. Distributed patternsS15Planned
  17. Kubernetes (k3s)S16Planned
  18. CI/CDS17Planned
  19. AWS and TerraformS18Planned
  20. ObservabilityS19Planned
  21. ReliabilityS20Planned
  22. NetworkingS21Planned

First numbers, not final ones

Provisional. Measured once while other work used about half of the machine's CPU. The suite will be re-run on a quiet machine.

691 req/s

Achieved at the 700 req/s step of the breakpoint test, the last step before 1% or more of requests failed. p95 there was 388 ms. At 750 req/s, 10.99% of requests failed.

k6 against the S1 API on one machine: Gunicorn with 4 Uvicorn workers and Postgres 17, with the load generator on the same host.

Requests per second: achieved against target

p95 latency at each step, in milliseconds

  • Measured
  • Target rate
  • Step where 1% or more of requests failed
Breakpoint test, by target rate in requests per second. Each step holds its target rate for 20 seconds. k6 gives up on a request after 5 seconds, so latency cannot read much higher than that.
Show the 24 steps as a table
Target req/sAchieved req/sp50p95FailedDropped
505013.6 ms41.3 ms0.00%0
10010014.2 ms39.2 ms0.00%0
15014916.0 ms691 ms0.00%22
20020017.9 ms67.7 ms0.00%0
25025017.3 ms74.7 ms0.00%0
30028534.8 ms941 ms0.00%294
35034826.7 ms383 ms0.00%40
40040020.7 ms66.8 ms0.00%0
45041342.8 ms988 ms0.00%742
50050045.9 ms232 ms0.00%0
55054943.8 ms208 ms0.00%0
600546132 ms786 ms0.00%1,082
650601272 ms694 ms0.00%975
700691137 ms388 ms0.00%160
750408325 ms5,097 ms10.99%6,840
8003671,350 ms3,057 ms3.27%8,621
850720572 ms1,137 ms0.00%2,576
900747363 ms1,095 ms0.00%3,031
950686796 ms1,609 ms0.00%5,259
1,000758741 ms1,379 ms0.00%4,818
1,050620743 ms2,641 ms0.13%8,548
1,100742817 ms1,708 ms0.00%7,146
1,1508181,847 ms4,375 ms3.13%6,617
1,2006742,913 ms5,001 ms7.78%10,474
The standard suite, whole-run figures
TestTrafficLengthRequestsAverage req/sp50p95Failed
BaselineLight, steady traffic1.0 min2,9995011.7 ms41.5 ms0.00%
LoadExpected normal traffic4.5 min44,91916614.4 ms113 ms0.00%
StressSteps up to three times normal5.0 min118,36739437.6 ms514 ms0.00%
SpikeA sudden burst to three times normal3.2 min26,97514233.1 ms637 ms0.00%
SoakModerate traffic for a long time20.0 min179,60915012.6 ms101 ms0.00%
BreakpointClimbs in small steps until it breaks8.1 min232,444481408 ms2,944 ms1.17%

How a stage closes

A git tag
Each finished stage is committed and tagged, so any step can be checked out and compared.
The same load suite
After S2, every stage re-runs the same k6 profiles and is compared with the stage before.
Written notes
What was tried, why, and what happened, kept in the repo next to the code.