Metro Smart Cities: real-time video analytics for airports

We came out of a university AI lab (see the other work) and entered a public tender for real-time deep-learning analysis over the CCTV network of Argentina's international airports. The other bidders were multinational integrators (EXA among them) proposing legacy systems; we proposed an AI-based architecture and won. Delivered to production in a year. The customer was Aeropuertos Argentina 2000, and the first deployment was Ezeiza International Airport.

The constraints: air-gapped. Sub-second latency from camera to detection. And the one that shaped everything: two physical servers.

One more constraint came from the commercial pitch: no new hardware. We ran on the cameras the airport had already installed, integrated with their existing Avigilon (Motorola Solutions) system. That is what removed the capital expense for the customer, and it is also why the hard problems in this system are the ones below. We did not get to pick the cameras, their placement, or their lenses.

~400 fisheye cameras plus three 30MP bullet cameras, all RTSP. A fisheye camera covers a whole hall, so one physical stream needs to become several logical viewports, each dewarped and analyzed independently. More logical streams than physical cameras, on hardware that was never going to grow. Roughly 200 cameras per server, and more logical streams than that.

No off-the-shelf stream processor fit. Spark's latency model is wrong for this. NiFi isn't customizable at the frame level. Flink is a different shape of problem. I tried each and built our own.

Physical/logical decoupling. One camera row fans out to N VideoStream(x, y, hfov) rows: viewports off a single RTSP source. Everything downstream operates on logical streams.

The thread pool, the core of the system.

A worker pool where each thread owns a slice of streams and round-robins them in a tight while(true) { grab() } loop, no sleep anywhere. The ratio that made it work was 15 streams per thread, found empirically starting from 1:1. Each stream carries a 15-slot ring buffer of grab outcomes; success rate below 0.8 marks it down, and a watchdog force-disconnects anything unread for 20 seconds.

The grab/retrieve split: the decision I'd keep. The worker thread only calls capture.grab(), which advances the stream without decoding. Decoding (retrieve()) happens on the gRPC request thread, only when a frame is actually asked for, against a VideoCapture with BUFFERSIZE=1. So ingestion cost scales with cameras; decode cost scales with demand. That asymmetry is what let 400 cameras fit on two machines.

Request path:

retrieve() → Mat → byte[] + BGR↔RGB swizzle → BufferedImage → dewarp → JPEG

Deployment.

24 services on Docker Swarm across two nodes, one of them the manager, 88 CPU and 133 GB declared: ZooKeeper ×3, Kafka ×3, schema registry ×2, Redis, Postgres, four videostream services at 12 CPU / 10 GB each (the heaviest boxes on the diagram), two videostream-gateway, two map-service, six PyTorch model servers pinned three per node, plus gateway, notification and metrics services. map-service is the key-value store that holds the state of every parking space in memory; the metrics engine does both the aggregated and the real-time cut, and the dashboard is the front end and back end over that. Data plane: cameras → videostream → gateway → map-service → model-pytorch, with Kafka fanning to metrics and notifications. Bare-metal DMZ on site; hybrid deployments for customers who allowed it.

Stack: Scala/JVM, OpenCV via JNI, gRPC, Kafka, PyTorch, BoofCV for dewarping, Docker Swarm, RTSP off the airport's Avigilon installation.

Won the tender against multinational incumbents. Shipped to production in Argentina's international airports, air-gapped. Team grew to ~10. Inbound interest from airports worldwide. Presented by Argentina's president and its minister of transport, and covered in the national press, including a piece in La Nación. The plan from there was the rest of the AA2000 group: 54 airports.

We made a large bet and we hit it. The no-new-hardware pitch was not a detail, it was the reason we won a tender against multinational integrators, and it committed us to making deep learning work in real time on somebody else's cameras, on two machines, with no network out. We delivered that. We delivered it days before the country went into full pandemic lockdown.

Across 2019 the computer vision systems processed on the order of 100 billion CCTV frames, about 270 million a day.

As of the February 2020 release, Ezeiza covered 1,800 covered and 3,500 uncovered parking spaces off roughly 200 cameras that were already on the walls; the deployment grew from there, which is why the camera count above is larger. The customer success story bills it as the first airport in the world with a real-time computer vision parking system, which is the company's claim rather than mine, but I have not found an earlier one.

Wound down in 2022: COVID eliminated airport and airline revenue and there was no adjacent market to pivot into on our timeline.

What I would do differently is not technical. Once the airports closed there was no demand to engineer against, and the right move was to stop spending: let the team go, myself included, take other jobs, and wait the pandemic out with the company intact. I kept building instead. In my defense the pandemic was genuinely unprecedented and I do not think the timing was predictable. The decision after it became clear was mine, and I was too slow.

The deeper gap was that we were engineers running a company. We did not know how to sell, how to run customer discovery, or how to raise, and all three were well outside where we were comfortable. I went and learned the last one properly. That shows up in Autonoma, which raised out of Argentina from one of the best funds in the world and from the people who back developer tools for a living.

The whole architecture, and the reason is that I had just left university. I believed microservices and Uncle Bob were settled truth rather than one set of tradeoffs among several, so the MVP went out as a microservices architecture that nothing about the problem required. Twenty-four services on two machines. Everything below this line is downstream of that one decision. It was an expensive lesson and it is the one that made me a better engineer: estimate the load first, then design for the load you estimated. Designing for maximum scale you do not have is not caution, it is a cost you pay every day in a system nobody can hold in their head.

The viewport blob. The dewarp viewport was Java-serialized straight into a Postgres bytea column; rows begin with \xACED0005, the JVM serialization magic. A JVM-only binary in a database that should have been readable by anything. The git log shows the repair: a JSON serializer, then a migration class. Should have been JSON from day one.

Nine encode/decode hops, two of them pure waste. The videostream service writes a JPEG with ImageIO.write; the gateway, one hop later, immediately reads it back with ImageIO.read to do work it could have done on the raw bytes, then base64-encodes it for the controller. Two adjacent services, encode then decode, for nothing.

The JNI tax. Crossing from OpenCV into Java with no zero-copy path meant a hand-written per-pixel BGR↔RGB swizzle on every frame. Plus one Mat, one byte[] and three BufferedImage allocations per frame per stream; the GC pressure was visible.

Thread-safety that cost the cache. The undistorter rebuilt its transform on every frame: FactoryDistort.distort(true, ...), where true is BoofCV's cache the LUT flag, so the 600×600 lookup table was recomputed each call. The stateful Scala original cached correctly but wasn't thread-safe; the thread-safe Java rewrite threw the cache away. Same algorithm, opposite tradeoff, and I picked wrong.

Three load balancers on one call path. nginx grpc_pass, a lookaside balancer with gRPC Next / HealthStream, and a RoundRobinServiceElection inside the gateway. Each added for a real reason at the time; together they're redundancy that nobody could explain.

Today I'd build it as: one process per stream, ffmpeg emitting raw frames, decode at request time, in Rust. No JNI boundary, no swizzle, no GC. A monolith on k3s rather than 24 services on two machines.