Cairn: real-time surface reconstruction on a phone

Objective

This project came from a discussion with a friend who does projection mapping and graphics work for a living. We were talking about whether there could be a non-proprietary way for someone to use a drone or an Android smartphone to map a surface with real-time feedback and results, without using specialised hardware such as depth cameras or LiDAR.

This was the best kind of nerd sniping, the one I could actually do something about, so I decided to build a small prototype.

I wanted a live 3D surface of whatever a camera is pointed at, built on the device holding the camera. No SAAS, no depth sensor, no desktop processing. Point camera, geometry. Since I don’t have access to a drone that I could crash, I decided to build a prototype on my phone.

The two targets overlap, but the drone brings difficulties of its own. Projection mapping needs the shape of the facade or wall before it can warp light onto it, and for anything larger than an indoor room, you cannot get that shape from the ground. You have to fly a camera up the building face, across the walls, around the corners, close enough that the details that affect projection quality survive into the model. All in all, the reconstruction problem is the same, just at a different range and a much larger surface area. What the two share is the need to reconstruct on the device, live. If the operator cannot see the model filling in while flying, they cannot tell which part of the facade was missed until they have landed and processed it, at which point the answer is another flight, while you rue the day you ever got a drone.

There are many techniques that would produce excellent reconstructions with a desktop GPU and a few minutes per scene, but my constraints are real time, one camera, on a phone or a small Pi computer, using only general-purpose sensors. Every choice below follows from that.

Representation: Gaussian surfels

A surfel (“surface element”, first described in Pfister et al., SIGGRAPH 2000) is the smallest useful piece of surface, a small disk with a position, a normal and a radius. A Gaussian surfel adds a Gaussian falloff toward the rim, so the disk is solid in the middle and fades out at the edge. That makes it the flat, oriented version of the fuzzy 3D blobs used by 3D Gaussian Splatting (3DGS; Kerbl et al., SIGGRAPH 2023), where a scene is represented as a few million such blobs and drawn by projecting each one onto the screen.

I’m basing this work on two papers, High-quality Surface Reconstruction using Gaussian Surfels (Dai et al., SIGGRAPH 2024) and When Gaussian Meets Surfel (GES for short; Ye et al., SIGGRAPH 2025). The splat maths itself comes from 2D Gaussian Splatting (2DGS; Huang et al., SIGGRAPH 2024), specifically the explicit form for the location of ray-splat intersections (don’t worry, we’ll get to it later on. Or not, if you decide this isn’t worth your time).

Surfels suit this problem for two reasons. First, a disk with a position and a normal is already a surface description, which is what projection mapping consumes directly. A radiance field, as used in splatting for instance, describes how the scene looks from every direction, and you would have to do some raymarching to find out where the wall actually is. Second, a flat opaque splat draws as two triangles under the GPU depth test, which keeps the nearest thing at each pixel and discards the rest. That matters more on a phone than on a desktop card, and illustrates why surfels are a somewhat natural representation for this problem. Mobile GPUs are tile-based, which means geometry is sorted into smaller screen tiles first, then each tile is shaded with its colour and depth sitting in fast memory next to the shader core. A fragment that is filtered by the depth test never gets shaded at all, while a transparent one has to be shaded and composited in order, so you pay for every layer of overlap and you sort the splats by depth every frame. Keeping them opaque avoids both.

There are other methods to accomplish the goal of this project, which make up the bulk of the literature. For instance, the seminal paper KinectFusion (Newcombe et al., ISMAR 2011) fuses depth maps into a truncated signed distance function (TSDF), creating a voxel grid where each cell stores its distance to the nearest surface, so the surface is wherever that distance crosses zero. SDFs are quite lovely, as anyone who has gazed in awe at Inigo Quilez’s articles knows, and I’ve had my battles with them, but they would not be ideal for this project. As mentioned before, a TSDF has to go through marching cubes or raymarching before it becomes pixels, and that extraction step is something that I just couldn’t figure out how to reliably do in real time, even if it might produce more accurate and stable results.

The implementation is Rust on both sides of a JNI boundary (JNI being Android’s bridge between Java/Kotlin and native code). I chose Rust for no real reason other than the fact that I like rust-gpu¹, which compiles Rust to SPIR-V, one of the shader formats Vulkan consumes, so the GPU kernels and the engine around them are the same language. A libhost.so engine handles tracking, pose, the surfel cloud and the Vulkan renderer. A thin Kotlin app handles the camera, the sensors and depth inference, which lives on that side because the Rust bindings for the inference runtime do not cross-compile to Android.

I decided later in the course of the project that the depth model should be the only thing that creates geometry, and that the phone’s inertial sensors (the IMU: gyroscope, accelerometer, magnetometer) should only assist with pose. I hadn’t considered whether sensor input would be all that necessary at first, but it’s what got me to a working demo. More on that under the pose section.

Depth: Depth Anything V2

You cannot get metric geometry (depth in actual metres, rather than relative) out of a single photograph without a prior on what the world tends to look like. This might seem obvious, but it’s a major topic of research. Monocular depth estimation is the part of this pipeline where I was most grateful for the fine work that researchers do all over the world, and where I found something immediately helpful. Depth Anything V2 (Yang et al., NeurIPS 2024) generalises well, has a metric variant and exports to ONNX, which is the universal format for neural network graphs. The phone runs that file with ONNX Runtime, through its onnxruntime-android package. I shudder to think how I would build the geometry to the required degree of accuracy without this model.

The model is a ViT (Vision Transformer), which in grug speak means the image is cut up and tokenised. ViTs can be pretty hefty, but three fairly standard techniques cut the inference time down to something useable:

1. Make it small. Because every token is compared with every other, attention grows with the square of the token count, and the rest with the count itself. Depth Anything V2 uses 14-pixel patches, so the default 518×518 input corresponds to a 37×37 grid (1369 tokens). I export the model at 252×252, creating an 18×18 grid (324 tokens), which comes out roughly 7× cheaper. At the original size a pass took several seconds, which would not be acceptable here.
2. Make it quantised. The weights are stored as 8-bit unsigned integers instead of 32-bit floats. On the phone, at 252×252 with two threads, a pass went from 499 ms to 273 ms, and against the float model depths come out 1.26% off on average. Not all of that error contributes equally. A uniform stretch, the whole scene reading nearer or further by the same proportion, is divided out, since the engine re-fits a scale and an offset against its own map every frame. Only the latter affects the result in the long-term.
3. Make it run on a separate thread, duty-cycled. The next inference starts only once as much time has passed as the last one took, so depth uses about half a core and a slow pass never stalls tracking and rendering.

I’m using a Redmi Note 14, with a Mali-G57 GPU. I spent a long time trying to figure out why inference was taking so long, even after quantisation and reducing the resolution. The answer was NNAPI, the Android Neural Networks API, which I was routing the model through.

NNAPI hands the model to whatever accelerator the phone exposes, be it a dedicated neural unit, a CPU, or the GPU. It can only reach hardware whose manufacturer has published a driver for it, and this phone has none, so with nothing to delegate to, it ran the model on its own CPU kernels. Those are slower than ONNX Runtime’s, and just not adding NNAPI to the Ort session setup took 623 ms off every depth pass on the quantised model, which was just a ridiculous loss in performance. This could have been detected much more quickly if I had remembered to check. Assuming makes an ass of you and me, and other such aphorisms.

This means that the Mali only ever rasterises surfels. It just sits there during inference, taunting me with its non addressability. I could just switch over to a different runtime, such as LiteRT or ExecuTorch. I have not tried either in this project, but I’m not sure if the headroom is there, since the shader cores are already drawing the map every frame. This is something to experiment with moving forwards.

I ship two calibrations of the model, indoor and outdoor, because absolute scale and depth quality differ a lot between those domains. The app picks between them on the first frame, by checking what fraction of that depth map reads beyond 12 m, and then keeps that choice for the session, since swapping calibrations halfway would shift the map’s scale under it. Collapsing the two into a single model is on the to-do list.

Fusion: from depth maps to surface

Each pass of the depth model gives one depth map, a single set of distances, measured from wherever the camera happened to be at that instant. Fusion is the step that merges those multiple maps into a unique map, based on the world coordinates rather than the camera’s, so that a wall seen from four angles ends up as one wall instead of four overlapping copies of it. Not every frame gets fused, since depth is much slower than the camera and some frames are thrown out by the gates below.

Every fused frame, the depth map is turned into 3D points. A pixel and the camera’s focal length and image centre give a ray out into the world, and the depth value says how far along that ray the surface sits. Every fourth pixel is used, and each resulting point either updates a surfel the map already holds or becomes a new one.

Scale is the difficult part. Monocular depth is only defined up to an unknown scale and shift, and the model’s estimate wanders frame to frame. So every frame I fit a correction against the geometry the cloud already holds, which means scale comes from the map rather than from the model’s raw output. That fit happens in inverse depth $1/z$, which is the space the model is actually linear in². It is smoothed with an exponential moving average (each new fit is blended 30% into the running one) so a single bad frame cannot smear the surface, and it falls back to the last good alignment when a fit fails.

Two gates keep obviously bad data out, since anything fused into the map stays there permanently:

• Motion blur gate. If the gyroscope has exceeded 1.5 rad/s (about 86°/s) at any point since the last fused frame, the frame is treated as too blurred to trust and skipped for fusion, while tracking and rendering continue through it. This is a somewhat arbitrary threshold, which I arrived at after experimenting with different values.
• Flying-pixel rejection. “Flying pixels” is the depth-camera term for spurious points suspended in mid-air at object boundaries. Monocular models spike at depth discontinuities, so a pixel whose neighbours all sit on the far side of a discontinuity is discarded rather than fused and carved back out later.

As previously mentioned, depth runs asynchronously and can lag by a second or more, so the engine remembers the pose of the frame whose pixels went to the model and fuses the result there, rather than wherever the camera has wandered to by the time it comes back. Fusing it at the current pose instead smears the geometry across everywhere the camera went while it was thinking.

Rendering: cheap surfels on mobile GPU

This is where I draw most heavily from 2DGS and GES. 2DGS is where the transform chain, bounding-box algebra and plane intersection come from, and it works out where a pixel’s ray meets a surfel’s disk. GES builds on that, and since its surfels are opaque it can draw them like any other geometry and let the depth buffer decide what ends up in front, so nothing ever needs sorting. Their rasteriser (the reference) is written in CUDA for desktop cards, which I simplified for usage in tiled mobile-GPUs.

Each surfel is drawn as a quad. Every pixel inside that quad measures how far it sits from the surfel’s centre, in units of the surfel’s own radius $\sigma$. That squared distance is $\rho$, and the weight the pixel gets from it is $g = \exp(-\rho/2)$, one at the centre and falling off toward the rim. To get this running in real time, and accepting a loss in reconstruction quality, I deviate from the reference in four ways:

• Cut the falloff early. I throw a pixel away as soon as its weight drops below half, at $g < 0.5$. Because the draw is unsorted and depth-tested, a faint pixel out in the tail still blends its colour in and writes depth, which then hides whatever was properly behind it, so you get a dark crescent around every surfel where the tails overlap. Cutting at half height leaves clean disks and lets the depth test do its job.
• Size the quad from that cut. If every pixel below half weight is thrown away, the quad only has to cover the disk down to half weight. Setting $g = 0.5$ and solving for $\rho$ gives a radius of $\sqrt{2 \ln 2} \approx 1.18\sigma$. The reference just uses a constant value of $3.4\sigma$, which is about eight times the area, all of it pixels that would die on arrival under my cut. There’s an option in there that derives the extent from the fragment cutoff instead, but it ships disabled.
• Clamp the radius. The radius can never go below $\sqrt{\ln 2} \approx 0.83$ px, for reasons described further down, and anything projecting wider than twice the screen is dropped outright. A surfel grazing the near plane, or one placed by a badly conditioned pose, comes out with absurd values for the pixel radius which rasterises the whole frame in dead fragments and flashes garbage across it.
• Hardware compositing. The reference walks through a list of surfels for each screen tile inside the shader, blending them one at a time. I just submit the quads and let the depth buffer handle it, which costs little on a tiler and means the GPU can throw a hidden pixel away before the shader ever runs.

Splat-plane intersection is worth a closer look, since it is where every pixel spends its time, and it gives me a chance to write down some maths. The transform chain results in a 3×3 matrix taking the surfel’s disk to the screen. My first approach was to invert it once per pixel, which was not a good idea. It is expensive, and it falls apart when a surfel is viewed edge-on. 2DGS avoids the inversion entirely by treating the pixel as the intersection of a vertical and a horizontal plane, pushing both into the surfel’s own frame. The point belonging to the set of both planes is the hit.

This is quite simple, with only 4 operations and a single edge case, $p_z = 0$, when the ray runs parallel to the disk. The result is already in the surfel’s own coordinates, so $\rho_{3\mathrm{d}}$ is already the correct $\rho$ and nothing needs rescaling.

There is a caveat, which is easily handled. A surfel seen edge-on, or one that projects to smaller than a pixel, can fall between the pixel centres and disappear, which looks like the surface flickering out in patches. This is fixed with a second distance $\rho_{2\mathrm{d}}$, measured the plain way in screen space from the projected centre with $\sigma = \sqrt{2}/2$ px, and takes

Since $g$ shrinks as $\rho$ grows, the smaller $\rho$ is the larger weight, so this is a floor rather than a blend. Nothing is ever drawn thinner than about a pixel, whatever the geometry says. It is also where the quad’s lower clamp comes from, since a tighter quad would clip away the pixels this exists to recover.

Pose: correcting drift against the map

This is where most of the time went, and the part I am least satisfied with. I welcome all criticism and suggestions here, because this solution is suboptimal and the result of some very frustrating chats with AI models.

Between visual corrections, the only thing tracking camera translation is the accelerometer, integrated twice (velocity, position). This is pretty much dead reckoning, and it’s an error-laden approach, in great part due to gravity.

An accelerometer measures specific force, which is acceleration plus a 1 g reaction it has no way of separating out, since, you know, we’re on Earth. To get the acceleration on its own, the engine rotates the reading into world coordinates using its current attitude estimate and subtracts a constant 9.81 m/s² pointing down. Both sides of that subtraction are about 9.81, and the value being kept is the difference between them, so the subtraction has to be as exact as possible.

If we tilt the attitude estimate by an angle $\theta$, the rotated gravity no longer lines up with the constant being subtracted, leaving a horizontal component $g \sin\theta$ which is indistinguishable from genuine sideways acceleration. At $\theta = 1°$ that is 0.17 m/s² the device is not actually experiencing, more than the accelerometer’s own bias contributes, so translation accuracy is set by how well the attitude is known rather than by how good the accelerometer is. Being a systematic bias, it does not average out either. Integrating twice turns it into a quadratic position error of $\tfrac{1}{2} g \sin\theta \; t^2$, which is two metres in five seconds at one degree.

Hold the phone still and none of that actually happens, because a stationary accelerometer reads gravity and nothing else, which makes it the attitude reference. The engine makes use of it to pull $\theta$ back toward zero whenever the total force looks like 1 g, and a zero-velocity update clears the integrated velocity once a few consecutive samples agree the device is at rest.

It should be apparent to the astute reader (yes, you) that scanning a room is not holding still. Once the phone is moving, the total force stops looking like gravity, the tilt correction switches itself off because it can no longer tell a lean from a push, and no static window comes along to clear the velocity. $\theta$ is unobservable for exactly as long as you are doing the thing the app exists to do, and the quadratic runs the whole time. Left uncorrected, the surfel cloud either slides away across the room or sticks to the phone and follows it around as if glued to the glass.

Now, by the time this drift matters there are already thousands of surfels sitting at fixed positions in the room, so rather than relying on the sensors to figure out where the phone is, I rely on the map. Each frame the engine picks out 150 distinctive corners and follows them from frame to frame with pyramidal LK (Lucas-Kanade method³). These are what is commonly known as tracks. Every track landing within three pixels of a surfel in the current view is paired with it, and that surfel’s world position is copied out and frozen. I call the pairing an anchor, a name borrowed from AR platforms, though theirs are a lot better.

Working out the camera from a set of anchors is a standard problem, PnP (Perspective-n-Point). Given points whose position in the room you know, and the pixels they appear at, PnP methods find the pose the camera must have had. Most methods find that pose from scratch, making increasingly better hypothesis until enough points are in agreement (RANSAC⁴). Mine doesn’t have to, since the IMU has already handed over a pose and we have only to correct it. So the engine starts there and takes ten steps, each one the small rotation and shift that most reduces the gap between where the anchors should appear and where their tracks actually are (this is just good ol’ Gauss-Newton, on SE(3)⁵).

Some anchors are wrong, whether a track slid onto a different piece of scene or the surfel behind it was noise. Past a threshold an anchor’s contribution stops growing with its error, so a handful of large errors cannot drag the answer around. That is a boilerplate Huber weight (Huber, 1964), and the anchors inside the threshold are the inliers. How many there are is the best measure of whether the answer means anything. Solving for the camera alone, against points held still, is motion-only bundle adjustment, which is what ORB-SLAM2 (Mur-Artal and Tardós, IEEE T-RO 2017) runs in its tracking thread.

That threshold is 4 px, which is pretty large, and from what I observed, has to be. An anchor sits wherever the depth model put the surfel behind it, and the model is off by 5–10% of the distance, which is 10 to 20 pixels once the viewpoint has moved. At 2 px every good anchor is an outlier and nothing is ever accepted.

There is a subtle issue here. The current pose is what decides which surfel a track pairs with.

Point the phone at a flat wall and let the estimate drift half a metre sideways. Every track pairs with a different surfel. This is not wrong, per se, they are all real bits of wall, and each one sits exactly where the drifted pose expects to find it. The solution converges, everything agrees, and the answer that comes back is the drift it was supposed to catch. The map cannot contradict this, because the pose that goes through it has the very error it was meant to find. This causes the cloud to stick in screen-space, without any indication that anything is wrong.

Some things came out of that:

Anchors are only created after the pose is corrected. An anchor created beforehand carries a frame of dead-reckoning lag, and since each generation is created at the pose the previous one holds, that lag compounds. This leads to a noticeable creeping of the map.

An anchor holds five frames before it is considered. A new anchor agrees perfectly with the pose that created it, because that pose is what picked it, so it has no information about where the camera really is. It still carries full weight, and holds the camera where it already is, drowning out older anchors that have genuinely drifted apart and should be considered. Five frames of track motion is enough to break that agreement. The window has not been tuned beyond that.

The solver needs to be tightly gated. The correction may disagree with the IMU’s prediction for the frame, but only by about three centimetres and a degree at one standard deviation⁶. Where the scene genuinely pins the camera down that costs nothing, since the pixel evidence outweighs this gate by orders of magnitude and real drift still washes out over ten frames or so. Where we need to be careful is when we have a flat wall. With everything in view lying in roughly one plane, sliding the camera sideways and rotating it slightly looks nearly identical to the pixels, so a whole bunch of answers fit equally well. If the solver is not gated, it picks a random member of that bunch every frame and the pose jumps back and forth. As it is, this is averaged out.

Anchors are only created while the pose is confirmed. With too few agreeing anchors in a frame, none are created. An anchor made at an unverified pose points at the neighbouring surfel, off by exactly the current error, and a batch of those will muddle the true pose and make the mistake permanent. There is really no way around this. Without confirmation the phone falls back to dead reckoning and the map stops growing, which is briefly wrong rather than permanently wrong.

Some anchors skip the map entirely. The four fixes above still leave an anchor’s world position coming out of a map lookup performed at the current pose. Depth frames give a second source for that position. When one comes back, every active track gets an anchor from the depth value at its own pixel, cast out along the ray and transformed by the capture pose. No lookup happens, so a drifted pose cannot steer the result onto a nearby piece of wall that fits it. These are the only anchors that can pull the pose back after a bad couple of seconds, and recovery runs on them. A depth frame arriving with no confirmed pose is aligned against the existing surfels to work out the pose it was taken at, and none of the anchors built from it derive from the old estimate.

Whether a correction is accepted comes down to evidence rather than size, requiring at least eight anchors for each correction and a quarter of them pointing to the same solution, and the correction goes in however large it is (I tried to have a majority rule, but this never hit, leaving out good solutions). Capping the distance is simple, tempting, and it failed, since any drift beyond the cap can never be undone and the map locks into its own mistake.

On top of all that is a small joint adjustment over the last eight frames. It re-solves each frame’s pose against the current anchor positions, then each anchor position against the re-solved poses, and repeats once. Anchors created at different moments carry different depth-model errors, and re-solving them together reduces the inconsistency between them. Each anchor is held within about 10 cm of where it was created, and the mean displacement of each round is subtracted, so the positions move relative to one another while the map as a whole stays still. Only the anchors are updated, and only 30% of the way. The live pose is given by PnP, since updating the map and the camera in the same step creates a feedback loop⁷.

Interestingly enough, this is a pretty old solution. Triggs et al., all the way back in the last millennium, describe bundled adjustment as “a naïve and often-rediscovered bundle method” belonging to “a family of simplistic and largely outdated strategies”. It’s always fun getting schooled by something that was written almost 30 years ago. It works for now, and I’ll find a better solution in the near future.

The tracker also requires some handling in order to not replicate this kind of error. Depth inference runs off the camera thread, so the tracker never sees a multi-second jump between frames. Across a gap that long, LK re-latches onto whatever has slid under a feature’s old pixel and reports success. This means that every track is followed back to where it came from and discarded unless it lands on its starting pixel. Without this handling, all 150 tracks agree that the camera has not moved, the anchors hand that agreement a full set of inliers, and the couch comes out gigantic and misshapen, like something out of Cronenberg.

Results

Guys, gals, and non-binary pals. After all the drudgery in this last section, you have all earned a breather, so here’s the demo. It is a capture of Cairn running indoors, live on my phone. I’m scanning my office, strafing along my couch (peak 90s, I know, but be respectful) and the surfel map is composited through the Vulkan pipeline over the live view. As the viewpoint swings around, the room stays put. The couch, the cushions, the table, the door, the cabinet and the sweet NASA print hold their positions and their shape. The map does not slide with the camera and the cloud does not split into duplicates, which are the two main issues the whole PnP section above exists to kill. The noise at object edges is what a sparse surfel map looks like up close.

Where that puts things:

• About 14 fps end to end on my old phone at the cloud sizes in the video (~18k surfels), covering camera, depth, tracking, fusion and render on the device. The frame rate sinks as the cloud grows, and the cloud grows too fast. See issues below.
• 150 tracked features held steady. Only the subset that is both mature and paired with a surfel votes, which on a good run is dozens of inliers, continuously, and that is what keeps the pose world-locked (fixed in the world as the camera moves, rather than fixed to the screen) between depth frames rather than only at them.
• Only using an RGB camera and the phone’s own sensors. No LiDAR, no stereo, nothing external doing the computation, which was the main objective.

It is still a proof of concept, so there are a lot of issues to handle.

There is no global optimisation beyond the joint adjustment mentioned above and no loop closure, so anchors seeded at a locally drifted pose keep that error for good.

Recovering from a total tracking loss is finicky. If every anchor is gone by the time a depth frame arrives, the engine tries to stabilise again by aligning that depth map against the map it already holds, and refuses if the two do not agree well enough. Mapping then stalls until you point the camera at something it recognises. Stalling construction is better than creating a wrong map, but waving the camera back and forth at landmarks is not a valid recovery strategy.

The biggest issue with the whole thing is what is called data association (Neira and Tardós, IEEE T-RA 2001), the step in fusion where a new depth sample is marked as revisiting an existing surfel (update it) or genuine new surface (add it). I currently do this by position, superposing a grid of cells on the map and restricting a sample to merge only with a surfel in its own cell. The cell size is not a constant, it is the median surfel radius of the first batch that fuses, which works out to about 2 cm at couch distance and would be an order of magnitude coarser on a facade.

In practice the merge almost never fires, because the per-frame scale correction shifts the whole depth map by more than a cell between fusions, so a revisited point lands in a new cell along the viewing ray and is appended as a duplicate. Each fused depth frame adds nearly all of its ~19k samples. Long scans balloon the cloud, and frame rate falls with it (measured 18k surfels running at 14 fps and 261k at 8). Widening the search to the 26 neighbouring cells does not help either, since the shift is larger than the whole neighbourhood. The approach is plain wrong and no tuning can save it.

The map’s scale is set only once, by the first depth frame that fuses, so a scan has to start pointed at the actual scene. A bad start (a distracted operator starting the capture with the phone facing the wrong way because he’s too busy watching the World Cup, for instance) leaves PnP without enough inliers to ever correct the pose, and the session never recovers. That is by design, since a frozen map is preferable to a corrupted one, but there is no warning for it yet.

The motion-blur filter is too blunt, having no knowledge of the exposure time that actually decides whether a frame is smeared.

Finally, the tuning for a lot of this is built around this one phone, and I have not set up a procedure to generalise this.

Given all of this, the hardest part, a real-time monocular map that stays put in world space, seems to be stable.

Future work

There is a tremendous amount of work left to do.

One possible answer for the issue of data association here might be projective data association, which goes back to Blais and Levine (IEEE TPAMI, 1995) and is what KinectFusion (Newcombe et al., ISMAR 2011) uses for its correspondences. Project the map into the camera, match each depth sample to the surfel landing on the same pixel, and allow a generous tolerance along the viewing ray, where the wobble lives, with a tight one across it. The rasteriser is already there, so rendering the map’s depth buffer from the current pose gives me that correspondence directly, one surfel per pixel, on the hardware already drawing the frame. This would probably be the quickest intervention.

The closest thing to what I want is RTG-SLAM (Peng et al., SIGGRAPH 2024), which adds a Gaussian only where a pixel is newly observed or badly explained by the current map, on a representation almost identical to mine, with one opaque Gaussian per patch of surface rather than a stack of translucent ones, for the same anti-bloat reason.

I could also move surfels rather than append them. Splat-SLAM (Sandström et al., 2024) does this for Gaussians. When depth is revised it projects each one into the frame and slides it along the viewing direction by the depth change. So a scale refit that moves the surface 5 cm along the viewing ray should slide the existing surfels 5 cm, rather than minting 19,000 new ones in front of them.

Currently, I’m working on a differentiable version of the renderer, which also runs backwards. This would allow me to render the surfels, compare against the camera image, and get gradients telling each surfel how to move and restretch itself to make the render match, which an Adam loop then applies. That is what 3DGS training is built on, and it would let me continuously refine the map instead of only accumulating data.

The kernels are running and the gradients check out against the captures, but I still haven’t quite figured out how to solve the ambiguity in this comparison. A surfel twice as far away and twice as wide covers exactly the same pixels. Nothing in that one image says which of the two is the real surface, so the optimiser can bring the difference between render and photo down to almost nothing while leaving the geometry as wrong as it found it. It only becomes a real constraint once the same surfel is seen from viewpoints far enough apart to disagree, which is adjacent to the problem that loop closure is targeted at, so the other stuff needs to be fixed first.

There is a more radical option, which I like, and that is to drop the fusion step entirely. MonoGS (Matsuki et al., CVPR 2024) is monocular with no depth prior at all, opting for optimising camera pose directly against the Gaussians and letting the optimiser itself decide where to add and remove primitives. That is where I’m taking the differentiable rasteriser, and the reason I treat the optimiser as a milestone for controlling density rather than for frame rate.

The thing I don’t really know how to bridge from these solutions is compute requirements. RTG-SLAM assumes a depth camera and a desktop GPU, Splat-SLAM runs a global optimisation backend, and MonoGS reports roughly 3 fps on a desktop card. I would be stripping these down rather than tuning them, but it all sounds very interesting.

On the algorithm side, the significant one is implementing robust global optimisation and loop closure. Recognising that the camera has come back to a place it has already mapped, and correcting the whole trajectory to agree with itself, would fix the anchor creation consistency issues discussed above.

There is also model unification, one depth calibration handling both indoors and outdoors instead of shipping two. Not only does this simplify the model inference, it would have the added benefit of making the binary smaller, on top of the quantisation.

The final challenge is actually getting off the phone and onto a drone. This phone build was always meant as a warmup for a Pi 5 or a Rockchip (or something smaller, depending on optimisations), with a real IMU and camera rather than a consumer phone accelerometer. There are many ways in which facade capture changes the problem, not all of which make it harder:

• The cloud growth will probably be a nightmare. A building has orders of magnitude more surface area than my office or my house, and captures will last minutes rather than seconds. The cloud size scaling stops being an inconvenience, and turns all of this into a no-go. That is why fixing data association is first on the list.
• Depth quality at range is the open question. Everything above was done at about 2-4 m. Monocular depth at 10–40 m, on a sunlit facade with shadows, alcoves, columns, etc, is a different beast, and I’ve only tested the outdoor calibration on the walls of my house. Depth Anything V2 might be able to generalise to this regime, but aspersions and Persians.
• More information. A drone actually flies real distances, which is exactly the input PnP and the global optimisation work best with. Many drones carry a barometer and navigation, which can anchor metric scale properly instead of relying on a single initial depth frame. Flying a building can also be made to close a loop by default, so loop closure stops being an addon and starts being the normal case.
• Prop vibration. The current gate rejects a frame whenever the gyroscope has spiked since the last one it kept. On a frame that buzzes continuously, the gate as written would reject everything, so it has to become exposure aware before it can work on a drone.

That’s pretty much a wrap.

If any of this is your kind of problem (on device vision, splatting, fusion, interior design) get in touch with me, I’d love to hear and learn from you. Please, it’s been weeks now and I’m in desperate need of human connection.

References

• Pfister, Zwicker, van Baar, Gross — Surfels: Surface Elements as Rendering Primitives, SIGGRAPH 2000.
• Kerbl, Kopanas, Leimkühler, Drettakis — 3D Gaussian Splatting for Real-Time Radiance Field Rendering, SIGGRAPH 2023.
• Dai et al. — High-quality Surface Reconstruction using Gaussian Surfels, SIGGRAPH 2024.
• Ye, Shao, Zhou — When Gaussian Meets Surfel: Ultra-fast High-fidelity Radiance Field Rendering, ACM TOG (SIGGRAPH) 2025.
• Huang, Yu, Chen, Geiger, Gao — 2D Gaussian Splatting for Geometrically Accurate Radiance Fields, SIGGRAPH 2024.
• Yang et al. — Depth Anything V2, NeurIPS 2024.
• Ranftl, Lasinger, Hafner, Schindler, Koltun — Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer (MiDaS), IEEE TPAMI 2020.
• Blais, Levine — Registering Multiview Range Data to Create 3D Computer Objects, IEEE TPAMI, 1995.
• Newcombe et al. — KinectFusion: Real-Time Dense Surface Mapping and Tracking, ISMAR 2011.
• Harris — The Mali GPU: An Abstract Machine, Part 2 — Tile-based Rendering, Arm, 2014.
• Matsuki, Murai, Kelly, Davison — Gaussian Splatting SLAM, CVPR 2024.
• Peng, Shao, Liu, Zhou, Yang, Wang, Zhou — RTG-SLAM: Real-time 3D Reconstruction at Scale using Gaussian Splatting, SIGGRAPH 2024.
• Sandström, Tateno, Oechsle, Niemeyer, Van Gool, Oswald, Tombari — Splat-SLAM: Globally Optimized RGB-only SLAM with 3D Gaussians, 2024.
• Kingma, Ba — Adam: A Method for Stochastic Optimization, ICLR 2015.
• Huber — Robust Estimation of a Location Parameter, Annals of Mathematical Statistics, 1964.
• Triggs, McLauchlan, Hartley, Fitzgibbon — Bundle Adjustment — A Modern Synthesis, Vision Algorithms: Theory and Practice, 1999.
• Mur-Artal, Tardós — ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras, IEEE Transactions on Robotics, 2017.
• Neira, Tardós — Data Association in Stochastic Mapping Using the Joint Compatibility Test, IEEE Transactions on Robotics and Automation, 2001.