Skip to main content
← Back to Writing
·4 min read

Processing LiDAR Data at Scale: Lessons from LiDAR Atlas

Processing LiDAR Data at Scale: Lessons from LiDAR Atlas

Magellan Stereo: Lessons in Processing Geospatial Data at Scale

My first experience processing geospatial data at scale was at JPL during grad school. Scott Hensley had designed a new algorithm, written in Fortran, to re-process all of the Magellan F-BIDR SAR imagery into the most accurate DEM of Venus to date. My job was orchestrating the run on a local Mac Pro (the generation that looked like a jet-black trash can) — a run that took over 30 days per iteration. You can read more about the project in our IGARSS 2019 paper, Magellan Stereo Revisited.

That project, and the analysis work by my advisor Daniel Nunes, sparked a passion for geospatial data that took me to Woven in Tokyo, where we extracted lane-level ("HD") maps from aerial and satellite imagery.

Batch Jobs

At Woven (a Toyota company), we pushed our jobs to AWS Batch and let it spin up Fargate instances from our Python-orchestrated pipelines.

Fast forward to founding Advanced Algorithms: I followed the same core pattern when building LiDAR Atlas. Natural iterations followed — orchestrating with Step Functions, adding error handling, and integrating a PostgreSQL database with PostGIS extensions to track run progress. For cost and control, we now run everything on EKS Kubernetes clusters with controllable instance types and optional Spot instances. This design has led to considerable cost savings and faster, more scalable runs.

The fundamental problem for GIS practitioners is that, for large areas and many kinds of operations, the compute and memory requirements become intractable on any local machine. We generally run about 4,000 CPUs per job, turning what could be a 30-day job into hours, or a 12-hour job into 20 minutes. Scaling geospatial workloads to the cloud enables insights at scales that simply weren't possible before.

Data Design

The system optimizations themselves have produced 5–10x efficiency gains over time, but by far the biggest lessons have come from iterating on the data design: when to merge data, when to divide it, and how big to make those divisions. Parallel processing is great when it's been optimized — and wasted when the divisions are too small or bottlenecks claw back the gains. These iterations were made across hundreds of runs spanning hundreds of thousands of square kilometers of LiDAR data.

Takeaways

Here are my big takeaways for processing data at scale:

  • Invest in the data index. The most important and most overlooked piece is the data index. You need to know exactly what data is where — both logically and geographically — along with projection, epoch, and other metadata. Spending more time on the index up front leads to big payoffs later. Our LiDAR Index and the infrastructure that maintains it are arguably half the IP on their own, and it pays off when you have a piece of data no one else does because you've been the most thorough.

  • Processing location matters more than processing cost. Co-locating your processing with the data, wherever possible, yields a massive latency edge.

  • Test your DAG designs before committing. For complex pipelines like the ones we build at Advanced Algorithms, there are usually multiple DAG designs that lead to the same output — but they are not all equal. Sometimes the less intuitive design is the better one.

  • Automate your checks. Scaling from 1 km² to 1,000 km² to 1,000,000 km² surfaces more and more edge cases. Debugging anomalies that show up in the final product always costs more than catching them earlier in the pipeline. The more ground-truth tests, histogram analyses, and coverage checks you can run up front, the better.

  • Build for visibility. Time spent on pipeline visibility pays for itself. The faster you can analyze and diagnose an issue, the more you can iterate — and the better your pipeline becomes. There is no such thing as one last bug.

GPUs Are the Next GIS Frontier: PCAX

At Advanced Algorithms, we build data pipelines that process geospatial data at scale and on demand — over 1,000,000 km² of LiDAR data for DEMs alone. As I target even greater efficiency and performance, GPUs are the next step. We've already implemented the foundations for CUDA-accelerated point cloud processing with native reprojection, geoid application, filtering, and gridding, and we're continuing to test, validate, and extend it. We've seen 10–20x performance gains over PDAL on older GPUs. We support FP64 and FP32, with FP32 mode falling back to CPU for precision-critical reprojection.

We call it PCAX — the Point Cloud Acceleration Framework.

If you have an application you think could benefit, reach out: support@adv-algos.com.

Kevin J. Cotton, Ph.D.