Descartes Labs Attains Rank 41 in TOP 500 with AWS Empowered Cloud-Based Supercomputing

Descartes Labs, Inc. has announced the latest achievement of cloud supercomputing with a TOP 500 run on AWS of 9.95 petaflops using virtualized Amazon Elastic Compute Cloud (Amazon EC2) instances. The TOP 500 organization utilizes the LINPACK Benchmark—an assessment that involves solving a compact system of linear equations—to rank the performance of the 500 most robust commercially available computer systems. Descartes Labs enhanced its previous cloud record of 1.926 petaflops set in the TOP 500 submission in 2019 (#136 ranking) with a 417% HPL performance development over two years. It has improved performance by a factor of 10 relative to the performance achievements by legacy, on-premises supercomputers in the same two-year tenure. For 2021, the overall computation took nine hours on average and performed 3 x 1020 total floating-point operations across 172,692 Intel Xeon Scalable Processor cores.

Descartes Labs requires cutting-edge cloud technology proficient in running High-Performance Computing (HPC) and Machine Learning (ML) workflows at scale to process the volume of global sensor data. This technology allows the Descartes Labs engineering teams to perform computation and data analytics to deliver critical business insights globally to customers in the defense and intelligence, agriculture, mining, and consumer packaged goods (CPG) industries. This capability is becoming significant for applications that depend on a combination of geospatial analytics worldwide. Examples are environmental monitoring, extreme weather tracking and forecasting, multimodal data fusion, and climate change analysis. An example of this assignment is the six petabytes of Sentinel-1 radar data recently processed by Descartes Labs to track deforestation at the continental scale, identify the high wildfire risk zones, and monitor regenerative agriculture practices.

Mike Warren, Descartes Labs Cofounder, and CTO said, "Having the immense computational capacity offered by AWS, connected to massive storage that's accessible at very high speeds. Under such conditions, you perform tasks that you couldn't do before in data analysis and understanding the reality of our planet or the universe as a whole. Our cloud usage today extends the range of analysis of earth observation datasets at the petascale, predominantly beyond just optical imagery into more esoteric types of earth observation such as radar, InSAR, and AIS data."

In association with the AWS HPC team, Descartes spun up the resources required for its TOP500 run using general-purpose, publicly available Amazon EC2 instances across C5/M5/R5 families. The credit goes to the scale and flexibility of the AWS Cloud; Descartes Labs could scale up infrastructure for this assessment without affecting its production workflows. Moreover, due to the flexibility of Amazon EC2's compute resources, Descartes was also able to assess different cluster sizes with multiple HPL runs spanning one petaflop to 8.5 petaflops, up to the final 9.95 petaflops result.

The data driving the computational needs is the petabytes of earth observation and ground sensor data moving at the speed of collection. The legacy supercomputer architecture is limited in terms of cluster size and input/output (I/O). With flexible computational capacity, connected to massive storage in the cloud, accessible at very high speeds, Descartes Labs is implementing science that no one could accomplish before.

The LINPACK benchmark utilized in this supercomputing demo helps estimate the computer's speed while solving "tightly coupled" real-world problems, such as weather forecasting. Descartes Labs can use the same systems to model the weather quickly, accurately, and over multiple locations simultaneously to enhance agricultural yield forecasts by manufacturing systems that can enable the LINPACK benchmark to perform optimally.

Such advanced science has contributed to Descartes Labs' partnership with customers like CJ OliveNetworks, an organization that recently migrated the complete IT systems of CJ Group Companies to AWS. The company leveraged Descartes Labs technology to procure an innovation project to forecast sugar input prices for CJ CheilJedang, which imports 600,000 tons of raw sugar every year. CJ has attained successful business innovation from this project via satellite imaging technology, climate, and vegetation data analysis.

The transition from on-premises to cloud computing mirrors the shift from just analyzing an organization's internal data to incorporating large volumes of geospatial and sensor data from the world beyond. Planetary scale data needs planetary-scale compute. Accordingly, the size of this "outside the firewall" data requires the kind of cloud infrastructure Descartes Labs has carefully developed and used for this most recent TOP500 run. The company looks forward to assists others do the same.

Warren says, "The cloud is where all the advanced technology appears first. You're seeing all the modern processors from Intel and AMD in the cloud before you have access to them in your on-premise supercomputers. The investment in networking technology to support these hyperscale applications in the cloud proves to be effective for legacy tightly coupled supercomputing applications. Now, the big drive is into exascale in HPC. However, parallelly, machine learning and deep neural networks make their way in other types of problems. I believe those worlds may either collaborate or drift further apart. And that's one of the big concerns for the next decade."

About Descartes Labs
Descartes Labs is a geospatial intelligence organization. It performs scientific analysis of geospatial, remote sensing, and diverse complementary data sets to enable workable sourcing best practices, commodity price forecasting, and efficient mineral exploration for leading CPG, Agriculture, and Mining organizations. Our SaaS platform automates the geospatial imagery analysis for our users, enabling planetary-scale analysis via artificial intelligence and machine learning. The company also supports a wide range of federal government efforts to curate, analyze, and provide unique, actionable insights from geospatial data. For more information, visit www.descarteslabs.com