On the Road to a Billion Acres of Data

The world’s population is forecast to continue growing for the foreseeable future. While this trend appears to have no obvious impact on Silicon Valley at first glance, the availability and cost of the food we eat every day is bounded by the sustainable capacity of our global agricultural ecosystem. In short, farmers must grow increasingly more food, despite gradual reduction in available farmland.

What if we could transcend agriculture from a patchwork of disparate farms to digitizing every inch of global farmland? What if we could improve sustainability, yields, and profitability via a deep understanding of comparative variability and productivity across agronomic variables and decisions? What if every field could be a test plot and every farmer could have the power of Internet-scale machine learning in their pocket?

From Walking to Brute Force

Farmers have walked and worked every acre of their fields for generations, knowing each pond and hill. They have progressively improved their equipment, from hand ploughs to modern precision farming equipment with GPS. For the next breakthrough, innovation must transcend individual fields. In the past decade, far from the heartland, Internet-scale data acquisition and machine learning has pioneered a different approach: sophisticated brute force.

One simple idea explains much of the past five years in the Valley: use large scale datasets, simple algorithms, and massive computing to solve problems and build products that were previously impossible. From voice recognition with deep learning to A/B testing with high-volume websites to online behavioral prediction with machine learning, the intuition is all the same.

The relevant question is how this brute force methodology from the Valley applies to a breakthrough in farm fields. The most obvious idea is to digitize the full spectrum of field agronomic data for every acre in each field at high resolution, including soil, atmospheric, imagery, crop, chemistry, and hydrology. Yet, the logistic challenges of this seem daunting: global growing regions have hundreds of millions of acres, data must be collected directly at the physical ground level in multiple passes per growing season, multiple tiers of remote imagery are required, farmland has minimal Internet connectivity, and the data storage footprint for this many geospatial data layers at high resolution is massive.

Closing the loop requires deriving novel insights and recommendations from this digital agronomic universe that predictably improve sustainability, yield, and profitability for farmers. In doing so, data will become as indispensable a tool for farmers as their equipment and will enable them to collectively meet the demands of a growing population.

Innovation Flywheel

From high resolution ground-level data acquisition to Internet-scale geospatial data processing in the cloud to easy-to-use mobile products, we condense all this complexity into a simple technical milestone: 1 billion acres of data. By this we mean scaling our acquisition, data pipelines, science, systems, applications, and infrastructure to process more than 1 billion acres of agronomic data originating from farmland across the globe. Doing so necessitates that we work across global farming regions, facilitate compatibility with varied modern farming equipment, and support the wide diversity of farming practices in use worldwide. Finally, everything must interoperate with and leverage all the innovation from the burgeoning digital agriculture ecosystem.

Our innovations progressing towards this mission are captured in our R&D pipeline, including progress across our Engineering, Science, and Analytics teams. At this time last year, we became the first digital agriculture company to share our innovation pipeline publicly. Today we announced, as we do twice a year, the research efforts advancing through our innovation pipeline that are driving us towards this goal. This is an exciting time for our teams to see their work improving farm practice.

Three simple mental models unify everything we do to achieve this breakthrough, echoing the familiar flywheel effect for data:

• Data in one place: hardware and software tools to unify and harmonize dozens of agronomic layers of soil, field, crop and atmosphere data that today is either not collected or dispersed across systems in varied incompatible formats
• Agronomic insights: multidimensional geospatial visualization and analytic reporting derived from the collection, cleaning, processing, and analysis of unified field data
• Actionable agronomic guidance: insights embodied in software tools and exportable to farm equipment that optimize yield potential, input efficiency, risk management, and sustainability

Although Climate is driving a similar Internet and IoT transformation as other industries have undergone, doing so is significantly more challenging for the agriculture industry for several reasons:

• No playbook: much of the established playbook of industry disruption techniques do not apply: agriculture data originates in the physical world; many global farming regions have minimal to no Internet connectivity; interoperability is required for varied equipment; and farming practices vary widely across crops and global regions
• Slow iteration cycle: experimentation is orders of magnitude slower in agriculture than other industries because agronomic growing cycles are long and there are comparatively few farmers
• Complex geospatial data: most agronomic data is geospatial and requires algorithmic intersection with both grids and vector polygons
• Full round trip: achieving scalable network effects requires pulling data through the full round trip from equipment to cloud and then back to equipment
• Trust: digital agriculture is unlike ordering food online or hailing a cab, as many agronomic decisions have significant financial consequences and are part of a carefully choreographed season of activities

Adding Our Voice to The Community

This blog will be one of our preferred mediums for sharing technical and scientific learnings along this journey. Subsequent blog posts will cover a variety of themes, including people highlights and expanding upon how we are solving challenging technical problems like:

• Hardware, embedded software, and cloud ingest for petabyte-scale IoT field data acquisition
• Statistics, machine learning, geospatial modeling, and field experiments to translate complex multidimensional agronomic data into actionable recommendations
• Globally distributed data ingest and processing pipelines to acquire, clean, normalize, rasterize, aggregate, and summarize data at up to inch-by-inch subfield resolutions
• Mobile and web apps that support multi-resolution geospatial visualization and ad hoc analytic reporting of many data layers, especially for use in field environments either offline or with intermittent connectivity

For me, the challenges we are tackling at Climate resonate with three professional passions honed from prior successes leading engineering and research teams at influential disruptive tech companies, most recently at Wealthfront and LinkedIn. First, helping to improve the lives and financial outcomes of many people through simple, yet sophisticated, digital tools. Second, solving real-world problems at the intersection of Internet-scale engineering with machine learning and applied mathematics. Lastly, driving the digital transformation of the agriculture industry, providing now my third opportunity to partake in industry revolution.

We intend this blog to foster transparency and collaborative dialog with our technical and scientific industry peers along with the broader community. Please reach out to us if you are interested in collaborating, learning more about these challenges, or perhaps joining us on the road to 1 billion acres of data.