LAS VEGAS, NEVADA – JANUARY 5: The Deer & Co. John Deere See & Spray technology is displayed … [+]
The John Deere Company wowed the crowds at the 2023 Consumer Electronics Show in January with their high-tech agricultural equipment. At their exhibit, heads craned upward in awe to take in the 120 ft boom of their precision spraying technology, straddled atop a massive tractor. At the CES 2022, Deere & Company introduced a fully autonomous tractor.
In the months since CES, we’ve seen multi-faceted challenges for companies seeking to transform road-running Automated Driving Systems (ADS) into a profitable business. ADS developer Embark announced a shutdown and Locomation appears to be on the same path. The mood of investors is uncertain, especially given troubles in the banking sector.
Against this backdrop, the off-road world is becoming increasingly interesting for companies developing autonomy. Caterpillar and Komatsu brought the first commercial ADS’s to mining operations well over a decade ago. At that time, although the tech was very expensive, a business case could be made for equipping the huge mine-hauling trucks at open pit mines.
Since that time, thanks to the tidal wave of AV development for passenger cars, trucks, robo-shuttles, delivery robots, and more, the tech cost has now come down to reasonable levels for other types of industrial operations. Plus, the tech robustness has progressed by leaps and bounds. The result? Use cases are expanding rapidly in areas such as agriculture and construction. For this article, I’ll dig into the Ag space to examine the linkages with on-road autonomy.
What’s Happening In Ag Now?
“Low tech” automated driving in the fields, powered by GPS and precision maps, has been in use for some years now, allowing tractors to dumbly trace a path. Thus, the drive-by-wire capability in modern tractors is mature. One could call this an SAE Level 2 system, as these tractors still need continuous human supervision. Enabling driverless operations by integrating sensors and software into these tractors is the new Holy Grail.
However, farm tractors don’t move just to get from one side of a field to another. They are performing tasks to cultivate and enhance the health of the green products in the soil. At speeds up to 12 mph, John Deere’s See & Spray™ Ultimate system uses cameras and processors mounted on the big boom to identify weeds and zap them with herbicide. The company says this approach can reduce herbicide application by more than two-thirds. Another product optimizes fertilizer usage by registering when individual seeds are planted and spraying the precise amount of starter fertilizer directly onto the seed as it goes into the soil. Deere says this approach can reduce the amount of starter fertilizer used by up to 60% and decrease the risk of fertilizer runoff into nearby waterways.
I reached out to AGCO, another agricultural equipment supplier, whose spokesperson made the distinction that “autonomy in agriculture is very different from on-road autonomy. It is about optimizing a job in the workflow of food production, such as planting, harvesting, and tilling, whereas on-road autonomy is about transportation.” AGCO provides both retrofit solutions and factory integrated solutions. AGCO notes that their tech is the result of addressing a wide variety of tasks “from those within specialty crops (vineyards, orchards) to broad acre production (corn, wheat) to generate the greatest yield possible.” For example, their SmartFirmer product from Precision Planting makes precision adjustments to the planter row-unit. As the system runs in the seed trench, it senses characteristics such as moisture and automatically changes the planting depth up or down to ensure the seed is placed with the right access to moisture for germination. Planting occurs without any operator input and with greater speed and accuracy than what could be done by a person.
At the vehicle level, AGCO says they focus on optimizing food production workflows by automating repetitive tasks. For instance, at the end of a row, a farm implement must perform a U-turn and align itself with the adjacent row. AGCO’s Fendt TI Headland product autonomously completes the “headland turning” process, including lifting and lowering of the implement, as well as completing the actual turn, without any input from an operator.
Ag and Startups
It’s a tough time to be a startup, but the situation appears to be brighter in ag than on-road if you want to be acquired. Being vertically integrated, Deere is in a unique position for acquisitions. For several years the company has operated their Startup Collaborator program, which selects 6-8 companies per year for collaboration. As noted by Julian Sanchez, Director of Emerging Technology at John Deere, “These connections help John Deere better understand opportunities to advance precision technology in agriculture and construction.”
Deere & Company selected eight companies for its 2023 Startup Collaborator program. To highlight a few relevant to this article, ANELLO Photonics is developing a low-noise and low-drift optical gyroscope sensor to support autonomy, while GrAI Matter Labs is developing artificial intelligence via “brain-inspired chips that behave like humans do” to optimize devices assisting humans in the field. Another selected company, Precision AI, is developing “artificial intelligence-powered agricultural drones for plant-level herbicide applications at broad-acre scale.” This sets up a technology competition between drones and ground vehicles: which is the best way to handle weeds, wheels or propellors?
Some of the Startup Collaborator players have gone on to be acquired. In 2021, Deere acquired ADS developer Bear Flag Robotics to enhance their driverless capability. Another member, LIGHT, was acquired by Deere in 2022. LIGHT’s camera vision technology plays a key role in the systems described above.
John Deere is not alone in the acquisition game. CNH Industrial recently acquired the precision agriculture firm Augmenta to expedite the growth of their Sense & Act Technology™. As fields are sprayed before and after harvest, this technology uses sensors to precisely direct and regulate spray volume through selective spraying capabilities.
But it’s not all about the Big Guys: the Ag space has significant independent startup activity as well. Late last year, Monarch Tractors delivered its first commercially-available electric driverless tractors to Constellation Brands. Carbon Robotics, based in Seattle, has raised $36 million for its self-driving weed-zapping machines using a unique “laser blasting” approach. Camera images are used to identify weeds, which are then precision targeted with 150-watt thermal bursts. Running driverlessly, they say the system can kill over 100,000 weeds per hour.
Makes me glad I’m not a weed.
Linkages Across Sectors
It’s worthwhile to look at linkages across the on- and off-road sectors. In many cases the engineers working on either discipline are hanging out in the same coffee shops. Caterpillar and Komatsu have offices in Pittsburgh, home to Aurora and others working on automated vehicles. John Deere employs 300 people in Silicon Valley, mingling with ADS developers, sensor vendors, chip makers, and next-gen drone technologists.
What are the commonalities for wheeled robots doing jobs in the outdoors, whether on-road or on-dirt?
Let’s start with what’s not common. Ag is a “semi-structured” setting in which operators have control of their surroundings. Street operations are structured but the actors are quasi-predictable, at best.
On-road robots, whether a small delivery bot or a tractor-trailer, must comply with rules of the road. Traffic police make sure you do so. The vehicle next to you is owned by somebody else – if something goes wrong and there’s a crash, litigation ensues. You’re stuck with the road as is, i.e. you cant change it or upgrade it. This contrasts with ag operations, which happen on land owned or leased by the robot operator. There are no rules of movement enforced by a government entity. All the vehicles around you are part of your operation and you have full control over their movements. But even though Ag vehicles can go wherever they choose, to do their job effectively their paths must be extremely precise.
The list of what is common is much longer.
Sensing The World
Sensing is fundamental. Regardless of the playground, the world must be viewed, measured, and ingested to understand the environment and other actors and how that affects the vehicle’s mission. Since ag vehicles are not “sharing the road” they don’t need to scan 360 degrees for vehicular threats, although they may be scanning for on-foot humans nearby.
Understanding the condition along the driving path is vital; terrain can be known due to mapping, but contours in the land can change with wind and precipitation. An autonomous truck on the road must understand what might bog it down (snow), but an autonomous tractor needs to avoid a literal bog that’s developed overnight.
Precise Maneuvering
Autonomy plus advanced sensing enables precision maneuvering. A novice over-the-road truck driver encountering a complex drop-off situation at a destination they’ve never been to before could struggle. Yet an automated truck with the sensors sufficient to understand the situation and maneuvering options can thread the needle and briskly get that trailer snug up to the dock.
In Ag, precision maneuvering can result in better use of water and how chemicals are being applied to the ground. The result? Crop yields go up, waste goes down. In addition, the vehicle must exhibit the precision necessary to avoid any damage to individual plants.
One wonders how Ag-level precision could help road maintainers better understand and treat their roads. Will we someday see micro-pothole-filler machines?
Running Around the Clock
Driverless operations bring a significant benefit for around the clock operations. On the road, human truckers must stay within hours-of-service regulations. Trucks are parked for many hours out of a day for driver rest periods, resulting in limited equipment utilization and longer delivery times. When driverless, trucks can run virtually 24 hours a day, stopping only for fuel and any needed maintenance.
Even though agriculture operations don’t place limits on human operators, the rotation of the planet does. Human-driven operations are generally not feasible in the dark of night. However, Ag equipment providers note that at key points in the planting or harvesting cycle, due to weather or seasonal changes, it is essential for their equipment to run non-stop to make the most of an ideal-conditions window. In some ways, this is similar to package trucks operating at max utilization in a seasonal holiday window.
Remote Support
It has become the norm for companies deploying automated trucks and robotaxi’s to have remote operations centers that ingest data from vehicles operating in logistics centers or on the public roads. This serves as quality assurance for their customer, in terms of both safety and operational efficiency. Also, there are times that the vehicle will encounter an unusual situation it doesn’t know how to handle, causing it to stop in a safe way, typically pulling out of traffic. An on-road example would be dealing with a construction zone with confusing temporary lane markings. The remote operator can view sensor data to understand the situation and provide instructions to the vehicle. Is there an analog for this in agriculture autonomy? I put this question to Jahmy Hindman, John Deere’s CTO, who said remote staff in an operations center monitor the data from planting operations to be able to assure the farmer that the planting was completed in the proper manner.
Remote support for on-road vehicles is usually envisioned as a beehive-type operations center handling many trucks. But it can be a little different down on the farm. As quoted in a recent Forbes article, Praveen Penmetsa, CEO and cofounder of Monarch, noted that “Instead of one driver sitting on a tractor going up and down a row, that same driver can manage three or four tractors at a time and finish the operation faster.” So, the “operations center” here is simply the guy that would have done the driving in the pre-driverless era.
Alternately, what if the farmer sets up their equipment to run all night, and turns in for a good night’s sleep, only to come out in the morning to see it stopped midway through the job due to tree falling onto the field? A more sophisticated remote operations center can detect such a situation and alert the farmer.
In the fields, basic driving occurs as well. Farm machines must move from parking areas to the fields, dealing appropriately with other vehicles and workers operating in the same space. Rather than apply costly autonomy specific to this situation, it may make more sense for a remote operator to directly drive the vehicle to its starting point in the fields.
Inter-Vehicle Coordination
During harvest time, Ag vehicles at times need to position themselves in coordination with other work vehicles, such as a harvester transferring grain or corn kernels to an adjacent haul truck. Depending on the terrain and soil conditions this could be straightforward or exceedingly tricky. While I don’t know of an analog in on-road logistics, here we see a similarity to construction operations such as a digger machine dumping material into a haul truck. Mr. Hindman noted that in serving both the agriculture and construction industries, this is the kind of overlap they are looking for in planning product introductions in both sectors.
Retrofit Autonomy
Farm equipment is designed to have a long lifetime. For the providers of technology, they have to think differently than the typical car- or truck-maker that brings out new tech only in the latest model. In discussing this with Jorge Heraud, VP Automation & Autonomy, John Deere, he stressed that vehicles which already have drive-by-wire and the necessary computing power can receive over-the-air updates to enable auto-driving functionality. Similarly, AGCO said their Fendt TI Headland is available on select new Fendt machines and can be installed through a software unlock on older Fendt machines.
The auto industry has begun to take this approach as well. Vehicles are being produced which are “hardware complete” for a particular level of autonomy, whether the software or regulations are ready or not. When the non-hardware issues resolve, a software upgrade can provide a new driving experience to the customer.
Beets Bring It All Together
While Ag operations may appear to be vastly different from over-the-road trucking, the two sectors have converged in a unique application of truck platooning technology. Kratos Defense & Security Solutions, Inc. has teamed with the Minn-Dak Farmers Cooperative (MDFC) to launch self-driving trucks, easing the truck driver shortage in ag hauling.
The retrofitted solution adapts “Leader/Follower” truck platooning to haul harvested sugarbeets between piling stations and a granulated sugar processing plant in Wahpeton, North Dakota.
The collaboration between Kratos and MDFC, one of America’s largest sugarbeet shareholder/grower cooperatives, was fostered by Grand Farm, a non-profit group focused on facilitating agriculture technology innovation headquartered in North Dakota.
Maynard Factor, VP of Business Development for the Kratos Unmanned Systems Division, said, “Kratos develops and fields transformative, affordable systems, platforms, and products for national security, and ensuring the agriculture supply chain using driverless technology directly aligns with our core company objectives. Our focus here is on the niche, short-haul trucking routes where Kratos’ technology is available today that can solve driver shortage issues impacting agriculture hauling capacity and, therefore, the supply chain.”
Significant effort, cost, and planning is required to ensure haul capacity meets national harvest quotas. Over 50,000 trucks a day can be deployed during peak sugarbeet harvesting season, and the Kratos Leader/Follower platoon is an enabling technology that the agriculture industry can now use for optimizing allocation of available labor to bolster the supply chain.
MDFC is owned by approximately 500 Shareholders/Growers who collectively grow just over 100,000 acres of sugarbeets and is part of the domestic sweetener industry. Mike Metzger, VP of Agriculture, said, “Our Cooperative’s goal is to take this technology to the next level by incorporating it into our commercial truck fleet that brings the sugarbeets from receiving stations to our factory for processing. It’s no secret that there is a gross shortage of commercially licensed truck drivers, especially in rural areas like ours. The deployment of driverless vehicle technology will undoubtedly help alleviate these labor shortages and improve the overall safety and efficiency of our fleet.”
Retrofitting driverless technology is an ideal solution for organizations like Minn-Dak that already have an existing fleet and logistics operations. It enables them to use their harvest trucks without having to invest in brand new “purpose-built” robotic vehicles. Additionally, the Kratos Leader/Follower platoon offers several advantages to logistics managers who can now pair available truck drivers with driverless trucks to enhance hauling productivity.
Summary
There are still strong and well-funded players bringing robotaxi and automated trucks to market. I remain bullish on the overall on-road autonomy industry, although the fate of specific AV startups remains to be seen.
How might the AV on-road guys start to play with the ag/construction guys and vice versa? As the layoffs mount up from companies like Embark – who couldn’t get their product to market before running out of money – could these engineers and programmers be re-purposed to Ag?
Potentially, tech from a struggling or failed startup in the on-road space could be re-purposed to Ag.
John Deere is playing the leading role in the Ag space. Startups can see a way forward with these types of companies showing an appetite for acquisitions. As I’ve said repeatedly since the tide turned last fall, the automated driving space has shifted to a deep pockets long view game.
The most striking commonality between these sectors? Lack of drivers to operate vehicles and machinery. These demographic shifts in our society show no signs of abating; indeed, they will probably get steadily worse. The technology enabling machines to take over tasks key to our economy is here to stay, and it is finding its way into many adjacent sectors. The interplay will be interesting to watch, let’s see what the next crop of autonomous products brings.
