Japanese auto giant Toyota Motor unveils their new violin-playing robot at the company’s showroom in … [+]
AFP via Getty Images
IoT or the Internet of Things has been widely deployed in the past decade as sensors became smarter, machine learning proliferated and advanced, access to WiFi, Bluetooth and other wireless communications became prevalent, and cloud storage and computing technologies matured. In general, IoT achieved intelligent networking of “things” that were typically static or stationary, through movement of data. The ongoing and imminent revolution is in the Autonomy of Things or AoT – which for purposes of this article is defined as autonomous movement of “things” or robots, either in public (mostly uncontrolled), semi-public (somewhat controlled, includes outer space) or private (highly controlled) spaces.
The most publicized examples of AoT are autonomous vehicles and trucks (AVs), and the role and necessity of LiDAR for enabling this application in public spaces has been reported extensively. The focus of the current article is to examine how LiDAR impacts AoT in other applications like construction, logistics, mining, manufacturing, agriculture, medicine and gaming. In these cases, the primary role of LiDAR is to enable perception for moving things – robotic arms, robots, machines, industrial equipment, tools, etc. Unlike AVs where the primary function of the LiDAR is to provide a unique perception modality to enable safe operation, these other applications have productivity, efficiency and precision positioning/guidance as their primary goals (Table 1). LiDAR enables these goals by providing 3D sensing of various objects or things in the environment. Gaming is slightly different in that the LiDAR senses the human body to adjust and cue the gamer – in this case, the “thing” is a human!
Table 1: AoT Applications and Role of LiDAR
Patience Consulting LLC
Over 75 LiDAR companies exist today, with at least $4B in equity funding. Six companies have gone public via SPAC (Special Purpose Acquisition Companies) mergers and command a market valuation of ~$14B. Additional captive LiDAR efforts exist within automotive Tier 1s (ZF, Valeo, Continental) and technology companies like Waymo, Aurora, Argo and Yandex. In general, these companies are focused on the AV and ADAS markets. However, as Tesla CEO Elon Musk recently admitted – AVs are a hard nut to crack (and this from a guy who wants to go to Mars!) because it needs to deploy in a public space, with significant levels of government regulation and human indiscipline – the safety thresholds are hard to achieve and convince customers, insurers and regulators about.
Most robotics and LiDAR companies are also focusing on near term applications of AoT – primarily in private or semi-public environments, where public safety is not the paramount concern, and regulatory and consumer acceptance hurdles are lower. The business model relies on a sale to a service business that can use AoT to address immediate problems of trained worker scarcity, worker safety, inhospitable working environments, higher productivity, lower costs, manufacturing precision and quality. The challenge in these cases is to ensure that performance is consistent with the application and pricing at lower volumes – generally, automotive requirements are much more demanding in some dimensions, and simply retrofitting an AV LiDAR to non-AV applications may not be practical in all cases. For example, since these applications typically involve things moving at lower speeds, long range and high detection speed may not be critical – on the other hand, angular and depth resolution may be very important.
A great example of AoT that is currently operational is mining, which occurs in harsh terrains and environments. Caterpillar understands the benefits of autonomy – higher productivity, higher levels of worker safety and lower costs. They made significant investments in providing autonomous equipment to their customers, ranging from drilling equipment to 600 ton haul trucks which have moved ~5 trillion pounds of material to date. These efforts paid off handsomely during the Covid crisis of 2020-2021 where trained worker supply was in short supply. Caterpillar customers were able to still continue their operations with close to full capacity. The LiDAR which was custom designed to meet harsh terrain, environmental and visibility conditions provides situational awareness and obstacle detection, as well as terrain mapping to plan tool movement.
Some of the LiDAR companies addressing the non-AV opportunities in a serious way include Quanergy, Cepton and Ouster.
Quanergy recently announced a collaboration for use of their LiDAR with Shanghai Rays Information Technology. It address AoT opportunities at the Port of Qingdao, one of the largest in the world. Applications include running mobile robots and AGVs (Automated Ground Vehicles) in the port area (controlled vehicle and pedestrian traffic, well lit), as well as positioning of crane and gantry end effectors relative to bulk cargo and containers. Traditionally, these automation tasks relied on the use of very simple laser sensors (single point or single line scanning) to detect obstacles or position the end effectors. Without high resolution 3D imagery, control actions were limited (stopping the crane or AGV in front of an obstacle rather than maneuvering around it, or being able to achieve very coarse positioning, which then needed manual intervention). Enzo Signore is Quanergy’s CMO, and comments: “Quanergy delivered the M-series LiDAR with the optimal level of performance in terms of range, resolution and range accuracy to fully automate the port’s manual bulk material operation in the terminals. This highlights the power of 3D LiDARs to enable levels of automation not achievable with the 2D LiDARs (simpler laser scanners) used earlier”. For other applications like mobile robots or construction equipment in rough terrains, Quanergy is targeting their solid-state LiDAR solution to deliver high reliability under severe shock and vibration, as well as improved range for obstacle avoidance and maneuvering.
Figure 1: Quanergy M-Series LiDAR Generates 3D Point Cloud to Visualize Ship Hatch, Shanghai Rays … [+]
Shanghai Rays
Similar to issues with safety and health of human drivers in long distance trucking, the control of loading cranes in busy port environments by humans is a difficult endeavor – weather, pollution, noise and safety are major concerns, and productive lifetime human operators is limited (15-20 years). This drove the development of crane automation technologies by Shanghai Rays, the prime contractor for deploying this solution at the Port of Qingdao (see Figure 1). According to Mr. Xu Lin, Project Manager at Shanghai Rays: “traditional solutions developed 5 years ago used laser scanners with limited performance, and therefore limited autonomy levels. Since 2018, the deployment of Quanergy’s M-Series LiDAR has provided the performance necessary to achieve full autonomy at a competitive cost point”. Mr. Xu indicates that the cm level depth accuracy, the stability of the LiDAR performance over time and weather, and reliability in harsh port environments were key factors in the selection of Quanergy’s M-Series LiDAR.
Cepton was created with a focus on developing a LiDAR for ADAS, and AVs are a natural extension. Outside of ADAS/AV, the company has a strong and growing presence in the areas of smart cities, smart spaces and smart industrials. According to the CMO, T.R. Ramachandran, “applications in the smart industrials sector include AGV operation in controlled spaces, sidewalk delivery robots, construction and agricultural automation and warehouse robots. In many of these cases, LiDAR is used for obstacle avoidance, guidance and precision positioning. Customers realized that Cepton’s ADAS LiDARs integrated with a perception suite works much better than rotational lidars” (Cepton’s LiDAR uses MMT or Micro Motion Technology, essentially a frictionless voice coil driven tuning fork). Leveraging the economies of scale for ADAS makes them incredibly affordable (Cepton just publicized their design-in with a major Detroit based automotive OEM for L2/L3 ADAS). According to Dr. Ahmad Kamal Aijazi, a senior researcher at the Pascal Institute (within University of Clermont Auvergne, France): “recently developed automotive grade lidars are well suited for deployment in industrial automation tasks, either in “brownfield” factories where dust, chemicals, pollution and hazardous substances make it dangerous and difficult for humans to operate, or in cases where high range precision assembly is required. Leveraging the performance and cost trajectory of automotive lidar will will make it easier for industrial robotics to improve on automation and productivity gains over time”.
Goetting KG is a Hanover, Germany based company that has been developing automation solutions since 1965. They specialize in using a system of sensors (some built in-house) to equip AGVs with adequate navigation and obstacle avoidance capabilities, enabling operation within controlled industrial environments. Europe deploys around 5K new AGVs/year. Indoor operation of such AGVs has been adequately managed through use of guided (optical or inductive) techniques, optical sensors and other location verification techniques. The real challenge that has emerged over the past two decades is in outdoor operation. LiDAR scanners have been used in conjunction with a forward bumper assembly to sense obstructions. Speeds are limited to 6 kmph (see Figure 2). This is primarily because worker safety standards require the perception system that detects small objects (3 cm) with very low reflectivity (1.5%). LiDAR scanners could do this at ranges of 5 m in good weather, which set constraints on maximum speed and need to integrate a bumper.
Figure 2: Left, Laser Scanner Equipped AGV with Bumper. Right, LiDAR Equipped Truck without Bumper
Goetting
Ongoing work to incorporate higher performance LiDARs allows for a 3X improvement in speed, operation in bright sunshine and rain, and eliminating the need for a bumper assembly which is expensive to customize, fabricate, assemble and maintain. Hans Goetting, CEO of Goetting: “The Cepton LiDAR provides 3 cm range accuracy, and the ability to image a 1.5% reflectivity target at a 20 m range – this allows us to increase the AGV velocity by 3X, increasing capital utilization significantly. The challenge is to do this without any safety compromises – this LiDAR enables our product to achieve this. Going forward, I see our solutions gradually penetrate the public driving space as it integrates with AI based perception and driving stacks”.
Ouster positions itself as a company serving a large base of users across many applications. They serve over 500 customers in areas ranging from AVs and trucking to mining, agriculture, construction, logistics, delivery robots, port automation and yard automation. One customer is Balyo (France), focused on forklift automation for construction, mining and warehousing. Mining applications are served in collaboration with major companies like Sandvik, and emerging Chinese leaders in autonomous mining like Waytous and TAGE. A key customer in the delivery robot space is Serve Robotics (formerly Postmates, a part of which was acquired by Argo.ai). Angus Pacala, CEO: “the requirements in these diverse applications can be just as complex as in automotive applications. Ouster’s products are built to a super set of performance requirements covering the corner cases of different applications, and customization for different applications is not required. The primary perception capabilities LiDAR offers are obstacle avoidance and precision positioning for increased safety and efficiency. In terms of costs, the superior capabilities of our LiDAR allows customers to replace multiple lower cost scanning LiDARs with a single Ouster LiDAR. This makes the overall price proposition for end customers compelling.”
Outrider is a pioneer in autonomous yard operations for logistics hubs, and an Ouster customer. In the United States, about 10B tons of freight are hauled per year by truck, and there are about 10 trailers for every truck. At human operated yards, over-the-road trucks wait on average 1 hour to pick up and drop off a trailer, resulting in ~ 12 million wasted days/year. Automation trims this by 5-10 minutes, realizing an additional 1-2 million productive days/year. This efficiency also has significant impacts on congestion and flow of traffic and goods. A fair amount of trailer swapping (sounds a bit weird!) needs to occur in these yards to ensure efficient flow and distribution of goods. Typical tasks in a distribution yard include parking, trailer unhitching and hitching, loading and unloading of cargo, and final exit of the loaded trucks onto public roads. The trailer hitching/unhitching is a key process and involves disconnecting/connecting pneumatic brake between the cab and trailer section. The Outrider System uses the Ouster LiDAR (mounted around the vehicle periphery) and other optical sensors in conjunction with custom perception and control software to automate this and other activities (Figure 3).
Figure 3: Outrider Fitted Yard Automation Vehicle with Handling Robot (Blue); The Outrider System … [+]
Outrider
Key LiDAR attributes include precise 3D perception, durability in harsh environments, dirt management and low costs. Andrew Smith, CEO: “Outrider transforms electric yard trucks into autonomous vehicles that couple tractor to trailer and precisely maneuver between dock doors, parking spots, and areas for over-the-road pickup. This offers 3 pillars of value this offers to our customers – efficiency, safety and sustainability – all enabling the Yard of the Future”.
AoT applications enabled by embedding LiDAR in smart glasses deserve a dedicated article (arriving soon). Paul Travers, CEO of Vuzix, a company that focuses on industrial smart glasses is bullish on LiDAR: “Once LiDAR equipped smart glasses are unleashed, they can be used in many operations that require knowing where things are spatially – from excising tumors to positioning applications in many commercial industrial automation tasks”. A teaser for a subsequent article is presented here, for orthopedic knee surgery. Figure 4 shows the CONOPS for the current version of smart glasses that enables this type of human-assisted surgical tool movement.
Figure 4: Tracking “QR codes” are placed on platforms that are connected to alignment and pin … [+]
Pixee Corporation
A better visualization can be seen in this video and in Figure 5 below:
FIGURE 5: Smart Glass Assisted Surgery
Dr Olivier Fontes, Clinic Via Domitia Lunel, France
With LiDAR capability embedded in the smart glasses, the QR code placement and prep work is not required, and the dynamic visualization enables significant more capabilities like real time information sharing of a 3D structural holographic map for collaborations, peer advice, and training of medical students and staff. LiDAR also allows for higher quality metrology on socket, balls and pin positions in real-time as the surgery proceeds, leading to more effective outcomes.
The LiDAR requirements here are very different from automotive and other harsh environment applications – very short ranges and narrow fields of view, high angular and range resolution, low SWaP (Size, Weight and Power) and minimal thermal dissipation. It’s not here yet, but brace yourself and don’t hurt your knees – it’s coming!
