US 12,139,271 B2
Fuel receptacle and boom tip position and pose estimation for aerial refueling
Deepak Khosla, Camarillo, CA (US); Leon Nguyen, Tarzana, CA (US); William Kentarou Kaneko Leach, Castro Valley, CA (US); Justin C. Hatcher, Sumner, WA (US); James L. Clayton, Meridian, ID (US); Yifan Yang, Kirkland, WA (US); Paul S. Idell, Thousand Oaks, CA (US); and Fan Hin Hung, Woodland Hills, CA (US)
Assigned to The Boeing Company, Chicago, IL (US)
Filed by The Boeing Company, Chicago, IL (US)
Filed on Dec. 29, 2021, as Appl. No. 17/565,437.
Claims priority of provisional application 63/134,085, filed on Jan. 5, 2021.
Prior Publication US 2022/0212811 A1, Jul. 7, 2022
Int. Cl. B64D 39/04 (2006.01); B64D 39/06 (2006.01)
CPC B64D 39/04 (2013.01) [B64D 39/06 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method of fuel receptacle and boom tip position and pose estimation for aerial refueling, the method comprising:
receiving a video frame;
determining via a first pipeline, within the video frame, a plurality of aircraft keypoints for an aircraft to be refueled, wherein the first pipeline performs operations including:
generating an aircraft keypoint heatmap based on the plurality of aircraft keypoints in a first region of the video frame containing the aircraft to be refueled, wherein each of the plurality of aircraft keypoints corresponds to a region of pixels in the aircraft keypoint heatmap having values dropping, according to a probability density function, with increasing distance from the center of the region of pixels; and
applying a first threshold to the aircraft keypoint heatmap to eliminate aircraft keypoints having a low confidence level;
based on at least the plurality of aircraft keypoints, determining via the first pipeline a position and pose of a fuel receptacle on the aircraft;
determining via a second pipeline a position and pose of a boom tip of an aerial refueling boom, wherein the second pipeline operates in parallel with the first pipeline, and wherein the second pipeline performs operations including:
generating a boom tip keypoint heatmap based on one or more boom tip keypoints in a second region of the video frame containing the boom tip, wherein each of the one or more boom tip keypoints corresponds to a region of pixels in the boom tip keypoint heatmap having values dropping, according to a probability density function, with increasing distance from the center of the region of pixels; and
applying a second threshold to the boom tip keypoint heatmap to eliminate boom tip keypoints having a low confidence level; and
based on at least the position and pose of the fuel receptacle and the position and pose of the boom tip, controlling the aerial refueling boom to engage the fuel receptacle;
generating a second video frame, wherein generating the second video frame comprises one or more of;
based on at least the position and pose of the aircraft, overlaying an aircraft model projection in the first video frame; or
based on at least the position and pose of the boom tip, overlaying a boom model projection in the first video frame: and
displaying the second videoframe with one or more of the overlay of the aircraft model projection or the overlay of the boom model projection.