US 12,234,028 B2
Automated air-to-air refueling using keypoint-based perception of aircraft flight control surfaces
Joshua Neighbor, Seattle, WA (US); and Daniel O'Shea, Seattle, WA (US)
Assigned to The Boeing Company, Arlington, VA (US)
Filed by The Boeing Company, Arlington, VA (US)
Filed on Jan. 12, 2023, as Appl. No. 18/153,934.
Prior Publication US 2024/0239513 A1, Jul. 18, 2024
Int. Cl. B64D 47/08 (2006.01); B64D 39/00 (2006.01)
CPC B64D 47/08 (2013.01) [B64D 39/00 (2013.01)] 20 Claims
OG exemplary drawing
 
1. An automated air-to-air refueling (“A3R”) system for use with a tanker having a refueling boom, the A3R system comprising:
a camera connected to the tanker in proximity to the refueling boom and configured to output a video stream of the refueling boom and a receiving aircraft (“receiver”) during an A3R process, wherein the receiver includes flight control surfaces, the flight control surfaces including flaps, elevators, rudders, and/or ailerons;
a human-machine interface (HMI) located aboard the tanker; and
an electronic control unit (ECU) in communication with the camera and the HMI, wherein the ECU is configured to identify keypoints on the receiver indicative of the flight control surfaces of the receiver, track corresponding positions of the flight control surfaces in real-time using the keypoints while performing an A3R process between the tanker and the receiver, predict a change in a three-dimensional (3D) position of the receiver, as a predicted 3D position, using the corresponding positions of the one or more flight control surfaces, and output a directional indicator via the HMI that is indicative of the predicted 3D position, the directional indicator alerting a boom operator of a need to perform a breakaway maneuver of the refueling boom or another evasion action.