I thought it would be fun to share the current status of my glass cockpit video overlay system. The graphics are generated by a set of open-source scripts written in python/opencv and are applied after the flight by automatically correlating the flight video with flight data log. Please notice the augmented reality items (like the flight path history and the sun/moon markers.) These are placed based on the onboard EKF and give an indication of how accurate (or inaccurate) the EKF is. Also, jump ahead a couple minutes and watch the fully autonomous landing into a fairly constrained area. Then jump another minute or two ahead where I pick up the airplane and show you the whole flight path in the video (again using augmented reality tricks.)
So, if I’m reading this right, this is all after-the-fact ‘effects’? No real-time translation of data sent to pilot?
Cool and all for youtube but, how does this help makers build a real system that helps the pilot fly?
Hi @Black_Merc , a couple comments in reply.
(1) The system is built around opencv/python which can take a live video feed. I’ve just never managed to put together an FPV system that feeds the live video into a computer for live frame capture. I’ve poked around … maybe a box that converts the video to a rtsp stream would be interesting. I just haven’t had the budget to try random things and some of this gear gets expensive to try if you aren’t sure it will work. Also, the system is pretty heavy to run onboard before transmitting video down. So in theory the system could be adapted for real-time use, but it would take some extra hardware and software glue and experimentation to put it all together.
(2) Even for the task of reviewing a flight, I’ve found the system really helpful for understanding the behavior and tuning of our EKF, evaluating PID gains, evaluating mission/task performance, etc. In the flights previous to this one shown, I had noticed some extra vibration/jello and it turned out to be a slightly bent prop shaft. I probably wouldn’t have even noticed except for reviewing the flight with this system.
For me, my main objective is to use this as a tool to evaluate and tune the performance of the systems and software I develop. For the missions I fly, I minimize the amount of hand flying. The FPV stuff is super cool, but I’ve never really done any of it myself.
Thanks for your comments and questions!
I would certainly love to know more about this system and the aircraft being used. Where can I get more information please?
Hi @Alf_Stockton , feel free to ask more question since I can’t possibly share everything in one short post. This flight was a joint research project between the entomology dept (insects) and the aerospace engineering dept. at the U of MN. We built a flying insect catcher out of an X-UAV talon. The autopilot hardware is built by a spin off company from our UAV lab called Bolder Flight Systems. The flight controller is linux-based (not pixhawk hardware). The flight control software/firmware is AuraUAS (all open-source) which can be found on github. One of the unique things (I think) about AuraUAS is that it is a hybrid C++/python app and much of the critical navigation/mission code is written directly in python and runs as part of the critical main loop at 100hz. The auto-land logic for instance is written entirely in python. Autopilots are difficult and mystical beasts, even when they are open-source, but my intention was to try to make this one simpler and easier to understand compared to most. Other’s will have to judge my success there. Hopefully the performance (as higlighted by the HUD overlay) speaks for itself. I think it’s a pretty cool system (but I’m biased) 
Right now it’s only ever flown on fixed wing aircraft and 99.999% of the world seems to have gone to mulitirotor, but maybe airplanes will come back into fashion some day?
Thank you. Excellent information. From what you have given me I can do something.
Great, if you run across any questions, feel free to holler … I can talk your ear off all day long about this stuff.
My background and therefore my interest is that I am a programmer, mainly C & Linux, a fixed wing RC pilot and dabble with Raspberry Pi & Aurdino. Combining the above to implement an autopilot would be very interesting.
I can be emailed @ alf at stockton dot co dot za if interested.