I just wanted to post a big thanks to Sandra Capri (Ambient Sensors) for the talk she did on pi day at the Embedded Linux conference. (And thanks to Drew for posting it here.) Based on some low hanging fruit from Sandra’s talk that I managed to understand, I made a ginormous improvement in the timing consistency of my own linux-based autopilot application. My results are too long to post here, so I wrote them up as a blog entry:
http://gallinazo.flightgear.org/uas/almost-but-not-entirely-real-time-process-control-on-linux/
My short summary: watch Sandra’s presentation if you haven’t, and then wow, the results running on my pocketbeagle were way better than I expected! Go linux!
Hey, great stuff. I’m just a little bit confused that you know some people but don’t seem to know Ardupilot can run on Linux, maintaining strict 400Hz mainloop on copter. It supports several boards, including BBB. I’m not saying you should be using it instead of writing your own (hey it’s fun to do that), but you could or feel free to take a look on the scheduler implementation for Linux boards, we set up the RT priorities since the beginning… And don’t forget to lock your pages on memory and allocate enough to not cause pagefaults later.
I have total respect for the great work the ardupilot (and px4) communities have done and are doing. These are ‘big tent’ projects that pride themselves on supporting every processor, every board, every sensor, every use case, idea, etc. This is great, and I’m really glad this approach is being embraced by a few groups. But in my opinion in leads to a lot of extra abstraction layers on top of abstraction layers, a highly complicated build system, and humans being what we are, we like to make elaborate things, and then add more stuff on top of that, and ultimately make things as fancy and impressive as possible. Again, this is all super cool and you guys are doing awesome things.
For myself though I wanted to go a different direction. I am trying to achieve high quality performance while keeping the code and system as simple as I possibly can. (I don’t know if I’ve succeeded with simplicity but I am trying.)
To this end I only support a small subset of hardware that I’ve decided has a good tradeoff between quality, cost, and performance. Large chunks of the core autopilot are written in native python – again, trying for code that is simple, concise, straightforward, understandable(?) … I also have avoided using a thread-based architecture. Instead I use a single ‘grand loop’ and non-blocking I/O. So single thread, heavy python use, university ekf, dual processor architecture (beaglebone + teensy) … crazy stuff, couldn’t possibly work! 
It’s easy to talk past each other with all this stuff because we all have our own ideas of what an autopilot should look like, what it should do, how it should be put together, etc. My ideas seem to be quite a bit different from what I’ve seen in the ardupilot/px4 communities, so I’ve been doing my own thing.
Historically I’ve been building linux-based autopilots since around 2005 when the U of MN EKF was far too heavy to run on any sort of embedded processor that wasn’t linux. Since then a lot has changed, but I like my approach and have good results with it I think.
Question: a 400hz main loop rate is great, but most of the commodity ESC’s I’ve seen can only take PWM rates of 50 hz (changing the throttle faster than that at best does nothing, or in my experience makes some of them croak.) Then ultimately you are changing the rotation rate of motors and propellers that have a lot of inertia. Does 400hz really buy you all that much, or was that more important for nuttx systems that have crazy-all-over-the-place timing jitter issues? Don’t get me wrong, being able to run your whole system at a rock solid 400hz is an awesome achievement. If you could see me right now, I’d be kneeling.
“but most of the commodity ESC’s I’ve seen can only take PWM rates of 50 hz”
wow, you’re living in a time warp. ESCs can do output rates of 32kHz these days using DShot. Even with the old fashioned oneshot125 you can do 4kHz.
We support oneshot125 with stable releases of ArduPilot now. We did our first flights with DShot last week.
The really cool thing about DShot is it is a digital protocol and supports telemetry coming back from the ESC. So you get RPM, temperature, current and voltage from all your ESCs. Very nice both for logging and potentially for control loops.
Oh, and before you tell me that at the RPMs that copters/planes run that running that higher output rates don’t matter, I’ll just let you know that there is plenty of evidence (both theoretical and experimental) that running output rates many times the prop speed does actually help.
yes, a fixed wing of the size you run does fly fine with 50Hz output. But if you started trying to fly a small plane or flying a copter then you’d want those higher rates. We recently flew a 98 gram fixed wing, and with something that small you really need high rates for both the servos and the ESC.
“Don’t get me wrong, being able to run your whole system at a rock solid 400hz is an awesome achievement”
actually, 400Hz is super slow for a copter these days. betaflight does 32kHz PID loops. We suspect the benefit becomes hard to measure above 4kHz with “normal” sized quadcopters. We’re aiming to get ArduPilot to 4kHz loop rates soon. With microsecond accuracy too.
As someone who does mostly fixed wing work, I was just asking the question about copter rates. I appreciate that you guys have put a ton of thought into it.
I’m working on a simple modular applicative stack on beaglebone and raspberry with python code. I’m used to SCHED_FIFO since my old work in C with the gnu pth on 166mmx, so … I’m interested in your approach, do you communicate about it, is it available and even free software ?
+Samuel Bucquet All this work is released under the MIT license. I have a collection of git repositories here: https://github.com/AuraUAS
I’m happy to discuss any of this. In fact, I’m happy if anyone out there finds any of this semi-interesting for 5 minutes.
So far I haven’t found python to be a bottleneck or limiting factor in any of the things I’ve been doing. The bottlenecks have been more in the realm of IO … things like uart speeds, micro sd card write speeds, etc. My goal is to do everything at a 100hz frame rate though … for people needing to do things at kHz rates, you probably don’t want to be looking over my shoulder. I also semi-cheat by offloading all the sensor IO to a teensy-3.2 and have written a simple arduino sketch to collect all the data in my system and send it down to the linux host. It’s a bit different approach from what I’ve seen elsewhere, but it has been working well for my needs. I like to think that I have two simpler apps running (distributed) versus one monolithic highly complex app. Plus I get to run linux and python on the main host system.
I’m not suggesting my approach is better than any other, or more or less appropriate. Mostly I’m trying to keep my life as simple as possible, and simultaneously trying to see what quality results (in terms of interval timing, EKF accuracy, flight precision, etc.) that I can achieve. I think I do ok… here is a video that shows 3 autolands.
I overlay a HUD that draws what the flight controller is thinking on top of the video (this is done in post processing.) If the hud (ekf) and augmented reality align perfectly with the video, things are working well. Where there are discrepancies, that shows errors in my system, errors in the ekf, etc. (Side note: I did a similar video with a px4 system about a year ago and the results were not as good as mine, so even though my system is far from perfect, I think it’s in the mix … for a couple reasons, I can’t share the px4 video though … if anyone wants to send me their own video + flight data for their system I could render out a video and compare results.)
https://www.youtube.com/watch?v=1fgVt5OetlI
(sorry for the long message, apparently I like to type!)
@clolsonus Hi, congrats on your project, this is far superior than anything i’ve done so far (and much more advanced) but I share the same vision about simplicity and modularity. And Python is the right tool for me.
Thanks for the work available with a free software licence, i’ll keep track of it 
My process exchange data via Redis streams (http://antirez.com/news/114) and what is not data oriented (like init or low level security, e.g. action oriented) is via xml-rpc, which is not perfect but very well supported and rather simple too.
I work on marine systems, so my RT needs is a little less drastic, and I enjoyed working with the bbb the last years till now.
I appreciate the discussion.
Cheers
@Samuel_Bucquet I hadn’t seen redis before…it looks pretty cool! I use something called a property tree which at first glance looks like a tiny subset of redis … I have a hierarchical tree of key/value pairs (an in memory db) that modules publish their important data (state) to, and then all the other modules can read out of there. It allows modules to be loosely and flexibly connected and is a simple way to exchange messages. (See aura-props in my AuraUAS github archive for more details on that.) This concept came out of the FlightGear (open-source) flight simulator project, and I have been using it ever since.
The property tree is not a true messaging system though, it’s more like a dead drop in a spy show (I must be watching too much netflix lately!)
A few years ago I worked with a small company to develop a marine UAV (salt water proof, tough skin, long endurance, etc.) under NOAA funding. The purpose was to do marine debris surveys. We never quite got everything to market and meanwhile the world probably passed us by. (And everyone switched over to multi-rotors anyway.) The NOAA work was largely based on open-source libraries and code and many things I carried over from my FlightGear flight sim work.
After I joined the U of MN UAV research lab, I wanted to save as much of that effort as I could, so I rescued all the open-source components, integrated python into the main loop, and then filled in most of the missing gaps with new python code. This is where AuraUAS came from. I also did a ground station from scratch. I use a python script on the ground as a data broker and web server, then the map and instrument panel and debugging screen are all served out as live html5 pages (so pick your favorite handheld device or laptop to use as the ground station interface.)
@clolsonus Yes Redis is the main component of my architecture (and I don’t have to maintain it), I tried first with my own solution like you, but mine were not sturdy enough for my needs, so I experienced with spreads, memcached, redis since 2014 and now with the new stream data support in soon to be released Redis 5.0.
My journey is a little similar to yours and I capitalize on my previous works too.
Today I love Python because It allows me to write all the tools around the main core (and the main core for now) like a launcher, a grapher (with graphviz) for the workflow of the data connected process, I can automate a lot of things pertaining to the software quality department.
@clolsonus
and a simple application example.
I stil have to publish a working version on github, but I keep modifying and changing some main components (for three years now, what a mess !), as you understand it is much of a research/studying process for me.
That’s why I’m very glad to be able to peek at your solutions and also that we made some similar choices.
Cheers!
@clolsonus Maybe we can pursue this discussion elsewhere (I have polluted enough your thread, sorry!)
@Samuel_Bucquet No worries, I do this to other people’s threads all the time … I’m also happy to chat via email, I’m sure my email address is all through the headers of the AuraUAS code and probably google-able as well, and I’m sure you could look up my university email too from http://umn.edu. It is fascinating to see the different ways people attack similar challenges and try to understand the core design principles and objectives and how this all came together into a final working solution.
@clolsonus “…
It is fascinating to see the different ways people attack similar challenges and try to understand the core design principles and objectives and how this all came together into a final working solution.”
My thoughts exactly !
I’ll keep going the discussion later (now is the end of my lunch time)
… and I’m just finishing breakfast here in MN right now.