Programming an Arduino In a Realtime Environment

[rgsparber]

Programming an Arduino gets challenging when real-time events are mixed in. Without care, the software can miss these events resulting in a flaky system. Push the button, nothing happens. Push it again, and maybe it works. Another jab and we are back to nothing happening.

I offer a programming technique, that does not rely on interrupts, to enable code to catch every button push while letting other code execute.

If you are interested, please, click here.


Your comments are welcome. All of us are smarter than any one of us.


Thanks,

Rick

[DIYer]

Thanks rgsparber! We've added your Arduino Asynchronous Event Execution to our Electronics category,
as well as to your builder page: rgsparber's Homemade Tools. Your receipt:

Arduino Asynchronous Event Execution by rgsparber

tags: Arduino

[metric_taper]

Rick, I worked on autopilots back at the start of my career. There the lead software architect did some testing to see how fast you needed to digitally scan a key press to accurately determine it was pressed. He found 30Hz was the scan rate. Every product I worked on, required the processor to perform continuous monitoring of the hardware. So the typical architecture had a running background task, and an interrupt timer, that started a foreground task. This would complete within this interrupt rate, or you would cycle slip, and that would never be release as fielded code (part of the software verification task).
So the foreground task would read the pilot input keypress hardware, and a little filter code, so you could determine it was pressed, then released (flags set,and cleared).
That's one of the problems with Arduino, there is an OS, but it's not really published (I've not taken the time to search and see what it is). All those time delay's are part of that OS.
Well, I did a search for "Arduino timer interrupt" and came up with this link:
https://www.arduino.cc/reference/en/...imerinterrupt/

You need to install that via manage libraries (tools menu).
If you click on read documentation for the timerinterrupt that brings up a github page, with sample software. I looked at examples/ISR_Switch. There the author put in debounce code. And the comments are exactly what you're wanting to have a task that is performed faithfully.

[rgsparber]

That 30 Hz figure is interesting. I suspect a lot of human factor testing went into that.

I have used interrupts on Arduino compatibles but try to avoid them.

Rick

[metric_taper]

That 30 Hz figure is interesting. I suspect a lot of human factor testing went into that.

I have used interrupts on Arduino compatibles but try to avoid them.

Rick

Not as much as you would think, the guy was and is the best system engineer I ever worked with. He was an EE by schooling. But he always did a full ground up design before ever starting to make the machine. Really good at thinking through all the pitfalls. You should have seen his revision control for software builds, as well the whole software architecture.

He set up a lab experiment from my recall of him telling how he arrived at that. But that is not what drove 30Hz, the innerloop stability control laws of the aircraft was the final design requirement. But he measured how fast you could put and remove your finger from a switch, and the length of time it was closed at the shortest human pressing was within the 30Hz. Human response is real slow.

Hardware interrupts can be a real hard thing to debug. In this case it's a built in timer of the Arduino that generates the interrupt. You just have to be sure to write the service routine that does not hang.

I recall a bug that did get fielded on one of the autopilot models. There was a hardware failure that caused a common open collector interrupt pin to be held low (single interrupt pin with multiple devices). The processor kept servicing the interrupt. There was a comment in the code that said "if it gets here, there's a failure". Well no code got written to do something. It returned from the interrupt, only to start the service routine. The only issue the pilot complained about, was a slow roll over of the aircraft, as the code that writes the sample and hold circuits was not being updated. So leakage current of the storage capacitor did it's thing. They figured out quickly they had a failure. The FAA regulations call this a slow over failure. When they happen for no reason, and never get figured out, accidents can happen. I think the 737 had this issue with a Honeywell system. Normally the pilot can see something is up as the attitude display shows the wings not level. But I understand they had a few that went overspeed before the pilot corrected the problem and lost aluminum from the wings. Night time flying, over cloud cover, or areas like northern Canada with few street lights, and a slow over that does not get noticed, as the pilot is sleeping.

[rgsparber]

Not as much as you would think, the guy was and is the best system engineer I ever worked with. He was an EE by schooling. But he always did a full ground up design before ever starting to make the machine. Really good at thinking through all the pitfalls. You should have seen his revision control for software builds, as well the whole software architecture.

He set up a lab experiment from my recall of him telling how he arrived at that. But that is not what drove 30Hz, the innerloop stability control laws of the aircraft was the final design requirement. But he measured how fast you could put and remove your finger from a switch, and the length of time it was closed at the shortest human pressing was within the 30Hz. Human response is real slow.

Hardware interrupts can be a real hard thing to debug. In this case it's a built in timer of the Arduino that generates the interrupt. You just have to be sure to write the service routine that does not hang.

I recall a bug that did get fielded on one of the autopilot models. There was a hardware failure that caused a common open collector interrupt pin to be held low (single interrupt pin with multiple devices). The processor kept servicing the interrupt. There was a comment in the code that said "if it gets here, there's a failure". Well no code got written to do something. It returned from the interrupt, only to start the service routine. The only issue the pilot complained about, was a slow roll over of the aircraft, as the code that writes the sample and hold circuits was not being updated. So leakage current of the storage capacitor did it's thing. They figured out quickly they had a failure. The FAA regulations call this a slow over failure. When they happen for no reason, and never get figured out, accidents can happen. I think the 737 had this issue with a Honeywell system. Normally the pilot can see something is up as the attitude display shows the wings not level. But I understand they had a few that went overspeed before the pilot corrected the problem and lost aluminum from the wings. Night time flying, over cloud cover, or areas like northern Canada with few street lights, and a slow over that does not get noticed, as the pilot is sleeping.

Your EE sounds like "an engineer's engineer." I can recall two or three in my entire career. The really good ones will share their thought processes.

My wife, a Computer Scientist, worked with a guy that would sit at his desk for hours on end and look at the ceiling. Eventually he would write some massive amount of code and it would work perfectly the first time. He drove his boss crazy.

As your story illustrates so well, some software bugs only become visible when there is also a hardware failure. It takes a profound understanding of the system to head off these kinds of problems.

As to the "if it gets here, there's a failure", I'm in the habit of typing "qaz" any place in my code that is left dangling. Periodically, I do a search for that unique string and clean them up. It has saved me many times.

Thanks for the great war stories.

Rick

[Claudio HG]

I developed a firmware for some distributed programmable controllers (named Automation Computer[s]), something vaguely similar to PLCs. The underlying system that control the process (look at it as an OS) have to manage many "simultaneous" events. There is a rule that say that for a sequential processor no simultaneity is possible as there is no event that can be really processed simultaneously, but we can frame "simultaneous" processes within a given time slot. Real time processes differs from non-real time not because they execute in actual real time, but within time slots that can be accettably deemed as "real time" for the whole system in which said process (or multiprocess) works.
So if for a system a time lag of say 10 milliseconds is negligible (think to something that have long reaction times), and your process is able to complete within that timeframe, then it can be considered "real time" under the perspective of such system.
The problem with Arduino is the lack of efficiency and some other subtle side effects due to the re-arrangement of the language that is based upon macros.
The mentioned programmable controllers was developed in C (and some assembly) directly on the "bare metal", an ATMega32 @16MHz clock.
From the product's specifications you can see the reaction time:

- temporal limits:
Output actuation over input transient: 0.69 ms [4]
Procedure execution, latency [1]: minimun 160 μs
Events cycle [2]: typical 160 μs
Remote execution [3]: minimun 20 ms (typical latency time about 9ms)

A note says that for each uninterruptible event the execution time increases by about 12 μs.

It works not much differently from what you did. In the main loop all inputs (including analog inputs) are checked and a register is set accordingly. Then for each flag in the register the related attached procedure is executed. It operates in what is technically known as cooperative multitasking, so it is up to the called procedure to yield the execution time to the micro OS. However because the user won't interfere with the OS, the user's program is checked against a service counter that may forcefully call the input collection subroutine so no events can be missed. Indeed the constant time of the inputs, which is a hardware thing due to the RC filters on the inputs, is slightly larger than the maximum time that may lasts between two input collection subroutine calls.
Also fast inputs, internal timers, and serial communication are all driven by interrupt, and a very small interrupt service routine record the data from those fast sources.

I have made a Code Framework to create asynchronous tasks easily, try to have a look at the videos in my YT channel Accidental Science.

[rgsparber]

Claudio HG,

Thanks for the tour of that project. It is a great example of a realtime sensitive system.

I've always worked in realtime critical systems and find them a lot of fun. If you ever made a long distant land line call, you likely went through a machine I helped to design. One machine I worked on had 2000 programmers that had to all play nice when it came to realtime.

Rick

[jdurand]

There's a few things that need actual simultaneous checking, I've seen cases where people thought "fast sequential is always good enough" when it's not. Pointed this out to a PhD or two at various jobs including at a national lab. Assuming expensive, single channel precision A/D conversion a bank of sample and hold circuits and an analog mux will get you there.