21

Today was a day at work that I felt like I made a significant contribution. It was not a lot of code. Actually it was a difference of 3 characters.

I am developing an industrial server so that my employer can provide access to their machines to enterprise industrial systems. You know, the big boys toys. Probably in fucking java...

Anyway, I am putting this server on an embedded system. So naturally you want to see how much serving a server can serve. In this case the device in more processor starved than memory starved. So I bumped up the speed of the serving from 1000mS to 100mS per sample. This caused the processor to jump from 8% of one core (as read from top) to 70%. Okay, 10x more sampling then 10x approx cpu usage. That is good. I know some basic metrics for a certain amount of data for a couple of different sampling rates.

Now, I realized this really was not that much activity for this processor. I mean, it didn't seem to me that it "took much" to see a large increase of processor usage. So I started wondering about another process on the system that was eating 60 to 70 % all the time. I know it updated a screen that showed some not often needed data from its display among controlling things. Most of the time it will be in a cabinet hidden from the world. I started looking at this code and figured out where the display code was being called.

This is where it gets interesting. I didn't write this code. Another really good programmer I work with wrote this. It also seemed to be pretty standard approach. It had a timer that fired an event every 50mS. This is 20 times per second. So 20 fps if you will. I thought, What would happen if I changed this to 250mS? So I did. It dropped the processor usage to 15%! WTF?! I showed another programmer: WTF?! I showed the guy who wrote it: WTF?! I asked what does it do? He said all it does it update the display. He said: Lets take to 1000mS! I was hesitant, but okay. It dropped to 5%!

What is funny is several people all said: This is running kinda hot. It really shouldn't be this hot.

Don't assume, if you have a hunch, play with it if its safe to do so. You might just shave off 55 to 60 % cpu usage on your system.

So the code I ended up changing: "50" to "1000".

Comments
  • 4
    So the moral of the story is ... You use less CPU if you use it less?
  • 6
    Usually, 20 display updates per second is insane for an embedded system because either the underlying data don't change that quickly, in which case the update rate is useless, or they do, but then you can't even read the display anymore. Once or at most twice per second is usually fine.

    Possible reason why this costs that much CPU time: if the display is attached via e.g. parallel I/O, there's usually busy waiting involved because you can't really do microsecond delays via a timer interrupt.

    Another trick is not always doing a full display update. If most of that won't change, you can instead do only a differential update.
  • 2
    If it’s mostly in a cabinet, why have it continuously update a screen at all?
  • 2
    @Root Production test, installation, troubleshooting, maintenance.
  • 1
    @Fast-Nop “push/hold button for stats” 🤷🏻‍♀️

    I realize that’s a bit silly, but if it makes that much of a difference...
  • 3
    @Root I wouldn't do that, for two reasons. First, it increases the part list and requires one more hole in the front panel plus some lasered label. Second, the software would behave differently when you look at it.
  • 2
    @Fast-Nop Didn’t mean it as a serious suggestion 😅
  • 3
    The size of contribution is not measured on loc but in result.

    Think of the cost of the comma vs dot that threw one venus sond off course, fixing that would have contributed leagues with a one char change.

    And your change probably prolonged lifetime of the device due to less heat.
  • 1
    Polling is a sign of a shitty programmer, shitty hardware, or both.

    That UI-updating code should be using interrupts or some other non-polling thing. Hardware state changes, update the UI. Maybe even implement "damage", so a hardware event that changes a given value only has to re-render and display section of the UI. Conceivably, the panel could run for hours without a rerender, or days with only parts of the screen being re-rendered.
  • 2
    That is an excellent example for workaround versus fix:
    Actually making the update code do nothing if nothing changed would have been a fix.
    Just reducing the update interval is a workaround.
  • 2
    @natesymer For embedded systems, you often don't have GPU-like stuff where you just DMA the data into the graphics memory.

    Instead, you have e.g. a 2x16 character text display, often interfaced via GPIOs. You can't do the delays via a timer interrupt because that would require a microsecond interrupt.

    @Oktokolo With the side effect that the timing becomes non-deterministic, depending on the displayed data. That may or may not be an issue.
  • 1
    @natesymer The screen is animated when an e-stop occurs. Something has to act as the time base for the animations.
  • 0
    @Demolishun Delay loops for I/O timing are one thing, but a delay loop of like 20ms in an animation should be done via checking the system time as given by the timer interrupt, and most importantly, returning if the target time hasn't been reached yet.

    That way, things stay responsive even if you don't have any OS, just system timer interrupt and main application loop. On top of that, depending on the controller, you can send the controller into sleep mode if the whole application loop has run through without actual activity and let it wake up through the next timer interrupt.

    You can have cooperative multitasking by that where the "tasks" are actually just entry functions called one after the other in the main loop. Long busy waiting doesn't work for that, but if each function stores the state "I'm waiting at this point", they can return when waiting.
  • 1
    @Fast-Nop There are no delays. Its a timer that causes an event every 1 second.
  • 2
    Sometimes... A workaround is perfection.

    Especially when you don't know how deep a non workaround must go.

    The workaround is explainable....

    It fires an event every X milliseconds.

    Depending on HW, depending on who sits in front of the display and so on a "change only if necessary" can become quite a headache.

    Anyone remembers the good old burn in on a monitor xD *scnr*

    Or Energy saving kicks in...

    Or there are other events chained up to an change of the display...

    A proper solution would require finding and testing all kinds of stuff, and suddenly a simple 3 line change ends up in a 5 day marathon.

    :)
  • 1
    Oh. And good work :) *cheers*
Add Comment