Project updates‎ > ‎

A generic C++ based framework for real time multitasking embedded systems - Part 2

posted Sep 3, 2010, 6:46 AM by Jean-Sebastien Stoezel   [ updated Sep 8, 2010, 8:49 AM ]

The logger, system monitor and the analog sampler




In part 1, I laid down the foundations of a a generic C++ framework for real time embedded systems. In part 2 I will add more functionality to it. I will introduce 2 more subsystems: the system monitor and the analog sampler.
As an example of how the framework can be used, I will then implement a data logging system called LogoDynamite.

The System Monitor

The system monitor is used to check the execution of the whole system. It periodically monitors the stack usage of running tasks, and checks if they are still running (not suspended). It also calculates the CPU load and checks the dynamic memory usage (heap). Based on the current health of the system, the system monitor could trigger a reboot to try and fix run time issues.

Calculating The CPU Load
The CPU load is calculated using an idle task. The idle task runs at the lowest priority, it only gets to run when no other task is active. A counter in incremented at each loop of the idle task. The more the counter is incremented, the less the CPU is loaded.
When no other task is running on the system, the idle task executes at maximum CPU time. Eventually it reaches a limit where it can only increment the counter that many times a second. This is the maximum counter value.
To provide an estimate of CPU load, the maximum counter value needs to be measured for a specific period of time. Once this value is recorded, the CPU load can be calculated by applying the following formula:

CPU_Load(%) = 100 - (Current_CPU_Count * 100) / Maximum_CPU_Count

Not that this formula is only valid if it is calculated at the same rate as Maximum_CPU_Count was measured at. Derating must be applied if the system cannot guarantee or does not calculate the CPU load at the same rate.

To calibrate the CPU load, follow the following steps:
- In the makefile: set RUN_MODE to RUN_CPU_LOAD_CALIB
- Clean and recompile the project
- In the logodynamite.txt conifguration file, set VERBOSITY to WARNING
- Run the generated firmware file and note the cpu_ref value displayed on serial port 0.
- In the makefile: set CPU_LOAD_REFERENCE to the cpu_ref you observed. Note that this value will be different based on builds and CPUs.

The maximum CPU calibration constant will then be displayed once a second on serial port 0:

SparkFun USB Bootloader v1.1
Boot up complete
No USB Detected
Root open
New firmware found
New firmware loaded
Boot Done. Calling firmware...



+----------------------------------------------------------------------+
|                                                                      |
|         ______________________________       . \  | / .              |
|        /                            / \        \ \ / /               |
|       |       LogoDYNAMITE!        | {========= >- -<                |
|        \____________________________\_/        / / \ \               |
|                                              . /  | \ .              |
|                                                                      |
|        V1.0 Jean-Sebastien Stoezel - js.stoezel@gmail.com            |
|        Release note: original release                                |
|                                                                      |
+----------------------------------------------------------------------+
CSystemMonitor::Task cpu_ref=079773

 Checking The Subsystem's Health

The health of a subsystem is assessed by checking its:
  • Stack usage (in percent): A 15-20% margin is desirable 
  • Running state: checks whether the task is suspended or not.

The health of the system is assessed by checking:
  • CPU load (in percent): this can be used to get a rough idea whether or not hard scheduled dead lines will be met. RMA is a good read on the subject.
  • Heap usage: dynamic memory allocation should not be used in embedded systems. In some cases it is tolerated at initialization only. It still a good idea to monitor this, just in case the code reviewed missed a call to a function that dynamically allocates memory. This is mostly used as a debugging tool.
Ideally the subsystem would reset the CPU if it had identified the system is not running in a healthy state. The system monitor could implement a scheme to toggle the CPU watchdog only when the system is in a healthy state.
For this reason, the system monitor should run at one of the lowest priorities, i.e. least important priority. This is because an infinite loop or longer loop occurring in a higher priority subsystem will always prevent the system monitor from running, and will always trigger a CPU reboot.

The implementation I propose here does not implement any kind of software triggered reset. System monitoring is provided for reporting purposes only.

If the verbosity level is set to 'INFO', then once a second the following report will be printed to serial port 0:

SparkFun USB Bootloader v1.1
Boot up complete
No USB Detected
Root open
New firmware found
New firmware loaded
Boot Done. Calling firmware...


+----------------------------------------------------------------------+
|                                                                      |
|         ______________________________       . \  | / .              |
|        /                            / \        \ \ / /               |
|       |       LogoDYNAMITE!        | {========= >- -<                |
|        \____________________________\_/        / / \ \               |
|                                              . /  | \ .              |
|                                                                      |
|        V1.0 Jean-Sebastien Stoezel - js.stoezel@gmail.com            |
|        Release note: original release                                |
|                                                                      |
+----------------------------------------------------------------------+
>
INFO,CSystemMonitor::Task name=analogSampler stack=77% is_suspended=0
INFO,CSystemMonitor::Task name=systemMonitor stack=91% is_suspended=0
INFO,CSystemMonitor::Task name=cpu_load stack=79% is_suspended=0
INFO,CSystemMonitor::Task cpu_load=40%
INFO,CSystemMonitor::Task heap=5104
INFO,CSystemMonitor::Task name=analogSampler stack=77% is_suspended=0
INFO,CSystemMonitor::Task name=systemMonitor stack=92% is_suspended=0
INFO,CSystemMonitor::Task name=cpu_load stack=79% is_suspended=0
INFO,CSystemMonitor::Task cpu_load=43%
INFO,CSystemMonitor::Task heap=5104
INFO,CSystemMonitor::Task name=analogSampler stack=77% is_suspended=0
INFO,CSystemMonitor::Task name=systemMonitor stack=92% is_suspended=0
INFO,CSystemMonitor::Task name=cpu_load stack=79% is_suspended=0
INFO,CSystemMonitor::Task cpu_load=40%
INFO,CSystemMonitor::Task heap=5104
...

This dump shows 3 tasks running with different level of stack usages. The CPU load and the stack usage is also printed.

Implementation
The system monitor is implemented by the CSystemMonitor class. It inherits all the benefits of the CSubsystem class defined in part 1. Since the system monitor is a periodic subsystem, the bulk of the implementation is handled by overriding the virtual method PeriodicHandler().

////////////////////////////////////////////////////////////////////////////////////////
void CSystemMonitor::PeriodicHandler(void)
////////////////////////////////////////////////////////////////////////////////////////
{
   uint8_t         cpuLoad     = 0;
   CSubsystem * p_Subsystem = 0;
   uint32_t i = 0;
   
   LOG_TRACE("CSystemMonitor::Task");
   
   p_Subsystem = CSubsystem::Get(i);

   while(0!= p_Subsystem)
   {
      LOG_INFO("CSystemMonitor::Task name=%s stack=%u%% is_suspended=%u",
                p_Subsystem->mp_Name,
                100 * (p_Subsystem->m_StackSize*4 -
                uxTaskGetStackHighWaterMark(p_Subsystem->m_TaskHandle)) /
                (p_Subsystem->m_StackSize*4),
                xTaskIsTaskSuspended(p_Subsystem->m_TaskHandle));
      i++;
      p_Subsystem = CSubsystem::Get(i);
   }
   
   // CPU load task
   LOG_INFO("CSystemMonitor::Task name=cpu_load stack=%u%% is_suspended=%u",
            100*(256*4 - uxTaskGetStackHighWaterMark(m_CpuLoadTaskHandle)) / (256*4),
             xTaskIsTaskSuspended(m_CpuLoadTaskHandle));
   
   cpuLoad = CpuLoadGet();
   LOG_INFO("CSystemMonitor::Task cpu_load=%u%%", cpuLoad);

   // Check heap size
   LOG_INFO("CSystemMonitor::Task heap=%u", xPortGetFreeHeapSize());
   
   CSubsystem::PeriodicHandler();
} // CSystemMonitor::PeriodicHandler



The PeriodicHandler method accesses the subsystems lists inherited from the CSubsystem class, and parses to check each subsystem's health.

The Analog Sampler

The analog sampler records analog values at a fixed rate, from the context of an interrupt. Samples are buffered and stored on the file system from the context of a task. This architecture allows for a much faster real time response, since minimal time is spent in the ISR context. Non real time- processor intensive tasks (like writing to the file system) are excuted from the context of a task. Samples are passed from the ISR to the task using a message-based system. Samples are stored to the file system asynchronously.

TODO

An Implementation Example: LogoDynamite

We use the frame work to implement a data logging system called "Logodynamite". This is a lightweight logging system, that stores analog data onto a microSD card using the FAT file system. Configuration of the logger is based on directives stored on a text file.

Configuration

LogoDynamite is configured using a text based method. Directives are stored on the microSD card, in the file called "dynamite.txt". If the Logomatic board boots up and not file is present on the card, the LogoDynamite software will create a default file.
By default settings are defined as follow:

FREQUENCY=100
P1=Y
P2=Y
P3=Y
P4=Y
P5=Y
P6=Y
P7=Y
P8=Y
VBAT=Y
MODE=ASCII
VERBOSITY=WARNING

These directives work as follow:
  • FREQUENCY: this is the ADC sampling rate, in Hz. By default, all the channels that are enabled are sampled 100 times a second. The maximum value for this field is 2000Hz. Valid values are multiple of 2000. 2, 20, 200, 2000 are valid, as well as 1, 10, 100, 1000. 13 is not a valid frequency.
  • Px: these are the channels to be sampled, P1 to P8. 'Y' means the channel will be sampled, 'N' means the channel will not be sampled.
  • VBAT: voltage battery. 'Y' means the channel will be sampled, 'N' means the channel will not be sampled.
  • MODE: 'ASCII' or 'BINARY'. If mode is 'ASCII', then samples will be stored in a CSV file. The first line of this file will contain the name of each channel enabled (P1-8 and VBAT). If mode is 'BINARY', samples for enabled channels are stored sequentially, 2 bytes at a time. In either mode, the values stored are in millivolt, hence the value 2990 means 2.99V.
  • VERBOSITY: this is mostly for debugging purposes. Valid values for this field are 'ERROR', 'WARNING', 'INFO'. See part 1 for more information on the logger.
They are some limitations with the configuration file:
  • Directives are case sensitive. 'p5' is not the same as 'P5'. All directives should be written in CAPS.
  • White spaces are not allowed. P1='Y' is correct, while P1 = 'Y' is incorrect.
  • Each line should only contain one directive

Performances

TODO

Caveats

  • In binary mode, the channel sequence is not obvious... Depending which channels are enabled, the sequence is as follow: 
  • There is no safety net for setting the sampling frequency. Hence you can overload the processor if you set a sampling rate to high and/or enable too many channels to be sampled. In this case the message "ERROR, CSubsystem::CommandGet no command available". The best way to find the maximum sampling frequency is to start by enabling all the channels to be sampled, specify a low sampling rate, set VERBOSITY to INFO. In this verbosity mode, the CPU load will be printed once a second. Increase the sampling rate until the CPU load reaches 50-60%. The CPU load should be less than 70% at any time, to insure real time responses.
Č
ċ
FW.SFE
(158k)
Jean-Sebastien Stoezel,
Sep 8, 2010, 8:48 AM
ċ
dynamite.txt
(0k)
Jean-Sebastien Stoezel,
Sep 7, 2010, 6:19 PM
ċ
logomaticv2-eframework-winarm-part2.rar
(471k)
Jean-Sebastien Stoezel,
Sep 8, 2010, 8:48 AM
Comments