We took two boards with Solo CPU and two boards with Quad Processor into the chamber testing. In first setup we mounted the isolation breakout only for two of them. The remaining two boards were testing without the serial console buffers. The test was running for over on hour in temperature -40°C or below. After we went under the temperature between -5 to -10°C the boards without isolation started failing. The board with serial console breakouts are working perfectly in every temperature.

Then we changed the setup - we took of the breakouts from 2 boards and mounted them on the failing ones. After we went to lower temperatures the boards with console isolation (which were failing in previous setup) now booted every time. Two boards without isolation (which working fine when isolation was mounted) started failing. This way we were sure that the isolation resolved the reset problem for every board. Here you can see the result from one of tested setup (boards 1 and 3 used isolation, boards 0 and 2 were without them):

iMX6 TinyRex reset testing

Board 0 with MAC 00:0d:15:00:d4:26

RST-ALL | RST-OK | RST-ERR

221 | 111 | 110

Board 1 with MAC 00:0d:15:00:d4:25

RST-ALL | RST-OK | RST-ERR

221 | 221 | 0

Board 2 with MAC 00:0d:15:00:d4:27

RST-ALL | RST-OK | RST-ERR

221 | 110 | 111

Board 3 with MAC 00:0d:15:00:d4:20

RST-ALL | RST-OK | RST-ERR

221 | 221 | 0

We also did some reset test with long off period in -60°C. Boards with isolation booted up into U-Boot every time.

Main script

We created a long running script for testing the reset. It repeatedly switches on and off the boards to simulate heavy reset conditions. As a result it outputs the statistics and logs into the separate files.

Script is starting by preparing the files which will be used during the test. Apart of summary log file it creates also one separate file to store the current U-Boot output for every board. In each loop are these logs overridden. It also setup variables and the statistics output.

Then the script goes into a infinite loop (it will be terminated by hitting Ctrl+C escape). At first all serial terminals are open - we use picocom terminal. They are opened in separate sessions using screen command. Then we switch on the power and wait for U-Boot to start. The script will then look if the log file contains the string U-Boot at least twice. This way we will know that the boot was completed (you can find string U-Boot in the start and also in the very end):

Script started on Fri 25 Sep 2015 08:31:02 PM CEST

picocom v1.4

port is : /dev/ttyUSB1

flowcontrol : none

baudrate is : 115200

parity is : none

databits are : 8

escape is : C-a

noinit is : no

noreset is : no

nolock is : no

send_cmd is : ascii_xfr -s -v -l10

receive_cmd is : rz -vv

Terminal ready

U-Boot 2014.10 (Jun 11 2015 - 18:26:04)

CPU: Freescale i.MX6SOLO rev1.1 at 792 MHz

Reset cause: POR

Board: MX6 Tiny Rex

I2C: ready

DRAM: 1 GiB

MMC: FSL_SDHC: 0, FSL_SDHC: 1, FSL_SDHC: 2

In: serial

Out: serial

Err: serial

Net: FEC

Hit any key to stop autoboot: 0

ethaddr=00:0d:15:00:d4:25

Rex U-Boot >

According to the U-Boot output script will update the statistics. To simulate the worst condition, we will close the picocom terminal in every loop. After the power socket is turned off, the loop starts another iteration.

If Ctrl+C is pressed, testing loop is terminated. In the escape function we close the screen session with opened picocom.

Below you can find the complete script:

#!/bin/bash

# Script used for reset testing of iMX6 TinyRex

# Parameters

# $1 number of board used in testing

# it is assumed that serial consoles are connected to ttyUSB0, ttyUSB1 and ascending

# Before running the script

# plug in the board into the Lan manageable power socket - use socket 4

# disable booting (use setenv bootcmd)

# connect board to control PC (Acer laptop) with a FTDI cable

# be sure all the scripts are located within "/home/fedevel/tiny-testing" directory

# picocom - serial console is run in bacground 'screen' session with output to uboot.log

# during the test we will close the terminal

echo "$(date +\%Y/\%m/\%d-\%T) Start iMX6 TinyRex reset testing"

cd /home/fedevel/tiny-testing/ # change path if needed

rm toggle_lamp 2> /dev/null # remove lock file for socket power switch if exists

num_b=$1

finish_test_now() {

echo "$(date +\%Y/\%m/\%d-\%T) Ctrl+C Detected: End of the test"

precced=0 # end of the loop

for (( i=0; i<$num_b; i++ ))

do

screen -X -S "picocom.$i" quit # quit picocom with associated screen

echo "Picocom.$i session closed" >> testing-reset.log

done

exit;

}

for (( i=0; i<$1; i++ )) # create and update all files

do

touch uboot.$i.log # create temp log file

cp /dev/null uboot.$i.log # clear log file

mac[$i]="00:00:00:00:00:00" # prepare variables for mac adresses

#variable for statistics

num_all[$i]=0

mun_suc[$i]=0

mun_uns[$i]=0

done

time_now=`date +%Y-%m-%d.%H:%M`

mv testing-reset.stat old-log/testing-reset.stat.$time_now # store old statistics if exists

touch testing-reset.stat # create new statistics file

mv testing-reset.log old-log/testing-reset.log.$time_now # store old log if exists

touch testing-reset.log # create whole log file

# prepare statistics

echo -e "iMX6 TinyRex reset testing\n" > testing-reset.stat

for (( i=0; i<$1; i++ )) # create and update all files

do

echo "Board "$i" with MAC "${mac[$i]} >> testing-reset.stat

echo " RST-ALL | RST-OK | RST-ERR" >> testing-reset.stat

echo -e " 0 | 0 | 0\n" >> testing-reset.stat

done

proceed=1

trap finish_test_now 2 # if Ctrl+C detected; exit the main loop

#turn off before start

bash ./toggle_lan_power.sh 1 off

sleep 5s

while [ $proceed -eq 1 ]

do

# open picocom in a detached screen window; print the serial console output to log file;

for (( i=0; i<$1; i++ )) # create and update all files

do

screen -dmS "picocom.$i" script -f -c "picocom /dev/ttyUSB$i -b 115200" uboot.$i.log

echo -e "Picocom.$i session opened" >> testing-reset.log

done

bash ./toggle_lan_power.sh 1 on # turn on the board

sleep 10s

for (( i=0; i<$1; i++ )) # create and update all files

do

# look for U-Boot string and count it - if occurred 2 times bootloader printed the whole message

res[$i]=`grep -ic "U-Boot" uboot.$i.log`

num_all[$i]=`expr ${num_all[$i]} + 1` # increment number of loops

if [[ "${res[$i]}" -eq 2 ]]

then

mun_suc[$i]=`expr ${mun_suc[$i]} + 1`

else

mun_uns[$i]=`expr ${mun_uns[$i]} + 1`

fi

# store temp files to complete log

cat testing-reset.stat >> testing-reset.log

cat uboot.$i.log >> testing-reset.log

echo "" >> testing-reset.log # add new line to the log file

screen -X -S "picocom.$i" quit # quit picocom with associated screen

echo -e "Picocom.$i session closed" >> testing-reset.log

done

# update statistics

echo -e "iMX6 TinyRex reset testing\n" > testing-reset.stat

for (( i=0; i<$1; i++ )) # create and update all files

do

temp=`grep -iw "ethaddr" uboot.$i.log`

if [[ -n "$temp" ]]

then

mac[$i]=`echo $temp | grep -o 'ethaddr=.*' | cut -f2- -d'='` # select mac address from u-boot print

fi

echo "Board "$i" with MAC "${mac[$i]} >> testing-reset.stat

echo " RST-ALL | RST-OK | RST-ERR" >> testing-reset.stat

printf " %6d | %6d | %6d\n\n" ${num_all[$i]} ${mun_suc[$i]} ${mun_uns[$i]} >> testing-reset.stat

cp /dev/null uboot.$i.log # clear temp log file

done

clear

cat testing-reset.stat

bash ./toggle_lan_power.sh 1 off # turn off the board

sleep 5s

done

Power switch control script

For switching on and of the LAN power sockets we used a script called Lampengeist created by rewoo. In the script we only changed config variables and added more testing outputs (or redirect some of them to our file instead of system log).

The use of this script is really easy. You just need to specify number of the socket to be changed and the desired new state:

./toggle_lan_power.sh 1 on

The log file will inform you about the update:

2015/09/25-20:32:07 State of 192.168.0.254 socket 1 toggled on by root

This approach have one disadvantage. We cannot check if the switch actually updates the socket state based on our request. It is done without no feedback. This way we encountered some loop numbers when the board was not properly supplied.

Here you can find our updated file:

#!/bin/bash

# This script toggles a socket on

# "energenie LAN programmable power connector"

typeset -a connectors=( 192.168.0.254 )

typeset -a passwords=( 1 )

typeset log_file="testing-reset.log"

typeset toggle="" state_dir="/home/fedevel/tiny-testing" lock_file="/home/fedevel/tiny-testing/toggle_lamp"

typeset -i i=0 socket=1

typeset -r connectors passwords state_dir

while [ -e ${lock_file} ]; do

if [ $i -gt 10 ]; then

logger -p user.error -t `basename $0` -s -- "Could not execute - lock file detected."

echo "Please contact administrator if problem exists for longer time." >&2

exit 3

fi

i=`expr $i + 1`

sleep 2

done

touch $lock_file

################# FUNCTIONS ###################

usage() {

cat << EOF

You called ${0} with unsupported option(s).

Usage: ${0} [1|2|3|4] <on|off>

Numbers 1 to 4 stands for the socket number. If no socket is given, it will

toggle socket 1 per default.

Please try again.

EOF

}

get_states() {

# get states of sockets

if [ $# -ne 1 ]; then

return 1

else

srv=$1

fi

states=( $(curl -f http://${srv}/status.html 2>/dev/null | sed -r "s/(.*)((ctl.*)([0|1]),([0|1]),([0|1]),([0|1]))(.*)/\4 \5 \6 \7/") )

}

toggle() {

local server="" str_state=""

local -i i=0 state sckt

if [ $# -ne 3 ]; then

return 1

fi

while [ $# -gt 0 ]; do

case $1 in

1|2|3|4)

sckt=${1}

shift

;;

"on")

str_state="${1}"

state=1

shift

;;

"off")

str_state="${1}"

state=0

shift

;;

*)

server="${1}"

shift

;;

esac

done

# poll status and toggle only if needed

get_states ${server}

if [ ${state} -ne ${states[$( expr ${sckt} - 1 )]} ]; then

curl -f -d "ctl${sckt}=${state}" http://${server}/ &>/dev/null

logger -p user.info -t `basename $0` -- "state of ${server} socket ${sckt} toggled ${str_state} by ${LOGNAME}"

echo -e $(date +\%Y/\%m/\%d-\%T)" State of ${server} socket ${sckt} toggled ${str_state} by ${LOGNAME}" >> ${state_dir}/${log_file}

fi

}

persist() {

# for cron job use only

# saves state of sockets

local state_file

local -i i=0 j=0

while [ ${i} -lt ${#connectors[*]} ]; do

state_file=${state_dir}/${connectors[$i]}

if (curl -f -d "pw=${passwords[$i]}" http://${connectors[$i]}/login.html 2>/dev/null | grep -q 'status.html'); then

logger -p user.info -t `basename $0` -- "Save states of ${connectors[$i]} to file ${state_file}"

# get states

get_states ${connectors[$i]}

echo "SavedStates=( ${states[*]} )" > ${state_file}

j=0

while [ $j -lt ${#states[*]} ]; do

j=`expr ${j} + 1`

toggle ${j} off ${connectors[$i]}

sleep 1

done

curl -f http://${connectors[$i]}/login.html &>/dev/null

logger -p user.info -t `basename $0` -- "States saved and all sockets switched off"

else

logger -p user.error -t `basename $0` -s -- "Login to ${connectors[$i]} failed."

fi

i=`expr ${i} + 1`

typeset +r state_file

done

}

recover() {

# recovers states from state file

local state_file new_state

local -a SavedStates

local -i i=0 j=0

i=0

while [ ${i} -lt ${#connectors[*]} ]; do

typeset -r state_file=${state_dir}/${connectors[$i]}

if [ -r ${state_file} ]; then

source ${state_file}

if (curl -f -d "pw=${passwords[$i]}" http://${connectors[$i]}/login.html 2>/dev/null | grep -q 'status.html'); then

logger -p user.info -t `basename $0` -- "Restore socket states from ${state_file} to ${connectors[$i]}"

j=0

while [ ${j} -lt ${#SavedStates[*]} ]; do

if [ ${SavedStates[$j]} -eq 0 ]; then

new_state="off"

else

new_state="on"

fi

j=`expr ${j} + 1`

toggle ${j} ${new_state} ${connectors[$i]}

sleep 1

done

curl -f http://${connectors[$i]}/login.html &>/dev/null

logger -p user.info -t `basename $0` -- "Socket states restored and switched on if needed."

else

logger -p user.error -t `basename $0` -s -- "Login to ${connectors[$i]} failed."

fi

rm ${state_file}

else

logger -p user.error -t `basename $0` -s -- "Could not read file ${state_file}"

fi

i=`expr ${i} + 1`

done

}

common() {

# common mode

local -i i=0

while [ ${i} -lt ${#connectors[*]} ]; do

state_file=${state_dir}/${connectors[$i]}

if [ -e ${state_file} ]; then

# state file exists -> do not toggle life, change in state file only

if [ ${new_state} = "on" ]; then

new_state=1

elif [ ${new_state} = "off" ]; then

new_state=0

fi

socket=`expr ${socket} - 1`

source $state_file

if [ ${SavedStates[${socket}]} -ne ${new_state} ]; then

SavedStates[${socket}]=$new_state

echo "SavedStates=( ${SavedStates[*]} )" > ${state_file}

logger -p user.info -t `basename $0` -- "Toggled state of socket ${socket} to ${new_state} in state file by ${LOGNAME}"

fi

else

if (curl -f -d "pw=${passwords[$i]}" http://${connectors[$i]}/login.html 2>/dev/null | grep -q 'status.html'); then

toggle ${socket} ${new_state} ${connectors[$i]}

# curl -f -d "ctl${socket}=${new_state}" http://${connectors[$i]}/ &>/dev/null

curl -f http://${connectors[$i]}/login.html &>/dev/null

else

logger -p user.error -t `basename $0` -s -- "Login to ${connectors[$i]} failed - Common mode."

fi

fi

i=$( expr $i + 1 )

done

}

############# END FUNCTIONS ##################

typeset -r curl_bin="$(which curl | head -n 1)"

if [ -z "${curl_bin}" ]; then

echo "Tool curl not found. Please install it."

exit 1

fi

if [ $# -lt 1 ]; then

echo "No action provided. What should I do?"

exit 1

fi

while [ $# -ge 1 ]; do

case ${1} in

"on"|"off")

new_state=${1}

mode="common"

;;

1|2|3|4)

socket=${1}

mode="common"

;;

"-r"|"--recover")

mode="recover"

;;

"-s"|"--save")

mode="save"

;;

*)

usage

rm $lock_file && exit 2

;;

esac

shift

done

case ${mode} in

"recover")

recover

;;

"save")

persist

;;

"common")

common

;;

esac

rm $lock_file && exit 0 || exit 1