Meeting Minutes: MAV

August 2020

Aug 27th

Juncheng Li

• Able to pick up the crank arm and bin at different orientation using the new pick and grasp algorithm.

• Adding validation test to the training network. It will use the same datasets as an input to calculate the accuracy of the result in order to evaluate the overall performance.

Arindam

• Able to get the network trained. The mean average precision on the test dataset is 90.2.

• Ran the network on the fetch robot. Some of the results are the following.

Aug 20th

Arindam

• Loaded some wrong parameters in the detection model, so the training didn't go as planned. It failed with error value going to infinity.

• Fixed the issue and now training again.

Wei Huang

• Add Kinect and other sensors onto turtlebot3.

• Can now project the depth image to 2D map.

Aug 13th

Arindam

• Classification training is complete. 3 versions of models are there, smaller one has 11 convolution layers 92% accuracy, biggest one has 17 convolution layers 97% accuracy.

• Training for detection now.

Juncheng Li

• Writing new pick algorithm which can make the gripper has the same orientation with the object.

• New algorithm has two parts. The first part is moving the arm a little bit above the object. The second part is to do the actual grasping which is moving the gripper down to the centroid of the object and then closes the gripper. If no IK solution find, it will change the gripper angle while maintain the same orientation with the object.

Aug 6th

Arindam

• Created the network and training it now.

Wei Huang

• Imported main dome into Gazebo.

• • Currently, different objects need to be export as .igs format separately. Then, modify and reassemble in SolidWorks before imported into Gazebo.

  • Still looking for an easier way.

Juncheng Li

• The MoveIt motion planning result will collide with the table.

• Writing code to scan the scene to generate a wide range of point cloud for occupancy grid.

July 2020

July 30th

Arindam

• • Created the dataset with all the objects.

• Wrote the code for the neural net, encountered some bugs, trying to fix those.

Wei Huang

• • Looked for algorithms to do automatic 3D mapping.

  • Now, still learning about VSLAM (Visual SLAM) and try to modify frontier-base exploration to combine with 3D mapping.

July 22nd

Arindam

• Almost done with the code to create the dataset by affixing these images with the masks on background images.

• There are some bugs which I am trying to fix.

Juncheng LI

• Create a Octomap for MoveIt planning scene.

  • Write the code transforming the pose from head_frame to base_link for MoveIt.

July 16th

Juncheng Li

• • Successfully recognize the crank arm as a whole unit. https://drive.google.com/file/d/1yiuW2a6TX9JxG5k18bTsYKlvWUJtcfRl/view?usp=sharing

  • Improved performance of multiple objects at the same scene. https://drive.google.com/file/d/1iP6iVPYHtW8efudfHu4GU8kkpxj1LoBC/view?usp=sharing

  • Able to use the prototype board to do the two way communication via Ethernet. https://drive.google.com/file/d/1GD8qpudx1pZzFYX0ZVWh4ccDkX3KlxXR/view?usp=sharing

Arindam

• Collected all the images of the objects needed. Created the masks for all of them.

• Writing the code to create the dataset by affixing these images with the masks on background images.

Wei Huang

• • Test with turtlebot3 and localization with AMCL package works fine.

  • Make two changes to bypass localization issue with Fetch.

    • Instead of using amcl package, use fake_localization to track robot's position.

    • Set global_frame in local costmap from /odom to /map. Also, reduce robot inflation radius to 0.3 to make it easier to pass narrow gap.

  • Now able to navigate through door and narrow gaps. Realized auto-slam with frontier-base exploration.

• • Successfully build the 3d map manually with fake_localization.

July 9th

Wei Huang

• Done serval tests and found out that robot's odometry is not good enough for localization. Looking for solution to improve odom.

• Implemented RTAB-Map package to generate 3D mapping. But the 3D point cloud currently is confusing.

Juncheng Li

• • Better performance for Bin recognition: https://drive.google.com/file/d/1OWPmz6mQA6mK32nigrdpnKZHj3zAVV1e/view?usp=sharing

  • Able to recognize the left part of the crank arm. False detection exist. https://drive.google.com/file/d/1OasKS35PQpIDCYhhNQYJjR_RrOaau2H5/view?usp=sharing

  • Can recognize the left part on the crank arm. Bad performance. https://drive.google.com/file/d/1XnCP8rze1YQqmKrnn4JWxLhvl4EMWPGo/view?usp=sharing

  • Working on recognizing the crank arm as a whole.

Arindam

• Created part of the dataset with the bins. Creating more images with the crank arm parts.

July 2nd

Arindam

• • Creating a dataset of images of the different objects to pick up in sequence.

Juncheng Li

• • Improved performance for Bin recognition. https://drive.google.com/file/d/1xVV4lQex_z5xUDhAVrI_ubNt2EHZiIK_/view?usp=sharing

• Affordance-templates have a lot of bugs for fetch robot. Trying to fix them.

Wei Huang

• • Completed frontier-base exploration in an open environment. https://youtu.be/eEGpGz41mGI

  • Currently not able to pass through narrow hallway or door. Possible reason.

    • Obstacles are inflated too much when building costmap.

    • Inaccurate localization using AMCL package. https://youtu.be/CvunAo6-Zfc

June 2020

June 24th

Juncheng Li

• • The Crank arm has bad performance in recognition.

  • Blue bin can only be recognized in certain orientation.

  • Need to change different configurations in lighting and 3d model.

Arindam

• • Picked up the bin with the arm.

  • Need to remove some of the false detections that appear sometimes.

June 18th

Arindam

• • Simplified the c++ code for pcl processing

  • There are some issues to make the python and c++ scripts work together. Trying to fix those.

  • The code is working in segments, but the complete pipeline is not ready yet.

Juncheng Li

• Router or fetch wifi adapter limited the data transmitting speed.

• The algorithm can work in real-time with Ethernet connection.

• https://drive.google.com/file/d/1kO2Lph1Araqle3XI76GPdFXXP8axeaCH/view?usp=sharing

• https://drive.google.com/file/d/1QXNWyDCs9CNWk_zcgOuJfojuCJNpyj1O/view?usp=sharing

June 11th

Arindam

• • Able to fit an unaligned (tight) bounding box to the blue bin. Will be needed for defining the proper pose for gripping.

  • Axis aligned bounding box Axis unaligned (tight) bounding box

Juncheng Li

• • Created a texture 3D mesh for crank arm.

  • Dateset acquisition and network training for crank arm.

June 4th

Juncheng Li

• • Able to recognize the position and orientation of the labeled spam.

  • Have issue with the processing speed.

Arindam

• • Able to detect the blue bins with image processing.

  • Trying to incorporate this detection with pcl.

May 2020

May 28th

Arindam

• • Was able to make a single bounding box around the blue bin.

  • Now trying to combine the depth and color image with it to isolate the unwanted objects.

Juncheng Li

• • Evaluating two ways to collect datasets. Successfully generate 20k images using NDDS.

  • Setting up the environments for Deep learning networks.

May 21st

Arindam

• • Was able to make some progress with the bounding box. The code seem to work properly when there are less objects in the scene, and when objects have a more continuous point cloud.

Juncheng Li

• • Understand different ways of collecting data for training.

May 15th

Arindam

• • Was able to segment out the table part from the scene. Now trying to separate all the individual objects.

Juncheng Li

• • Looking at different papers on Pose estimation.

May 8th

Arindam

• • Understood part of the point cloud data processing. But not there yet.

May 1st

Juncheng Li

• • Successfully pick the crank arm from the CMM machine.

  • Trying to pick the crank arm from the Bin.

Arindam

• • Trying to get the ar code position in to the camera image from rviz.

  • Still working on the point cloud data

April 2020

April 24th

Arindam

• • Was able to pick up the blue bin using the AR tag position.

  • Reading up on the point cloud documentation.

Ryan

• • Successfully loaded in a Fetch with a Turtlebot

    • ROS Melodic only supports Turtlebot3, which are very small (~15 cm tall), so not useful for tandem operations

  • Completed first version of Bottomley presentation & will update with recommendations

Naveed

• • Pix4D results are promising - periodic photos seem to be more efficient than video

    • Pix4D flight planner can be used with DJI, has photo planning functionality

• Can export data into .obj files, but limited by free license currently

  • options: $45/month for Pix4Dmodel, ~$250/month for full feature suite (more analysis tools and export formats)

Juncheng Li

• Able to integrate the point cloud system with the manipulation system.

April 17th

Ryan

• • Ran into conflicting namespace issues when loading multiple Fetch/Freight robots into the simulation

    • Additionally, creating a robot inside a namespace structure causes Gazebo to crash on startup

    • May be a limitation of underlying Fetch code -- not entirely sure

  • Successfully loaded in multiple Turtlebots -- maybe this is the direction to take?

Juncheng LI

• • Was able to identify the crank arm and putting the binding box around it in real scene.

  • The Fetch can reach that position and orientation.

  • Looking at the Ethernet communication for end-effector.

Arindam

• • Trying to get the arm to pick up a blue bin.

  • Better to use point cloud for this purpose. So looking into that.

April 10th

Arindam

• Was able to pick and place the crank arm on the new cardboard stand.

• Videos fetch_pickup_and_put_back_crank_arm.mp4 and testing_with_cardboard_stand.mp4 uploaded in the server.

Juncheng Li

• • Collected couple point cloud scenes using the real robot.

  • Run the point cloud pipeline on the scene in simulation environment.

  • Was able to put the binding box contains pose information around the crank arm.

April 3rd

Juncheng Li

• • Try to sampling the 3D crank model into point cloud.

  • Have trouble converting the ply mesh file into the pcd file for point cloud library.

Ryan

• • Got videos of the robots in action; saved to the server

  • Working on getting correct paths for autonomous navigation

Arindam

• Made a model stand for the crank shaft.

Naveed

• testing Pix4D mapper, still not creating model as we'd expect

  • • need to fiddle with the settings more

March 2020

March 27th

Juncheng Li

• • Use PCL library for the segmentation, cropping, down-sampling, feature extraction on the saved point cloud data.

  • Scan through the dataset to find the nearest neighbor to do the feature matching using the extracted feature label ros bag.

Ryan

• • Created new world file based on test bed's current layout; added crank arm and stand

  • Gained control of Fetch arm

March 20th

Arindam

• • Tried to estimate the angle at which the object is gripped by the fetch robot, using image processing. But it is not very accurate.

  • Designed a stand to hold the crank arm with guided slots, so that the arm can get back to its proper orientation once placed on it.

Juncheng Li

• • Look at the point cloud data acquisition, segmentation, binding box and matching features.

  • The point cloud recognition can be done from the 3D model or point cloud data of real scene.

  • Look at the end-effector communication with computer and ROS interface.

March 13th

Arindam

• • Got the video feed from the camera, trying to run opencv functions on that. But the frame rate is very slow via ssh connection.

Juncheng LI

• • Create a loop of series action command from navigation to dropping the crank arm.

Naveed

• • continued troubleshooting on motor problems with steve

    • problem is not in Taranis, receiver, or motor controller.

    • looking into motors, some strange current draw behavior shown on diagnostics

March 6th

Ryan

• • Mapped simulated facility with Fetch robot

  • Can successfully navigate to arbitrary points using the map

Naveed

• • 3d printed several of the new panels but they take up way too much filament

    • going to redesign for decreased volume

• replaced motor controller and receiver, have to pair to controller

Arindam

• • Creating a single python script that incorporates all the functions of the fetch and can work with namespaces.

Juncheng LI

• • Combine the navigation function with the rotation/linear adjustment to reach the AR-tag.

February 2020

Feb 28th

Ryan

• • Fixed Gazebo model so others can now use it

    • Put a ZIP folder with all STLs on the server at /Current Projects/robots_for_factory_tasks/simulation

    • Included a readme with instructions on how to implement

  • Successfully added Fetch robot to simulated environment, still working on controlling it for mapping

Juncheng LI

• • Combined the AR tag code with the arm code.

  • Working on motion planning of the robot arm.

  • Created an AR tag leader and follower system, but we do not need that in the demo.

Arindam

• • Able to control the arm given a position and orientation.

• Able to get the current position and orientation of the end effector.

Feb 21st

Arindam

• • The arm of fetch robot is working, was able to auto-tuck it.

  • Working on how to make the end effector follow a trajectory.

Juncheng LI:

• • Fetch and Freight can work independently going to Way-points.

  • The camera can find out the position/orientation of the AR tag relative to base link.

  • Try to make fetch robot follow the AR tag.

Ryan

• • Added missing components and doors to Gazebo model

  • Swapped out models for colored variants for ease of visualization

Feb 14th

Ryan

• • Switched to newer ROS Melodic distribution

    • PX4 and IRIS Gazebo simulation seems to be working now

  • 90% complete on Gazebo model of IMI test bed main room

Arindam

• • Got the fetch and the freight robot working together. Example video uploaded to the server.

  • Working on getting the arm working.

Feb 7th

Arindam

• • Got two turtlebots running together simultaneously using ros namespaces.

  • Stopped the default roscore from running on the fetch and freight robots.

Juncheng Li:

• • Fetch and Freight navigation in RVIZ using the map that we build.

  • Looking into different kinds of quick change end-effector.

Naveed Riaziat:

• evaluated new image processing techniques using GRIP and alternative line tracking algorithm

  • received new parts to add panels and replace motor controller

  • considerations for getting a new IR camera

    • steve needs the flir camera back...are there good alternatives to consider?

    • timeline? might be some time before things are back on track and ready to use thermal imaging

  • contacted "dslrpros" for quote/feedback on drone options they offer

  • https://youtu.be/BDh1lZikd-A

  • promising output video from matrice 200

  • DJI flight hub will do flight data collection and tracking

Ryan

• Successfully extracted STL files from Revit building model

• MAVROS installation still not working; attempting switch to ROS Melodic to resolve issues

January 2020

Jan 31st

Arindam

• • Modified the code for the turtlebots to work on the fetch.

Juncheng LI

• • Build a map of the facility using real fetch robot.

  • Try to do navigation using that map.

Jan 24th

Ryan

• • Contacted Kevin; he doesn't have individual models but does have a Revit file for the whole building

    • Can be exported as one big STL, but I feel like Gazebo would choke on it

  • PX4 successfully running; linking to Gazebo for simulation is a struggle

Juncheng LI

• Fetch- Using Open-SLAM package to create a "fake" map of the playground environment in rviz by moving fetch robot around with keyboard

• Navigation- Using the fake map i saved and do the simple way-point navigation in rviz.

• Will run this on the real robot and build the map of the facility and start navigation.

Naveed Riaziat

• Finding thermal camera drone options

  • compiled list of industry leaders (DJI and Parot options)

• Communicated with Steve about replacement parts

  • sending replacement motor controller

  • sending file for new side panels - should have more space for accessible electronics

• undocumented behavior still has not been resolved for HDMI to CSI bridge

  • going to test on new rasPi and alternative camera

• miniArgo stalled testing progress until new parts come in, shifting focus to camera debugging and drone

Jan 17th

Arindam

• • Got the turtlebot working to move specific distance as specified by the user.

  • Modifying this code to make it follow a trajectory.

Ryan

• Successfully created a ROS package for simulation use, importing necessary Fetch and Turtlebot packages

• Loaded in Fetch robot to "manufacturing floor" environment

• IRIS drone simulation will require bridging gap between PX4 autopilot and ROS - I don't know enough about this to make it happen

December 2019

Dec 13th

Ze

• • Synchronized LiDAR timestamp with odom timestamp in post processing

  • Having problem with LiDAR extrinsic transformation

    • Physically measured transform does not align LiDAR data

• Worked with Naveed on the new Duke Energy robot

  • Successfully captured IR camera images with the HDMI adapter

  • Set up the new upboard

    • Can't drive motors

    • The motor driver would not connect to usb

    • Swapped the driver with the one on argo mini and motors still does not run

      • Could be software settings with the 24V battery setup

    • The onboard RPi appears to be dead

Dec 6th

Arindam

• Made updates in the rebuttal documents of the RBC paper.

• Working on the UNET model.

Naveed

• Reworking multithreading for linetrack algorithm

• Connected IR camera to Pi via CSI

• Ordered new UpBoard to replace burnt one

• Need to relabel voltage rails for clarity

Ze

• Met with Raja and discussed the implementation of IMU and GPS

• Still working on pointclouds registration

Ryan

• Now able to add STL models to Gazebo map

• Anything in particular you're looking for?

• Looked into 3D mapping with TurtleBot

• Scan angle and odometry inaccuracy will cause this to fail, as well

November 2019

Nov 22nd

Ze

• • Finished the code to read LiDAR frames from PCAP files and generating pointclouds using pointcloud library

  • Working on reading pose data and registering all pointclouds together

Ryan

• • Received Bottomley scholarship

• A summary of research is needed to register for ME 498

  • Description of research, student's role, deliverables, performance metrics, etc. need to be established

  • Practiced PBE Lab test w/others

Naveed

• • Training for PBE

  • Received Bottomley Scholarship - need to register for ME 498 (week of dec 2)

  • Designing mounting plate for all electronics, will need to order acrylic sheet and hardware to laser cut and mount

    • Necessary for outdoor/moving test because there are not enough accessible and secure mounting points for

      • camera

      • computers

      • realsense

      • thermal camera

Nov 15th

Ryan

• • Environment objects are limited to online Gazebo database

  • Investigating 3D point cloud mapping for better results than 2D

Ze

• • Looked into the wheel odometry integration code

    • Too much integration causes more drift

    • Drift issue should be fixed by using LiDAR data to correct pose

Naveed

• • found cables for CSI-bridge, need to validate

  • ported nearly all electronics from old miniArgo

    • wasn't able to run tests in PBE yet, still need to improve mounting of electronics and cable managment

Nov 8th

Ryan

• • Created building interior Gazebo environment

  • Mapping too imprecise to create a navigable map of complex environments

    • Believed that it relies too much on odometry to determine position, fix unknown

    • Possible alternative solution: increase sensor scan angle?

Arindam

• • Made the prelim presentation.

Naveed

• • Mounting Realsense camera, begin to interface with UpBoard

  • Installed UpBoard and wired to new electronics

  • Will install and test Thermal Camera with new HDMI-CSI converter

  • Aiming to run tests with regular webcam (as before) in PBE this week

Nov 1st

Naveed

• • Evaluating CSI-2 vs CSI capture card - unclear which standard the Pi/UP board uses (how many pins for FFC cable?)

  • https://www.mouser.com/ProductDetail/Auvidea/70501?qs=sGAEpiMZZMuDw7xUFNwm7OMzTlXceEzJyDo7uCublDA%3D

  • todo on new miniArgo:

    • install kill switch on remote

    • move UP board and test code

  • printed tracking camera mount

    • install on front, taking care to avoid arm mounting area

    • integrate tracking and localization into code

Ze

• • Designed and printed an angle-adjustable holder for the camera

  • Calibrated the camera at ACRE

    • Fixed camera hot plugging issue

  • Recalibrated LiDAR at Mann Hall

Arindam

• • Made connectors for the odroids to power them from turtlebots

  • Installed all the required packages on the odroids after erasing the emmc cards. Kept backup of the cards.

  • Working on a ros script to make them follow trajectories

October 2019

Oct 18th

Ze

• • Finished the script to capture time tagged images using the upward-facing camera

    • Set exposure time to a fixed value

    • Tested at ACRE

      • Scaned the field 3 times

      • No significant drift though out the experiment

Arindam

• • Finished the first draft of the prelim report

Naveed

• • Capture Cards - Worth being ordered to validate. On paper, latency and format should be adequate

  • Designing test for antenna

    • considering using a recycled TV antenna for baseline feasibility test

• miniArgo

  • reseraching controls methods for trank-tread drive ROV

Oct 11th

Arindam

• • Submitted the poster.

  • Started writing the report. Almost done with first chapter.

Ze

• • Wrote a script to directly log the odom data

  • Test the code by scanning the High Bay area

    • The pose has significantly less drift on flat ground

  • Working on the upward facing camera

    • Once finished, will head to ACRE to test

Naveed

• Submitted IAC poster

• developing BOM for antenna

• composite capture card research (as Ze proposed)

Oct 4th

Arindam

• • Was able to move the robot hand. Fixed the error.

    • But the hand movement is not very consistent. Sometimes it picks up the block, somtimes it cannot.

• Updated the plan of study.

  • Need an example prelim report to check the format.

• Checked the link for the parts. Have some questions about these.

Ze

• • Tested all motors

    • Confirmed the front motors were heat damaged

    • Ordered new motors xm430

  • Moved tracking camera outwards a little bit to improve tracking

  • Updated wheel odom calibration file for tracking camera

Naveed

• IR replacement option:

  • https://www.flir.com/products/blackfly-s-usb3/ (Camera - $365)

  • https://www.flir.com/products/computar-a4z2812cs/ (Lens - $90)

• Exploring use of Antenna to detremine location relative to powerline

  • https://courses.engr.illinois.edu/ece445/getfile.asp?id=5212

  • Previous research shows it is feasible to couple with the magnetic field

  • Inexpensive, but does need relatively large antennae pair (determine direction of B-vector by ortho-components)

September 2019

Sep 27th

Ze

• • Integrated wheel odometry into tracking camera

    • Seems to be able to improve the z drift at Herrick lab

  • Working on motor overheating issue

Sep 20th

Arindam

• • Submitted the paper. Intel gave the permission.

  • Trying to move the arm of the robot.

    • Got errors probably because of the ros version.

    • Uninstalled ros kinetic and replaced with ros indigo but problem still persists. So reverted back to ros kinetic.

    • Now trying to write a separate script by using different examples and help files to move the arm.

Ze

• • Did another test at ACRE with Raja

    • Two front motors intermittently stopped working during the test

      • Could be caused by overheating as motors returned to work after a while

  • There is still z-axis drift in the pose data

    • Reconstructed point cloud looks good after noise reduction

      • Side view is not as good as the front view. Raja is investigating this.

  • Redesigned and printed the chassis

  • Made a power adaptor to run Agbug and LiDAR on a single battery

  • Working on integriting wheel odometry into tracking camera to improve the drifting

Naveed

• collected video of powerlines in dirrect sun, hidden in shrubbery, and complex junctions

  • results are overwhelmingly positive, good detection even when under and through branches (implies heating is not solar)

• working on printing intel camera mount

  • communicating with BIDC to see if CF printer is available to increase rigidity

  • otherwise, ME printers will be an option (PLA)

• IR camera: current lens will not work on usb/ethernet FLIR body

  • having trouble consistently switching camera from write (capture) mode to read (USB host) mode

Sep 13th

Arindam

• • Received some edits from Jeremy, included those in the paper.

  • Able to navigate the fetch robot in the simulated environment using keyboard commands

Ze

• • Built a logic inverter cable to send ROS timestamp to LiDAR

  • Printed parts to fix Agbug

  • Performed an indoor experiment

    • Haven't heard back from Raja about the 3D reconstructrion results

  • Still working on the FPV preview issue

Naveed

• • finalizing design to mount intel realsense camera to Argo

  • accumulating powerline photos in different areas/lighting conditions

    • preliminary results are good: very little processing required to create distinct line in various lighting conditions

• streaming options:

  • hdmi capture card: $250, latency, can use current camera

    • does high latency matter? slow response time system might no be disturbed too much

    • may be valuable to do some sysID before making this judgment

  • flir blackfly s USB3 - $365 flir camera with usb interface + ~$90 for zoom lens

• todo: feedback control: (realsense camera + IMU, encoders)

Sep 6th

Arindam

• • Included all the edits in the 3rd draft of the paper.

  • Have some doubts about the naming conventions of the images.

  • Brushing up the old ROS concepts.

Ze

• • Tested Agbug at acre with Velodyne LiDAR

  • Chassis, LiDAR and tracking camera all worked as expected

  • LiDAR appears to be having a minimum detecting distance

  • Tracking camera had some significant drift

  • FPV camera was not sending onboard video stream but was able to record

  • Still having trouble to sync LiDAR with ROS timestamp, could be a hardware issue

Naveed

• • exploring IR options for live streaming video to computer

  • planning on testing IR camera on powerlines in sunny weather conditions this weekend

  • designing and installing tracking camera onto curent ArgoMini

  • waiting on larger ArgoMini to run outdoor tests

August 2019

Aug 30th

Arindam

• Finished the second draft of the paper.

  • All references included.

  • Paper is within 6 pages.

• Brushing up the old ROS concepts.

June 2019

June 18th

Daniel

• • Final Defense Preparations

    • Finished a full draft of dissertation, planning to schedule formatting consultation this week

    • Still haven't heard back from Prof. Chiu...

    • Planning for a practice presentation next Tuesday @ 1:30 (during our normal meeting time)

• All packages received from last purchase order:

  • Power connectors (XT-60, 5.5x2.5 barrel, JST)

  • WiFi adapters (regular, high-gain)

  • 5V regulators

  • FSRs

June 11th

Daniel

• • Sensor Mounting Tests @ S-BRITE

    • 1 fully successful mount on June 6

    • 7 fully successful mounts on June 11

June 4th

Daniel

• • Sensor Mounting Tests

    • IBC got stuck to wall and crashed on front boom

      • Broken force sensor, boom-prop, and push-to-release-mechanism (PTRM)

  • Rebuilt everything and had several successful flights

    • Previously had some issues with PTRM not toggling (4 failed mounting attempts)

      • During the rebuilding process, discovered an issue with the mechanism and fixed it so it toggles smoothly now

    • Performed 11 successful sensor mounting tests, with 1 failed test where PTRM did not toggle

      • Min/Max/Avg placement error: 1.0 / 22.0 / 9.1 cm

• Writing

  • Still working on analyzing/plotting all flight data

  • Writing about target tracking method and localizing without Vicon

May 2019

May 21st

Ze

• • T265 odom vs. Vicon

    • Vicon total distance traveled: 47.3m

    • T265 total distance traveled: 49.5m

Apr 11th

Daniel

• Localization

  • • Initial analysis of position estimation with EKF2 is promising, generally under 10cm error

    • Working on adapting state machine code to work with the new target tracking method

      • Goal is to do some hand-held tests of the state machine code by this weekend

    • Looking into methods for automatically detecting whether the I-BC is aligned with the clicked target point

      • Currently planning to use a virtual reference point based on the measured offset between the camera and end effector

• ROVer

• • All components received

  • Ubuntu 18 is installed on NUC and UP Board, now they just need all the usual software packages to be installed (ROS, OpenCV, etc.)

Naveed

• • Argo Tests

    • Ran non-live test in PBE with faux powerline

    • Image Processing works very well on low resolution image, robust up to low angles of missalignment

      • outputs angle of line to robot, centroid of line to controller

    • Thermal imaging

      • Flir tests on large power lines (by Grand Prix track)

      • very strong lines against relatively mixed condition sky

      • still struggling to find a robust way to stream video to controller

  • Next step: running driving tests on Argo

    • safety concerns preventing "alive" tests

Ze

• • Creating pointclouds of PBE lab from the depth images taken for the VIP class

    • Having issues converting Vicon Euler angles to quaternions

Apr 4th

Daniel

• Localization

  • • Re-recorded data with correct ROS topics

• • Finishing up script to calculate OF and target position estimation errors

    • Writing to CSV file for plotting, etc.

  • Planning to set up state machine to align in front of clicked target

    • Initial tests will be hand-held to test settling logic, etc.

  • Looking to revisit possibility of running D435 on Odroid so don’t have to change I-BC computer

• PBE Lab

  • • Song has moved in some furniture and Dr. Davies would like us to make a procedure for setting up experiments

      • May want to cut zip-ties on netting in SouthWest corner of PBE

      • Would like to use our carpet to help with acoustic dampening, aesthetics, etc.

      • Moved all boxes to High Bay and posters between inner/outer PBE walls

      • Planning to move force plates to NorthEast corner

Naveed

• Image Processing

  • 3 Distinct Methods - HoughLines, "Arindam Method", simple findContour

    • Hough Lines - Strong ability to find dominant lines

      • poor noise rejection

      • need to implement location masking

    • Arindam Method

      • finds contours effectively, but struggles with thin lines

      • tuning of contour detection may prioritize thin objects vs. boxes

      • center of mass calculation is not robust

    • findContour

      • high false positive rate

      • thresholding may aid remove false positives

• Argo Bot

  • Run test line follows in lab today

March 2019

Mar 28th

Daniel

• • Aero drones

    • Prepared video recording script for Video Drone Class on Friday

    • Made a ros version of the recording script that can also record Vicon data

• Localization

  • Recorded data from a few flights with Ze. Clicked a few different points in the lab to track

  • Working on analyzing the data to calculate the error in the optical flow position estimation and the 3D target point estimation

• ROVer project

  • We've started receiving the CPU components we ordered. They are in an Amazon box on my desk in HLAB 1024.

    • So far we have: Barrel connectors, Arduino Uno, SSD & RAM for NUC, USB OTG cable

Mar 21st

Daniel

• • Aero drones

  • Got PX4/Gazebo simulator working

  • Nadia is now working on simulations for auto takeoff and landing of multiple drones.

  • Localization

  • Can “track” a user click in an image

• • Simultaneously calculates desired yaw heading for perpendicularity

  • Tracking script compatible with R200 and D435

  • Works with Vicon or optical flow as position estimation source

  • Optical flow does drift a lot sometimes depending on conditions, so multiple clicks may be required

  • Now ready to update state machine code to generate waypoints

  • May need to change flight control firmware on I-BC...

Ze

• • Worked on the Botshot project

    • Converted Lidar's data from polar coordinates to cartesian coordinates

    • Used hough transform to find straight lines

    • Successfully measured the hoop distance

  • Got a Jetson tx2 for the project

    • Did a benchmark

      • The cpu performance is about the same as Odroid

Naveed

• • Controlled Argo over PySerial

  • Started looking through Sabertooth software package

    • allows for various reconfigurations of control logic

    • safety logic allows for controller take over - no universal e-stop

• FLIR Camera

  • Windows/Android/iOS only?

    • may need to mess with linux drivers to get it functioning on Odroid

• Exploring E-field measurment for line follow

Mar 14th

Ze

• • Still working on finding continuous surface in depth images

    • Now using the method of surface normal

• • • Tested on depth images captured on flying drone for VIP class

      • Results are really bad as R200 images have low depth resolution and quite a lot noise

      • Trying to update the kernel pattern for gradients so that it can work with realsense cameras

Mar 7th

Daniel

• • Aero Drones

    • Resolved issue with OFFBOARD switch on transmitter

    • Successfully flew autonomously around a square path in PBE

    • Connected both Aero drones to QGC simultaneously

    • Working on streaming Vicon data to both drones simultaneously

• Localization

  • Updating wall angle tracking script to work with R200 and respond to user click

Ze

• • Working on using gradient of depth image to find continuous surface

February 2019

Feb 28th

Daniel

• Aero Drones

  • • Got EKF2 working with Vicon data

    • Attempted 1st autonomous flight, had issue with transmitter switching to OFFBOARD

    • Will attempt autonomous flight again, probably Monday

• Localization

  • • Got green circle tracking file from I-BC, working to run it on D435 with Iris

    • Next step: flight test to determine position and orientation hold capabilities with optical flow only

• ASME Papers

  • • Submitted IDETC, finished JMR review

Naveed

• • Continued OpenCV tuning

    • Line detection works reasonably but false positive rejection needs work

    • Working with webcam, better info once faux powerline and camera are in

  • ROS

    • Line tracking node created and can communicate with motor controller listener

    • Some bugs with repeat/unwanted messages

Ze

• • Wrote the code to create pointclouds from given pose and RGBD images

    • Tested on nyu dataset

    • To be tested in PBE lab with vicon pose

Feb 21st

Naveed

• • Line Follow

    • Experimenting with different edge detection algorithms and OpenCV schemes

    • Trying to determine best method for discerning desired heading from dominant edges

  • Up Board Setup

    • SSH and VNC setup on up board and personal computer

• Power Line Info

  • • Rough estimate ~1" diameter at 6' height should be comparable in size ot actual powerline

    • found black, flexible 1" OD PVC tubing to span in hallways or office?

Ze

• Installed px4flow and lidar on iris

• Did a test fly without the odroid and realsense

  • • Altitude hold and position hold seem to be very stable without the payload

Daniel

• ASME Papers

  • • About 1/2 way through review, planning to finish by tonight

    • Compiled RA-L reviews and highlighted main comments, will look at modifications after I finish JMR review

• • Aero Drones

    • Updated the flight controller to latest PX4 (1.8.2)

    • Successfully streamed Vicon data to drone, but basic control became unstable

    • Currently using LPE estimator, with Alt Ctrl somewhat unstable

    • Discovered that we need to publish to a different topic for mocap to work on EKF2

  • Tours

    • Recruitment and MEP tours went well

    • Things are set up for Girl Scouts, what time today?

Feb 14th

Naveed

• • Getting up to speed with Upboard and TX

  • Power line simulation in Gazebo struggles to run on my laptop, maybe try moving to desktop

    • Running line follow on ground works, trying to reposition kinect model to look upwards and run same code

    • increase straight line preference

    • consider using GPS, accessing preloaded map and localize within

  • Thermals in Gazebo are not feasible

    • flir camera tests will have to be done by hand

  • Testing

    • purchase tubing for running across high bay to simulate powerlines

    • powerlines against sun

    • simulating wind in Gazebo

Daniel

• Intel Aero

  • • Finished software install (Ubuntu, ROS, OpenCV, etc.)

    • Working on Vicon setup for autonomous flights

    • Working on accessing front and down facing cameras

• IDETC submission

  • • Converted to ASME style

    • Planning to re-read reviews and edit content as necessary to address concerns

• Localization:

  • • Next step: reuse green circle detector to hold position in front of wall, and maintain perpendicularity

• Office PC:

  • • Finished Ubuntu install

Feb 7th

Daniel

• Found a VNC server that can display OpenCV videos and capture keys

• Flew Iris+ with D435 video displaying through VNC

• Refocused PX4Flow for next flight test w/position hold over carpet

• Calibrated D435 Depth measurements

  • This improved wall angle measurements

• Need to submit request for waiver of 2-semester rule to Graduate School

Ze

• Continue to read through the SLAM book

• Started exploring the px4flow sensor

Jan 31st

Daniel

• • Indoor localization:

  • Fixed issue with D435 not running on battery power (changed 5V regulator configuration)

  • Iris+ weight current payloads and no battery is about 100 g more than a plain Iris+ with battery, so we may be exceeding the 400g spec, but it hovers at about 60% throttle, so it should be ok for testing

  • Planning to calibrate depth measurements under 2 meters to improve accuracy

  • For wall detection, I may just measure Continue to read through the SLAM book wall angle using a fixed rectangular region in the camera frames (e.g. in the center of FOV).

Arindam

• • Ran the algorithm on the training images. Some images are well segmented, but many of them are having the background as well. Trying to fix that.

Jan 24th

Daniel

• Localization without Vicon

  • Mounted UpBoard and D435 on IRIS+ for flight testing

    • 3D-printed front-mount bracket

      • VHB tape was too flexible

      • Moved camera further forward to help with overall COG

  • Working on wall-angle detection code to maintain perpendicular heading relative to wall

    • Able to measure angles with decent accuracy at close range

    • Detecting the wall may be somewhat intensive, but may be able to merge with target tracking?

  • Likely need to calibrate D435 depth measurements like we did with the R200

• Intel Aero Drones

  • Performed initial flight tests, both drones work

  • PX4 Firmware should be updated eventually

  • Nadia is planning to put Ubuntu on Aero Compute board and begin developement

• Collaboration w/Prof. Culurciello's Lab

  • Juan emailed me about a new intern working on the project with AirSim and Pixhawk, etc.

  • R200 depth data wasn't accurate enough for their algorithms

    • Would like to get some new sample videos with D435 to see if that helps

• NRI 2.0 Proposal

  • Got some more detailed information from Yuanzhi and Tianyi about their module

    • Working to flesh out the collaborative navigation section

Arindam

• • Got some results for the segmentation on the sample images.

• • But there are a lot of images where it is not producing good segmentation.

  • Trying to make it work entire training set.

Ze

• • Reworked Agbug's code

    • Opencv's imshow is too slow. Now directly write to linux's framebuffer to show images. No need for x server.

    • Using multiple cameras simultaneously

      • Takes YUYV raw images from cameras

      • Automatically detects new cameras

      • Switches camera views from the transmitter

      • Dropped resolution to 640 x 480 to increase FPS

December 2018

Dec 18th

Arindam

• • Trying to install tensorflow on odroid. Some error is coming while trying to import into python. Fixing that.

Ze

• • Not able to directly write to frame buffer.

  • Tried to tweak picamera library

    • Can show the webcam images as overlays on top of the pi camera images

      • Max 20FPS

      • Occasionally memory error

  • Switched to OpenCV to show the image

    • Able to get both cameras to work at the same time and switching between each other

    • Now showing images relies on X server

      • Might cause some issue if the program starts before system login

Daniel

• Met with Prof. Ramani's group, discussed goals/areas for new project

• Installed OKVIS on Upboard, but don't have proper calibration for time-synced IMU to test

• Working on attaching Upboard + RealSense to IRIS

  • Added new power distribution board and attached Serial4 FTDI adapter

  • Working on full setup with Lidar, PX4Flow, Upboard and RealSense to test real-time visual servoing without Vicon

• Replaced thermocouple on Makerbot, test print worked well

Dec 4th

Daniel

• • EPRI-UAS Workshop went well, got some good insights from the utilities industry and made some good contacts

  • VINS-Mono SLAM

    • Got VINS-Mono installed and tested with UpBoard, D435 & Pixhawk IMU

    • Localization seemed very unstable on live camera data (may be an issue with sensor calibration/configuration)

    • CPU usage was almost 100% on all four cores, so likely wouldn't be able to run other ROS packages simultaneously

    • EuroC dataset seemed to work fine, but with a massive time lag ~ 60 seconds

Ze

• • Rewrote video recording code. Unified all cameras and moved everything to opencv.

  • Using separated threads to pull images

    • Fixed fast-forwarded video problem

    • Sync frames to have a steady FPS

  • Trying to directly write to framebuffer

Arindam

• • Modified the segmentation part to find out more pure segments.

  • Removed the autoencoder plots as segments are more pure now.

November 2018

Nov 27th

Arindam

• • Made some changes in the code to make the overall detected segments as pure as possible.

  • Matched texture with the heat map to find out the segments which can be included in the output annotation label.

Che Kun

• • Performance with carpet texture improved significantly for pure rotation

  • Able to complete a square path by moving slowly and smoothly - Primary bottleneck is the rotation aspect

• • Multiple overlaying arrows is when localization is lost and regained.

Ze

• • Swapped Agbug files to new SD card

  • Compiled and installed OpenCV

  • Now save video files directly on SD card

Nov 20th

Ze

• • Able to record videos with USB cameras

  • Need to use opencv for better integration:

    • Will be able to switch video sources and add text overlays

    • Had issues to compile

Arindam

• • Filtered the segments with the Hue channel and that gave better results in purifying the contours from unwanted parts.

  • Will apply the texture filtering now.

Che Kun

• • ORB SLAM 2 performance on rotations is fairly poor.

  • Likely due to featureless environment. To investigate further

  • Half circle test

• • Worked with Daniel to integrate ORB SLAM with Robot Localization

  • TFs integrated

  • Also facing issues with rotation.

Nov 13th

Daniel

• Pose estimation with robot_localization package

  • Created ROS driver for MPU 9250 which publishes sensor_msgs/Imu topic

  • Set up MAVROS with Pixhawk to stream Imu topic from Pixhawk as well

  • Tested EKF fusion of Turtlebot 2's odometry with the Pixhawk IMU topic (using robot_localization package)

    • Performs pretty well within a few meters, but after traveling 10+ m, some error starts to accumulate

    • Working on incorporating ORB_SLAM2 data into the estimator, and then will measure the error with ground truth

      • Che Kun has already made a ROS package to publish ORB_SLAM2 data in the necessary topics, just need to work with him to get all the data into the right format to work together.

Arindam

• • Using rank of matrix to find the good matches in the texture of the image. But this is not giving the complete result.

  • Trying to find better alternative metrics for this.

Nov 6th

Che Kun

• • ORBSLAM2

    • Square loop test (1m x 1m)

      • Robot seems to lose tracking during pure rotation

    • Writing wrapper to output image with ORB features over ROS topic to aid debugging

  • Camera mount for RealSense on TB3 design done

    • To print and test

Arindam

• • Trying to find the best texture in order to filter out the unwanted textures by using the heat map.

October 2018

Oct 30th

Ze

• • Fixed USB flash drive mounting issue

  • Fixed video naming issue (now base on file sequence)

  • Added a ROS param to start/stop recording

  • Assigned a switch on the transmitter to trigger the ROS param

Che Kun

• • ORBSLAM2

    • Managed to get RGBD code working by using a launch file

    • Did a 2m straight line test for visual odometry for TB2

    • Estimated pose seems to be relatively accurate

    • Going forward 2m: Translation: [1.983, -0.356, -0.051] [x, y, z]

    • Returning back to start point: - Translation: [-0.041, -0.147, -0.003]

• • • Note: I took the images in portrait but they were auto-rotated when I uploaded them

Daniel

• • Found some 9-axis IMUs in the AgBug parts box (Sparkfun MPU9250 breakout)

    • Got data streaming through Arduino Uno

    • Will make a ROS node to publish the data as "sensor_msgs/IMU" ROS topic

  • Looking at using "robot_localization" ROS package as a starting point for implementing a 6D Pose estimator using EKF

Arindam

• Trying to seperate the excess background parts from the segments using the heat maps and the texture images from the first layer of the network.

Oct 23rd

Che Kun

• ORBSLAM2

  • Added scaling factor for monocular code

  • Able to visualize without Pangolin viewer in Rviz on a separate computer

  • Tested RGBD code with depth stream: Seg fault. To debug

  • Modified cover to place RealSense

Arindam

• • The vectors corresponding to each segment is a 3 dimensional vector (as we have 3 classes). Converted them into 2 dimensions by an autoencoder to visualize them better.

Daniel

• RealSense D435

  • Solved issue with converting depth images to laserscan

  • Got rtabmap_ros working with D435. Quality seems ok, but need to do more tests to see how accurate/robust it is compared to Kinect

• SLAM

  • Continuing to search literature for best camera-based localization setup.

  • Planning to implement Extended Kalman Filter using IMU from old AgBug, or Pixhawk to estimate 3D pose

    • • Goal is to potentially use that in combination with a SLAM approach to improve performance

Ze

• • Warpped the pi cam streamer with ros

  • Measured power consmption:

    • Idling: 8 ~ 10W (FPV 3W, Pi & OpenCR 6W)

    • Running: 17 ~ 20W

  • Field tested range:

    • FPV: 160m(0.1 mile) line of sight (FPV signal dropped below 40%)

    • Radio: >= 160m

  • Video uploaded to spira server under agbug folder

Oct 16th

Arindam

• • Plotted the vectors of each of the segments of each image individually along with the mean vectors to see how well the segments are representing the detected class.

Che Kun

• Able to control/teleop TB3 with Upboard

  • Fixed issue due to OpenCR firmware, ROS TB3 code mismatch

• Modifying/Designing cover for AgBug to place RealSense camera

Daniel

• New UP Board Setup

  • • Got Ubuntu 16, ROS, OpenCV, etc. installed on new UP Board

    • Installed drivers for Kinect and RealSense D435

      • Basic Turtlebot Demos working with Kinect (teleop, follower)

      • Able to access color, depth and pointcloud streams from D435 using both OpenCV and ROS wrapper

• • SLAM

    • Found some papers comparing different SLAM versions and sensor types

    • Planning to evaluate various SLAM algorithms using D435 to see if it is good enough to use as a direct replacement of Vicon

Oct 9th

Che Kun

• Finished ROS wrapper/code to output camera pose from ORB SLAM as an odometry message

  • • Able to visualize in Rviz

• To look into robot_localization ROS package for sensor fusion between visual odometry and wheel odometry

• Debugging Upboard to OpenCR interface issues for TB3

• • Unable to teleop control it

Ze

• • Fixed the auto startup

  • Swapped the FPV module and added a heat sink

  • Tested, having some hiccups but overall working well

Daniel

• • Finished RETH and IAC posters

  • SLAM

    • Looking at different options for 3D SLAM, starting with R200 camera

    • Found some parameters to tune for rtabmap, planning to try on Turtlebot

Arindam

• • Retrained the network using some background images.

Oct 2nd

Che Kun

• Wrote ROS wrapper for ORB SLAM2 to output camera pose as a TF message

• To write code to convert TF message into odometry message for integration with AMCL

Daniel

• Continuing SLAM Literature Review and considering most viable options for I-BC localization without Vicon

  • Current options I've considered

    • 1. Use PX4 optical flow on Pixhawk with real-time visual servoing based on forward-facing camera

         • Update rate may be too slow, and may not have enough features during manipulation

      2. Implement some SLAM algorithm with forward- or down-facing camera, possibly fused with IMU data

         • Latency may be an issue if we obtain IMU data from Pixhawk and then try to time-sync with camera data

         • May also be limited by computation power of SBCs

      3. Combine downward and forward-facing cameras to improve Option 2 in various environments

• • • • Again, may be limited by computation power of SBCs

• • Planning to try out some existing SLAM options from OpenSLAM and look into the robot_localization ROS package for state estimation

• Visited the Bechtel Innovation Design Center

  • Considering different material options for gripper fingers to make them more resilient

Ze

• • Merged ROS files for AgBug and and created a launch file

  • Setup the system to automatically launch ROS during startup

    • ROS will start by itself but does not have access to system files(ttyAMA0 and ttyACM0)

September 2018

Sep 25th

Daniel

• IAC

  • Mostly finished with Quad Chart, deciding on figure(s) to use

  • Planning to update I-BC poster with door opening figures from recent submission

• Optical Flow Flight Testing

  • Flew several times over the carpet with the Iris+

  • Position holds pretty well, with occasional cyclic drifting

  • Added ESP8266 to IRIS+ to stream Vicon and calculate absolute positioning error

• • Documentation Round-up

    • Arindam and I consolidated the documentation for the Odroid software installation into a single Markdown file

      • Also installed software on one of the spare Odroids so we will have a reliable base for testing SLAM, etc.

    • Plan is to create similar '.md' files for all other lab documentation and store on SPIRA in a single location

  • SLAM / Localization without Vicon

    • Reviewing literature on SLAM & brainstorming ideas for I-BC localization without Vicon

Che Kun

• • ORB SLAM 2 works on Monocular live camera feed

    • Investigating camera depth image stream

      • Depth stream does not seem to be published as a ROS topic

      • To check with launch files on ODROID

• • TB on straight path in HLAB room

  • To Do

    • Write ROS wrapper/code to output pose/odometry from ORB SLAM

Ze

• Made a USB receiver dongle to use on the turtlebot

• Fixed the motor issue

  • Re-plugged all the wires

  • Reset the OpenCR board several times

    • All motors run now but the OpenCR board randomly throws a failed to connect error

• AgBug is fully functional now

• To do

  • • Clean up the catkin work space

    • Create a launch file for AgBug to automatically start everything

Arindam

• • Separated out the most important feature from the image using the heat map.

  • Working on overlaying it on the segmented image to find out more segments.

Sep 18th

Ze

• Can publish cmd_vel message to Agbug using the remote control now

• Can publish front camera image now

• A few issues

  • Motors won't run

    • Might try re-flashing the OpenCR board

    • Sometimes RPi won't automatically connect to WiFi after startup

Che Kun

• Installed and tested ORB SLAM 2 on Intel Joule

  • Working with live camera data

  • To test on robot to see trajectory generated

Daniel

• Optical Flow

  • • Got about 12' x 18' of carpet (extra length cut on each 4' roll)

    • Flight tests:

      • Tested Iris+ Position control using PX4 v 1.8.0 and PX4 optical flow sensor

      • Holds position pretty well, with some occasional oscillations on the order of ~0.5 m

    • Streamed Vicon to the Pixhawk (for logging only) to compare with optical flow pose estimate

      • Having some issues with the quaternion conversion to Euler angles (yaw)

• • Tours

    • Two Turtlebots ready to go, one with UpBoard, one with Odroid

Arindam

• • Implemented a graph cut algorithm for segmenting the input image.

  • Checking the response of the network on each of these segments and grouping the important ones together to create an outline around the object.

    • A bounding box can be extracted from this outline.

Sep 11th

Ze

• Wrote a cpp code to transfer SBUS signals to turtlebot velocity commands

  • • Tested on turtlebot2 using the SBUS receiver and FTDI232

    • To be tested on Agbug

Che Kun

• Trying to setup Intel RealSense D435 on Intel Joule

  • • Debugging errors during kernel patch

• Setting up ORB SLAM on Intel Joule

  • • OpenCV 3.x bugs - Debugging missing shared libraries

    • ORB SLAM tested on OpenCV 2.4.x and Ubuntu 14.04 (On their wiki). OpenCV 3.x not mentioned

    • Approach 2: Try ORB SLAM 2 which is mentioned to be compatible with OpenCV 3.x and Ubuntu 16.04

Sep 4th

Daniel

• Door Opening

  • • Compiled video of door opening with mini quad fly-through

    • Updated OverLeaf project to v2

      • Working on writing everything up

• UWB Project

• • Comparison experiments to highlight the efficiency of the Ke/Tianyi's UWB-surveying ROS node

    • Built an arena in the PBE lab using posters and cardboard boxes

    • Autonomous exploration still struggled, so manually created a 2D map of the PBE using gmapping

    • Wrote program to search the full space

    • Ready to test Ke/Tianyi's UWB-surveying ROS node in the space

  • Added Vicon markers to TurtleBot for experiments

  • Still need to write a timing script and set up a Vicon stream to simulate proximity-based UWB detection

Arindam

• • Planning on using the network to detect qr codes first. Just to see if it is able to create boxes for less complex objects.

  • Creating the dataset for that.

Che Kun

• Install/Setup ORB SLAM on RPI3

  • Memory issues: Insufficient RAM to compile C++ code (Internal Compiler Error)

    • M1: Use another SDcard with more memory space (Create swap file)

      • Con: RPI3 only has USB 2. Not ideal for RealSense (R200/D435) due to limited bandwidth

    • M2: Try ODROID/Up Board

      • • Setup TB3 files on ODROID. To test interfacing with OpenCR board

August 2018

Aug 28th

Arindam

• • Network is having some trouble training. The training error is not reducing. Modifying the network to tackle that.

Daniel

• Door Opening

  • • Successfully flew mini drone through door with I-BC flying in corner of the lab

      • See video uploaded to YouTube

    • Got information about the fan in the PBE lab

      • At 2 meters from the fan, wind speed is only about 4 mph, so probably need a larger fan.

      • Also, need to set up obstacles in PBE lab for UWB experiments, so probably will hold off on wind experiments for now

• • UWB Project

  • Tested several autonomous exploration ROS packages

    • • explore_lite and frontier_explorer are two of the more stable packages, but they still struggle to give consistent results and crash pretty frequently. If we have larger obstacles in the PBE lab, then explore_lite may be able to work well.

  • Tested QR code reading packages in ROS

    • • Most packages just extract the text without giving the 3D position of the code itself

      • Also, detecting QR codes requires either additional cameras (e.g. facing left/right) or a more complex/exhaustive search path through the map, which will likely take too long to implement given the pending deadline

• • • Helped Tianyi debug some issues with detecting the desks/obstacles in their lab with the Kinect (modified the depth-to-laserscan package), and sending waypoints in rtabmap. Got their Turtlebot to successfully travel to desired waypoints even without a complete map.

    • Met with Ze, Ke, Tianyi and Muzhi and decided to revise the experiment

      • Instead of scanning the space for QR codes, we’ll do two related experiments:

        1. Efficiency test

           • Follow a predetermined path through the map and discover UWB devices based on proximity (e.g. distance reported < threshold), and measure how long it takes to locate all devices (or, see if it fails in some instances)

           • Use Ke’s UWB registration algorithm to discover/localize the UWB devices and compare that time and success rate

        2. Accuracy test

           • Place UWB devices in various locations and measure the positioning error of Ke’s algorithm

Ze

• Solved Linux custom baud rate serial communication issue

  • • Need to set all flags(c_cflag, c_lflag, c_cilag) for driver to work properly

• • https://www.cmrr.umn.edu/~strupp/serial.html

Aug 21st

Daniel

• • Door Opening

    • Cut out back of electrical enclosure

    • Opened the door about 65˚ using a yaw setpoint of 20˚

    • Attempted many times to fly mini drone through doorway with I-BC in the air, but the turbulence/wind was too much for the mini drone to overcome.

      • We do have some videos recorded of the mini drone flying through the door without the I-BC in the air

  • UWB Project

    • Got the localization working with a Kinect and rtabmap_ros (SLAM package), brought the kinect and a Turtlebot to Tianyi and Muzhi, so they can test the UWB integration

    • Built the 3rd Turtlebot and mounted our second Kinect sensor on it to test other algorithms

      • e.g. Google Cartographer

    • Waiting to hear from Tianyi about what aspect we can help with next...

Arindam

• • Found a bug in the code because fo which the training was not giving proper results.

  • Retraining the network with the artificial and natural images again.

Aug 14th

Arindam

• Installed pytorch on Odroid successfully.

• Having some issues with the combined training still.

Aug 7th

Arindam

• • Training for the network is still going on.

  • Facing some issues with installing Pytorch package on the Odroid. Working with Juan and Manav for that.

Ze

• • UWB Project

    • Dumped and edited a Tango param YAML file so that the frameIDs could be renamed before the Tango ROS streamer starts

    • Connected the Tango to Turtlebot navigation stack

      • Having some noise issue. Trying to play with the setting

      • The static TF publisher needs to be tuned

  • Debugging the SBUS receiver script

  • Finished designing the top shell for Agbug

Daniel

• • UWB Project

    • Working with Ze to get Tango SLAM to work with Turtlebot's path planner

    • Will meet with Tianyi/Muzhi to integrate the path planning to navigate to the physical UWB locations

    • As a backup for Tango SLAM, I've been working on getting the RealSense drivers for the D435 to work with Odroid

      • Drivers have now been installed on Odroid, just trying to test them with OpenCV to see if they are working correctly

  • Door Opening

    • Has been somewhat on hold while working with the UWB project, but the OverLeaf document has about 2-3 pages of content about the gripper and image processing from my prelim document

July 2018

July 31st

Daniel

• I-BC Door Opening:

  • Adjusted I-BC yaw to 5, 8, then 10 deg. and opened the door ~60 degrees

    • May try 15 deg. yaw, but I'm thinking 60 deg. may be close to the max for this method

  • RA-L/ICRA paper

    • Started writing (transferred some sections from ch. 4 of my prelim document as a starting point)

    • Arindam will go over what I wrote about the image processing and flesh out that section

• UWB Project

  • Met with Tianyi & Prof. Ramani on Saturday

    • Working on a 10-day plan to get all sub-system experiments done, followed by 10 days of integration and refinement, and then 10 days for write-up

    • Planning experiments to quantify accuracy of Turtlebot navigation using Tango SLAM

      • • Also looking to potentially improve the localization accuracy using the UWB, but still not sure how

• • • • Prof. Ramani offered Ze and I temporary workspaces in the C Design Lab while we work to integrate the UWB nodes with the Turtlebot, so we may bring the Turtlebots to ME for a few days

Ze

• • Redesigned and 3D printed a chassis for Agbug

  • Wrote a script to read SBUS receiver signal

  • Mounted and wired all core components on Agbug

  • Tested FPV transmitter

July 24th

Ze

• Debugged way point publisher

• Tested agbug

  • Video stream, drivetrain and LiDAR worked

  • GPS and sensor probe not working

  • Need FPV and radio receiver

• Created bottom shell for agbug

Daniel

• I-BC Flight Tests:

  • Modified BoomCopter code to change yaw during door opening

  • Vicon had to be recalibrated (possibly due to HVAC repair?)

  • Ready for flight tests as soon as tours are done today

• UWB Project:

  • Looking at related work to see what has already been done and how we can contribute to the field

  • Tianyi brought a Tango device (Asus Zenfone) and the RealSense D435 for us to test w/Turtlebot

Arindam

• • Created some dummy images and their corresponding bounding boxes.

  • Trained the network on that.

  • Now, need to create and train another network that will take in the localized images and output bounding box information.

Ganesh

• • Finished documenting the GEP PAGE workshop and Gazebo SITL simulation

  • Working on the AirSim project document.

July 17th

Arindam

• • Created a fresh dataset of images that has a custom random object in it. Will start training with this dataset.

Ganesh

• Finished setting up for Prof. Eugenio's project

• they are planning to get the same hardware

• preparing a guide for the set up.

Ze

• • Finished the coding for way point selecting for path planning

    • To be tested

Daniel

• Met with Prof. Ramani's group, got homework to do lit. review of SLAM/Path Planning

• Taking time off with family this week, but plan to do flight tests on Friday with Arindam

July 10th

Ze

• • Tested rtabmap_ros with kinect2. Printed a holder to keep the sensor staying upright.

  • Trying to find a method to make move_base planner use unknown cell for path planning

Daniel

• • UWB Collaboration Project

    • Met wit Tianyi to discuss SLAM strategy & answer ROS questions

    • Found ‘frontier_exploration’ ROS package

    • Working on a custom ROS node to compute and send exploratory waypoints given a known UWB location

  • Door Opening

    • Fixed the gripper, ready for next test flights, planning to include yaw motion during door opening

  • RealSense Tuning

    • Got the RealSense to work outdoors, recorded some videos for reference

    • Arindam added sliders to tune parameters as needed

Arindam

• • Localized the objects in the images using the gap layer.

    • Now modifying the network to make some bounding box predictions.

  • Will be flying the iris for the ai project.

Ganesh

• • Finished writing the script and tested it

    • MAVROS - RC data - rospy

    • AirSim - Images - AirsimAPI

• Workshop: Finished setting up 21 robots in groups of 3. Tested.

  • Will set up the remaining stations this week

  • Will go over the other course materials

July 3rd

Daniel

• Door opening

  • Completed several flight tests, but the door still only opens about 10 cm, looking at videos to see what is causing the early release.

  • Will perform more flight tests after repairing one of the compliant fingers (broke during a test yesterday)

• Imported the RA-L paper template into OverLeaf

• RealSense tuning

  • Found some parameters to try for outdoor use, will test this week

• Localization w/out Vicon

  • Got IRIS 2 set up with PX4Flow, LIDAR, and latest PX4 firmware

  • Made some texture patterns for the floor in PBE that give optical flow 100% quality

  • Trying to get PX4 to switch to position hold with just optical flow (no Vicon), but need to debug an issue with the compass causing a failsafe

• Collaboration w/Prof. Ramani's group

  • Need to meet with them to get more specifics about what is needed before we can proceed on our end

Arindam

• • Improved the accuracy of the network to 93% from 87%. Stopped Training and working with the Global Average Pooling (GAP) layer now.

  • Will do some flight test this week for AI+ project.

Ze

• • Created a new launch file for rtabmap_ros to work with kinect2

    • Trying to change the parameters to get a better mapping result.

June 2018

June 26th

Ganesh

• • Have some Unreal Engine environments that we could use for the project - meeting Juan and his team this thursday

  • Reading up on how to create our own robot

• Workshop: Soldered 6 robots - the 2 sample QR holders printed - will go today to pick it up and check on the chassis using the velcro -Will place an order for that to be printed as well

• Will goto 1030 to test the robots and install the software in the computers this week.

Daniel

• Battery recommendation for Intel Aero:

  • Turnigy 4S 4000 mAh 30C Lipo, Amazon

    • LiPo Balance Charger, if needed: SkyRC iMAX B6AC v2 (same as we use in our lab), Amazon

  • SLAM & Motion planning

    • Mapped out the Turtlebot's navigation ROS launch files and configuration files

      • Learned how to change path planning algorithm parameters and tested with autonomous navigation demo

    • Installed SVO monocular SLAM and got it running on Odroid

    • Got rtabmap_ros working on Turtlebot/Odroid, working on getting Turtlebot navigation demo to work

      • rtabmap_ros has a demo using Google Tango on Turtlebot, so this may be a solution for our work w/Prof. Ramani's group

Arindam

• • Created the model for classifying the images with Global Average Pooling layer. Training is still going on.

  • Need to check how well it can localize, once the training is over.

  • Will be flying the drone with obstacles to get the training data.

Ze

• Tested VISLAM on Snapdragon Flight

  • • Newest Machine Vision SDK is not compatible with the official demo

      • Modified the demo code to work with an old SDK(1.0.2)

    • The "SLAM" only gives pose and odometry and relies heavily on IMU data

• • Trying to modify the rtabmap launch file to work with Kinect2

    • Currently having some issue with the TF topics

June 19th

Ze

• • Looked through snapdragon flight user guide and developer guide

  • Reinstalled firmware and ROS

  • Tested all debugging tools and ROS

  • Applied for Machine Vision SDK license

  • Looking into camera and IMU APIs and trying to write a ROS publisher node

Daniel

• • Got pull sensor data from 2 more door opening flights

    • Still need to analyze the data, but at first glance it seems to have much more detail

  • Learning about the path planning / SLAM pipelines in ROS

    • Mapped HLAB 1st floor and tested autonomous navigation

    • Looking into switching the planning and SLAM algorithms for Turtlebot

  • Tours:

    • Made separate power connectors for the Odroid on the Turtlebots

Ganesh

• • Built a custom environment with AirSim with a working Quadrotor model and tested - can now buy any environment and integrate the simulator in it.

  • Fixed the issues with HITL and got it working - https://photos.app.goo.gl/F7pPxbTjvddSWdFq6

• Received the packages from Sparkfun and Polulu - can begin soldering - Amazon yet to arrive - expected tomorrow

  • Meeting the 3D printing team at WALC to finalize things - They approved the design for the holder as it is.

Arindam

• • XYZ position data from vicon is synchronized with the video frames. Need to set up some dummy obstacles and perform a flight test.

  • Working on the bounding box code.

June 12th

Ganesh

• The environment file downloaded from Windows wasn't opening on the version of Unreal Engine on my Linux partition

• Met with Juan's group. They suggested coming to their lab and working on the server when Juan is present. Going to their lab today in the afternoon to set things up.

• Simultaneously working on setting it up on my Linux partition as well, to check the feasibility of setting everything up on a new laptop - probably get an idea of required GPU specs

• Workshop: When can we expect the 3DP filament to arrive? I'm planning to do a test print of the QR holder using my June material quota at WALC and find out if any design changes need to be done.

Daniel

• Finished Prelim Exam

  • Planning to do door-opening flight tests with the new pull sensor, and look at ROS packages for collaboration w/ Prof. Ramani's group

  • Also planning to test out some monocular SLAM algorithms on RealSense/Odroid

Ze

• • Tested rgbd slam in the office room with kinect. It did not work very well without external odometry.

  • Started working with snapdragon flight. Installed all the environment software.

  • Looked through the SVO paper. Trying to run the code on snapdragon board

Arindam

• • Fixed the errors on the gpu and started training the yolo network.

  • Will be working on synchronizing the vicon x, y, z coordinates with the video frames on Odroid.

June 5th

Ganesh

• Trying to do HITL on Linux

• Had some trouble downloading the environment on Epic Games using Linux - Lukas suggested doing it on windows and copy the files

• GPU was not being used for the engine - fixed that

Daniel

• Preparing for Prelim Exam

  • Finished Prelim Report

  • Working on Presentation

  • Looking into using WebEx in Eisele Conf. room

Arindam:

• • The commands from the teleoperated controller are synced with the video frames from realsense.

  • Tried running the model on paperspace gpu. Got an error that gpu is running out of space. Working on that.

Ze

• • Looked into two slam packages: rtabmap and rgbdslam.

  • Tested kinect v2 with rtabmap_ros

May 2018

May 8th

Daniel

• Force sensor replacement arrived

  • • Removed old pull force sensor

      • Clear plastic layer was delaminated from back of sensor: I assume this is the cause of failure...

    • Made a new cable for the pull force sensor that can be more easily detached from the end-effector side in the event of another failure

    • Verified that the pull sensor is now working as expected:

      • Response of pull and push force sensors looks about the same (raw analog output to terminal)

• • Door closing algorithm

    • Was stuck on best way to calculate and store actual handle position after collision

      • Came up with an idea to use an existing projection function to get the right coordinates

      • Will put all of the door closing code into a new file separate from working door-opening code

  • Will perform some flight tests with working force sensor to see if the door-opening logic needs to change and/or if the door opening can be improved with better pull force data

  • Working on compiling previous work into prelim proposal document

Apr 2018

Apr 23rd

Ganesh

• Finished importing and creating the boom-copter Tricopter

  • Created the model with all the plugins, links and joints

  • Was crashing all the time, has to play around with the inertial model and collision models to simpler geometry to achieve takeoff

  • Tested flight - no issues - works perfectly

• Tested the boom-prop separately - uses two channels (one for forward motion and other for reverse

• Daniel wrote a code to check if the tricopter accepts commands

  • Got MAVROS running - Have access to all the data on board - have flight logs saved after every flight (can be used for debugging)

  • Given a target X, Y and Z, the copter takes off, transitions into off board mode and goes to the target position

  • Works perfectly

• Added a camera

  • Coded the camera with properties into the model file

  • Other suggested methods did not work

  • Image gets published to a topic

• Addition of camera - caused the copter to oscillate and crash - even when added along the center of mass - couldn't takeoff and hover in place

  • Manipulated the different parameters

  • Solved that issue and was able to takeoff and hover

• List of sensors:

  • IMU - data can be accessed through MAVROS - Logs saved for each flight to visualize and for diagnosis

    • Has a error modeled into the plugin to simulate sensor errors in real life.

  • Camera - available raw image

  • GPS - ROS topic - has a error modeled into it to replicate positioning errors

  • Lidar - can add to get depth data

  • Intel Real-sense - Found a plugin for the Intel Real-sense hardware - not compatible with the version of gazebo - will try it on another laptop before changing the version (if possible)

Daniel

• Compiled the door-detection position estimation error in a single document

  • Errors are similar to what we had for sensor mounting

  • Average error:

    • Total: 5.8 cm

    • X: 5.3 cm, Y: 0.012 cm, Z: 0.021 cm

• Made poster for RoSe-HUB

Arindam

• • Tested the yolo v2 code for training.

  • The training error reduces with the iterations which means the model is learning the features. So all parts of the code is working properly.

  • Need to test the model on the Amazon cloud services.

Che Kun

• • To redesign linear actuator to be more robust with grooves/slot.

  • Characterization Experiments for

    • Max Incline

    • Max obstacle size

    • Battery life

• Maximum Speed Calculations

  • • Theoretical Maximum Speed (No load): 0.3488 m/s

      • Calculation = 0.05461 m * 8.063421135 rad/s

    • Experimental: 0.3 m/s

    • Calculations based on motor with higher no load speed 77 rpm

    • Based on motor with lower speed. As there are 2 different sets of motors used on TB3

• • http://www.robotis.us/dynamixel-xl430-w250-t/

    • Speed Limit is limited due to motors used.

  • Trying to setup OpenCV 3.3 to utilize Deep Neural Network (DNN) module

    • Setup in progress

Apr 16th

Ganesh

• • Finishing up the boom copter model.

    • Some issues with COM and axes for rotation

    • Fixed those

    • Added a tiltrotor variables in the model file => +/- 15 degrees

    • will add the model to the simulator and set it up

  • Working on the 'world' file with the door

Arindam

• • Understood the yolo code and rewrote several parts of the scripts.

  • Prepared the datasets so that they can be read by the model properly.

  • Will start the training. If it trains properly that means the codes are all working properly.

Daniel

• • Flight tests

    • Successfully opened the door in 11 different flights

    • Had a few failures due to gripper positioning

      • Typically happens if the battery is getting low and position-holding capability decreases

      • Also can struggle if door handle is grabbed at the top

    • Currently analyzing the data...

Che Kun

• • Laser Filter: Implemented

    • Pre-Filter

• • • Post-Filter

• • Case for linear actuator to prevent it from falling

    • To fine-tune. Print 1 more iteration

• Characterization Tests

  • Max Speed: 0.3 m/s

  • Max Incline: 35 degrees

• Use Case: Under-Canopy Plant Phenotyping

• https://dailyillini.com/news/2018/01/23/university-startup-develops-5000-agricultural-robot/

  • • Plant Phenotyping Lab (Purdue ABE Faculty)

  • https://abephenotyping.weebly.com/

Apr 9th

Ganesh

• Got hold of the parameters and list of code changes in the package to get tricopter working

  • Implemented it using a test model of tricopter

  • Works :D - has a tiltservo - takes off fine

• • Next steps:3

    • Verify working of the model and find bugs if any (tiltrotor code is not available as an official release - not tested yet - in experimental stage)

    • Add boom prop

    • Add depth camera

    • Import environment

    • Test

Arindam

• • Reading the papers on bounding box detection in neural networks.

Daniel

• • Flight Testing

    • Finished integrating the box detection image processing into the state machine

    • Performed several flight tests

      • 3 tests successfully opened the door

        • Force sensor only worked for one of these flights

      • The remaining tests had various network lag issues

    • Planning to perform several more flight tests for validation of the error data

  • Static Thrust tests

    • Set up and performed thrust tests for 16" vs. 15" propeller with Che Kun

Che Kun

• • Added undercarriage

  • Tested LiDAR: Only linear actuator part detected.

• • Teleop to control linear actuator: Works, but sluggish - To tune pub/sub

  • Setting up udev for USB devices connected - To debug with Daniel

  • Point to point navigation ok

    • • Recovery behaviour affected as it cannot rotate due to linear actuator behind.

Apr 2nd

Tianqi

• belt length 221.65mm, find a belt might fit:

• https://shop.sdp-si.com/catalog/?cid=p348&filter=a17%3A2%3A225.0&sort=undefined&view=table

Ganesh

• A model for QUAD-Plane with tiltrotor was added to the package - I made the boomcopter model (tri) following the same steps (Changes made to the files of the package)

  • Created a make variable

  • Added a custom mixer file <--

  • Added a parameter file <--

  • Modified the physics and the SDF file

  • Modified the mavlink file to add a case for tricopter

• Still not working perfectly - most likely some issue with mixer or the parameters specific to the simulator.

  • Waiting for the tricopter parameter update inside the package

  • Developers of the package are still finishing the model

Che Kun

• AgBug Packaging

  • LiDAR: 3D printed standoffs to raise LiDAR height

  • Managed to place USB hub under LiDAR. Saving space and enabling lipo batteries to be placed on top.

  • Cleaned up cables. Cables are below LiDAR detection plane. To validate with testing.

  • Cameras

    • RPI Camera (Front facing): Tested and working with ROS on RPI3, designed camera mount for it.

    • USB Camera (Downward facing): Tested on ROS on VMware

      • Unsure of optimal placement.

  • Current Weight:

    • AgBug: (1.768 kg with 2 cameras)

    • Cage: 0.309 kg

• To Do

  • Systems test check if everything is working

  • Change to 16" diameter propellers on Boomcopter

  • Add undercarriage to shield RPI, OpenCR boards from dirt, dust from the ground.

Daniel & Arindam

• Calibrating Realsense door detection program

  • Went through several iterations of the code to make it multi-threaded

  • Frame rate is now ~14 FPS

  • Added a formula to center the X error

  • Working on formulas to offset Y & Z errors so they are centered around 0

Mar 2018

Mar 26th

Ganesh

• Mainly worked on setting up the tiltrotor tricopter

  • Recreated the boomcopter (without boomprop) - just a tiltrotor tri - with the latest sitl_gazebo package

  • Made series link Base_link>motor_2>rotor_2 (rear)

  • 'rotor_2' moves even when 'motor_2' is active.

  • Created make variables for tricopter inside the package

  • Setup mixers and parameters for this setup

  • Still not working completely - one of the developers said that he will update the package with some values of the parameters specific to tricopter in a couple of days

Che Kun

• Mainly worked on packaging/streamlining AgBug

  • Reduced 1 module

  • 1.61kg

Daniel

• Modified state machine code to use Realsense camera instead of webcam

• Updated simulator to test/calibrate the Realsense image-tracking code

• Planning flight tests for this week to calibrate Realsense data & open door

Arindam

• • Measured distance from the size of the box by image processing (beyond 2m from wall).

  • Distance measurement switches to realsense within 0.5m to 1.5m from the wall.

Mar 19th

Ganesh

• Direct PWM test for each motor is not supported anymore (>1.5)

  • Unable to give PWM inputs directly to check the bi-directional motor

  • Made 2 models: One with bidirectional motor with a single plugin and another with 2 plugins for each direction (2 channels)

• Made the Enclosure with a door hinge - Need to attach handle - need link for the handle

• Upgraded to Ubuntu 16.04 and ROS Kinetic.

  • Was facing build errors with latest firmware and simulation packages

  • Installed all the required dependencies

  • Some new changes in the simulation package - was able to arm the rotor with series link

  • there is a chance that the tiltrotor could be simulated - simultaneously working on that

Tianqi

• • Latest flight test