Available Research Tools

• DARPA Open Catalog Open Source Tools
  • SMACCMPilot UAV
• Open source software engineering tools
• DOD Tool List
• Verification Tools
• Code Based Test Generation Tools

Deep Learning Tools for SE

• Summarizing Source Code using a Neural Attention Model - CODENN@UW
• Convolutional Attention Network for Extreme Summarization of Source Code@Edinburgh (other tools)

Testing Tools (automated-test-tools)(opensourcetesting)(Code Based Test Generation Tools):

• White Box Testing:
  • Symbolic execution
    • Java: JDART@NASA (based on JPF) CarFast@UIC CodePro Analytix@Google WISE@UCB LIME@HUT Dsc@UTA jFuzz@MIT/JPF NASA Java PathFinderLCT@HUT UTIDA@UIUC Sireum/Kiasan@KSU (manual) MUSE@Lugano CUTE/jCUTE@UIUC
    • C#: Pex@MSR FSE SEViz@BME
    • C: LAV@BG (based on LLVM) SymDrive@UWisconsin S2E@EPFL (based on KLEE) ZESTI@IC KLEE@Stanford KLEE-FP@ICL Bugst Otter@UMCPTRACER@NUS LCT@HUT Crest@UC Berkeley Score@KAIST Splat@UCLA Slang-Analyzer@CAS Explode@Columbia CUTE/jCUTE@UIUC SUDS@Seattle U. Catchconv@Berkeley Loopfrog@USI RELAY@UCSD SAGE@MSR
    • C++: https://github.com/ahorn/native-symbolic-execution-clang Slang-Analyzer@CAS Klover@Fujitsu
      • Ruby: Derailer@MIT
    • PHP: Stranger@UCSB
      • Python: PyExZ3@MSR Chef@EPFL
      • Lua: Chef@EPFL
    • Binary (Tool List): Binsec/SE@Inria Ponce@Salesforce Angr@UCSB FuzzBall@UCBerkely Fuzzgrind@SecurityLabs PeachFuzzing Libdft@Columbia Miasm Metasm
    • JavaScript: Jalangi@UCB&Samsung Lambda S5
  • Constraint solvers
    • General: Green@Stellenbosch&UNL Z3OpenSource@MSR Z3@MSR Crest+Z3@UIUC STP@MIT Yices@SRI CVC3@NYU Simplify@UCD Boolector@JKU
    • String: Rex@MSR Kaluza@UCB Hampi@MIT DPRLE@UVA Z3-str
    • Relational Logic: Kodkod@MIT
  • Random testing:
    • Java: tpalus@UW Randoop@MIT YETI@U. York Eclat@MIT Tautoko@Saarland U. AgitarOne JUnitFactory Crap4j JCrasher@Gerogia Tech FB Crasher@Gerogia Tech
    • C#: Randoop@ABB Randoop@MSR
    • C: Ballista@CMU Fuzz@Wisconsin
    • Ruby: RushCheck Rantly
    • Fuzz Testing: SecFuzz@ETH
  • Evolutionary testing:
    • Fittest
    • Java: EvoSuite@U. Saarland eToc@ITC Testful@Politecnico di Milano
    • C: Austin@UCL FlagRemoval@UCL
  • Bounded exhaustive testing:
    • Java: Korat@UIUC ASTGen@UIUC UTIDA@UIUC JWalk@Sheffield
  • Fault-oriented testing
    • Java: CnC@Gerogia Tech DSD-Crasher@Gerogia Tech jMoped@München
  • Capture/replay:
    • Java: ReCrash@MIT
  • Misc.:
    • Java: JExample@U. Berne UnitChecking@Charles
    • C++: VeriSoft@Bell Labs Verisoft Automatic Interface (VAI) Framework Dejagnu@GNU
    • JavaScript: WebAppWalker@UCB MasculietJS@Qualcomm EventTracer@IBM (WALA for JavaScript) Jalangi@SRA Artemis@Brics CrawlJax JSure
  • Performance Testing
    • Java: SpeedGun@UCB WISE@UCB WAIT@IBM LagHunter@USI Pounder@USI FlyBy@USI LiLa@USI LagAlyzer@USI Trevis@USI Other Tools@USI
    • C#:
    • C/C++: Trend-Prof@Stanford
    • Binary: Coz:Casual Profiling@UMass aprof@uniroma
  • Behavior-Driven Testing
  • Cucumber
• Combinatorial Testing
  • M[agi]C@FBK Casa@UNL ACTS@NIST Hexawise
• Concurrency Testing:
  • C#: CHESS@MSR
  • Java: PENELOPE@Illinois PECAN@HKUST IMUnit@UIUC pbbench@GIT DetCheck@UCB RoadRunner@UCSC CalFuzzer@UCB CUTE/jCUTE@UIUC MultithreadedTC@U. Maryland TestCon@U. Queensland
    • Mutation testing: ConMAn@UOIT Nester NinjaTurtles (old project web)
    • C/C++: Maple@UMichigan VeriSoft@Bell Labs Verisoft Automatic Interface (VAI) Framework
• Spec-based Testing (projects, projects):
  • General: ISTA@DSU
  • Java: JUMBL@UTK Korat@UIUC ModelJUnit@waikato MMUnit OSMO Tester ParTeG JavaTESK@ispras
    • C: CTESK API Sanity Autotest@ispras
    • C++: C++TESK@ispras API Sanity Autotest@ispras
  • C#: NModel for C# Spec Explorer@MSR FSE
  • JML: JMLUnitNG@UWTacoma JET@UTEP UTJML@UTEP JMLTT@INRIA
  • UML: CowSuite@ISTI UCTSystem@IRISA UMLTesting@CoroladoStateU
  • Others: Mothia@CNR hol-testgen@Brucker Congu for Checking Algebraic SpecsUBET@Bell Labs TorX@Wwente Uppaal TRON@AalborgU. Atasyn@UCLA Dataabstr Charon@U Penn CharonTester@U Penn Charon SAL-atg@SRITesting tools Bomotest AutoFocus Nimbus TVG TGV UMLAUT Triskell AFTG tool demo at NIST Basset@UIUC
  • Mobile App/System Testing/Analysis
    • Android Static Verification: Infer@Facebook (null pointer exceptions and resource leaks) GUIErrorDetector@UW (invalid thread access) [ISSTA'12] Thresher@Colorado (not maintained)
  • Android Static Analysis Frameworks: GATOR@OSU [ICSE'15] SPARTA@UW (type checking) [CCS'14] Amandroid@KSU [CCS'14] SOOTForAndriod@UL
    • Framework model: Driodel@Colorado [SOAP'15]
    • Taint Analysis/Information Analysis: DroidSafe@MIT [NDSS'15] FlowDroid@EC (project web) (other tools) [PLDI'14]
    • Source/Sink Inference: Susi@EC SPRIDE (project web) [NDSS'14]
    • Analysis Complications
      • Inter-Component Communication Analysis: IC3@PSU [ICSE'15] (based on COAL) IccTA@Luxembourg [ICSE'15]
      • Java Reflection/Intents: SPARTA@UW (type checking) [ASE'15]
  • Android APK Tools: ApkTool Smali/BakSmali APK-Analyzer
  • Android Test Generation:
    • Symbolic Execution: Acteve@GIT [FSE'12]
    • Event Exploration: DroidBot@PKU DroidBox@Lund SwiftHand@UCB [OOPSLA'13] A3E@UCR [OOPSLA'13] Dynodroid@GIT (online version) [ESEC/FSE'13] LeakDriod@OSU [ISSRE'13]
    • Model-based/Combinatorial Testing: TEMA@TUT [12 pubs] M[agi]C@FBK [ISSTA'12]
    • Andriod Runtime Verification/Dynamic Analysis: Ella@Stanford CopperDroid@RHUL [NDSS'15] DriodRacer@IISC [PLDI'14] DroidScope@Syracuse (binary analysis) [ISSTA'14] Dexposed VdbOnAndroid
    • Andriod Record/Replay/Debugging: IDA Pro (Android Debugging) RERAN@UCR [ICSE'13] GROPG@UTA [ICSE'12 NIER]
  • Android Testing Frameworks: Calabash-Android
    • Android Benchmarks: DriodBench@Darmstadt
  • A Collection of Android Security Tools
• Regression Testing/Change Impact Analysis/Refactoring: DeltaDoc@UVA DiffCat@McGill iregression RefFinder/VDiff/LSdiff @UTAustin ReAssert@UIUC Chianti@Rutgers JDEvAn@UAlberta JDEvAn Viewef@UAlberta Diff-CatchUp@UAlberta AutoCVSCommit@UAlberta ReBA@UIUC MolhadoRef@UIUC RefactoringCrawler@UIUC
  • Compiler testing
    • C: Csmith@UUtah
    • Syntax: SynTESK@ispras
    • Static semantics (context conditions): SemaTESK@ispras
• Mutation testing
  • Java: jMINT@UIC Javalanche@SaarlandU muJava@GMU exman@UOIT ConMAn@UOIT
  • C#: CREAM@WUT
  • Configurations/Logging Debugging
    • Java: Relogger@UCB Confalyzer@UCB Confanalyzer2 Conspellchecker@UCB
• Profiling
  • C#: Prof-It ProfileSharp SlimTune
  • Windows phone application testing
    • Testing frameworks: XnaMobileUnit
    • Subjects: Hawaii applications
• Database application testing: SQLFpc@UNIOVI Other Tools@UNIOVI
• Testing Dataflow programs: SEDGE@UTA
• Grammar-Based Testing: yagg@CWM Quest@Saarland
• Combinatorial Testing: CASA@UNL FireEye@NIST (available in ASE server)
• GUI Testing: GUIERRORETECTOR@UW Guitar@Maryland (GUITAR@U. Maryland) Sikuli@MIT
  • DB Testing (projects): SQLMutation SQLTest DOT@Man Open source testing tools for SQL
  • Scripting Language Testing: LectroTest for Perl
  • AOP Testing/Verification (projects): Bugdel Debugging Tool@Titech Rejuvenate Pointcut AJTE aUnit Flextest Eos@Iowa State JLO abc Prism aspect minerPrismCRAFT Celadon Change Impact Analysis@SJTU RaceJ@McGill rejuvenate-pc@Lancs
  • Web Testing/Verification (projects): Soabench@Lugano Google JS Test BITE@Google Qualitybot@Google WebDiff@GIT Firecrow/Firedoc/phpModeler@FESB Amnesia@GIT Process verifier@UCLA EclipseWTP Blade@ASU LAPSE@Stanford Magenta BPEL4WS LTSA Plug-In bpel4ws Tool4BPEL Eclipse BPEL MSCplugin Web Service Analysis Tool SUT: XML Schema Unit Test Astro suite Web service regression testing toolkit Xmltester Open source testing tools for XML WSUnit WebInject SOAPUI JMeter PushToTest Open source web services tool in Java Open source web testing tools in Java Apache Axis Axis2 Apache Web services project
• SOA Testing/Verification: Soabench@Lugano worldtravel@CNR RoleCast@CNR PUPPET@CNR TAXI@CNR
• Firewall Testing/Verification (projects): Firewall analysis tools@MSU (available in ASE server) ITVal@WM Firewall Policy Advisor@Depaul Firewall tools
  • Policy Testing/Verification (projects): RBAC@Open U. mohawk@MIT SPoX@UTDallas Ur/Web@Harvard Policy testing tools@NCSU (available in ASE server) Margrave@Brown RW model checker
  • Coverage Measurement: PolyFlow@Avaya Jusc@NCSU (available in ASE server) InsECT@Gerogia Tech Coverlipse/Kempka muJava@GMU NCover for C# Prof-It for C#
• Runtime Verification: BIT-Wrapper@VT
  • VM Testing: KEmuFuzzer@UMINI
• HPC Testing (projects):
• Embedded System Testing (projects):
• Security Testing/Debugging (projects): Malfor@Saarland U.
• JavaScript Analysis Tools: SAFE@KAIST Clematis@UBC EventTracer@IBM (WALA for JavaScript) Jalangi@SRA Artemis@Brics TAJS@Brics JSRefactor@BricsCrawlJax JSLint, JavaScript Lint JSure
• Servlet Analysis Tools: ServletAnalyzer@Brics
• Benchmark generators: RUGRAT@UIC
• Debugging Tools: Hierarchical Delta Debugging Framework GenProg@UVA BugAssist@MPI HeapDBG@IMDEA Delta Debugging@Saarland U. HDD@UC Davis Malfor@Saarland U. PlanB@UCLA Juzi@UT Austin FlowCheck@MIT Bug Isolation@UWisconsin DIDUCE@Stanford Repair@UCI Automatic Bug Repair@UNM SOBER@UIUC Ample@Saarland Lizard@Vishwakarma Debugging tools: listed in Zeller's book "Why does my program fail? - A Guide to Systematic Debugging"
  • JavaScript Tools: Crawljax Querypoint-Debugging@EPFL
  • Repairing Tools: Pachika@Saarland U.
  • Dynamic Slicer: JavaSlicer@Saarland U.
  • Purity Tools: JPure@Saarland U.
  • Concurrency Tools: Recon@UW (C/C++/Java) CLAP@HKUST (C/C++) LEAP@HKUST (Java)
• Human-Computation Tools
  • AutoMan@UMass
• Synthesis Tools:
  • iSynth@EPFL Examplary@UVA
  • Generic Instrumentation Tools: Synoptic@UW ExtendedReflection@MSR Dila@Rutgers U. Sofya@UNL InsECT@Gerogia Tech
  • Java bytecode instruemntation: ASM
• Text Mining/NLP Tools:
  • Word Vectors: Glove@Stanford word2vec@Google
    • Sequence to Sequence: TensorFlow@Facebook Keras
  • etcML@Stanford
  • Recodoc@McGill
  • Cognitive Computation Group UIUC tool Page
  • See more tools at Text Analysis for Software Engineering Software Resources Other Resources Also Here
• Machine Learning Tools:
  • Vowpal Wabbit@MSR
• Specification Inference/Code Mining Tools (projects) (mining tool resources): Priming@Technion PEC@MSR Contractor.NET@UBA Synoptic@UW Tark@MSRI Tautoko@Saarland U. LSC Miner@NUS Nort@NUS Objectflattener@UNIMIB Abadu@Saarland Lever@UIUC TrendProf@Berkeley Hear@UCDavis Google Eclipse Search Programming By Sketch@Berkeley API Birthmark@Saarland Sourcerer@UCI Sourcerer@UCI Code Codegenie@UCI Quack@MIT Fossology.org
  • Mining execution traces: usemetainv4daikon@UOregon GetMetaInv@UOregon Daikon@MIT usemetainv4daikon@UMass getmetainv@UMass DIDUCE@Stanford kBehavior@UNIMIB Terracotta@U. Virignia Invgen@UVA Dinv@UBC
  • Mining version histories: Spdiff@DIKU Evolizer@UZurich SemDiff@McGill LSDiff@UTAustin TeMo@Delft eROSE@Saarland
  • Mining static code bases: PEC@MSR API Explorer@McGill Brittle Parameter Typing@ETH ArgOrder@ETH DECKARD@SMU Tikanga@Saarland U. CP-Miner@UIUC LtRrules@OhioU Prospector@UCBerkeley JavaSketch@UCBerkeley Strathcona@Calgary Sniff@UCBerkeley DSFinder@Waterloo
  • Searching code: Code Hint@UCB Sando@ABB Sniff@UCB CriticAL@Clarkson Portfolio@CWM CLAN@UIC CLAN2@UIC Tools@UCI Datasets@UCI Sourcerer@UCI Code Conjurer koders-loganalysis@UCI Tools@CMU (Tool List) Tools@McGill
    • Open source tools: C# and VB.NET Code Searcher - Using Roslyn Code-Conjurer@UMannheim Sourcerer@UCI ack lxr ctags Portfolio C/C++@WM CLAN@WMOpenGrok@Oracle gongzui cs2project for C# (blog) ohloh@BlackDuck (codesight for enterpirse search) Semantic Design Search Engine
    • Commerical webs/tools: ohloh krugle code sight@blackduck codebase
    • Open source code search engines: google code search opensearch@krugle JExample Jarhoo Javadocs Spars Nullege SearchCode symbolhound grepcode snipplr codesearch.debian
    • Logs/Data Set: Data sets koders-loganalysis@BlackDuck/UCI UCI DataSet
    • List: Tool List Top 10 code search engines Top 8 code search engines Top 15 code search engines Source code browsing tools
    • Discussion: C# Internal Use Code Search Tools
  • Doc inference: failuredoc@UW
  • Concept Analysis Tools: Colibri-Java@Saarland Colibri/Concepts@Saarland KABA@PASSAU
  • Visualization Tools: HeapDBG@IMDEA iSpace@vDarmstadt MVT@HUT EPOSoft@Saarland CCVisu@SFU
  • Clone Detection Tools:
  • Java: Deckard@UCDavis SourcererCC@UCI CodePro Analytix@Google
  • C: Vulpecker@HUST Deckard@UCDavis SourcererCC@UCI CCCD@RIT MeCC@HKUST
    • C++: Vulpecker@HUST SourcererCC@UCI
    • PHP: Deckard@UCDavis
  • Various: CCFinder@Osaka
• Case Study Tools: Quayizer@McGill
  • Static Analysis Tools:
    • Multiple Languages: Sonars VisualCodeGrepper
    • Generic Java Static Analysis: Shipshape@Google Wala@IBM Research Crystal@CMU Doop Eclipse JDT Partial Program Analysis@McGill Crystalsaf@CMU Soot@McGill TACLE@OSU BCEL Bddbddb@Stanford bddbddb@Stanford PQL@Stanford Idus@KSU (slicing) JSA@Brics Purity Analysis Tool@MIT JOP Java-XML@Japan Julia@Spoto Jad Jclasslib
    • Generic C/C++ Static Analysis: VCC@MSR Grammatech CodeSurfer (available in ASE server) Eclipse CDT CIL@Berkeley TXL@UWaterloo TXL course
    • Generic Binary Analysis: Vine/Tamu@UCB BAP@CMU Malware@CMU Jakstab@TUM PIN@Intel Valgrind FlowCheck@MIT Fjalar@MIT Kvasir@MIT REDUX Pin@UCorolado Tracer@SUNY SB
    • C# Tools (projects): CCI@MSR Boogie@MSR Spec#@MSR MBEL NModel for C# MSR Phoenix open source testing tools for C# TestGen.Net (download) NUnitGen Addin dynamo NCover Reflector .Net tools FxCop exfinderreflector
      • Tools using CCI@MSR: EFlowMining CCIContribute HeapDBG@IMDEA CCIMetaData CCIAST CCISamples VCC Spec# AutoPex Injectors CircularDependency MindMunge Prevail PexMutator@NCSU BV&LC@NCSU
      • Tools using Roslyn: C# and VB.NET Code Searcher - Using Roslyn
    • PHP Static Analysis: Phantm@EPFL PHP-front Resin@MIT RIPS PHP_CodeSniffer
    • Python: IncPy@Stanford SlopPy@Stanford
    • Java Static Bug Finding: Checker@UW JikeRVM Research Archives Plural@CMU FindBugs@Maryland LAPSE@Stanford Mobius@U. College Dublin JML@Iowa Stae U. ESC/Java 2 SQLGuard@OSU Security tools Security tools Static analysis tools Security tools
      • Java Reverse Engineering: PINOT@UC Davis (Design Pattern Recovery)
      • Java exceptions: TurboJet@Lehigh
    • C/C++ Static Bug Finding (tool collection) : SMACK@UUtah HAVOC@MSR Driver Development Kit/SLAM@MSR Splint@Virginia Airac5 Psi@UCLA
    • C/C++ Concurrency Analysis: STORM@UUtah RELAY@UCSD CHESS for Win32@MSR
      • Java Concurrency Analysis: Vyrd@Koc
    • Other applications: KABA@PASSAU Relo@MIT StringBorg
      • Java Model Checker: NASA Java PathFinder NetStub@UCSB Forge@MIT Bandera@KSU Bogor@KSU
    • C/C++ Model Checker: Satabs@U. Lugano JForge/JMLForge@MIT Blast@UCBerkeley MOPS@UCDavis MAGIC@CMU BOOP VeriSoft@Bell Labs Verisoft Automatic Interface (VAI) Framework Focuscheck@Iowa State U. Cadena@KSU
    • General Model Checker: Alloy@MIT ArCats@NTNU CWB-NC LTSA-MSC@Imperial College PAT@NUS GMC@Sunysb CPROVER@Oxford
    • Embedded System Analysis: TALIRO@ASU Static Stack tool@Utah UPPAAL Avrora@UCLA PlayGame@U Penn MIST@UPenn
    • Software Engineering Static Analysis Tools: Tools@McGill Tools@UBC Tools@MIT
  • Experimental Subjects:
    • Benchmarks Links Rutgers Benchmarks Collection NIST Bib PROMISE SAMATE@NIST
    • Mining: Tools@UCI Datasets@UCI UCI Source Code Data Sets
      • Java: SIR@UNL SF100Benchmark iBug (AspectJ code base and its bugs) TSAFE@MIT Stanford SecuriBench SQL Injection Application Testbed@GIT Web application security contest
      • Java GUI: Java GUI Applications
    • C/C++: GenProg@UVA SER@UCR SIR@UNL SBBASE Test Subjects C++ Extractor Test Suite (CppETS) MIT Lincoln Labs buffer overflow C subjects BugBench@UIUC WiSA format-string exploit benchmarks Software Engineering Experiments on the Web
      • Java Concurency Bugs: pjbench@GIT
      • C Concurrency Bugs: Collection of Concurrency Bugs@U. Michigan Concurrency Bug Benchmarks@UW
      • C#: MSIL/Phoenix Benchmarks
      • AspectJ: ApsectJ Benchmarks AspectJ examples
      • JML: Electronic Purse
      • Spreadsheets: EUSES Spreadsheet Corpus
      • HPC: HPC libraries installed on different HPC platforms at ORNL Matrix Benchmarks links at HPCS Open MPI
  • Others projects: SymExec@UTAustin Mulsaw@MIT TGEN@U. Arizona Algebraic Spec@U. Colorado SpecMining@UCBerkely Hastings@Stanford U. WebAppSleuth@UNL Jist@UPenn Skoll@U. Maryland Gamma@Georgia Tech Evolve@Georgia Tech Galileo@UNL PRESTO@OSU OOTesting@George Mason U. AGENDA@Polytech U. DIATOMS@U. Pitts SymExec@CASSymmetryTesting@INRIA Explain@CMU SOBER@UIUC Opera@UIUC
  • Available executables: Spec#@MSR FSE Spec Explorer@MSR FSE Pex@MSR FSE Zing@MSR Z3@MSR AETG@Telcordia ConTest@IBM Research Agedis@IBM Research et al. IBM Alphaworks-Eclipse IBM Alphaworks-Java UBET@Bell Labs
  • Commercial tools: Agitar Agitator Parasoft Jtest Silvermark Test Mentor Reactive Reactis
• Industrial/government projects: Pex@MSR FSE MSR TVM IBM Research Verification IBM Research SoftEng Leakbot@IBM Research Avaya Labs Research NIST Random Number Generation and Testing