Elements: Can Empirical SE be Adapted to Computational Science?

Award #1931425

A common concern within the Computational Science (CS) community is that computational scientists are not trained in Software Engineering (SE) and hence might create less-than-good software. This concern is unfounded.

Computational Scientists study micro events within atoms that add up to predictable properties of macro materials (e.g. the sun). Similarly, empirical software engineers study micro patterns within software projects, to learn predictable properties of those projects.

Hypothesis: Integrating SE practices will help computational scientists produce better science (e.g. more reliable, more reproducible, & more efficient).

DR. TIM MENZIES: Email /  LinkedIn /  Website

HUY TU: Email /  LinkedIn /  Website

RISHABH AGRAWAL: Email /  LinkedIn /  Website

  1. Defect forecasting
  2. In our study conducted on 21 CS projects, we found that in case of majority of them (15), adapated SE methods dominate traditional SE methods for defect forecasting.

    • Red: Standard Empirical SE (perform badly on Compute.Sci. projects)
    • Green: After we made it better

  3. Identify Heroes / Antiheroes
  4. In our study conducted on data of more than 3000 developers across 59 CS projects, we found that non-experts and novices are able to contribute to the software development without causing more bugs than existing heroes. Non-hero CS developers introduced only slightly higher number of bugs than hero developers (1.09-1.15 times), as compared to (1.3-1.9 times) in case of SE projects.

    • Idea: Identify effective and ineffective contributors. When to reject patches from “untrusty” sources?
    • Value: Focus energy on top contributors. Automatically assign review tasks to best hero available. Identify contributors that need additional skills development.

  5. CS projects: More granular, build faster
  6. In our study conducted on data of more than 3000 developers across 59 CS projects and comparing with 1000+ SE projects on github, we observed that CS developers commited, closed issues, deployed/released, and tagged more while the projects are shorter in duration.

Which of the following is useful to you?
Or... what else should we do?


  1. Effort forecasting
    • Idea: Predict next month’s work due to issues, bugs and enhancement requests.
    • Value: Increase effectiveness of planning and staff allocation. Improve project estimates.

  2. Defect Forecasts using these attributes

  3. Mailing list genie
  4. A friend who tells you where this question has been answered before

    • Idea: Automatically triage mailing list questions and support requests using prior responses.
    • Value: Reduce support efforts. Improve user experience and community engagement.

  5. Reviewer selector
    • Idea: Automatically identify best reviewer for a peer review.
    • Value: Optimize reviewer selection. Speed up the development cycle with fast feedback.

  6. Test case prioritization
    • Idea: Minimize time to failure for PRs by reordering test cases dynamically.
    • Value: Speed up the development cycle with fast feedback. Find new issues faster.

  7. API Usage Assistant
    • Idea: Jump-start new users with tailored code completion recommendations.
    • Value: Onboard new users quickly and make their first steps the right ones.

  8. API Usage Metrics / Idioms
    • Idea: Characterize your API surface with regards to usage from the community. Find low/high areas of usage and common usage patterns.
    • Value: Prioritize engineering effort based on actual API usage. Identify areas for new examples and documentation. Receive indirect feedback on API design.

  9. Automated Changes
    • Idea: Automatically propose bug fixes and other changes.
    • Value: Reduce developer effort to fix bugs.
Data snapshot

We collected basic measures for over 100 projects and test case / build time metrics on over 50.

  1. Publications
  2. The Changing Nature of Computational Science Software
    Huy Tu, Rishabh Agrawal, Tim Menzies

    In this paper we seek quantitative evidence (from dozens of Comptutational Science projects housed in Github) for 13 previously published conjectures about scientific software development in the literature. In all, we explore three groups of beliefs about (1) the nature of scientific challenges; (2) the implications of limitations of computer hardware; and (3) the cultural environment of scientific software development. We find that four cannot be assessed with respect to the Github data. Of the others, only three can be endorsed. Our conclusion will be that the nature of Computational Science software development is changing. Hence the tools we should develop to support software-based-science, need to change as well.

    Better Data Labelling with EMBLEM (and how that Impacts Defect Prediction)
    Huy Tu, Zhe Yu, Tim Menzies

    Standard automatic methods for recognizing problematic development commits can be greatly improved via the incremental application of human+artificial expertise. In this approach, called EMBLEM, an AI tool first explore the software development process to label commits that are most problematic. Humans then apply their expertise to check those labels (perhaps resulting in the AI updating the support vectors within their SVM learner).

    The source code we used for better labeling can be found here:- SillyWalk


  3. Computational Science projects data
  4. We collected data of 678 computational science projects and identified 59 good ones which met our selection criteria for analysis.

    Selection criteria
    Commits >20
    Duration >1 year
    Issues >10
    Programmers >7
    Releases >0
    Selected projects
    Project Name Link
    abaco https://github.com/TACC/abaco
    abinit https://github.com/abinit/abinit
    apbs-pdb2pqr https://github.com/Electrostatics/apbs-pdb2pqr
    blis https://github.com/flame/blis
    cctools https://github.com/cooperative-computing-lab/cctools
    changa https://github.com/N-BodyShop/changa
    clowder https://github.com/ncsa/clowder
    cpptraj https://github.com/Amber-MD/cpptraj
    cyclus https://github.com/cyclus/cyclus
    elasticsearch https://github.com/elastic/elasticsearch
    forcebalance https://github.com/leeping/forcebalance
    galaxy https://github.com/galaxyproject/galaxy
    GooFit https://github.com/GooFit/GooFit
    hoomd-blue https://github.com/glotzerlab/hoomd-blue
    htmd https://github.com/Acellera/htmd
    hubzero-cms https://github.com/hubzero/hubzero-cms
    hydroshare https://github.com/hydroshare/hydroshare
    irods https://github.com/irods/irods
    lammps https://github.com/lammps/lammps
    luafilesystem https://github.com/keplerproject/luafilesystem
    luigi https://github.com/spotify/luigi
    madness https://github.com/m-a-d-n-e-s-s/madness
    MAST https://github.com/uw-cmg/MAST
    mdtraj https://github.com/mdtraj/mdtraj
    MetPy https://github.com/Unidata/MetPy
    mpqc https://github.com/ValeevGroup/mpqc
    ndslabs https://github.com/nds-org/ndslabs
    nwchem https://github.com/nwchemgit/nwchem
    ompi https://github.com/open-mpi/ompi
    openforcefield https://github.com/openforcefield/openforcefield
    openmm https://github.com/pandegroup/openmm/
    openmmtools https://github.com/choderalab/openmmtools
    OpenMx https://github.com/OpenMx/OpenMx
    orca5 https://github.com/RENCI-NRIG/orca5
    parsl https://github.com/Parsl/parsl
    pcmsolver https://github.com/PCMSolver/pcmsolver
    plumed2 https://github.com/plumed/plumed2
    plumed2 https://github.com/plumed/plumed2
    psi4 https://github.com/psi4/psi4
    pymatgen https://github.com/materialsproject/pymatgen
    pyscf https://github.com/pyscf/pyscf
    QCFractal https://github.com/MolSSI/QCFractal
    quantum_package https://github.com/LCPQ/quantum_package
    radical.pilot https://github.com/radical-cybertools/radical.pilot
    RMG-Py https://github.com/ReactionMechanismGenerator/RMG-Py
    signac https://github.com/glotzerlab/signac/
    signac-flow https://github.com/glotzerlab/signac-flow
    TauDEM https://github.com/dtarb/TauDEM
    trellis https://github.com/trellis-ldp/trellis
    Trilinos https://github.com/trilinos/Trilinos
    tripal https://github.com/tripal/tripal
    WorkflowComponents https://github.com/LearnSphere/WorkflowComponents
    Xenon https://github.com/NLeSC/Xenon
    yank https://github.com/choderalab/yank
    yt https://github.com/yt-project/yt
    galaxy https://github.com/galaxyproject/galaxy
    dealii https://github.com/dealii/dealii
    foyer https://github.com/mosdef-hub/foyer

    Details for the rest of 678 projects can be found here


  5. Tools
  6. We have used the following tools in our work:-

    Git miner

    Git Miner is a python based application to mine GitHub repository.

    UNDERSTAND

    Understand provides you with pertinent information regarding your code. Quickly see all information on functions, classes, variables, etc., how they are used, called, modified, and interacted with.

    Commit Guru

    This tool is used to view automatically generated reports of a repository's commits