Fig 1.
An example scholarly neural network system diagram, from Maharjan et al.[3].
Fig 2.
Fig 3.
Three example NN system diagrams from ACL 2017, with captions from original papers, licensed under CCBY4.0.
Clockwise from top left: [19–21] (IDs numbers as in the diagram corpus of [18]: 186, 189, 174).
Table 1.
Interview participant summary.
Table 2.
User tasks reported by each participant when reading diagrams. Y indicates a “top three most important task" and N indicates “do not do at all". Some participants did not select precisely three tasks, as explained in Sect 4.3
Fig 4.
The two parts of the experiment, completing the communication loop and assessing NN system diagram usability, while facilitating remote administration.
The method is based on accessibility guideline effectiveness investigation [81], with an ecologically valid context and additional quantitative and qualitative measures. The bottom row relates to readership, the middle row to the authorship or diagrams themselves, and the top row to the framework (though some aspects are shared across different streams).
Fig 5.
Number of different digital diagram creation tools reported as being used by each participant.
Table 3.
Participants expressed a spectrum of opinions, of differing confidence, on whether a precise depiction is meaningful. P7 made two different comments, and P3 and P9 did not make comments mapped to this theme
Table 4.
Overall opinion of example diagram: + = like, - = dislike, blank = neutral.
Table 5.
Summary of interview findings related to the usage of scholarly neural network system diagrams.
Table 6.
Proposed framework for improving neural network system architecture diagrams, as presented to participants during evaluation.
Table 7.
Evidence from the interview study supporting each of the proposed guidelines in the framework.
Fig 6.
“Before framework” = left, “After framework” = right.
Fig 7.
“Before framework” = left, “After framework” = right.
Fig 8.
“Before framework” = above, “After framework” = below.
Table 8.
Count of participants recommending to keep each guideline. Most participants thought most individual guidelines should be kept.
Table 9.
Count of participants preferring each version of the diagram. Diagrams B1P4, B2P3 and B2P4 were not edited.
Fig 9.
Good diagrams as perceived by readers is correlated with those readers providing a good overall text summary according to the diagram author.
Table 10.
Cumulative probabilities, for diagram sample size (n) and simulated number of post-framework preferred N, with probability p of being a sufficient sample size to demonstrate significance.
Table 11.
Framework violation count, for each quartile of number of citations.
Fig 10.
Scatter plot of number of citations versus NN system diagram guideline compliance (as a quantitative proxy for “how good the diagram is").
LOESS curve for locally weighted smoothing is in blue, and the function is in red.
Fig 11.
Number of citations for papers containing system diagrams, grouped by level of guideline compliance.
Table 12.
Correctness in partitioning top and bottom cited papers based on diagram only. Framework compliance outperforms the best performing expert assessment, in addition to consensus-based measures.
Table 13.
The framework for improving neural network system architecture diagrams, incorporating evaluation results.