Walk the hierarchy
Every circle is a cluster of supercomputer jobs with similar I/O behaviour. The top circle is the whole dataset; each split below it separates jobs that behave differently. Select a circle to see what sets that group apart.
Cluster Hierarchy
Thinned for PreviewFrom raw records to groups you can explain
Keep the measurements that describe behaviour
Darshan writes one log per job with several hundred counters. DataExplorer keeps the POSIX I/O counters and job metadata, and derives the user and application from each log.
344 measurements per job, of which 43 describe I/O behaviour and are used for clustering.
Remove records that cannot be characterised
Malformed records are dropped, and so are jobs too small to say anything about: a job that moved under 100 MB has no meaningful I/O pattern.
941,364 logged jobs in, 110,421 jobs analysed.
Put every feature on a comparable scale
Volumes that span many orders of magnitude are log-scaled. Detailed counters become shares of a total, for example the share of reads that were sequential.
14 log-scaled volumes and 29 share-of-total features.
Group similar records at every level of detail
HDBSCAN, a density-based method, builds a full hierarchy without being told how many clusters to find. Splits that separate very little are folded away so the picture stays readable.
1,871 clusters, from the whole dataset down to groups of a few jobs.
Say what each group is
For any cluster DataExplorer reports who and what is in it, how its feature distributions look, and which features sit furthest from the dataset average.
Users, applications, six volume features and five ratio features per cluster.
Compare groups side by side
Open up to four clusters next to each other, recolour the hierarchy by any feature, search for a user or application, and drill into individual records.
All of it runs in the browser against the live service.
Four questions, any tabular dataset
DataExplorer is configured per dataset by a short profile: which columns are categories, which are volumes, which are ratios. No code changes are needed to add a domain.
How does the data group?
A hierarchy of clusters, from the whole dataset down to tight groups, with no labels and no preset number of clusters.
What characterises a group?
The features that sit furthest from the dataset average, and the mix of categories inside the group.
How do groups compare?
Up to four clusters side by side, with the same charts and the same colour coding.
Which records deserve a closer look?
Drill from a cluster into its individual records with linked, filterable parallel-coordinate views.
Clusters that are one application
Applications were never used for clustering, only I/O behaviour. Even so, many of the finest clusters turn out to be at least 60% a single application: the same code tends to do the same I/O.
Select a row to open that cluster in the preview above.
| Application | Jobs | Share of cluster |
|---|---|---|
| Loading… | ||
The demonstration dataset
Darshan is an I/O characterisation tool that runs on production supercomputers. This dataset holds the Darshan logs of jobs run on the Theta system at the Argonne Leadership Computing Facility from 2017 to 2020. HPC I/O experts used an earlier version of this tool, Gauge, to study the same workload.
Dataset Profile
Bring your own data
DataExplorer adapts to a new dataset with a profile, not a rewrite. Talk to our team about your data.
Request a Briefing →