Skip to content

Commit

Permalink
Update doc
Browse files Browse the repository at this point in the history
Signed-off-by: Chen Wang <[email protected]>
  • Loading branch information
wangvsa committed Jan 4, 2025
1 parent e988db4 commit 3c4bfdf
Show file tree
Hide file tree
Showing 2 changed files with 41 additions and 112 deletions.
103 changes: 41 additions & 62 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
@@ -1,80 +1,59 @@
|build|

Recorder
Overview
========

**A Multi-Level Library for Understanding I/O Activity in HPC
Applications**

Quickstart
----------

**1. Build**

Recorder requires MPI and HDF5 to build. Please make sure they are installed
before building Recorder.

Run the following to fetch, build, and install Recorder.
In this example, we will install Recorder under its srouce directory.
You can change the install localtion by modifying ``$RECORDER_INSTALL_PATH`` below.

.. code:: bash
git clone https://github.com/uiuc-hpc/Recorder.git
cd Recorder
export RECORDER_INSTALL_PATH=`pwd`/install
git submodule update --init --recursive
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=$RECORDER_INSTALL_PATH
make
make install
We believe that multi-level I/O tracing and trace data analysis tool can
help end users understand the behavior of their application and I/O
subsystem, and can provide insights into the source of I/O performance
bottlenecks.

Recorder is a multi-level I/O tracing framework that can capture I/O
function calls at multiple levels of the I/O stack, including HDF5,
MPI-IO, and POSIX I/O. Recorder requires no modification or
recompilation of the application and users can control what levels are
traced.

**2. Trace an application**
Publications
------------

Make sure ``$RECORDER_INSTALL_PATH`` points to the location where you installed Recorder.
Now let's run an application with Recorder to collect it traces.
We will use an example MPI program as our targeting application.
Go to the Recorder's source directory and run the following.
Recorder has undergone significant changes since the last
"Recorder 2.0" paper. We have incorporated a new pattern-based
compression algorithm, along with many new features.
We are preparing a new paper that will describe all these
changes in detail.

.. code:: bash

cd test && mpicc test_mpi.c -o test_mpi
mpirun -np N -env LD_PRELOAD $RECORDER_INSTALL_PATH/lib/librecorder.so ./test_mpi
`Wang, Chen, Jinghan Sun, Marc Snir, Kathryn Mohror, and Elsa
Gonsiorowski. “Recorder 2.0: Efficient Parallel I/O Tracing and
Analysis.” In IEEE International Workshop on High-Performance Storage
(HPS), 2020. <https://doi.org/10.1109/IPDPSW50202.2020.00176>`__

# On HPC systems, you may need to use srun or
# other job schedulers to replace mpirun, e.g.,
srun -n4 -N1 --overlap --export=ALL,LD_PRELOAD=$RECORDER_INSTALL_PATH/lib/librecorder.so ./test_mpi

Once completed, you will see a folder named ``recorder-yyyymmdd``. This is a parent folder
that contains all traces folders generated on *yyyymmdd*.
Inside it, you will find the actual trace forlder of the application you just run.
The folder name follows the format of ``HHmmSS.ff-hostname-username-appname-pid``.
I/O Patterns and consistency requirements
-----------------------------------------

**3. Examine the traces**
`Traces from 17 HPC applications <https://doi.org/10.6075/J0Z899X4>`__

Recorder provides several useful tools under ``$RECORDER_INSTALL_PATH$/bin`` for analysis purposes.
The traces were collected using an old version of Recorder. The current
version uses a different trace format. To read those traces please use
Recorder 2.2.1 from the
`release <https://github.com/uiuc-hpc/Recorder/releases/tag/v2.2.1>`__
page.

*recorder-summary* can be used to reports high-level statistics:

.. code:: bash
$RECORDER_INSTALL_PATH/bin/recorder-summary /path/to/your_trace_folder/
*recorder2text* is used to convert the Recorder-format traces to plain text files.

.. code:: bash
$RECORDER_INSTALL_PATH/bin/recorder2text /path/to/your_trace_folder/
This will generate text fomart traces under ``/path/to/your_trace_folder/_text``.
You will see *N* [pid].txt files, where *N* is the number processors you run your application.
Each of these txt files contains the traces generated by one processor indicated by the [pid].
The following paper presents a comprehensive study on the I/O patterns
of these applications. Additionally, it investigates the consistency
requirements of these applications, confirming experimentally that
POSIX consistency is rarely required by HPC applications.

`Wang, Chen, Kathryn Mohror, and Marc Snir. “File System Semantics
Requirements of HPC Applications.” Proceedings of the 30th International
Symposium on High-Performance Parallel and Distributed Computing (HPDC).
2021. <https://dl.acm.org/doi/abs/10.1145/3431379.3460637>`__

.. toctree::
overview
:maxdepth: 1
:hidden

quickstart
build
usage
features
Expand Down
50 changes: 0 additions & 50 deletions docs/source/overview.rst

This file was deleted.

0 comments on commit 3c4bfdf

Please sign in to comment.