PetaVision Output (PVP) file specifications

The PetaVision library uses a single file extension for model outputs, *.pvp. This extension includes a header that houses information about the data contents as well as the data. There are six different types of PVP files, for the different types of networks in PetaVision. Each type can represent its data using one of four different data types, that vary in precision. There are analysis scripts written in Matlab and Python that parse these files and convert them into workable data structures.

Types of PVP files

1. PVP_FILE_TYPE
   • No longer used.
2. PVP_ACT_FILE_TYPE
   • Used for storing binary sparse activity file
3. PVP_WGT_FILE_TYPE
   • Weight file type, only used for HyPerConns that are not KernelConns
4. PVP_NONSPIKING_ACT_FILE_TYPE
   • Used for storing dense activity matrices
5. PVP_KERNEL_FILE_TYPE
   • Weight file type for KernelConns
6. PVP_ACT_SPARSEVALUES_FILE_TYPE
   • Used for storing floating point sparse activity files

Data types for PVP files

1. PV_BYTE_TYPE
   • char (8-bit precision) type
   • Used in PVP_WGT_FILE_TYPE if the writeCompressedCheckpoints flag is true
     • Currently only used in file outputs, the weights are still float-precision internally. We are currently implementing the capability for PetaVision to use byte-precision internally as well as in writing out files.
2. PV_INT_TYPE
   • int32 type
   • Used for PVP_ACT_FILE_TYPE
3. PV_FLOAT_TYPE
   • float32 type
   • Used for PVP_WGT_FILE_TYPE and PVP_NONSPIKING_ACT_FILE_TYPE
4. PV_SPARSEVALUES_TYPE
   • Alternates between int32 (index) and float32 (values)
   • Used for PVP_ACT_SPARSEVALUES_FILE_TYPE

Each PVP file has a header associated with it.

PVP header fields

• datasize
  • The number of bytes used by one value of the given datatype
• datatype
  • An int32 that corresponds to one of the PVP data types, numbered above
• filetype
  • An int32 that corresponds to one of the PVP file types, numbered above
• headersize
  • The number of bytes used by the header
• kx0
  • No longer used
• ky0
  • No longer used
• nb
  • An int32 that corresponds to the number of batches used. Note that this parameter is for the end user only, as OpenPV never reads this parameter when reading files.
• nbands
  • Activity: An int32 that corresponds to the number of frames in the pvp file.
  • Weights: An int32 that corresponds to the number of arbors.
• nf
  • The number of features in the model
• numparams
• numrecords
• nx
  • The spatial extent of the layer in the x direction, will always be = nxGlobal.
• nxGlobal
  • The spatial extent of the column, which defines the global space.
• nxprocs
  • How many processors MPI used in the x direction. This field is no longer used.
• ny
  • The spatial extent of the layer in the y direction, will always be = nyGlobal.
• nyGlobal
  • The spatial extent of the column, which defines the global space.
• nyprocs
  • How many processors MPI used in the y direction. This field is no longer used.
• recordsize

File structures

Syntax is:

dataField [(numberElements, Type), (numElement, Type), (numElement, Type), ...]

Note that header may or may not have extra fields, but readPvpHeader should handle it. recordSize is in the header

• PVP_ACT_FILE_TYPE

Header[(1, 80 bytes)]
time[(1, float64)]
numActive[(1, uint32)]
data[(numActive, uint32)]
time[(1, float64)]
numActive[(1, uint32)]
data[(numData uint32)]
...

• PVP_WGT_FILE_TYPE Extra header is nxp, nyp, nfp, wMin, wMax, numPatches

[For every frame]
Header[(1, 80 bytes)]
Extra Header[(3, int32), (2, float32), (1, uint32)]
[For every arbor]
[For every proc]
[For every patch per proc]
shrunkenPatchNxNyOffset[(2, uint16), (1, uint32)]
data(nxp * nyp * nfp, dataType)
[end for every patch]
[end for every proc]
[end for every arbor]
[end for every frame]

• PVP_NONSPIKING_ACT_FILE_TYPE

Header[(1, 80 bytes)]
time[(1, float64)]
data[(recordSize, dataType)]
time[(1, float64)]
data[(recordSize, dataType)]
...

• PVP_KERNEL_FILE_TYPE

[For every frame]
Header[(1, 80 bytes)]
Extra Header[(3, int32), (2, float32), (1, uint32)]
[For every arbor]
[For every proc]
[For every patch]
shrunkenPatchNxNyOffset[(2, uint16), (1, uint32)]
data[(nxp * nyp * nfp, dataType)]
[end for every patch]
[end for every proc]
[end for every arbor]
[end for every frame]

• PVP_ACT_SPARSEVALUES_FILE_TYPE

Header[(1, 80 bytes)]
time[(1, float64)]
numActive[(1, uint32)]
[for numActive]
data[(1, uint32), (1, float32)]
[end for numActive]
time[(1, float64)]
numActive[(1, uint32)]
[for numActive]
data[(1, uint32), (1, float32)]
[end for numActive]
...