RIB-in-.zip Support

RIB-in-.zip Support

January, 2012

Introduction

RIB has been a reasonable choice for representing renderable geometric data because it:

  • supports a rich collection of geometric primitives with a focus on final-rendering quality, while also supporting pre-tessellated representations.
  • supports both composition and deferred-loading through Delayed Read Archives and Procedural Primitives.
  • supports a reasonably compact representation for animation.
  • supports a standard representation for shading and lighting (through RiSurface, RiLightSource, RiShader and RSL).
  • supports association of arbitrary attributes through primitive variables and user attributes.
  • supports just-in-time modification and filtering via RenderMan Interface Filters (RIF).
  • is required by RenderMan.
  • is documented and reasonably standard.

But with the growth of CGI pipelines has also grown the number of RIB files. A single shot might be comprised of thousands, or even millions of RIB files; this can tax even the most robust of network file systems. What's missing from the RIB file format is a means by which a group of RIB files can be associated efficiently. If we could group all the potentially moving pieces associated with a character or set into a single logical collection we might ease the burdens imposed by RIB on complex pipelines.

Is this really a missing feature of RIB? Both .zip files and .tar (or .tgz) files have long been used to collect groups of files into a larger entity. Both of these formats qualify as open source standards and might serve our needs for a container of RIB files. For our purposes, the .zip file format has the advantage over the .tgz file in that it supports direct reading of arbitrary subfiles within the collection.


RIB-in-.zip Files

RIB-in-zip is a simple but powerful solution to problems associated with managing a large number of on-disk RIB files.

  • simplifies asset management, asset groups can be combined into a single .zip file and managed atomically.
  • atomization of RIB components means small units that are extremely efficient to render:
    • delayed read-archives mean minimal memory footprint
    • re-entrant procedural primitive for reading .zip files is thread safe and scalable (multiple threads can read in the same .zip file with no locks).
  • gentler on network files systems, which suffer from poor performance when many files are opened.

Writing RIB into a .zip File

RIB authors should understand that a .zip file is merely a stream-oriented collection of individually compressed subfiles, which can be efficiently appended but inefficiently pruned, reorganized, etc. Thus, the one constraint imposed on the RIB writer when writing to .zip files is that a single subfile must be written "in one go". If your RIB generation stream is recursive and parent RIB streams are held open while writing child streams, this would not be the case.

Here is a legal multi-RIB stream (indentation for illustrative purposes only):

// first we attempt to create a .zip container file
RiBegin("/a/path/to/your.zip")
  // our first RIB in .zip
  RiBegin("renderman/rib/job/first.rib")
  // more Ri calls here
  RiEnd()
  // our second RIB in .zip
  RiBegin("renderman/rib/job/second.rib")
  // more Ri calls here
  RiEnd()
// we close the .zip and write its table of contents
RiEnd()

And here is an illegal sequence:

RiBegin("/a/path/to/your.zip")
  RiBegin("renderman/rib/job/first.rib")
    // more Ri calls here (->first.rib)
    // our second RIB in .zip (nested)
    RiBegin("renderman/rib/job/second.rib")
      // more Ri calls here (->second.rib)
    RiEnd()
  // more Ri calls here (->first.rib)
  RiEnd()
RiEnd()

We relax this restriction via the addition of per-subfile buffering. Here, the entire contents of a RIB stream are buffered and emitted at RiEnd. As you can imagine, this feature should be used with care or the memory consumption of your RIB generation session will be unbounded.

Here's a legal nested sequence with a buffered outer stream:

RtInt one=1, zero=0;
RiBegin("/a/path/to/your.zip")
  RiOption("rib", "int fullybuffered", &one);
  RiBegin("renderman/rib/job/first.rib")
    // more Ri calls here (->first.rib)
    // our second RIB in .zip (nested)
    RiOption("rib", "int fullybuffered", &zero);
    RiBegin("renderman/rib/job/second.rib")
      // more Ri calls here (->second.rib)
    RiEnd()
  // more Ri calls here (->first.rib)
  RiEnd()
RiEnd()

Issues, Limitations

  • The request to create a .zip file is signaled by the file extension passed to RiBegin.
  • Since we open/create a zip file with RiBegin, it's possible that Ri calls will be made before the first named subfile. We create a default subfile a.rib (following Unix a.out convention) to accumulate unqualified Ri requests.
  • Currently we only create a new .zip; append isn't yet implemented.
  • Subfile editing/replacing isn't supported. Existing zip tools (e.g. zip, unzip) can be used for this purpose.

Reading

.zip as searchpath entry:

Option "searchpath" "archive" "testzip:@:testzip/archive.spheres.zip"

Inter-zip or Extra-zip references (! notation follows jar/zip URI convention)

ReadArchive "archive.cones.zip!archive/driver.rib"

Intra-zip references:

ReadArchive "archive/firstcone.rib"

Appendix

  • zip vs tgz vs tar

    1469002253 Jan 27 16:46 a.zip
    1468786910 Jan 27 17:06 a.tgz
    6158317568 Jan 27 16:58 a.tar
    

See Also