Serialization format#

ng::serialization (serialization.h) writes a graph to a std::vector<char>, ng::deserialization (deserialization.h) reads it back. The GraphManager uses them to write the .graph files.

ng::serialization::Serializer s(graph);      // throws if the graph has no qualifier
s.serialize_graph();
const ng::serialization::buffer_t& bytes = s.get_buffer();

ng::deserialization::Deserializer d(bytes);
d.deserialize_graph();                       // throws std::runtime_error if the buffer is invalid
ng::Graph copy = d.get_graph();

Everything is saved#

The graph is saved completely: every socket with all its attributes (name, ID, type, active flag, value, allowed values, linkable flag, callbacks), every node, link, parameter and result. Loading a graph doesn’t consult the NodesRegistry: a graph can be loaded even if its node types are not registered in the program (but it can’t be executed).

Encoding rules#

  • Base types are written as raw bytes, in the native endianness, without prefix: len_t (uint64_t, 8 bytes), bound_t (uint8_t), bool (1 byte), socket_t (4 bytes).

  • Strings are [len_t n+1][n characters]['\0']: the length includes the null terminator.

  • Fixed arrays (identifiers, qualifier chunks, version) are [len_t N][N bytes].

  • Compound objects are prefixed by their total length in bytes, prefix included: [len_t L][...].

  • Collections are [len_t L][bound][len_t count][items...][bound]. The bound byte identifies the content and is written at both ends.

Bounds#

Bound

Value

Content

graph

21

Start and end of the whole graph

whitelist

19

Allowed values of an input socket

callbacks

17

Qualifiers of callbacks

sockets

15

Sockets of the graph, or socket IDs of a node

nodes

13

Nodes

links

11

Links

interface

9

Parameters or results

Layout#

Graph
  len_t            total length of the buffer
  bound_t          21
  qualifier        [L][chunk 0][chunk 1][chunk 2]        each chunk: [len_t 28][28 bytes]
  version          [len_t 3][major][minor][patch]
  string           app name, always "nodes-graph"
  string           graph name
  set<link>        bound 11
  map<node_id, Node>                         bound 13
  map<socket_uuid, InputSocket>              bound 15
  map<socket_uuid, OutputSocket>             bound 15
  map<socket_uuid, InterfaceProperties>      bound 9    parameters
  map<socket_uuid, InterfaceProperties>      bound 9    results
  bound_t          21

socket_uuid          [L][node_id: len_t 32 + 32 bytes][socket_id: len_t 24 + 24 bytes]
link                 [L][socket_uuid to][socket_uuid from]
Node                 [L][node_id][qualifier][string name][set<socket_id> inputs, bound 15]
                     [set<socket_id> outputs, bound 15][bool is_proxy]
socket base          [string name][socket_id][socket_t type][bool active]
(no own prefix)      [set<qualifier> on link added, bound 17][set<qualifier> on link broken, bound 17]
InputSocket          [L][socket base][string value][set<string> allowed values, bound 19]
                     [bool linkable][set<qualifier> on value changed, bound 17]
OutputSocket         [L][socket base]
InterfaceProperties  [L][socket_id alias][string show_as]

Checks done when reading#

Every read goes through ReadHead::read, which throws if the buffer is too short: a truncated or corrupted file throws std::runtime_error, it never reads out of the buffer. In addition, the reader checks:

  • the total length equals the size of the buffer;

  • each length prefix equals the number of bytes actually consumed;

  • each bound has the expected value, at both ends of each collection;

  • the number of items of each collection (duplicate keys are detected);

  • array lengths, string null terminators, booleans being 0 or 1;

  • the app name;

  • the version: the file must not be more recent than the parser (parser_version()). Older files are accepted.

The tests flip every byte of a valid buffer and truncate it at every length to check that the reader always throws std::runtime_error or returns a graph, under AddressSanitizer.

Changing the format#

  1. Change the writer (serialization.cc) and the reader (deserialization.cc) symmetrically.

  2. Increase parser_version() in serialization.h.

  3. If the old files must stay readable, branch on the version read in Deserializer::deserialize_graph.

  4. The round-trip tests (tests/serialization) compare the reloaded graph with CmpGraph: a field forgotten on one side makes them fail if CmpGraph compares it, so update CmpGraph too.

The files are not portable between machines of different endianness (see Known limitations).