# 4 Pickles for Persistent Data Structures

Applications often require to store their state on file and load the saved data later. Oz supports this by pickling of data structures: Data structures are made persistent by writing them to files.

## 4.1 Stateless, Stateful, and Sited Nodes

Values, or more precisely nodes,1 in Oz are either stateless or stateful:

Stateful

  Basic data structures that are stateful include cells, variables, and ports. Since objects, arrays and dictionaries are conceptually composed of cells, they are stateful as well.

Stateless

  Stateless data structures are literals, numbers, records, classes, chunks, and procedures.

In addition, nodes in the store can be sited: the node is specific to a particular site; it is a site-bound resource. For example, classes for files (`Open.file`) and widget classes for graphics (for example, `Tk.toplevel`) are sited.

Only stateless and un-sited nodes can be made persistent by pickling.

## 4.2 Loading and Saving Values

After executing the following statement

> `X=a(proc {$ Y} Y=X end 1 2 f:X)`

`X` refers to a record node. The node can be saved or pickled to the file `test.ozp` by executing

> `{Pickle.save X 'test.ozp'}`

Pickling traverses the entire graph reachable from the root node (which is referred to by `X` in our example), creates a portable description of the graph and writes the description to a file.

The pickled data structure can be loaded by

> `Z={Pickle.load 'test.ozp'}`

Now `Z` refers to a graph which is an isomorphic clone of the graph that has been saved. For our example this means: what can be reached from `X` and `Z` is equal. For example

> `X.1==Z.1`

evaluates to `true`. In fact, `X` and `Z` cannot be distinguished.

Loading of pickles works across the internet: it is possible to give a url rather than just a filename. For example, if you have a public html directory `~/public_html` and you move the pickle file `test.ozp` there, everybody can load the pickle across the internet. Suppose that the url of your public html directory is `http://www.ps.uni-sb.de/~schulte/`, then the pickle can be loaded by

> `Z={Pickle.load 'http://www.ps.uni-sb.de/~schulte/test.ozp'}`

## 4.3 Example: The Data Base Revisited

To extend the data base we developed in Section 2.3 with persistence, we just add two procedures to load and save a data base and extend the export specification accordingly. The toplevel structure of the functor definition is as follows:

<PDB.oz>=
  `functor
  import Pickle
  ` <Export specification for PDB.oz>`
  ` <Body for PDB.oz>`
  end`

The functor imports the system module `Pickle`. The export specification is just extended by the fields `load` and `save`.

<Export specification for PDB.oz>=
  <Export specification for DB.oz>`
  load: Load
  save: Save`

The body for `PDB.oz` is as follows:

<Body for PDB.oz>=
  <Body for DB.oz>`
  proc {Save File}
  {Pickle.save {Ctr get($)}#
  {Dictionary.toRecord db Data}
  File}
  end
  proc {Load File}
  I#D={Pickle.load File}
  in
  {Dictionary.removeAll Data}
  {Ctr init(I)}
  {Record.forAllInd D
  proc {$ K E}
  {Dictionary.put Data K E}
  end}
  end`

`Save` takes as input the filename of the pickle, whereas `Load` takes the url from which the pickle can be loaded.

When using the persistent data base, it has to be kept in mind that it does not offer concurrency control: Simultaneous add and remove, as well as load and save operations performed by several threads might leave the data base in an inconsistent state. In ??? we will develop the data base in a data base server that also allows for concurrency control.

Note that since we only extended the functionality the functor provides, all programs that used the non-persistent data base could, in principle, still use the persistent data base with out being recompiled. We say could because the implementaion of the persistent database is named `PDB.ozf` rather than `DB.ozf`. However, you can give it a try and simply rename `PDB.ozf` to `DB.ozf`: all applications based on the the non-persistent implementation will continue to work as before but now using the persistent implementation (though without actually taking advantage of the persistency).

## 4.4 Pickle Compression

Pickles can also be compressed so that they occupy less space on disk. For example, a compressed pickle for `X` can be written to file `testz.ozp` by

> `{Pickle.saveCompressed X 'testz.ozp'` `LevelI``}`

`LevelI` is an integer between `0` and `9` specifying the compression level: the higher the value the better the compression, but the longer pickling takes. A value of `0` gives no compression.

Compression time and ratio depend on the data being pickled. The compression ratio might vary between 20 and 80 percent, while compression at level 9 is usually less than 2 times slower than using no compression.

1. A node may also be an unbound variable, i. e. a value that is not yet determined