What is Prevayler?

Prevayler is an object persistence library for Java. It is an implementation of the System Prevalence architectural style, in which data is kept hot in Memory with changes journaled for system recovery.

Prevayler ' s architecture is illustrated in the diagram shown here. Prevayler [1] serves as a transactional barrier for the business objects [2] of your application, held in Memory. You encapsulate all modifications of your business objects into instances of the Transaction interface [3], much like a " command " pattern (though different from a command in some details of the pattern). Whenever you ask Prevayler to execute a transaction on your business objects [4], Prevayler first writes the transaction object to a journal [5] so that data is not lost if your system crashes. Prevayler can also write a snapshot of your entire business object graph [6] as often as you wish. Prevayler uses the latest snapshot together with the journals to automatically recover your business objects from disk [7] on application startup by restoring the snapshot and then re-executing every transaction that was originally executed after that snapshot was taken.

Why would I use Prevayler?

Well, here are a few of the reasons why we use Prevayler:

1. It ' s extremely simple. There ' s no separate database server to run.

2. It lets us program with real objects. We can choose whatever object models, data structures, and algorithms are appropriate to our particular application domain. Prevayler doesn ' t require our business objects to extend any base class or implement any interface in order to be persistent.

3. It ' s test-friendly. Since we ' re programming with real objects, we can properly encapsulate logic together with the data it uses, making unit tests cleaner, easier to write, and faster to run. The restrictions that Prevayler does impose on us - - keeping our code deterministic, for example - - are consistent with the disciplines that test-driven development teaches us anyway.

4. It makes threadsafety easy. Prevayler makes transactions run sequentially, so we don ' t have to worry about typical multithreading issues like locking and consistency (unless we really want to get tricky, and Prevayler lets us do that, too).

5. It ' s extremely fast. Everything runs in Memory, so the only disk access is streaming transactions to the journal, which we ' ve optimized to a peak rate of up to a thousand transactions per second on desktop hardware. And read-only queries don ' t even have that overhead, so they run instantaneously, hundreds of thousands per second.

What rules do I have to follow?

Prevayler is extremely simple, and imposes very few restrictions on your object model. The only restrictions are to make sure that transaction execution is deterministic and repeatable, so that the exact state of your business objects can be recovered from the transaction journal.

1. All modifications to your business objects must be encapsulated in transaction objects.

2. All your transactions must be completely deterministic. For example, don ' t call System.currentTimeMillis () from within a transaction (or within any business object code called indirectly by a transaction), because it will give a different answer every time the transaction is run. (If you do want to know the time, Prevayler provides an " official " timestamp, assigned when the transaction is first written to the journal. See the Transaction interface.)

Note that any hardware input or output is inherently nondeterministic, and therefore violates this rule. Do all I / O outside of Prevayler transactions.

3. All your transactions and business objects must be serializable, so that they can be written to the journal and snapshot. (You can also configure your own serialization mechanisms if you don ' t want to use Java serialization.)

4. Transactions can ' t carry direct object references (pointers) to business objects. This has become known as the baptism problem because it ' s a common beginner pitfall. Direct object references don ' t work because once a transaction has been serialized to the journal and then deserialized for execution its object references no longer refer to the intended objects - - any objects they may have referred to at first will have been copied by the serialization process! Therefore, a transaction must carry some kind of string or numeric identifiers for any objects it wants to refer to, and it must look up the objects when it is executed.

Download

Download Prevayler 2.3, containing code, documentation and demos.

You can also join the Prevayler mailing lists.

If you have any questions or problems using Prevayler, please post on the prevayler-discussion mailing list, not the comments section here - - you'll get a much quicker response.

See: Links, Prevalent Hypothesis, Baptism Problem, Memory Technology, Prevayler Examples

[[This new wiki is still under construction. An earlier draft of this front page is here.]]

The essential principle relating to the " baptism problem " is that business objects within Prevayler's protection should never leave Prevayler's protection. Abiding by that principle lets Prevayler take care of things like concurrency and transaction recovery consistently.

The implication of that principle is that the safest style of Prevayler usage is to always use Queries and Transactions for any access to the business objects, and never pass business objects as the results of queries or held in transactions. Prevayler makes sure that a Transaction will never run concurrently with a Query, simplifying the threadsafety of your application.

If you were to return a business object as the result of a query (or access one directly through the prevalentSystem () call), your first challenge would be that you are now responsible for more careful threadsafety, because Prevayler can no longer prevent transactions from running concurrently with other accesses to that object. You can do that if you like, just be aware of the consequences.

If you do go ahead and extract a direct reference to a business object, the real meat of the " baptism problem " is that you really cannot pass that object back in through a transaction and expect it to work, because during recovery from the transaction journal, each transaction is deserialized from the journal, which necessarily creates a new instance of the object rather than referring to the one protected inside Prevayler. This is a particularly troublesome problem because it might seem to be working fine as the system is running, but will break if you shut down the system and restart it from the journal.

(As it happens, Prevayler's internal pipeline always deserializes a copy of each transaction anyway, even when running for the first time, which helps to catch this kind of problem earlier.)

All this means is that transactions should carry some identifier of the business objects they want to interact with, rather than direct references to the objects, and then look up the object during execution within Prevayler.

As a special case, immutable objects such as String may be fine to pass in and out - - as long as you are not using their object identity (pointer equality) for anything meaningful - - because threadsafety is taken care of by the very fact that they ' re immutable.

As another special case, a transaction which is adding a new object may safely carry a business object, because there's no issue with object identity - - you're not trying to interact with an existing object. Even if the object is copied, it's okay, because you're treating it as a new object anyway. Just don't keep any references to that object outside of Prevayler - - once you've sent it in through a transaction, if you want to interact with it, you'll need to either use a Query or get it back out with the prevalentSystem () call in order to get the ultimate " real " object.

It's named like that because most new users end up having this problem when creating their first prevalent system, and after they're " baptized ", they rarely run into it again.

All your transactions must be completely deterministic, including any business object code called indirectly by a transaction. A transaction must always have exactly the same effect on the business objects whether it is being executed the first time or executed again later during journal recovery.

What does that mean in practice? Here are some things to avoid:

• Don ' t call System.currentTimeMillis (), because it will give a different answer every time the transaction is run. If you do want to know the time, Prevayler provides an " official " timestamp, assigned when the transaction is first written to the journal. See the Transaction interface.
• Don ' t access any files or sockets, because hardware I / O is inherently nondeterministic. Even if you ' re reading a file that always has the same contents, you could get an unexpected exception. Even if you ' re just writing to a log file and not depending on the results, again you could get an exception. (Doing any hardware I / O can also be arbitrarily slow, and you want your Prevayler transactions to run instantaneously.) You can, of course, do I / O outside of Prevayler and pass the actual data to your business objects through a transaction.
• Don ' t use the default Object.hashCode () implementation, because it relies on the current low-level Memory address of the object, which will be different the next time the system runs.
• Don ' t use the iterators of Hashtable, HashMap, or HashSet. They return entries in an arbitrary order that may be changed by some serialization mechanisms. LinkedHashMap is a good alternative, as it guarantees that the iteration order will be the same as the order that items were added. TreeMap and TreeSet keep entries sorted, so they ' re also okay.
• Don ' t use Random without giving it a seed. It gets its seed from the system clock. SecureRandom is even worse, because it reads from hardware devices to get true random input. Instead, you can use Random or SecureRandom outside of Prevayler and pass random values or seeds in through a transaction.

This new Prevayler site is under construction.

In the meanwhile, you can download Prevayler 2.3, containing code, documentation and demos.

You can also join the Prevayler mailing lists.

The following classes have a main () method and can be run directly. They are self explanatory.

1. org.prevayler.demos. demo1.Main - A tiny application that generates random numbers and stores them using Prevayler.
2. org.prevayler.demos. demo2.Main - A bank application using Prevayler.
3. org.prevayler.demos. demo2.MainTransient - Exactly the same bank application running only in RAM.
4. org.prevayler.demos. demo2.MainReplicaServer - Exactly the same bank application with transparent replication enabled.
5. org.prevayler.demos. demo2.MainReplica - An application that connects to the MainReplicaSerever above and transparently replicates the bank application.

Related projects that may prove useful to Prevayled applications

Queries

The JXPath Component

http://commons.apache.org/jxpath
http://commons.apache.org/jxpath/users-guide.html

Related articles

Java Object Querying Using JXPath - by Brian Agnew (08/03/2006)
http://today.java.net/pub/a/today/2006/08/03/java-object-querying-using-jxpath.html
Java object queries using JXPath - Query complex Java object trees using the XPath expression language - by Bart van Riel (21/03/2007)
http://www.javaworld.com/javaworld/jw-03-2007/jw-03-jxpath.html
JXPath Tutorial - by Bart van Riel
http://www.tfo-eservices.eu/wb_tutorials/pages/jxpath-tutorial.php

Groovy

An agile and dynamic language for the Java Virtual Machine
http://groovy.codehaus.org/Mixed+Java+and+Groovy+Applications
Mixed Java and Groovy Applications
http://groovy.codehaus.org/Mixed+Java+and+Groovy+Applications
Collections guide
http://groovy.codehaus.org/Collections
Maps guide
http://groovy.codehaus.org/JN1035-Maps (search for Grouping!)
Interception guide
http://groovy.codehaus.org/JN3515-Interception (intercepting BO methods?)

Collections

GNU Trove

High performance collections for Java
http://trove4j.sourceforge.net

Apache Commons Collections

Commons-Collections seek to build upon the JDK collections classes by providing new interfaces, implementations and utilities.
http://commons.apache.org/collections

The Google Collections Library

A suite of new collections and collection-related goodness for Java 5.0, brought to you by Google. (Still Alpha)
http://code.google.com/p/google-collections

Articles / Tutorials

Prevalence: Transparent, Fault-Tolerant Object Persistence by Jim Paterson (06-08-2005) _{[www.onjava.com]}

Preclipse Plugin by a team studying Computer Science at the University of Hannover (~ March 2004 ) _{[preclipse.sourceforge.net]}
( Do check the example-tutorial pages! )

An introduction to object prevalence by Carlos Villela (01-08-2002) _{[www-128.ibm.com]}

Serialization Throughput by Ward Cunningham (30-12-2001) _[c2.com]

Discussions

DB snapshotting, reporting, partial prevalence, AOP prevalence by Klaus Wuestefeld - forum discussion (26-01-2004) _{[article.gmane.org]}

3251 Times Faster than MySQL, 9983 Times Faster than ORACLE by Klaus Wuestefeld - forum discuss ion (May 2002) _{[www.advogato.org]}

Object Prevalence by Klaus Wuestefeld - fo rum discussion (~ Xmas 2001) _{[www.advogato.org]}

Serialization

Debugging Serialization Errors by Bob Lee (26-02-2007) _{[crazybob.org]}

Ensure proper version control for serialized objects by Alan Hui (27-02-2006) _{[www.javaworld.com]}

JDC Tech Tips - Serialization by Stuart Halloway (29-02-2000) _{[java.sun.com]} (In the real world, Versioning, Secure data, Complete rewrite)

Always Declare Serial Version Uid by Ward Cunningham (14-07-2005) _[c2.com]

From Three minutes with Google's Eric Schmidt :

" At Google, for example, we found it costs less money and it is more efficient to use DRAM as storage as opposed to hard disks - - which is kind of amazing. It turns out that DRAM is 200,000 times more efficient when it comes to storing seekable data. In a disk architecture, you have to wait for a disk arm to retrieve information off of a hard-disk platter. DRAM is not only cheaper, but queries are lightning fast. "

http://archives.cnn.com/2002/TECH/internet/02/01/interview.eric.schmidt.idg/index.html

From Java 2 Platform, Standard Edition v 1.4 Datasheet :

" 64-bit support provides Java technology developers with near limitless amounts of Memory for high-performance, high-scalability computing. While previous J2SE releases were limited to addressing 4 gigabytes of RAM, version 1.4 allows Java applications to access hundreds of gigabytes of RAM. This enables developers to drive more applications and very large datasets into Memory, and avoids the performance overhead of reading data in from a disk or from a database. "

http://java.sun.com/j2se/1.4.2/datasheet.1_4.html

From MetaRAM Bets On High Capacity Memory Breakthrough by Andy Patrizio (February 25, 2008):

" MetaRAM, founded by Fred Weber, a former chief technology officer at AMD, is going public today with its MetaSDRAM. The technology doubles the Memory on a Memory stick by using cheaper Memory components. "
" The MetaSDRAM chipset is a controller that sits between the chips on a DRAM stick that enables up to four times more mainstream DRAMs to be integrated into existing DIMM stick without the need for any hardware or software changes. As a result, 8GB and 16GB DRAM sticks are possible for a fraction of their normal price. "
" Weber estimates that by 2010, 2GBit DRAMs will be available in mass quantities just as 1GBit DRAMs are available now. At that point, MetaRAM can produce 16 and 32GB Memory controllers for roughly the price of the 8GB and 16GB products it will produce now. "

http://www.internetnews.com/hardware/article.php/3729911/MetaRAM+Bets+On+High+Capacity+Memory+Breakthrough.htm

Pricewatch - System Memory

http://www.pricewatch.com/system_memory /

 

Non-direct RAM

Such solutions will allow for very fast virtual memory access.

From Gear6 satiates hungry apps with 500GB RAM monster by Ashlee Vance (August 28, 2007):

" At 250,000 IOPS, Gear6 boasts that it can beat out a high-end NAS (network attached storage) system from, say, NetApp by 5X. The company also says that its half a millisecond latency easily beats out the typical 2 to 8 milliseconds of latency experienced by customers with demanding, clustered software. All told, Gear6 promises a 10X to 30X performance improvement for applications with large data sets. "
" You all knew that someone would pump out MAS (memory attached storage) boxes sooner or later. There's just too many tempting uses for such a system - like slapping a database in memory - to ignore the concept. And it seems very likely that Gear6 will face some direct competition soon enough. "

http://www.theregister.co.uk/2007/12/10/amd_violin_memory/print.html

From Violin plays AMD for memory monster by Ashlee Vance (December 10, 2007):

" The company sells a Violin 1010 unit that holds up to 504GB of DRAM in a 2U box . Fill a rack, and you're looking at 10TB of DRAM. "
" An AMD Opteron processor-based server connected to a HyperTransport technology-enabled Violin Memory Appliance will have both directly connected memory and Extended Memory resources. Directly connected memory can be selected for bandwidth and latency while the Extended Memory can be much larger and located in the Memory Appliance. Applications such as large databases will benefit from the large-scale memory footprints enabled through Extended Memory. "
http://www.theregister.co.uk/2007/12/10/amd_violin_memory/print.html

The terms defined in this glossary are hereby placed in the public domain:

1 - System Prevalence

Transparent persistence, fault-tolerance and load-balancing of the execution of the business logic of an information system through the use of state snapshots as well as command and query queueing or logging.

Prevalence of object-oriented systems.

Software library or component that provides prevalence for information systems.

Prevalence layer for object-oriented systems.

16-Oct-2001 - Klaus Wuestefeld.

(copied from: http://sourceforge.net/docman/display_doc.php?docid=7291&group_id=36113)

[[the following is still under construction]]

Benefits

• Power - You are free to implement your business logic and store your business data using any syntax, data-structure and algorithm your programming language supports: SQL, pure OO, functional, reactive functional, whatever. The Prevalent System pattern is orthogonal to that.

• ACID Transactions

• Persistence - Prevalent System state is persisted transparently without the need of a DBMS.

• Performance - There is no system architecture faster and more performance-scalable than System Prevalence, period. Queries on Prevalent Systems are 3 to 4 orders of magnitude faster than using a relational database, for example, depending on the RDBMS you choose to compare. Details below.

• Stability - System Prevalence is based on the simple and proven concepts of transaction logging and system state snapshots. That makes Prevalent System implementations straightforward and stable.

• Load Balancing

• Fault Tolerance - If one machine in your Prevalent System crashes, no big deal. The system will prevail.

• Local Mirroring - It is trivial to create local mirrors of any Prevalent System, so that your users can experience a rich user interface (not browser limited), without the need of clumsy remoting middleware such as CORBA, RMI or SOAP, and with zero latency for queries.

• Simplicity - The concept is so simple it actually enrages some people.

Applications

In stand-alone mode: Document editors, CAD / CAM / CAE, personal productivity apps, embedded apps, SCADA, devices, games.

In single server mode: websites, e-commerce, business systems.

In clustered server mode: Critical non-stop systems, heavy-duty websites.

In local mirror mode: Multi-user CAD / CAM / CAE projects, multi-user document editing, workflow, small business systems, multi-user games, sovereign apps.

How it Works

Let us now see exactly how each of the benefits above is achieved:

[Under construction]

Limitations

• All the business data in your system must fit in RAM. That is precisely the Prevalent Hypothesis.

[Under construction]

Problems

[Under construction]

Comparison to Traditional Database Architectures

[Under construction]

" That all your business data fits in the RAM you can afford "

Does it make sense to do Reporting directly against OBJECTS?

There are pros and cons to that:

Pro 1) Queries will run MUCH faster than through a database connection. Every Date or every
BigDecimal value present in every column of every record of every JDBC ResultSet is an object and
has to be marshalled. That is extremely costly.

Pro 2) You can use all the querying logic and calculations already available in the business logic
of your system without having to duplicate that in your Reporting SQLs.

Pro 3) Because of the above, you never need to break your system's encapsulation. Most people
couldn't care less about this but I believe encapsulation breaking through " shared " databases to
be the main cause for system integration and maintenance nightmares.

Con 1) You have to use an OO query layer that works with POJOs or a Reporting tool that runs on
objects rather than table records (or write your own layer / tool).
Alternatively you write a bit of extra code to access the existing business logic methods.: -)