Update documentation to reflect changes. --- cur-root/doc/guide.lyx | 22 ++++++++++ cur-root/doc/guide.txt | 56 ++++++++++++++++++-------- cur-root/doc/libjio.3 | 3 + cur-root/doc/libjio.lyx | 94 +++++++++++++++++++++++++++++++++++++++++++- cur-root/doc/libjio.txt | 101 +++++++++++++++++++++++++++++++++++++++++++----- 5 files changed, 246 insertions(+), 30 deletions(-) diff -puN doc/libjio.lyx~doc_update doc/libjio.lyx --- cur/doc/libjio.lyx~doc_update 2004-07-13 18:16:48.000000000 -0300 +++ cur-root/doc/libjio.lyx 2004-07-14 23:14:27.947656832 -0300 @@ -129,7 +129,8 @@ This warnings are no different from a no The transaction file \layout Standard -The transaction file is composed of two main parts: the header and the payload. +The transaction file is composed of three main parts: the header, the payload + and the checksum. \layout Standard The header holds basic information about the transaction itself, including @@ -138,7 +139,9 @@ The header holds basic information about two parts: the first one includes static information about the operation (the lenght of the data, the offset of the file where it should be applied, etc.) and the data itself, which is saved by the library prior applying - the commit, so transactions can be rollbacked. + the commit, so transactions can be reapplied if necesary. + The last part is just a 32 bit integer with the checksum of all the previous + data, used for integrity verification during the recovery process. \layout Section The commit procedure @@ -201,7 +204,7 @@ Write the header Read all the previous data from the file \layout Itemize -Write the previous data in the transaction +Write the data in the transaction \layout Itemize Write the data to the file @@ -325,6 +328,91 @@ In any case, after making the recovery y transaction atomicity was preserved. \layout Section +Advanced flags +\layout Standard + +The library allows to set flags to transactions in order to support special + features and behaviour changes that might be useful in special cases. + In this section, we describe the most relevant ones. +\layout Subsection + +Avoid rollbacking +\layout Standard + +If you are completely sure that you will never need to rollback a transaction, + there is one flag, +\emph on +J_NOROLLBACK +\emph default +, that will tell the library to avoid reading the rollback information from + the file when applying a transaction. + It can be useful when transactions are very very big, or there are several + memory constraints, or reading is really synchronous. + It is also very very dangerous because if for some reason the transaction + fails to apply you will not be able to recover it. +\layout Subsection + +Skip locking +\layout Standard + +In some cases, you might not want the library to lock the file itself, because + you need to do it yourself. + For this cases, the flag J_NOLOCK makes the commit procedure skip locking + regions. + You need to be quite careful with this flag because if you don't take good + care of locking, it will lead to corruption. +\layout Subsection + +Lingering transactions +\layout Standard + +We call lingering transactions to a small but intresting variant of the + regular transactions described throughout this text. +\layout Standard + +If we go back at the commit procedure, we will see that first we save all + the data to the transaction file, then write the file, and finally remove + the transaction file, so data gets written twice synchronously. +\layout Standard + +The problem with this approach is performance: it's quite slow because all + the writes and seeks involved. + Besides, it makes no use of the OS write caching capabilities, and it optimizes + for the uncommon case of a crash. +\layout Standard + +Lingering transactions is a special way of dealing with the transactions + we have already seen. + After writing the transaction file and making sure it has hit the media, + the data is already safe. + So then we write to the real file, but this time +\emph on +asynchronously +\emph default +, and let the OS perform the write caching and defer the real operation + to the media. + Then, instead of removing the transaction file, we leave it. + At this point, we know the transaction file is safe, but as the real file + has not been synchronized yet, the data state is still uncertain; however, + if we crash, there will be enough data to recover. +\layout Standard + +Usually, OS do write caching and delay the proper write to the media and + perform it when the time is right or when it's forced by a fsync(), so + the performance goes up a lot. +\layout Standard + +In this mode, you should call jsync() frequently, which calls fsync() on + the file making sure the data is safe, and after that removes all the lingering + transactions. +\layout Standard + +The downside of lingering transactions are the additional space needed to + hold them, and the fact that if you crash there will be more transactions + to reapply, and might take longer. + But if you jsync() often, that shouldn't be noticeable. +\layout Section + UNIX API \layout Standard diff -puN doc/libjio.txt~doc_update doc/libjio.txt --- cur/doc/libjio.txt~doc_update 2004-07-13 18:16:48.000000000 -0300 +++ cur-root/doc/libjio.txt 2004-07-14 23:14:58.365032688 -0300 @@ -10,9 +10,13 @@ Table of Contents 3 The commit procedure 4 The rollback procedure 5 The recovery procedure -6 UNIX API -7 ACID (or How does libjio fit into theory) -8 Working from outside +6 Advanced flags + 6.1 Avoid rollbacking + 6.2 Skip locking + 6.3 Lingering transactions +7 UNIX API +8 ACID (or How does libjio fit into theory) +9 Working from outside @@ -86,8 +90,8 @@ working on them. 2.1 The transaction file -The transaction file is composed of two main parts: the -header and the payload. +The transaction file is composed of three main parts: +the header, the payload and the checksum. The header holds basic information about the transaction itself, including the ID, some flags, and @@ -97,7 +101,10 @@ two parts: the first one includes static about the operation (the lenght of the data, the offset of the file where it should be applied, etc.) and the data itself, which is saved by the library prior -applying the commit, so transactions can be rollbacked. +applying the commit, so transactions can be reapplied +if necesary. The last part is just a 32 bit integer +with the checksum of all the previous data, used for +integrity verification during the recovery process. 3 The commit procedure @@ -135,7 +142,7 @@ way, inside jtrans_commit(): * Read all the previous data from the file -* Write the previous data in the transaction +* Write the data in the transaction * Write the data to the file @@ -214,7 +221,81 @@ remove the journal entirely and let the a new one, and you can be sure that transaction atomicity was preserved. -6 UNIX API +6 Advanced flags + +The library allows to set flags to transactions in +order to support special features and behaviour changes +that might be useful in special cases. In this section, +we describe the most relevant ones. + +6.1 Avoid rollbacking + +If you are completely sure that you will never need to +rollback a transaction, there is one flag, +J_NOROLLBACK, that will tell the library to avoid +reading the rollback information from the file when +applying a transaction. It can be useful when +transactions are very very big, or there are several +memory constraints, or reading is really synchronous. +It is also very very dangerous because if for some +reason the transaction fails to apply you will not be +able to recover it. + +6.2 Skip locking + +In some cases, you might not want the library to lock +the file itself, because you need to do it yourself. +For this cases, the flag J_NOLOCK makes the commit +procedure skip locking regions. You need to be quite +careful with this flag because if you don't take good +care of locking, it will lead to corruption. + +6.3 Lingering transactions + +We call lingering transactions to a small but +intresting variant of the regular transactions +described throughout this text. + +If we go back at the commit procedure, we will see that +first we save all the data to the transaction file, +then write the file, and finally remove the transaction +file, so data gets written twice synchronously. + +The problem with this approach is performance: it's +quite slow because all the writes and seeks involved. +Besides, it makes no use of the OS write caching +capabilities, and it optimizes for the uncommon case of +a crash. + +Lingering transactions is a special way of dealing with +the transactions we have already seen. After writing +the transaction file and making sure it has hit the +media, the data is already safe. So then we write to +the real file, but this time asynchronously, and let +the OS perform the write caching and defer the real +operation to the media. Then, instead of removing the +transaction file, we leave it. At this point, we know +the transaction file is safe, but as the real file has +not been synchronized yet, the data state is still +uncertain; however, if we crash, there will be enough +data to recover. + +Usually, OS do write caching and delay the proper write +to the media and perform it when the time is right or +when it's forced by a fsync(), so the performance goes +up a lot. + +In this mode, you should call jsync() frequently, which +calls fsync() on the file making sure the data is safe, +and after that removes all the lingering transactions. + +The downside of lingering transactions are the +additional space needed to hold them, and the fact that +if you crash there will be more transactions to +reapply, and might take longer. But if you jsync() +often, that shouldn't be noticeable. + +7 UNIX API We call UNIX API to the functions provided by the library that emulate the good old UNIX file @@ -235,7 +316,7 @@ list them here for completion: * jclose() -7 ACID (or How does libjio fit into theory) +8 ACID (or How does libjio fit into theory) I haven't read much theory about this, and the library was implemented basically by common sense and not @@ -282,7 +363,7 @@ Let's take a look one by one: syncronous I/O, data is safely written and can be recovered after a crash. -8 Working from outside +9 Working from outside If you want, and are careful enough, you can safely do I/O without using the library. Here I'll give you some diff -puN doc/guide.lyx~doc_update doc/guide.lyx --- cur/doc/guide.lyx~doc_update 2004-07-13 18:16:48.000000000 -0300 +++ cur-root/doc/guide.lyx 2004-07-14 23:06:47.499655568 -0300 @@ -480,6 +480,28 @@ You can also do this manually with an ut jiofsck \emph default , which can be used from the shell to perform the checking and cleanup. +\layout Subsection + +Lingering transactions +\layout Standard + +If you need to increase performance, you can use lingering transactions. + In this mode, transactions take up more disk space but allows you to do + the synchronous write only once, making commits much faster. + To use them, just add +\family typewriter +J_LINGER +\family default + to the jflags parameter in +\family typewriter +jopen() +\family default +. + It is very wise to call +\family typewriter +jsync() +\family default + frequently to avoid using up too much space. \layout Section Disk layout diff -puN doc/guide.txt~doc_update doc/guide.txt --- cur/doc/guide.txt~doc_update 2004-07-13 18:16:48.000000000 -0300 +++ cur-root/doc/guide.txt 2004-07-14 23:15:05.762908040 -0300 @@ -12,6 +12,7 @@ Table of Contents 5.1 Interaction with reads 5.2 Rollback 5.3 Integrity checking and recovery + 5.4 Lingering transactions 6 Disk layout 7 Other APIs 7.1 UNIX API @@ -56,12 +57,12 @@ represent an ordered group of operations The act of committing a transaction means writing all the elements of the list; and rollbacking means to undo a previous commit, and leave the data just as it was -before doing the commit (while all this definitions may -seem obvious to some people, it requires special attention -because there are a lot of different definitions, and it's -not that common to see "transaction" applied to file I/O -(it's a term used mostly on database stuff), so it's -important to clarify before continuing). +before doing the commit.While all this definitions may seem obvious to some +people, it requires special attention because there are +a lot of different definitions, and it's not that +common to see "transaction" applied to file I/O (it's a +term used mostly on database stuff), so it's important +to clarify before continuing. It's important to note that the library not only provides a convenient and easy API to perform this kind @@ -141,15 +142,27 @@ Let's put it all together and code a nic program (return values are ignored for simplicity): char buf[] = "Hello world!"; + struct jfs file; + struct jtrans trans; + + jopen(&file, "filename", O_RDWR | O_CREAT, 0600, 0); + jtrans_init(&file, &trans); + + + jtrans_add(&trans, buf, strlen(buf), 0); + + jtrans_commit(&trans); + + jclose(&file); As we've seen, we open the file and initialize the @@ -161,7 +174,7 @@ with jtrans_commit(), and finally close 5 Advanced functions -5.1 Interaction with reads +5.1 Interaction with reads So far we've seen how to use the library to perform writes, but what about reads? The only and main issue @@ -176,11 +189,10 @@ This set of functions are very similar t (read(), readv(), etc.); and in fact are named after them: they're called jread(), jreadv() and jpread(); and have the same parameters except for the first one, -which instead of a file descriptor is a file structure (in -fact, this set of functions is a part of what is called -the "UNIX API", which is described below). Bear in mind -that transactions are only visible by reads after you -commit them with jtrans_commit(). +which instead of a file descriptor is a file structureIn fact, this set of functions is a part of what is +called the "UNIX API", which is described below. +. Bear in mind that transactions are only visible by +reads after you commit them with jtrans_commit(). 5.2 Rollback @@ -222,6 +234,16 @@ You can also do this manually with an ut jiofsck, which can be used from the shell to perform the checking and cleanup. +5.4 Lingering transactions + +If you need to increase performance, you can use +lingering transactions. In this mode, transactions take +up more disk space but allows you to do the synchronous +write only once, making commits much faster. To use +them, just add J_LINGER to the jflags parameter in +jopen(). It is very wise to call jsync() frequently to +avoid using up too much space. + 6 Disk layout The library creates a single directory for each file @@ -264,11 +286,11 @@ the manual page to see the details, but like their UNIX version, only that they preserve atomicity and thread-safety within each call. -In particular, the group of functions related to reading -(which was described above in "Interaction with reads") -are extremely useful because they take care of the locking -needed for the library proper behaviour. You should use -them instead of the regular calls. +In particular, the group of functions related to +reading (which was described above in [sub:Interaction-with-reads]) are extremely +useful because they take care of the locking needed for +the library proper behaviour. You should use them +instead of the regular calls. The full function list is available on the man page and I won't reproduce it here; however the naming is quite diff -puN doc/libjio.3~doc_update doc/libjio.3 --- cur/doc/libjio.3~doc_update 2004-07-14 23:19:42.570826840 -0300 +++ cur-root/doc/libjio.3 2004-07-14 23:20:49.426663208 -0300 @@ -22,6 +22,8 @@ libjio - A library for Journaled I/O .BI "int jtruncate(struct jfs *" fs ", off_t " lenght " ); +.BI "int jsync(struct jfs *" fs " ); + .BI "int jclose(struct jfs *" fs " ); .BI "void jtrans_init(struct jfs *" fs " ,struct jtrans *" ts " ); @@ -190,4 +192,5 @@ albertogli@telpin.com.ar. .BR pread (2), .BR pwrite (2), .BR ftruncate (2), +.BR fsync (2), .BR close (2) _