Five Nines
Randy Guck, chief scientist for StarTeam and CaliberRM, gave a great talk on increasing uptime for StarTeam servers. Most of it was fairly industry-standard stuff: backups, recovery plans, failover, clustering, etc. The cool part was his description of how the new StarTeam 2005 vault works.
First, the old vault (VaultI). Each file in the repository is stored in the vault archive. The initial file is stored in full, and revisions are stored as deltas (except for binary files). That composite file may also be compressed, based on filename extension. When you check out a file, the server looks up the correct offset into the composite file in the database, reconstructs the file, and places it in the cache, where it is then streamed to the user. If I need a different revision of that file, we go through the same process all over again. The fact that you need the database to be in exact sync with the files at all times, because of the offsets and deltas, is why StarTeam has to be locked to perform a backup.
Here's how VaultII works: Each instance of a particular file is stored on it's own, with a filename that is also it's md5 hash. The database then tracks each revision of a file by its hash, rather than by a name and offset. The server, when the file is written, automatically determines if compressing that file would have any benefit and chooses the compression type on a file by file basis, so you'll have files in the vault that are just "md5", and some that are "md5.gz".
But doesn't this mean we have a lot more files? For any given set of file contents, there is only one file in the vault. So, if you have the same file checked in to multiple projects, or if you check in identical revision content, the database points to the same vault file because the md5s all match. Their research with customer-donated databases found that 20-40% of files in the vault end up being redundant, so a lot of the space concerns are offset. From a user perspective, this means that, at most, the file may have to be decompressed before sent over the network (unless the user has link compression turned on).
How does this help with backups? Well, when you do a backup under the new system, you first do a backup of the database (which can be done while the db is live). With that snapshot, you can then copy/replicate/backup the vault archive at will, since any new content that goes to the vault is not referenced by the database. Sure, you may have a few extra files in your vault backup that aren't in the db, but that's ok. If you had to restore from the backup, the user would have to check in any work that occurred since the backup, but the server would notice that those file contents were already in the vault, so it links the file that's already on disk, and you don't end up wasting space (unless a user doesn't bother to check in something that was lost in a failure, but that's his fault).
But, there's lots of revisions we don't use. Defaultworld.wld in Biomek, for example, has over 900 binary revisions, and we don't use that file any more. Isn't this wasteful? Turns out they've added a feature where you can ask the server to archive old revisions. The admin can then burn those to disk and store them away. If a user tries to reference that revision, they get a message saying that they need to get the admin to restore that particular revision to the server. The admin restores the file, and the user is happy, all live and without downtime.
I've got some good pictures of this in the slides, and I'll cover it in more detail if people want. I'm going to a "Birds-of-a-Feather" session for StarTeam users next, and hopefully I can get some more detail out of the devs. There's a meet the StarTeam developers session tonight as well.
(Oh, and if you have questions, throw them in here as comments, and I'll try to answer them.)
First, the old vault (VaultI). Each file in the repository is stored in the vault archive. The initial file is stored in full, and revisions are stored as deltas (except for binary files). That composite file may also be compressed, based on filename extension. When you check out a file, the server looks up the correct offset into the composite file in the database, reconstructs the file, and places it in the cache, where it is then streamed to the user. If I need a different revision of that file, we go through the same process all over again. The fact that you need the database to be in exact sync with the files at all times, because of the offsets and deltas, is why StarTeam has to be locked to perform a backup.
Here's how VaultII works: Each instance of a particular file is stored on it's own, with a filename that is also it's md5 hash. The database then tracks each revision of a file by its hash, rather than by a name and offset. The server, when the file is written, automatically determines if compressing that file would have any benefit and chooses the compression type on a file by file basis, so you'll have files in the vault that are just "md5", and some that are "md5.gz".
But doesn't this mean we have a lot more files? For any given set of file contents, there is only one file in the vault. So, if you have the same file checked in to multiple projects, or if you check in identical revision content, the database points to the same vault file because the md5s all match. Their research with customer-donated databases found that 20-40% of files in the vault end up being redundant, so a lot of the space concerns are offset. From a user perspective, this means that, at most, the file may have to be decompressed before sent over the network (unless the user has link compression turned on).
How does this help with backups? Well, when you do a backup under the new system, you first do a backup of the database (which can be done while the db is live). With that snapshot, you can then copy/replicate/backup the vault archive at will, since any new content that goes to the vault is not referenced by the database. Sure, you may have a few extra files in your vault backup that aren't in the db, but that's ok. If you had to restore from the backup, the user would have to check in any work that occurred since the backup, but the server would notice that those file contents were already in the vault, so it links the file that's already on disk, and you don't end up wasting space (unless a user doesn't bother to check in something that was lost in a failure, but that's his fault).
But, there's lots of revisions we don't use. Defaultworld.wld in Biomek, for example, has over 900 binary revisions, and we don't use that file any more. Isn't this wasteful? Turns out they've added a feature where you can ask the server to archive old revisions. The admin can then burn those to disk and store them away. If a user tries to reference that revision, they get a message saying that they need to get the admin to restore that particular revision to the server. The admin restores the file, and the user is happy, all live and without downtime.
I've got some good pictures of this in the slides, and I'll cover it in more detail if people want. I'm going to a "Birds-of-a-Feather" session for StarTeam users next, and hopefully I can get some more detail out of the devs. There's a meet the StarTeam developers session tonight as well.
(Oh, and if you have questions, throw them in here as comments, and I'll try to answer them.)
0 Comments:
Post a Comment
<< Home