StaticMethod » data http://www.staticmethod.net Things programmers might find interesting Thu, 04 Mar 2010 20:05:58 +0000 http://wordpress.org/?v=2.9.2 en hourly 1 Google’s BigTable http://www.staticmethod.net/2009/04/30/googles-bigtable/ http://www.staticmethod.net/2009/04/30/googles-bigtable/#comments Thu, 30 Apr 2009 14:13:38 +0000 Chris http://www.staticmethod.net/?p=12 Google’s BigTable is (was?) their approach to storing and accessing petabytes of data in a decentralized and highly-available way. At the most basic level it’s a properties system of keys and values, but goes way beyond that. It also provides mechanisms for spreading the data across thousands of machines while giving programmers an easy API for editing and reading data from the tables. Here’s their description of it:

Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.

The paper isn’t too long (14 pages or so) and isn’t too hard to digest. You can read it here

]]> http://www.staticmethod.net/2009/04/30/googles-bigtable/feed/ 0