Serving technology enthusiasts for more than 25 years.
TechSpot is the place to go for tech advice and analysis.
Editor’s take: Colossus: The Forbin Project is a story about a supercomputer that becomes sentient, enslaving the human race. Colossus also refers to the storage platform on which Google’s internet services are stored. Although we don’t yet know if Google was inspired by the classic sci-fi film, the connotations still exist. Google revealed in a blog post that some of the “secrets” hidden behind Colossus – a massive network infrastructure described by the company as its universal storage platform. Colossus has a robust design, is scalable, easy to use, and can be programmed. Google said that the massive machine uses magnetic hard disks, which are still in use and evolving.
Colossus is the engine behind many Google services including YouTube, Gmail and Drive. The platform evolved from the Google File System, a distributed storage solution for managing large data-intensive applications. This made things easier to manage. Google surprised many by supercharging Colossus with an exclusive cache system that relies on solid-state drives.
Google creates one Colossus File System per cluster in a Data Center. These clusters are powerful and can manage multiple exabytes, with two filesystems in particular hosting more than 10 Exabytes each. The company claims Google-powered apps or services will never run out of space in a Google Cloud Zone.
Data throughput is impressive in a Colossus File System. Google claims that “regularly” the largest clusters can read up to 50 terabytes of data per second and write up to 25 terabytes.
Storing the data in the correct place is crucial for achieving such a high level of performance. Colossus users can specify whether their files should be stored on an HDD, SSD or both. However, most developers use an automated solution called L4 distributed SSD cache. This technology uses machine-learning algorithms to determine what policy to apply to certain data blocks. The system will eventually write any new data onto the HDDs.
L4 caching technology can (partially solve) this problem over time, by observing I/O pattern, segregating the files into specific “categories,” & simulating different storage locations. Google’s documentation states that these policies include “place on SSD for one hour,” “place on SSD for two hours,” “don’t place on SSD.”
and”place on SSD for one hour,” “place on SSD for two hours,” When simulations accurately predict the file access pattern, a small amount of data is placed on SSDs to absorb the majority of initial read operations. The data is eventually moved to cheaper storage (HDDs), to minimize hosting costs.
“As the basis for all of Google and Google Cloud, Colossus is instrumental in delivering reliable services for billions of users, and its sophisticated SSD placement capabilities help keep costs down and performance up while automatically adapting to changes in workload,” said the company. “We’re proud of the system we’ve built so far and look forward to continuing to improve the scale, sophistication, and performance.”
