a‎ > ‎

j

Facebook feeds open source community with 'mcrouter'

Summary: Facebook engineers boasted that mcrouter can also work in an Amazon Web Services setup based on Instagram's deployment.

Rachel King

By Rachel King for Between the Lines | September 15, 2014 -- 17:55 GMT (10:55 PDT)

Follow @rachelking SHARE: SUBSCRIBE TO: Networking TOPICS: Networking, Apps, Cloud, Data Centers, Data Management, Web development 2

By Rachel King for Between the Lines | September 15, 2014 -- 17:55 GMT (10:55 PDT)

zdnet-facebook-mcrouter

Facebook is kicking off the week with a number of new bullet points on its open source agenda.

Introduced at the @Scale technical summit in San Francisco on Monday, the social network is targeting engineers who build or maintain systems that are designed for scale.

More about Facebook's infrastructure:

  • Facebook details two years of work to turn on default HTTPS
  • Facebook explains how 'TAO' serves social workloads, data requests
  • Facebook devs explain how it maps user connections to other 'entities'
  • Facebook translates natural language interface under Graph Search
  • Facebook engineers reveal how Parse fits into Platform, B2B strategies
  • Facebook releasing new Social Graph database benchmark: LinkBench
  • Facebook reveals the makings behind App Center recommendation engine
  • Understanding Unicorn: A deep dive into Facebook's Graph Search

The first project is dubbed "mcrouter," a memcached protocol being unleashed under an open-source BSD license.

Facebook itself uses mcrouter for managing all traffic to, from, and between thousands of cache servers across dozens of clusters distributed in Facebook data centers around the world.

The Facebook Engineering team posited in a blog post on Monday that the code will "help many sites scale more easily by leveraging Facebook’s knowledge about large-scale systems in an easy-to-understand and easy-to-deploy package."

Most web-based services begin as a collection of front-end application servers paired with databases used to manage data storage. As they grow, the databases are augmented with caches to store frequently-read pieces of data and improve site performance. Often, the ability to quickly access data moves from being an optimization to a requirement for a site.

Instagram has also been using mcrouter for the last year even before transitioning to Facebook's infrastructure.

Thus, Facebook engineers boasted that mcrouter can also work in an Amazon Web Services setup and not just projects based from Facebook's own data centers.

Facebook also partnered with Reddit, which is currently running mcrouter in production for some caches as part of a limited beta test. Developers and engineers can look on Github for the mcrouter source code now.

At peak scale, the Menlo Park, Calif.-headquarted company boasts mcrouter can handle close to five billion requests per second.

The second announcement continues on Facebook's commitment to the open source community and Open Compute Project with a new collaborative community in Silicon Valley.

Talk Openly, Develop Openly (TODO) was formed for the purpose of sharing and addressing challenges encountered by different tech companies in consuming and running open source software. Companies already signed on Box, Dropbox, Facebook, Google, GitHub, Khan Academy, Square, Stripe, Twitter and Walmart Labs.

The TODO initiative is still in the early stages of planning, but the door (or online form) is open for applications to join the group.

James Pearce, who runs the open source team at Facebook, also revealed in a blog post that "we want to help create a roadmap for companies that want to create their open source programs but aren't sure how to proceed."

Image via Facebook

Topics: Networking, Apps, Cloud, Data Centers, Data Management, Web development

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

#auto

Subpages (7): 4 8 a d e m y
Comments