Google's browser and applications chief Sundar Pichai will replace Rubin, bringing the firm's mobile software, applications and Chrome browser under one roof.
Larry Page, Google's chief executive and co-founder, credited Rubin for evangelizing Android more than half a decade ago and said the decision to switch was Rubin's. Page was mum on Rubin's future role in a blogpost on Wednesday announcing the switch.
Rubin built Android into a free, open-source software platform now used by most of the world's largest handset manufacturers, from Samsung Electronics Co Ltd to HTC Corp.
Android is now installed on roughly two-thirds of the world's smartphones, supplanting Apple Inc. at the pinnacle of the fast-moving mobile arena.
Android tablets are also expected to overtake Apple's iPad in terms of shipments in 2013, IT research house IDC predicted on Tuesday.
"Going forward, Sundar Pichai will lead Android, in addition to his existing work with Chrome and Apps. Sundar has a talent for creating products that are technically excellent yet easy to use-and he loves a big bet," Page wrote in the blog post.
"Andy's decided it's time to hand over the reins and start a new chapter at Google. Andy, more moonshots please!"
Mr Pichai, who began his engineering training in India before moving to the United States, aggressively pushed Google's Chrome browser in 2008, when Microsoft Corp's Explorer lorded over the market. Chrome now commands a roughly 35 percent market share according to Web traffic analyzers Stat Counter.
He is also credited with the development of some of the company's more successful apps, such as Calendar and Gmail.
"While Andy's a really hard act to follow, I know Sundar will do a tremendous job doubling down on Android as we work to push the ecosystem forward," Page said.
Shares in the company slipped 0.2 percent to $825.83 in afternoon trading.