To accelerate our innovation in search engines, we have found it
necessary to build an infrastructure to provide many large-scale data
processing and data management capabilities. The infrastructure we are
currently building, called WebStudio, allows researchers and engineers
to easily implement and test their ideas or algorithms at web-scale
without worrying about low-level system issues. With this
infrastructure, we are able to apply web-scale data mining technologies
to understand web pages and queries and use the extracted information to
improve the performance of our current search engine. In this talk, I
will introduce the projects we are working on at MSR Asia related to the
development of this infrastructure and shared some important lessons we
have learned. |