Real-Time Inverted Search by Bryan Bende
Video from DevIgnition 2014 - Elephant Talk! See devignition.com/ for details.
Building real-time notification systems is often limited to basic filtering and pattern matching against incoming records. Allowing users to query incoming documents using Solr's full range of capabilities is much more powerful. In our environment we needed a way to allow for tens of thousands of such query subscriptions, meaning we needed to find a way to distribute the query processing in the cloud. By creating in-memory Lucene indices from our Solr configuration, we were able to parallelize our queries across our cluster. To achieve this distribution, we wrapped the processing in a Storm topology to provide a flexible way to scale and manage our infrastructure. This presentation will describe our experiences creating this distributed, real-time inverted search notification framework.
Bryan Bende is a Senior Lead Engineer in the Strategic Innovation Group at Booz Allen Hamilton. He focuses on building solutions for clients involving big-data, distributed systems, and information retrieval. Bryan received a B.S. in Computer Science from the University of Maryland at College Park, and a M.S. in Computer Science from Johns Hopkins University.