Also, I have a question for you regarding some claims I have heard someone make about the ability to alter their search ranking in LinkedIn – I was hoping to point you towards a video and either debunk or confirm what this person is claiming.

Looking forward to your response. Thanks!

This is not to say that one can’t do prefix/wildcard-based searching on a Lucene-backed search system with a large number of documents, in a performant way. It’s just that doing so while also serving a heavy load of textually-simpler (but also taking into account the multiple language preferences of our users, and as mentioned before, the social-graph component) more popular queries with low latency, in the same system is highly nontrivial, and having multiple systems serving the same data in different ways poses its own resourcing challenges.

Allowing much more advanced control over query relevance is more a question of “what Products LinkedIn should provide”, and is not really my bailiwick, but also digs into the question of who LinkedIn builds products for: we obviously try to serve our Power Users, who do sourcing for a living, but LinkedIn is not just for them – it’s for everyone who wants to take control of their career as if it were a small business, for hiring managers who aren’t search-engine experts, for people looking to connect with former coworkers and clients. The average user, while wanting the search system to take their “intent” into account, may not have the time or inclination to spend a lot of time learning query syntax or a new UI to plug in how much they care about each term in their query.

On the other hand, users in the past decade have been trained by Google to assume that the search engine will be “smart enough” to know what they mean without them being very specific. Similarly, at LinkedIn, we do a lot of work with offline data mining to do things figuring out that when a user searches for “VP IBM”, they’re looking for someone with the *title* VP and the *company* IBM, even without specifying ccompany:IBM AND ctitle:VP (because a miniscule fraction of our userbase uses the query-field based syntax we expose that you are familiar with). Similarly, since the typical user is looking for people who currently do that thing, instead of in the past, we dynamically turn VP IBM into (ccompany:IBM AND ctitle:VP)^current_boost OR (pcompany:IBM AND ptitle:VP)^past_boost OR (VP AND IBM)^body_boost, where current_boost > past_boost > body_boost are boost parameters we need to figure out based on how well it serves our users (of course, there’s more going on here as well: when someone puts in “Dell” – are they looking for Michael Dell, the CEO, or are they looking for someone *at* Dell – we do some fancy magic to figure out the relative probabilities of both, if the user doesn’t specify by using the company or name fields, and adjust the boosts accordingly: (lname:Dell^last_name_boost)^last_name_probability OR (ccompany:Dell^current_boost OR pcompany:Dell^past_boost)^company_probability and other things like this).

But since you really want more control over the kinds of searches you can do on the site… well, I can’t say anything now, but just wait a few weeks or so, you’ll start to get a taste of some of the stuff our Search Team has been brewing for quite some time, and I hope you’ll like it.

I have had the privilege of working with software and database engineers responsible for an enterprise portal (involving search) and a 40+ TB data warehouse for a Fortune 50 company, so although I am clearly non-technical, I can appreciate (although certainly nowhere near your level of understanding) the challenges associated with searching profiles that are being updated in real time, and I am aware that prefix searching can be a significant drain on resources (although it can be implemented with extremely fast search execution).

I do have a few questions for you if you have the time – I’d love to tap your significant experience and insight on a couple of things:

#1 When you sort results by keyword, does LinkedIn search performance still suffer from the challenge of taking into consideration the searcher’s personal network in terms of determining relevance? If sorting results by keyword only is NOT encumbered by the searcher’s connections when considering relevance, could it be less resource-intensive to offer more search functionality such as prefix and configurable proximity? If so, perhaps offering a truly separate keyword-only search and results sorting, unaffected by the user’s network/connections, could offer users more advanced search functionality that could otherwise be too complex/resource dependent when sorting by what LinkedIn terms as “relevance” – which incorporates both keyword and connections. Thoughts?

#2 If not for the real-time updates/search aspect, does enabling wildcard/prefix search still pose as serious of a resource drain for searching LinkedIn? There are databases of well over 40M records where new records are being added every minute (but each record is typically not altered after being entered) that seem to effortlessly handle queries with 10+ prefix/wildcard terms. What in your opinion poses the biggest challenge for LinkedIn in offering prefix searching?

I’m glad you find the idea of offering users the ability to boost specific search terms interesting, and I do agree that most people are unaware of this kind of search functionality. However, that fact alone should not determine whether not not such a feature is offered. In fact, being able to offer search term boosting could be a nice way to differentiate LinkedIn search, in that I am not aware of any social network, social media application, online resume database, or Internet search engine that offers this feature. Bragging rights, if you will (e.g., the Ferrari of people search at no additional cost). Plus, booting terms is not rocket science – I am confident that many users will be able to easily take advantage of the ability to control the relevance of their own results based on what they feel are the most important search terms.

Ultimately, information management is all about retrieval – data is worthless without the ability to find exactly what you need when you need it. Basic keyword search with Boolean logic (AND/OR/NOT) is a approach that only allows for basic retrieval, which is (IMO) intrinsically limited and imprecise when it comes to retrieving relevant results and is prone to a large percentage of false positive results. I believe that most people only know how to use basic keyword search because they have not been offered anything else.

When users with premium access to LinkedIn can view 300, 500, 700, or 1000+ results, I’m pretty confident most of them aren’t really viewing past the first 100 – 200, as most people don’t have the time to do so. As such, to truly provide value to your paying customers, it is critical for users to be able to take some control over the relevance of their search results so that the first 100 to 200 are in fact the most relevant to them – in other words, that the the first 100 – 200 results are the ones that most closely match the INTENT of the user’s search, not just the keywords. Search terms, in and of themselves, do not determine relevance – the search engine does – and the search engine does not and can not “know” the intent of the user. I can tell you from personal experience that boosting (controlling specific term relevance weighting) and configurable proximity search can play a HUGE role in users being able to take true control over the relevance of their results.

@Phil – that is simply not true. I just ran a simple search of the type you indicated this morning — chief AND (oncologist OR neurologist) — under LI Advanced People Search just to make sure I’d have distinct results and it definitely pulls up both expected result sets.