Back in March 2010, I had the distinct honor of delivering the keynote presentation at SourceCon on the topic of resume search and match solutions claiming to use artificial intelligence in comparison with people using their natural intelligence for talent discovery and identification.

Now that nearly 2 years has passed, and given that in that time I’ve had even more hands-on experience with a number of the top AI/semantic search applications available (I won’t be naming names, sorry), I decided it was time to revisit the topic which I am very passionate about.

If you’ve ever been curious about semantic search applications that “do the work for you” when it comes to finding potential candidates, you’re in the right place, because I’ve updated the slide deck and published it to Slideshare. Here’s what you’ll find in the 86 slide presentation:

• A deep dive into the deceptively simple challenge of sourcing talent via human capital data (resumes, social network profiles, etc.)
• How resume and LinkedIn profile sourcing and matching solutions claiming to use artificial intelligence, semantic search, and NLP actually work and achieve their claims
• The pros, cons, and limitations of automated/black box matching solutions
• An insightful (and funny!) video of Dr. Michio Kaku and his thoughts on the limitations of artificial intelligence
• Examples of what sourcers and recruiters can do that even the most advanced automated search and match algorithms can’t do
• The concept of Human Capital Data Information Retrieval and Analysis (HCDIR & A)
• Boolean and extended Boolean
• Semantic search
• Dynamic inference
• Dark Matter resumes and social network profiles
• What I believe to be the ideal resume search and matching solution

The trouble with search today is that people put too much trust in search engines – online, resume, social, or otherwise.

I can certainly understand and appreciate why people and companies would want to try and create search engines and solutions that “do the work for you,” but unfortunately the “work” being referenced here is thinking.

I read an article by Clive Thompson in Wired magazine the other day titled, “Why Johnny Can’t Search,” and the author opens up with the common assumption that young people tend to be tech-savvy.

Interestingly, although Generation Z is also known as the “Internet Generation” and is comprised of “digital natives,” they apparently aren’t very good at online search.

The article cites a few studies, including one in which a group of college students were asked to use Google to look up the answers to a handful of questions. The researchers found that the students tended to rely on the top results.

Then the researchers changed the order of the results for some of the students in the experiment.  More often than not, they still went with the (falsely) top-ranked pages.

The professor who ran the experiment concluded that “students aren’t assessing information sources on their own merit—they’re putting too much trust in the machine.”

I believe that the vast majority of people put too much trust in the machine – whether it be Google, LinkedIn, Monster, or their ATS.

Trusting top search results certainly isn’t limited to Gen Z – I believe it is a much more widespread issue, which is only exacerbated by “intelligent” search engines and applications using semantic search and NLP that lull searchers into the false sense of security that the search engine “knows” what they’re looking for.

This is Your Search Without a Brain

It’s easy to see why people and companies create search products and services using semantic search and NLP that claim to be able to make searching “easier” – they are looking to sell a product  based on the value of making your life easier, at least when it comes to finding stuff.

If you take a look at some of the marketing materials for intelligent search and match search products, you’ll find value propositions such as “Stop wasting time trying to create difficult and complex Boolean search strings,” ”Let intelligent search and match applications do the work for you,” and “A single query will give you the results you need – no more re-querying, no more waste of time!”

I love saving time and getting to what I want faster, but my significant issue with “intelligent search and match” applications is that they try to determine what’s relevant to me.

And that’s a rather large issue, because only I know what I am looking for.

It’s critical to be reminded that the definition of ‘relevance,’ specifically with regard to information science and information retrieval, is “how well a retrieved document or set of documents meets the information need of the user.”

The only person that can make the judgment of how well a search result meets their information need is the person conducting the search, because it’s their specific information need.

Any reference to “relevance” by a search engine, whether it be Google, Bing, LinkedIn, Monster, etc., is based purely on the keywords, operators, and/or facets used.

Search engines don’t know what you want – they only know what you typed into or selected from the search interface.

Poor use of keywords, operators or facets will don’t stop you from getting results. All searches “work,” as I am fond of saying – but the quality or relevance will likely be low.

Of course, that assumes that the person conducting the search is actually proficient at judging the quality or the relevance of the results – comparing results to their specific information need and experimenting with different combinations of keywords, operators and facets to look for changes in relevance.

Related Does Not Equal Relevant

I personally never implicitly trust first page or top ranked search results online, nor top ranked results on LinkedIn, Monster, or anywhere I search. Some of the best search results I have ever found were buried deep in result sets – far past where most people would typically review, and essentially in the territory of results the search engine deemed least “relevant.” Pshaw!

One reason for this is because I understand that any search engine I use, no matter how “dumb” (straight keyword matching), or “intelligent” (semantic/NLP), they can only work with the terms I give it. What do you think the  most “intelligent” search engine can do with poor user input?

When it comes to searching, unfortunately everyone’s a winner, because every search “works” and returns results.  The problem is that few searchers know how to critically examine search results for relevance.

Regardless how how “intelligent” a search engine might be, it can only try to find terms and concepts related to my user input.

This is an often overlooked but critical issue – just because terms might be related, it does not mean they are relevant to my information need.

It certainly helps to understand that some of the most relevant search results can’t actually be retrieved by the obvious keywords, titles or phrases, or even those that a semantic search algorithm deems related to them. In fact, some of the best results simply cannot be directly retrieved – see my post on Dark Matter for more information on the concept.

However, to appreciate the concept that no single search, no matter how enhanced by technology, can find all of the relevant (by human standards and judgment) results available to be retrieved, you have to know a thing or two about information retrieval in the first place.

And if you already lack the ability to critically judge search results and evaluate them for relevance, how can you be expected to be able to evaluate and critically examine the search results returned by intelligent search and match applications?

The “black box” matching algorithms of intelligent search and match applications pose significant issues to users in that searchers have absolutely no insight as to why the search engine returns the results it does. Without this, what option does a user have other than to implicitly trust the search engine’s matching algorithm?

Searching Ain’t Easy

Who says search has to be easy anyway?

Just because you might want it to be, should it be? Does it have to be?

Let’s face it – a lot of people look for the easy way out. The sheer volume of advertisements pushing diet supplements that claim you can lose a ton of weight without having to watch what you eat and exercise is evidence that people want to get the results they want without working for them.

You know the best way to lose weight? A healthy diet combined with regular exercise. The problem is that eating healthy and exercising regularly is that it requires discipline and hard work.

I’m not saying there isn’t a better way to search – I am a fan of Thomas Edison’s belief that “There is always a better way.”

However, I believe that the better way, specifically when it comes to information retrieval, involves discipline and the hard work of people using critical thought in the search process – not short-cutting or completely removing it from the equation.

And I am not alone.

There is already considerable work being done to create new kinds of search systems that depend on continuous human control of the search process. It’s called Human-Computer Information Retrieval (HCIR) - which is the study of information retrieval techniques that bring human intelligence into the search process.

Truly intelligent search systems should not involve limiting or removing human thought, analysis, and influence from the search process – in fact, they should and can involve and encourage user influence.

When you break it down, the information retrieval process has 2 basic parts:

1. The user enters a query, which is a formal statement of their information need
2. The search engine returns results

The key, in my opinion, is that the search engine should return results in a ”Is this what you were looking for?” manner and allow you to intelligently refine your results, as opposed to a “This is what you were looking for” manner.

There’s a BIG difference.

The former begs for user influence and input, the latter does not – it makes the assumption that it found what you wanted

The bottom line is that no matter what you are using to search for information, only you know what you’re looking for and therefore judge the relevance of the search results returned.

Intelligent search isn’t easy, because you actually have to think before and after hitting the search button.

The Intelligent Search Process

As I have written before, searching should not be a once-and-done affair – there is no mythical “once search to find them all.”

Searching is ideally an iterative process that requires intelligent user input.

Here is an example of an intelligent, iterative search process applied to sourcing talent:

1. Analyzing, understanding, and interpreting job opening/position requirements
2. Taking that understanding and intelligently selecting titles, skills, technologies, companies, responsibilities, terms, etc. to include (or purposefully exclude!) in a query employing appropriate Boolean operators and/or facets and query modifiers
3. Critically reviewing the results of the initial search to assess relevance as well as scanning the results for additional and alternate relevant search terms, phrases, and companies
4. Based upon the observed relevance of and intel gained from the search results, modifying the search string appropriately and running it again
5. Repeat steps 3 and 4 until an acceptably large volume of highly relevant results is achieved

Anyone can enter search terms and hit the “search” button, but not everyone can effectively and intelligently search.

Until you’ve witnessed intelligent and iterative search in action, you likely wouldn’t know the difference between “great” search results, “good” search results and “bad” search results.

It’s as dramatic as the difference between and experienced professional offshore fisher, a recreational fisher, and someone going offshore fishing for the first time.

The ocean holds the same fish for everyone fishing it. While a first-time or recreational fisher can get lucky every once in a while, only a person who really knows what they’re doing can get “lucky” on a consistent basis and catch the fish  the recreational fisher only dreams of catching.

Final Thoughts

The ability to enter in some search terms and click the “search” button doesn’t convey any supernatural search ability, but it does certainly make people feel like they are good at searching, because unless you mistype something, everyone’s a winner.

Ultimately, search engines of all types retrieve information, but information requires analysis, and only humans can analyze and interpret for relevance.

Eiji Toyoda, the former President of Toyota Motor Corp., has observed that “Society has reached the point where one can push a button and immediately be deluged with…information. This is all very convenient, of course, but if one is not careful there is a danger of losing the ability to think.”

Critical thinking is perhaps the most important skill a knowledge worker can possess.

The reason why so many people stink at search is because most people simply don’t think before or after they search, and they place too much trust in the machine.

Additionally, the quality of the search terms/info entered directly affects the quality of the results. “Garbage in = garbage out” certainly applies here. And effective searching is rarely a “once and done” affair – the ability to critically evaluate search results for relevance and successively refine the search criteria to increase relevance is the key to true “intelligent search.”

“Black box” matching algorithms can be wonders of technology and engineering, but they pose significant problems in that searchers have absolutely no insight as to why they return the results they do, and in many cases, the engineers creating these semantic/NLP matching algorithms assume they know what their users are looking for better than the users themselves. I’m sorry if I am the only person offended by such an assumption.

Okay, I’m not sorry.

I love technology, and I use and have used some of the best matching technology available, but also I know it’s not a good idea to try to limit or remove intelligent critical thinking from the search process and completely replace it with matching algorithms.

The term human–computer information retrieval was coined by Gary Marchionini whose main thesis is that “HCIR aims to empower people to explore large-scale information bases but demands that people also take responsibility for this control by expending cognitive and physical energy.” (emphasis mine)

For those who simply want information systems to magically provide them with the most relevant results at the click of a button, you should take special note of the fact that experts in the field of HCIR do not believe that people should step out of the information retrieval process and let semantic search/NLP algorithms/AI be solely responsible for the search process.

If you want to get better search results, use the latest technologies, but don’t put too much trust in the machine.

Instead, put some skin in the game, take responsibility for the search process, and expend some cognitive energy critically thinking through not only your search input, but also the results for relevance.

“In the age of information sciences, the most valuable asset is knowledge, which is a creation of human imagination and creativity. We were among the last to comprehend this truth and we will be paying for this oversight for many years to come.” — Mikhail Gorbachev, 1990

Strictly For the Search Geeks

Check out this HCIR Challenge, and at least read the  introduction which compares and contrasts precision vs. recall, and references iterative query refinement.

I’ve read a few articles recently talking about IBM’s Watson and how the technology they developed may be the harbinger of unemployment for people in many professions.

Here’s one from Fortune magazine, asking if IBM’s Watson will put your job in jeopardy.

Here’s another suggesting that those who train others in Internet, social media, ATS, and resume database sourcing techniques and strategies will be eventually eliminated by semantic search solutions.

Watson Winning at Jeopardy isn’t Surprising

First, let’s first recognize that it’s an apples to oranges comparison between Jeopardy and sourcing/recruiting.

The ability to quickly research and answer trivia questions (or provide questions for the answers, in the case of Jeopardy) is a far cry from having to boil a hiring need (skills, capabilities, and specific responsibilities in specific industries and environments) down to a series of queries to mine flawed and incomplete human capital data (i.e., resumes and social media profiles) in order to return people who have a high probability of not only being qualified for the position, but also interested in the job (i.e. “recruitable”).

With trivia, all of the facts and information are readily accessible, completed and identifiable on the Internet, or in the case of Watson, saved on a multi-TB hard drive array.

It’s not really shocking that a highly specialized $900,000,000 to $1,800,000,000 (estimated 3 year cost of developing Watson) NLP (Natural Language Processing) computer can sort through 200 million pages of structured and unstructured content, including the full text of Wikipedia, to retrieve information faster than a human relying on memory alone.

Why is anyone surprised that Watson spanked people?

I wasn’t.

However, I can’t pass up the opportunity to point out that Watson did make mistakes – here’s one example in which Watson thought the answer was “Who is Picasso?” when the correct answer was “What is modern art?”

Who knew that the err is human, as well as inhuman?

The Unique Challenge of Human Capital Data

Unlike finding the answers to trivia questions, when it comes to finding and identifying qualified and talented people based on their resumes and social media profiles and updates, the information is often incomplete, and in many cases, critical bits of identifying data are simply not present.

For example, how do you find someone with Spring MVC experience when many people don’t mention it on their resume, nor on LinkedIn, Twitter, blogs, etc.?

I recently gave the world a tiny glimpse into the Dark Matter of LinkedIn – direct keyword, title, and even concept/relational search methods, used by humans or algorithms, can only retrieve results based on existing text.

Quite simply – if the text isn’t there to be retrieved or analyzed, a semantic search/NLP algorithm can’t do anything with it.

Good sourcers really do “read between the lines” of both the job description and requirements as well as the human capital data they are searching for and analyzing.

There is much more to high-level sourcing than keyword and title search/match.

There have been semantic solutions on the market for quite some time that can do keyword, title and concept matching reasonably well (as well as some that claim to, but don’t). The issue with those solutions that no one seems to (or wants to) realize is that they have limitations – they find some matches, exclude some, and bury others.

The real question is who, how, and why are some matches found and ranked highly, while others are excluded, and others ranked lowly but actually represent the best talent?

What Do I Know?

I have hands-on, practical experience (read: trying to find people to fill real jobs) with many of the “top shelf” semantic search applications out there, specifically designed for human capital data, so when I write or speak on the matter of semantic search, I’m not throwing around empty opinions.

I’ve seen what these solutions can do, and I’ve also directly experienced their limitations, including what they simply can’t do.

Unlike many people who write on the subject of semantic search, I have to personally find people and help others find and recruit talented, qualified candidates with highly specialized skills and experience within 24-48 hours of receiving a client request on a daily basis. If semantic search solutions (including the one I have access to) could speed up that process and help me find more and better candidates faster – trust me, I would use them!

I’ve witnessed a sourcer with 9 months of total experience find better qualified matches (and faster) than a big-name semantic search solution in front of one of the senior technical managers responsible for developing the product. It was eye-opening and even somewhat confusing for them, to say the least.

I’ve also spoken with sourcing/recruiting managers at Fortune 500 companies who have evaluated leading semantic search solutions and they passed on purchasing them because the solutions did not find more and better results faster than their sourcing/recruiting team.

Ultimately, it’s not about humans vs. technology – it’s about results.

The Solution is Part of the Problem

I’ve found the creators of semantic search products don’t seem to like it nor do they seem to really listen when you point out the flaws and limitations of their creations – and I’ve had exchanges with people who hold patents in this space.

I’ve also gotten the sense from talking with semantic search solutions providers that some of these folks believe that sourcers, recruiters and HR professionals don’t (and/or can’t!) really understand semantic search and more complex information retrieval strategies.

To their credit, if their perception (based on experience or otherwise) is that recruiters and HR professionals struggle with Boolean search – the most basic query “language” – why wouldn’t they assume that the average recruiter could not possibly understand and appreciate what’s going on “under the hood” of semantic search solutions?

However, it is folly to apply that stereotype to all sourcing, recruiting and HR professionals – there are plenty of us who actually know more about the specific challenges posed by human capital data and the practical needs and concerns of recruiting organizations than the people who are developing the solutions that we are supposed to intrinsically trust to automatically find the best people available.

I don’t hate – I appreciate semantic search. I simply want these solutions to live up to their hype. Semantic search vendors – listen to your current and potential customers – they just want your product to work better!

I’d like to extend an open invitation to any semantic search/NLP vendor – I will happily evaluate your product and make suggestions for improvements…for free! If you’re very confident in your solution, I’ll also write a review online. If you’d rather not have your product exposed publicly, I can also evaluate products privately. I really do want to accelerate the efficacy of semantic search applications for sourcing and recruiting!

I also want to educate others who may be buying these kinds of solutions so they are more knowledgeable and informed as to the pros and cons, capabilities and limitations of these solutions, and not sold simply on impressive sales pitches, techno-speak and “see how many results?” demonstrations. If you’re a potential customer of semantic search solutions, please be sure to include your best sourcers/recruiters in the evaluation process – if the only people who are evaluating a semantic search solution are HR, management, and procurement professionals who don’t actually search for top talent on a daily basis and won’t be using the proposed solution, you can easily be sold on a product that doesn’t actually work as well as you might think based on the sales presentation.

If you’re looking to buy a new flat screen TV or car, anyone can read reviews online, test drive them and compare them to competing products.  I find it interesting (and telling!) that you can’t do the same thing when it comes to recruiting and HR software.

When you buy a house – you get it inspected by a specialized professional before you buy it so you really know what you’re getting beneath the surface. Before you buy a semantic search solution, you should have it evaluated by a person who specializes in human capital information retrieval (who is also ideally a neutral third party!).

Are you attending or thinking about attending SourceCon 2010 in San Diego in March?

I am going to be the keynote speaker for the event, and I will be presenting on Artificial Intelligence vs. Human Cognition when it comes to sourcing and matching resumes.

If you’re curious to know what kinds of things I’ll be addressing during the session, here is a sneak peek:

1. The intrinsic and often overlooked challenges associated with sourcing resumes
2. What artificially intelligent semantic search and match applications claim to do and how they actually work
3. The limits of artificial intelligence
4. What people can do that semantic search applications cannot
5. The 5 levels of semantic search
6. The 5 levels of secondary/e-sourcing
7. What I believe would be the ideal candidate sourcing/talent identification solution

If you’ve ever wondered about the fantastic claims that some of the semantic search application vendors on the market make as to how their solution can mimic a senior recruiter when finding candidates, then you will be very interested in hearing what I have to say about the reality of what they can do.

If you’re a sourcer and you’re concerned that your role/position might eventually be replaced by sourcing software, you will be encouraged by my analysis and supporting arguments that explain why the abilities of creative and investigative sourcers will always be in demand – tomorrow and 50 years from now.

I hope you will be able to attend SourceCon 2010 – I know I’m looking forward to it!

If you’re unable to attend, the good news is that the presentations will likely be streamed. Additionally, I plan on posting my expanded slide deck, including all talking points – so you won’t be stuck staring at some pretty pictures wondering what the heck I talked about.