Patent rankings: The numbers game v. patent quality
Posted by J Matthew Buchanan at April 20, 2005 10:51 AM
Every year, several lists are published that purport to rank various types of organizations on some type of innovation scale. The lists provide great headline material (“Top ten universities announced;” “Most innovative companies named,” etc.), but not much else. It seems there’s a list for every type of entity (see, for example, these lists for businesses, universities, states, law firms) and the media, including the popular media, gobbles them up, year, after year, after year.
Unfortunately, these lists boil down to nothing more than a numbers game. Here’s the basic formula for all numbers-based lists:
1. Count the number of patents granted to an organization in a given year
2. Rank the organizations in a particular group (universities, companies, law firms, etc) based on the numbers.
Many people blindly believe that, in the patent world, more is better. This includes many business people (ever look at an annual report of a publicly-traded innovation-based company? Some business folks are very proud of their “patent numbers”). Unfortunately, the numbers game is contributing to the quality problem. Many believe that organizations playing the numbers game are primarily responsible for the perceived drop in patent quality of late.
An interesting aside: How’s this for irony…in a recent call for reform of the US system, IBM Vice President of Technology and Strategy Irving Wladawsky-Berger, whose company sits atop the USPTO patent rankings year after year, stated flatly that “any idiot can get a patent for something that should never be granted a patent.”
Numbers-based rankings do nothing but feed the numbers game monster (well, ok, there is the “catchy headline” thing…).
Here’s a seemingly simple rethink: If patents are to be used as a measure of innovation, shouldn’t patent quality be the primary focus?
I’ve been trying to rethink this one for awhile. Turns out, its not so simple. How do you rate patent quality? Does it require a detailed analysis of each and every claim from each and every patent? To be accurate, does it require perfect knowledge of the prior art? Does it require a full legal analysis of validity and scope?
My rethinking led me to conclude that the determination of a quality score is a difficult task. No matter how its determined, though, a quality score should be a reflection of the scope of a claim or claims relative to the invention and the applicable prior art. Considering this, its easy to see why the simple two-step formula of the numbers game is so popular. Its easy! Determining quality is hard (and likely very expensive)!
I don’t know what the solution is. But, I do believe that we, as patent professionals, need to downplay the importance of the numbers game and stress the importance of quality over quantity. So, the next time you read one of those catchy headlines, take it with a single grain of salt.
Another aside: I am aware of at least one system that purports to use a quality-based analysis: the Intellectual Property Quotient (IPQ) system of PatentRatings LLC. I have no experience with the IPQ system and would love to hear from anyone who does.
J. Matthew Buchanan 
Stephen M. Nipper 
Douglas Sorocco 
Trackback
You can ping this entry by using http://www.phlexability.com/mt/mt-tb.cgi/40 .
Excellent post over at rethink(ip). They make the extremely valid point that having a lot of patents does not mean you have quality patents. For the analysis that I do, I try to go at least one layer below the raw numbers (e.g., best companies at get...
Trackbacked from Numbers vs. quality on Patent Chronicles.
Comments
Douglas Sorocco Says:
April 20, 2005 12:26 PM
Patent Ratings LLC is highly reputable and has a wonderful system in place for rating patent quality.
Of course, Dunlap Codding & Rogers was ranked 2nd in 2004 for mechanical patents - so I am a bit biased. =)
If you look at the results from 2004, however, it was the smaller prosecution boutique firms who were wiping up the floor in the "quality ratings". The authors surmised that the smaller boutiques could devote the time to really crafting quality applications while the bigger firms saw them as loss leaders for the more lucrative litigation practice. Often they are given to lower level associates who have no clue how to prep quality applications and are given little or no training or supervision.
The 2004 article from Patent Ratings is available at the PHOSITA blog.
Pete Medley Says:
April 20, 2005 05:52 PM
R. POLK WAGNER and GIDEON PARCHOMOVSKY have an interesting article here
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=582201
that suggests that the best metric of a patent lies in the number of related patents and not in quality of individual patents.
From the abstract: "We find that for patents, the whole is greater than the sum of its parts: the true value of patents lies not in their individual worth, but in their aggregation into a collection of related patents, a patent portfolio."
Steve Says:
April 21, 2005 11:41 AM
A couple of thoughts:
Another mechanical way to rank patent quality is by number of cites as prior art. Cites in patents by common assignees or inventors are discounted by some amount to prevent gaming the ratings system.
I'm not thoroughly famliar with the Patent Ratings system, but it appears to relate the quality of their rating with the maintenance life of the patent. However, the patent life often depends on the rate of change in the technology, rather than patent quality. For example, one might expect a given food processing patent to have a longer commercial lifetime than a software patent, but the software patent might have higher commercial value during its short life.
Which brings up the question: how should we define quality? Should it be the commecial value, immunity from litigation, clear expression of the invention? I just started reading this interesting blog, so I might have missed an earlier discussion of quality.
Brandon Williams Says:
April 23, 2005 11:56 AM
Since the patent system was created to protect commercial rights, shouldn't the most important criteria be "license-ability" of the patent? The quality of a patent will ultimately be tested and validated through licensing agreements and the IP commercializtion market. It is always when money is involved that people care to test the validity and value of the IP and its protection. I am just in the business of monetizing IP, so maybe my view is slanted.
John Koepke Says:
December 27, 2006 03:09 PM
One must understand the context when assessing patent "quality". At a minimum, patent quality can be used at to describe legal quality, underlying technology quality, or "license-ability".
Assessing legal quality requires one to consult an attorney.
Assessing technology quality involves understanding how interested are the most relevant technologists/scientists in the technology. Bibliometrics (the science of measuring the quality and strength of texts and information)is typically utilized here.
Legal quality and technology quality need not be correlated. For example, a seminal invention (high technology quality) can be drafted as a patent very poorly (low legal quality).
Assessing "license-ability" typically involves some understanding of the first two as well as an understanding of the relavancy to current and future products/markets and their associated revenues.
"Quality" can be assessed but depends upon context.