Also, large cache can actually diminish performance. The larger the cache, the longer is spent examining the TAG something-or-other, which is the lookup table for the cache. Larger cache means either a larger table, or larger cache segments, which means that swapping data into and out of the cache takes longer.

Either way, there's a point at which the cache lookup time becomes longer than a simple access time.

Performance vs. cache size is a steep curve which plateaus rather steeply--or whatever the opposite of steep would be in this case. Perhaps, plateaus sharply.