Want rarity ranking directly inside of OpenSea? Download our Extension:

See in chrome web store
Frosty.so

How our Rarity Scanner is so Fast

12/29/2021Nikita Jerschow

Arguably the most important thing in the NFT space right now is access to information, specifically access to information before anyone else has it.

That's why we've spent the last few days investing in our backend infrastructure, to make sure we have the fastest reveals on the market.

In order to understand how our rarity scanner works, and why it's faster - we first need to understand what rarity is, and what makes most rarity reveals so slow.

What is Rarity?

Many NFTs that follow the ERC-721 Standard have what are called properties. These properties usually define how the NFT looks, and sometimes affect how the NFT can be used (for example in a game).

As a result, properties that show up in fewer tokens have become more rare, while common properties have become less rare.

As the NFT community developed, different strategies were developed to assign rarity scores to NFTs in a logical way, based on its properties. Where tokens with rare properties are ranked higher than those without. As a result, rare NFTs have become highly recognized and sought after in the NFT community.

How do we get Rarity?

Because rarity is based on how often a tokens properties appear in the collection, the only way to accurately calculate rarity is to download all of the properties of every NFT in a collection. Once you have them all, you can run standard rarity calculations which count up the number of times certain properties appear in the collection. Once you have the collection-wide property info, you can assign each NFT a rarity score, where higher is better.

Here you can start to see one of the main bottlenecks when it comes to calculating rarity. Collections will often times have upwards of 10,000 tokens in them, and downloading several thousand metadata files is painfully slow, even on a fast internet connection.

In a space with such high rewards for knowing rarity early, there hasn't been any standardization when it comes to calculating rarity. What that means for you is that it's still the wild west, and finding ways to get rarity seconds before anyone else is the difference between flipping for 1 ETH, or flipping for no ETH.

So how is Chilly tools faster?

Before beefing up our infrastructure, we were getting rarity just like every other tool on the market. One centralized server which handles everything:

  • Property downloading
  • Rarity calculation
  • OpenSea events fetching
  • Serving rarity results via frontend

This system does OK, it certainly does better than OpenSea, or finding rarity manually, but it's far from the limit. The main bottleneck is still downloading the properties.

Since NFTs are a global game, the closer you can get to the data source, the faster your download is. Our server was in the US, which meant that if an NFT collection goes live in Europe, so much more time is spent waiting for properties than if, say, the NFT went live in Los Angeles.

This is certainly true for web2 infrastructure, and even more so for web3 infra. If an NFT is uploaded to IPFS (~50% of collections are doing this, which is the right way in our opinion), the consequences of being far from the source are magnified by the nature of a peer-to-peer network.

So what's the solution?

Simple: Spin up servers around the world, split the downloading work between them, and let the far-away servers handle the data processing and calculation.

We spun up 6 servers globally (to start). Now if we want to get rarity on an EU-based NFT, our closest servers will collaborate & split the downloading effort between them, the result? A >5x speed-up for reveals.

Here's what our reveal process looked like before: Reveal scanner before update vs. after: Reveal scanner after update

We're still far from where we want to be reveal time wise, but we're improving daily. Check out our Discord for a live estimate of our reveal times.

-Nikita


An investment in knowledge pays the best interest. Subscribe today to stay in the loop on the fastest ways to build.