Add proof batch verification API

[georgepisaltu]

This PR adds the API for verifying proofs in batches. The purpose of this is to perform the verification of N proofs in less time than it would take to verify each proof independently.

For this purpose, any of the 2 batch verification functions added (batch_validate or batch_is_valid) will suffice.

There is also a new API - batch_step - to create a batch verification step. At this moment, this requires all of the components normally used in a single proof verification: the proof itself, the ring root, the context of the proof and the message.

[davxy]

I don't fully understand the API design choices, particularly:

• The need for a new associated type
• The batch_step

IMO we need something like the following (members should not change in the context of one batch validation call)

struct BatchItem<T> { proof: T::Proof, message: &[u8], context: &[u8] }

GenerateVerifiable {
    ...
    fn validate_batch(batch: impl IntoIterator<Item = BatchItem>, members: &Self::Members);
}

[georgepisaltu]

The new associated type is based on the discussion around a new cryptographic primitive to optimize the batch verification. In this sense, I understood that the step would be an intermediate representation of a proof along with other auxiliary components. Jeff said these steps don't have to all be from the same ring; they could all belong to different rings and use different ring roots for their proofs. This is the reason for the associated type (we don't know what this step looks like right now) and the primitive we want is more powerful than your suggested API would allow.

[georgepisaltu]

Also I want to add that the API presented in this PR is intended as an objective for the batch verification primitive which will be developed for ring-vrf and which promises significant improvements in terms of compute time for many proofs. We don't just need a function to verify multiple proofs in the same way we do it today, we need the performance improvements, otherwise this would have been a simple for loop in our downstream code.

[davxy]

@georgepisaltu I understand what you mean about batching :-) You'll likely need to call into ark-vrf, so I'll need to expose batching support there.

That said, I'm not sure batching across different rings can be done in a single call. The interface I proposed is meant for batching within a single ring commitment (aka the “root” here). I'll check with Jeff about multi-root batching (and here I mean one batch call for proof spanning multiple ring commitments); I don't see how this could be possible under the currently used ring proof scheme. But maybe this is something I'm not aware of.

Perhaps you're (he's) actually talking about delayed batch verification? For example, buffering signatures for multiple roots potentially across several calls (e.g. saving the signatures relative to one root in the state - i.e. sometging like Map<RingRoot, Vec<Signature>>), then verifying all buffered signatures at once on some condition - i.e. for each key in the map perform a batch verification. That approach could make sense but this is eventually not required in this interface and shoud be implemented in the pallet

cc @burdges what the api exposed by ring proof looks like? Is there some code or a spec draft to look at? Changing Verifiable trait api without this info may end up being not the best strategy. Also, what about this "cross commitment batching"?

[burdges]

Alright, a powerful batch verificaiton interface usually looks like this:
https://docs.rs/schnorrkel/latest/schnorrkel/struct.PreparedBatch.html

You need everything together simultaneously, with which you create the batch proof, and you place the batch proof into the block, instead of the individual proofs. This allows the batching to save blockspace. It changes what a block is though.

In other words, a blockchain could support flexible "transformation" or "compliation" steps, not unlike how we transform the parablock for the RC by attaching the PoV data, except crypto batching would come earlier, before even being shared with other collators. There are various other sorts of steps like this one could explore, like steps that "compile" blocks for multi-threaded execution. In this, you need transformed crypto to still be bound by whatever the user did, which require care.

We do not have some flexible approach like that of course. Instead, we have this ridgid notion of extrensics going into blocks, and then the block being "done". We've plenty of block space though, so we care little about the tx being 200 bytes bigger or saller.

If we focus upon CPU time, and ignore blockspace, then we could do batch verification from the original tx sent by the users, without doing any preprocessing, and simply accumulate the proofs behind the scense. This is much more friendly to substrate's limitations.
https://gist.github.com/burdges/079d24dba55e5033117d8a3b7f26ca4f

We're not done yet though, because batch verification remains a computation on the whole block, while substrate extrinsics can only safely do computation upon one transaction, almsot exactly the same limitations as smart contracts. As a rule, blockchain projects alter their computation by writing into storage, but this winds up pretty wasteful, in terms of excess hashing and even PoV size for us.

We do only have one thread though, so we could probably just use static muts and unsafe code, like this in my gist:

static mut WEB3SUM_RVRF_DEBTS: Option<Web3SumPlonkishAcc> = PlonkishAcc::nonbatched();

In theory, this should be None when run from the memepool, and Some(..) when run inside a block, provided the right call get made in on_initialize, and it'll be secre if the right call gets made in on_finalize.

tl;dr. At its simplest, the interface is merely a "start batching" call in on_initialize and a "check & close batch" call in on_finalize. It could hide the accumulation type in unsafe code that requires single threaded execution, instead of exposing the accumulation type.

[davxy]

@burdges we need punctual (and short - like yes/no) answer to this question:

"the ring proof batching primitive provided by the low level w3f batching scheme allows to batch across multiple ring commitments?"

that was my question.

What you're talking about makes sense in the context of the pallet. Accumulation of batches should be done in the pallet logic (pallet has storage and on_initialize/finalize hooks) .

This is a trait interfacing over the low level crypto primitive. Doesn't have any knowledge of blocks, storage, on_initialize, etc

[gui1117]

What you're talking about makes sense in the context of the pallet. Accumulation of batches should be done in the pallet logic (pallet has storage and on_initialize/finalize hooks) .

This is a trait interfacing over the low level crypto primitive. Doesn't have any knowledge of blocks, storage, on_initialize, etc

We need to have the concept of batch verification available to us in the trait, otherwise we just can't use the trait and there is no point for us in this trait.

What I see is that we want to call is_valid in the validation function in the transaction pool. And we call is_valid_defer_in_batch(..) -> SomeArtefacts during the execution of the transaction, then at the end of the block execution we call defered_batch_valid(Vec<SomeArtefacts>) -> bool. So ensure our block is correct. (in substrate we would call it in the confusingly named on_finalize hook).

If it is more handy we can also do is_valid_defer_in_batch(..., artefacts: &mut SomeAccumulatedInPlaceArtefacts), defered_batch_valid(SomeAccumulatedInPlaceArtefacts) -> bool (and potentially better for memory footprint).

I agree there plenty more we can do with block compilation. But in our usecase we can just have different code-path between transaction validation and transaction execution and it will work 100% for our specific usage.

EDIT: in the snippet sent by Jeff, the new associated type SomeAccumulatedInPlaceArtefacts would be this PlonkishInner.

[davxy]

We need to have the concept of batch verification available to us in the trait, otherwise we just can't use the trait and there is no point for us in this trait.

Maybe I was not super clear.

I'm not saying that you don't need an interface for batching here. You need it!

What I'm saying is:

1. Accumulation of batch entries should not be exposed here. Is the pallet that accumulates proofs and is in charge of calling the batch_verify exposed by this trait (that merely forwards the batch for verification to the lower level crypto)
2. The verify batch call should be over the same ring root if the low level primitive does't allow batch verification over multiple roots in one shot. That is why my api proposal takes a vector of proofs and the ring root as a separate parameter (ie the root is not repeated for each proof - as it is the same value if the lower level crypto doesn't support batching over multiple roots, which is what I suspect)

[gui1117]

yes I read too fast.

1. Accumulation of batch entries should not be exposed here. Is the pallet that accumulates proofs and is in charge of calling the batch_verify exposed by this trait

Yes it can be also good, but it seems having an accumulated batch type could be better for memory usage than having to keep all the proof until the final verification. But both solution are good to us, as long as it cuts the verification time, we can afford the memory footprint without issue I think.

2. The verify batch call should be over the same ring root if the low level primitive does't allow batch verification over multiple roots in one shot

I don't know the answer myself but in the past I got this answer:

And after Jeff explained a bit about the implementation, it is in the Crypto implementation matrix channel

[davxy]

@gui1117 Let's wait for @burdges' response, but I don't see how this could work with the current ring-proof scheme.
But again, a yes/no response is preferred (and possibly see a spec) :-)

Otherwise you're just going to repeat the same RingRoot for each BatchItem. Which is sub-optimal

[burdges]

"the ring proof batching primitive provided by the low level w3f batching scheme allows to batch across multiple ring commitments?"

Yes, I wrote up https://gist.github.com/burdges/079d24dba55e5033117d8a3b7f26ca4f to batch across really any pairing-based-plonk-like protocol. If I added another lengths: Vec<(u16,u16)> then would not even care if you prove the same type of statement, but I made lhs_size and rhs_size constants so only the same type of proof works. Also the g2s being constant means you can only use one trusted setup here.

There are likely further optimisations based upon specific extra assumptions like the same ring commitment, and their proof size savings could be interesting, depending upon the current proof size, but..

Our immediate problem is simply the crazy CPU time of doing pairings, which can be addresses by simply batching the pairings together. Also, the MSMs involved in the verifier are like 5 points each on one or both sides, so saving their CPU time seems much less important, especially since my gist batches those MSMs into one.

[davxy]

moved the discussion to the crypto chan