@needmonero90 is this the plot you asked for yesterday?

Block propagation time as a function of time (colored by block size, yellow indicating 50+ kB)

Also, this is ... interesting

I wanted your chart, the distribution, to be timeseriesed over 1 month chunks or so

Also

I'm playing antichamber in Discord atm, Justin is watching, if you want to come :D

@n3ptune helped me figure out how to extract some more data around the miner transactions : )

Cool, I might hop on discord in a sec. Might keep you muted, nothing personal.

Im guessing the lowest one is solo miners.. maybe jtgrassie can shed some light on which line corresponds to which mining software

lowest lines will be solo miners (just tx pub key in extra, 32 bytes)

nodejs-pool and monero-pool use an extra 17 bytes, so 49 bytes.

of course, anyone could add other stuff in but that should be the usual.

(thats all for the later block heights anyway)

(of and of course those numbers I just gave are without the tag bytes)

so minimum will be 33 bytes (solo miner) and typical pool 51

atoc is the guy who was interested in matching, right? and he's not on right now. :\

needmonero90: i've never enjoyed antichamber, it reminds me too much of a nightmare

would it be 52 bytes? to account for the extra nonce length

The bottom line looks like it is below 33 bytes.. is someone burning coins, or maybe using a custom software to sign transactions without the transaction pub key?

almost certainly the latter, not the former, if those are the only two options

they have been mining blocks since the very beginning

there seem to be 2 main groups with sub-33 extra sizes

Hi suraeNoether , I am here now. I typically see the log on mattermost.

I hope you're feeling better. I saw that you were sick.

Yep. Good to catch you

Are these the updated changes?

So right now, testingSimulator.py appears to be passing all but one test. unfortunately that test is testing a function that is a composite function, and both of the functions it calls seem to be passing unit tests

Yeah

I see. Thanks

Hmm okay

Which means that my unit tests for either the make_coinbase function or make_txns function in Simulator is failing to catch an edge case

Do you have an idea of what that failing edge case may be?

Make_coinbase seems fine. Since make_txns just calls make_txn a bunch and make_coinbase seems fine, I think it's make_txn

I suspect that I am not computing my predictions for the tests correctly. namely, it appears as if the number of red edges being added isn't what I expect them to be based on the state of the system before they are added

Which also points to make_txn

As the problem child

At any rate, I am working on that today. As soon as that problem is fixed, I have some ready to go code to push that will play the tracing game and output the results to a human readable file

I see. Also, do you work out of matching-mojo?

That's my most up-to-date presently

There are many branches and I'm not sure what each are for.

yes I noticed that

I may do another branch before I pull it back onto my main branch. But right now matching-mojojo is what I am working on

Ok cool. I will also see if I can run those tests

and see what you're seeing

probably a little later though, but i will keep you updated on it

what's exciting about this is that we will be able to upload a couple of simulated ledgers to GitHub with some ground truths, and then if anybody else comes along with new matching methods, they can test their method against these ledgers and generate an accurate confusion table. Sort of like unit tests

No problem... A brief warning though

Ah yes that would be cool

The way that I wrote my tests was that I have at least one basic test touching each function. Most functions have exactly four tests... One starting from a totally empty ledger, one starting from a simulated ledger, and one of each of the above that runs sample sizes and does repeated tests of the first two. I recommend uncommenting my unit test skip decorators for the test functions that you aren't

investigating immediately... This will save an enormous amount of time every time you execute testingSimulator

And I also recommend just uncommenting the skip decorators for all of the repetition tests

ok will do

this is also an open invitation to anyone who knows python: this final bug is ... bugging me. *sunglasses*

looks like people mining with the gui have 33 bytes in the tx extra, at least currently based on this guy reddit.com/r/Monero/comments/eq8h2s…_block_using_the_gui_thanks_randomx

Here's a zoomed in copy

Confirming - the bottom row of dots is 33 kB

Oh, let's get the hashrate of solo miners

Since we activated RandomX, six blocks have been produced by solo miners using the Core software

(or, technically, by software that mimics the Core software)

6/(2015615-1978433)*1000000000

161 kH/s by solo miners

That will have *huge* error bars and a left wall.

yep, that's an estimate from censored data

Yea, 6 is a hilariously small sample size

if we are curious, i can formulate a BLUE

but with 6

we may as well go blue ourselves

'censored'?

emailed u Isthmus

seems to imply nearly all blocks are pool mined

at least 6 implementations too

6 implementations with significant hashrate, 4 implementations with a little bit of hashrate, and a few solo mined blocks

are there any outliers flying high above those?

No outliers, here's the big picture:

didnt you mention something about 60kB tx extras?

oh 60 bytes of padding lol

Here's one with 99 bytes of padding - xmrchain.net/tx/6598c15d6557a0759f5…2171a0696c05524529d18dd0cedb884456c

it's an extra nonce with 64 bytes, where only 8 bytes are used

60 bytes is quite alright. Was the kB a repeated typo ?

Yea, I was task switching between block propagation (kB scale) and tx_extra (B scale)

that one is also 56 bytes of padding

Although there have been kB scale tx_extras for transactions IIRC

FWIW, a pool could just regenerate the coinbase output to differentiate between miners.

did you mean a different tx with 99?

Gets rid of the extra mangling.

use subaddress maybe? lol

IIRC mining to subaddresses required lots of changes to monerod so wasn't merged.

or yeah could just do different transaction pub keys for each miner

Could you just assign a different stating point in the nonce range for each miner?

I don't think it's needed though, pools already keep track of template to user.

Isthmus doesnt work for >1000 miners

Yes, you'd have to double check the reply's in the range though.

Otherwise I can connect to you 1000 times, and every time I find a block, ever miner of mine claims they found the same :)

Damn, I love git bisect. It turns a slow aggravating slog into a slow ok slog.

The nonce range is waayyyy bigger than that

the nonce itself is 4 bytes

thats why pools use extra nonce to differentiate workers

Ah yep, it is a bit tight

A fast RandomX miner with 5,000 H/s searching for a block for *20 minutes* could cover almost 2^23 nonces

So you could comfortably accommodate about 2^(32-23) ~ 512 miners

Too small for big pool

Although if this is the main reason that pools are doing all kinds of weird and trivially fingerprint-able stuff, let's just make the nonce range 8 bytes.

2^(64-23) would accommodate an astronomically large number of miners

> 1,000,000,000,000

I think the main reason is that it was coded that way at first and everyone just reused the same thing.

So all it takes is someone to write patches and PR them to whoever maintains the code now.

And then maybe a bit of arse kicking or public mocking to get pools ops to pull.

Once we extend the nonce range to 8-bytes, to accommodate 1000000000 workers per pool, is there any valid use case for unvalidated tx_extra in the miner transaction?

Since the value of the nonce itself proves to the pool who mined the block (if that matters)

Also, pool-mined blocks would then be indistinguishable from solo-mined blocks

I think it was hyc who pointed out it's always good to have a little wiggle room. That's why tx extra hasn't had hard scrutiny sofar

if the nonce was bigger it would still be fingerprintable, since each implementation would divide the nonce into different sizes for their workers

If the intent of extra is allowing unexpected uses, asking whether there's a valid use case for it is... not helpful.

I feel like the best route is to enforce TLV in the extra field, enforce type sorting, and roll out that update with best practice guidelines. That way when pools update their software, they can at the same time harmonize with other implementations.

Nonce range should never be divided up deterministically. Pick random starting points for the miners (and reassign any obvious collisions)

good point

(best privacy practice for both pool and solo miners)

jtgrassie: do you feel like patching existing pool code (I think there are 3) to avoid the extra nonce thing ? :)

could just pick a random starting point, then deterministically increment for each worker

Ah yea, I wrote it up in an old articlle

If you are a miner that is using custom/obscure software that mines with an unusual nonce space search pattern (e.g. down from the maximum, or up from the midway point), then you should switch to random sampling so that your blocks blend into your transaction trees and the overall blockchain. Note that for each new block, you only need to call random() once for the first nonce, then you can iterate

sequentially in either direction — this will entirely remove statistical links between your blocks’ nonces.

I like the Zcash idea of having a consensus-enforced fixed-size encrypted memo field is great to allow users to safely innovate applications without damaging privacy on the base layer

koe: explain details how will you split N bytes of nonce in non deterministic way between pool workers?

Unvalidated plaintext memo field is antithetical to fungibility

cohcho it can be deterministic. Randomly select the first miner's starting point, then for each other miner increment the starting point by 10mill or something

only works if nonce rises to 8 bytes

a typical miner will only do 1mill or less hashes per block

you will have lines with 10mill width at block_nonce/height chart

nope, since for each block the offset is random

there would be lines if each miner is always mining from the same starting point, but if the entire pool starts at a random point for each block then nonces will be randomly distributed

8_bytes_nonce % (2^32) will have 10mill width *

this way current nonce distribution will be transformed into 4 LSB

and tx_extra will be represented by 4 MSB

You can concatenate tx_extra to current nonce and plot it

I think that makes sense to me. Easy change without updating header format

It should be randomly distributed

Yeah, let's just randomly distribute across the whole range though. Not 4B + 4B or else I'll have two fields to fingerprint on (individual worker search algorithm based on nonce side, pool assignment algorithm based on the extra side)

does this sound right: in other words, random select a pool starting nonce, then for each miner increment that nonce by 10mill. For each miner, take the 4MSB of their starting nonce and put in tx_extra, and the 4LSB to put in their header nonce.

pool starting nonce: 8bytes

Currently 4 bytes of nonce is miner's space to iterate and whole tx_extra is for pool only

It's already 8 bytes (at least in monero-pool) plus 8 bytes reserved for special need

some pools write their domain name into tx_etxtra

^ Isthmus see if you can fingerprint the different extra length by running ASCII on some fields

ew. Marketers :S

@koe I think I get what you're describing, but I don't quite understand the benefit of splitting it into two fields.

"...see if you can fingerprint the different extra length by running ASCII on some fields" uh oh, I'll check that out

the benefit of splitting is no need to update the block header format

it's still version 1 for all I know

the final aim is to have completely random tx_extra represented as TLV and force (or ask somehow) pool workers to choose starting point randomly

aren't workers given a blob, which has the miner tx extra field smushed inside it?

so workers arent constructing the blog

Oh, I meant theoretical reason. I don't mind updates for improvements

blob

yes, it's hidden behind tree_root_hash, they can only change nonce

But they can iterate nonce in defferent ways

ah do pool workers run different software?

theoretical not that I can see

Might be easier to force pools to standardize by updating the block format. We see 12 implementations in use today, not sure how many will update if we soft standardize

(no one in case of soft)

hehe exactly

pools haven't even updated to random nonces, and we've known about that for over a year

I agree to update at least my own

But don't understand yet how to blend in the crowd in the best way

:- )

simple question: github.com/jtgrassie/monero-pool/blob/master/src/pool.c#L2326

monero-pool has tuple with 4 values: the last are rarely used and for special need to support proxy that will distribute work further

Should it be seet to zero or removed when not need (reduce tx_extra to 8 bytes) ?

be set*

the last 2 *

pools should store information about workers on their own servers, not on the blockchain

more memory (or starage in this case) allows programmer to think less about efficiency during memory allocation

also second element in tuple is being iterated one by one from zero

Is it better to generate it randomly?

Ah @koe that's the succinct heart of the matter!

Lets say pools get 4 bytes in the extra nonce to prevent worker overlap. To start a new block, randomly generate an extra nonce and assign to worker 1 (server side assignment). Increment by one bit for each additional worker. When sending out a job, either randomly assign the worker a header nonce to start mining from, or have the worker randomly

select their own starting nonce.

This is making my brain melt

I might be hallucinating, but I think a few patterns show up very often

That should be sufficient to make everything appear random, while limiting pools to just 4 bytes per block instead of using the extra field to store info related to their business.

Should pool check explicitely that 4 bytes random tx_extra are uniq or rely or random generator?

you're hallucinating lol

because in case of overlap two workers will do the same work

Are there any mapping functions with 4 bytes output that guarantee uniq output for each uniq input?

only do one random number for tx_extra, then increment it for each additonal worker. Worker 1 = 3145, worker 2 = 3146, etc. New job: worker 1 = 889281 worker2 = 889282. That's in the extra field

identity

except identitiy :)

Header nonce can be random without worry of overlap since the extra nonce is different for each worker

starting point should be generated after each new block only once at pool start?

yes

yes 1st or yes 2nd?

each new block

ok, that's easy

What's about workers?

They receive start_x, end_x in my case

workers receive blob which contains the extra nonce computed by pool and assigned to them; they EITHER: receive random header nonce to start mining from , OR generate their own random nonce to start from (up to implementer)

there is xmrig-proxy that do the following:

receive only block template with 4 bytes nonce and split it into 256 ranges ( 3 bytes each) between at most 256 workers

How workers of this software should start iterating?

hold up y'all

2004429

not hallucinating

tx_extra reads:

�PeL�F�`L`,4R��/?�SD���minexmr.com

Yes, I was meaning minexmr

You were right :)

there were 10 blocks in row today by this pool

whats the length?

let me find height

92 or 96 probably

62 *

it varies probably

62

But now that I'm looking at the decoded data they're only responsible for about 2/3 of those len(tx_extra)==62 blocks

cocho each miner random selects a starting point RAND(3 bytes). If they roll over, just start at 0 (within the 3 byte range they are allowed)

until the first repeat of nonce ?

and then go to sleep until next job or request new one (that's not supported yet)

should be no repeat since the 4th byte differentiates each worker

and 3 bytes is 16mill so IDK what super powered worker you have

I mean 3 bytes can iterated within few hours

by slow worker

in worst case

in practise it will not be

I just need correct general solution to cover all cases

I guess go to sleep, if acting within the limits of that implementation

what do they do right now? same thing

IN my case sleep, don't know about other software

probably repeat again

So why not the new output thing ? It saves memory, removes potential patterns in favour of encrypted info, so seems better, no ? What is the catch >

new output thing?

new output to pool's wallet with 0 length nonce in tx_extra vs the same output with different >0 length nonce?

Send every a template with a different output.

computation overhead is low?

every miner.

I'm not sure.

lower than set 16 bytes and rerun tree_root_hash?

Probably not. Is this a bottleneck somehow ?

might be more intense for pools to generate so many curve points

Or is this "if I get one extra cent profit, I'll cause somene else millions in losses because I'm a human" situation ?

if someone prefers the best efficiency with anonimity tradeoff they will care

What thing guarantee that every new output will not repeat previously generated?

What's the length of random data in new output?

it's an ECC curve point, so 32 bytes

random

bit less than 255 bits technically

Is it safe to generate 1st one randomly and the next by iterating one by one?

I suppose no :)

i think..

yes?

um

no probably not a great idea, since it can reveal the pool's address

And each new output will consume 32 bytes of precious OS entropy, right?

It's too expensive probably

ok the random space of a curve point is between 2^253 and 2^252

err random size, this isn't important. What's important is that it costs a couple curve operations to generate a one time output address.

also in that 4 elements tuple: 1st element 4 bytes generated once at pool start and will be constant until restart/crash

Is it better to replace them with random?

It's called "instance id" in code

probably better to be random, since if certain workers are better than others their instance ID will appear more often

1st element will be constant for all tx_extra generated by pool instance*

can probably just roll a random offset for each block, and add the real instance ID to that offset for sending out a job. That way you only have to store the random offset temporarily while the job is alive

can add the same offset to all chunks of 256 workers

or 255 workers I guess

256 get real koe

256 workers is different software; Chain of workers is pool (e.g. monero-pool with 4 elements tuple ( 4 bytes instance id, 4 bytes worker specific nonce, 4 bytes reserved, 4 bytes reserved) -> xmrig-proxy ( represent nonce as 2 elements tuple ( 1 byte worker specific offset, 3 bytes nonce for worker) -> worker that iterates within 3 bytes

(and monerod at the beginning)

It's 256 since there are 256 uniq offsets (0 including)

moneromooo: thing is with extra nonce, it's not just different miners. proxies also use the extra nonce. so whilst a pool could give each miner a separate coinbase output, it doesn't solve for pool proxies, or multiple workers.

If the last level in chain generate completely random tx and all previous propagates only the first level address Then it will work

but up the chain the pool needs to be able to recreate and validate

it will have overhead to generate tx and the last level should back propogate actually generated block template

and nothing down-chain should be able to set the output

which is why what you are suggesting is flawed

OK, I can guess what a proxy does based on that statement. Fair enough.

I'm also not quite clear on the goal - to stop using tx extra for miner txs?

why is this important?

mostly the goal is to make miners more uniform

since there are clearly 6+ separate groups of miners

but why?

same reason anything in Monero - indistinguishability, fungibility, reduced fingerprinting, and so on

but the very nature of tx extra is anyone can use it already. so unless you are talking about restricting tx extra wholesale, it seems a mistake to focus on just miner txs

That sounds a lot like a fallacy.

I think the point is to improve on the current situation

which has developed due to lack of common guidelines about how to implement the tx extra field. Everyone just does what feels right to them

moneromooo: but tx extra is freeform and anyone can put what they want in it.

miners are the focus since they have specific needs

but standard tx are also part of the story

IIRC there were arguments against validating tx extra.

hyc suggesting enforcing TLV format, which retains the open endedness while encouraging uniformity

perhaps moneromoo can recall the reason why we didn't want to do tx extra validation?

I seem to recall the conversation coming up when discussing payment ids.

Depends who. Me, because it's meant to be arbitrary.

Some others want it gone.

Some want it parsed (a bit).

I don't think parsing just a bit is good.

me neither

its either arbitrary or not

Though some kind of generic parsing so that unknown stuff may be skipped seems like a good idea.

Just structural though.

We'd have to do away with the 255 limit. Make it a varint.

I don't think there is a 255 limit from a consensus pov ?

There isn't.

it is howver used for a number of different puposes already. Trying to restrict it just from a miners pov seems misguided.

You're confusing things.

There is changing how pools do things and restricting the use of extra.

The above was about the first one. The second may or may not happen, independently.

the two are related

I think it's ok for the tx_extra field to permit opt-out privacy, but it would be better if it encouraged remaining private by default (in terms of implementation). Currently there are no restrictions, no guidelines, and so even implementers who DONT want opt-out privacy, are stuck with no reasonable path.

I hope you're not claiming that relatedness implies implication, right ?

no

I'll assume not. So yes, they're related.

which totally contradicts the Monero reason d'etre

as it is, tx extra is used for tx pub key, multisig, subaddresses, pools, merged mining implementations...

from my point of view, I agree with hyc that enforced (sorted) TLV would go a long way to privacy by default, without any real restraint on opt-out privacy

and along with sorted TLV should be explicit guidelines on how to implement the extra field to be the same/similar as other implementations

TLV is some sort of chunk based structure, right ? Like PNG ?

type-length-value, it's similar to how the field is already parsed

it's just, if you want to add something random you must prefix it with a type and length

and by sorted, I mean each type is sorted by value. Currently some implementations dont sort, but the main wallet does.. ??

sorry just sort the types, not the values..

the implementation guidelines would include a list of common/standard types. Public keys, for example

to verify the field, read it following TLV expectations (byte 0 is a type, byte 1 is a length, bytes 2-8 are a value, byte 9 is a type,...). If the last byte isn;t the last byte of a value, reject the field.

or, if the type's arent sorted in ascending order, reject the field

IIRC I sort based on likelihood of usefuless. So probaby txkey first :)

yeah

Though the wallet probably has to parse it all anyway, due to some stupid old bug.

and maybe reserve some types for core updates (idk if that's a good idea)

types can easily be larger than 1 byte, tag=255 then interpret next byte as an extension of the tag (by addition)

jtgrassie what's your perspective on sorted TLV?

"which is why what you are suggesting is flawed" <- Anyone with (sec_view, pub_spend) can verify distination of coinbase tx and create it. So if pool shares it then all the next levels will be able to create/verify uniq tx locally.

jtgrassie: Can you help me understand what is flawed exactly here or privately?

moneromooo: if pool doesn't support proxies then with each new block you will need 1 uniq tx per connected worker (e.g. 60e+3 connected to minexmr.com now , so 60e+3 per 120 seconds of uniq txs)

60k ??

Damn, decentralization is coming ? :D

yes, 60k workers ~10k miners

60k workers of ~10k miners

It's the worst case now probably for single real world pool

average miner has 6 workers?

moneromooo it's at least better than ASICs, since if a pool misbehaves then miners can jump to other pools. Each miner has individual decision making power

"and nothing down-chain should be able to set the output" <- This is required due to lack of ability to verify destination of tx submitted at intermediate level. But with (sec_view, pub_send) this ability will be added

(sec_view, pub_send) is called tracking key in cryptonote whitepaper and allows to track destination. It isn't obsolete info at the moment, right?

no that's accurate

koe: I have nothing against TLV, its not much different to how it is currently.

cohcho: a proxy does not have the pools secret keys

(sec_view, pub_send) != (sec_view, sec_send), (sec_view, pub_send) can be sent to proxy instead of address

Do you use see any security problems in it?

you see*

a pool may not wish to disclose their private view key

At least one shared

It's required only for those who supports proxies

Others can't keep it private

And proxies don't help in decetralization

thats a naive statement

they neither help nor hinder

With an assumption that pool will share that pair of keys There is no problem, right?

forcing pools to disclose private key is going to be a non-starter, IMO

and tx extra moved to a strict TLV scheme, why not simply allow a 4 byte type field pools can use?

^and if tx

I just wanted to know that there are no security problems in that approach with uniq tx without tx_extra

I don't know percent of pool users that uses xmr-node-proxy instead of xmrig-proxy

If it's almost zero then it's not a problem

both of those proxies have plenty of users

pools encourage using a proxy for miners with lots of machines as it takes load off the pool server.

256 reduction factor from xmrig-proxy is acceptable too

the "security problems" is a design which forces disclosure of a secret key

jtgrassie: are you specifically not adding "view" or it's assumed by default?

the fact its a view has no bearing. Its a private key that enables disclosure of private data.

I would be happy if pool shared it's tracking key so that I can recheck all found blocks

It's ok for public pool but not for private

whilst arguably a pool has nothing to hide, it feels like a flawed assumption

is there any reason for a pool to make anything public at all? miners can verify their payout based on hashrate vs global hashrate

and if tx extra is going to remain free to use (even if using TLV), there is no need for a change.

with view_key it will precise check not probability based

the question is if any miners care

since miners already mine, despite no view key

seeing what payments a pool recieves does not enable one to determine they have been paid correctly for thier work.

true^

to know that you need to know every other miners work submitted.

nothing shared < pool's blocks can be tracked < pool's submitted shares can be tracked < ...

It's getting closer to ideal with amount of shared info

so you are advocating banning of using tx extra for anything non-standard?

Without huge proxies and tx_extra used for nonce I'll be less distinguishable among other miners.

which then prevents people using it for things like merge-mining or HTLC or anything else?

If there are other ways to reach the goal Let's discuss it

what I'm saying is, you are trying to suggest a forced ban of pools using tx extra, which means you are saying nobody can use tx extra for anything other than monero defined uses.

This argument looks solid, previous one about problem with sharing sec_view wasn't.

I agree to not implement that approach with uniq tx unless the last problem exists.

forcing pools to disclose private view key is a non-starter IMO. That's an unfair enforced policy.

and if the goal is reduced information disclosure, you've achieved the exact opposite.

All pools (except only one) pastes height of their blocks, so fingerprint their coinbase tx via side channel

With published sec_view_key it will be fingerprinted but explicitely on blockchain

I don't know what is worse