03:08:11 <Isthmus> https://usercontent.irccloud-cdn.com/file/hRJWQdEK/image.png
03:08:26 <Isthmus> @needmonero90  is this the plot you asked for yesterday?
03:09:02 <Isthmus> Block propagation time as a function of time (colored by block size, yellow indicating 50+ kB)
03:09:58 <Isthmus> Also, this is ... interesting
03:10:00 <Isthmus> https://usercontent.irccloud-cdn.com/file/g2Ms1LsZ/image.png
03:10:00 <needmonero90> I wanted your chart, the distribution, to be timeseriesed over 1 month chunks or so
03:10:01 <needmonero90> Also
03:10:21 <needmonero90> I'm playing antichamber in Discord atm, Justin is watching, if you want to come :D
03:10:40 <Isthmus> @n3ptune helped me figure out how to extract some more data around the miner transactions : )
03:11:14 <Isthmus> Cool, I might hop on discord in a sec. Might keep you muted, nothing personal.
03:11:57 * Isthmus notes that the second plot is the length of the tx_extra field within each block's coinbase transaction
03:31:03 <koe> Im guessing the lowest one is solo miners.. maybe jtgrassie can shed some light on which line corresponds to which mining software
06:58:15 <jtgrassie> lowest lines will be solo miners (just tx pub key in extra, 32 bytes)
06:59:04 <jtgrassie> nodejs-pool and monero-pool use an extra 17 bytes, so 49 bytes.
06:59:57 <jtgrassie> of course, anyone could add other stuff in but that should be the usual.
07:01:19 <jtgrassie> (thats all for the later block heights anyway)
07:02:57 <jtgrassie> (of and of course those numbers I just gave are without the tag bytes)
07:10:29 <jtgrassie> so minimum will be 33 bytes (solo miner) and typical pool 51
16:10:04 <suraeNoether> atoc is the guy who was interested in matching, right? and he's not on right now. :\
16:10:37 <suraeNoether> needmonero90: i've never enjoyed antichamber, it reminds me too much of a nightmare
16:12:20 <koe> would it be 52 bytes? to account for the extra nonce length
16:13:21 <koe> The bottom line looks like it is below 33 bytes.. is someone burning coins, or maybe using a custom software to sign transactions without the transaction pub key?
16:14:14 <suraeNoether> almost certainly the latter, not the former, if those are the only two options
16:14:48 <koe> they have been mining blocks since the very beginning
16:15:58 <koe> there seem to be 2 main groups with sub-33 extra sizes
16:19:23 <atoc> Hi suraeNoether , I am here now. I typically see the log on mattermost.
16:19:34 <atoc> I hope you're feeling better. I saw that you were sick.
16:19:47 <suraeNoether> Yep. Good to catch you
16:20:43 <suraeNoether> https://github.com/b-g-goodell/mrl-skunkworks/commit/06f93587e2d40d21b50cdf48846fc5c75b58d4fb
16:21:21 <atoc> Are these the updated changes?
16:21:30 <suraeNoether> So right now, testingSimulator.py appears to be passing all but one test. unfortunately that test is testing a function that is a composite function, and both of the functions it calls seem to be passing unit tests
16:21:34 <suraeNoether> Yeah
16:21:58 <atoc> I see. Thanks
16:22:03 <atoc> Hmm okay
16:22:13 <suraeNoether> Which means that my unit tests for either the make_coinbase function or make_txns function in Simulator is failing to catch an edge case
16:22:54 <atoc> Do you have an idea of what that failing edge case may be?
16:23:06 <suraeNoether> Make_coinbase seems fine. Since make_txns just calls make_txn a bunch and make_coinbase seems fine, I think it's make_txn
16:23:42 <suraeNoether> I suspect that I am not computing my predictions for the tests correctly. namely, it appears as if the number of red edges being added isn't what I expect them to be based on the state of the system before they are added
16:24:03 <suraeNoether> Which also points to make_txn
16:24:11 <suraeNoether> As the problem child
16:24:51 <suraeNoether> At any rate, I am working on that today. As soon as that problem is fixed, I have some ready to go code to push that will play the tracing game and output the results to a human readable file
16:25:06 <atoc> I see. Also, do you work out of matching-mojo?
16:25:15 <suraeNoether> That's my most up-to-date presently
16:25:25 <atoc> There are many branches and I'm not sure what each are for.
16:25:31 <atoc> yes I noticed that
16:25:49 <suraeNoether> I may do another branch before I pull it back onto my main branch. But right now matching-mojojo is what I am working on
16:26:24 <atoc> Ok cool. I will also see if I can run those tests
16:26:35 <atoc> and see what you're seeing
16:26:51 <atoc> probably a little later though, but i will keep you updated on it
16:26:52 <suraeNoether> what's exciting about this is that we will be able to upload a couple of simulated ledgers to GitHub with some ground truths, and then if anybody else comes along with new matching methods, they can test their method against these ledgers and generate an accurate confusion table. Sort of like unit tests
16:26:59 <suraeNoether> No problem... A brief warning though
16:27:16 <atoc> Ah yes that would be cool
16:28:36 <suraeNoether> The way that I wrote my tests was that I have at least one basic test touching each function. Most functions have exactly four tests... One starting from a totally empty ledger, one starting from a simulated ledger, and one of each of the above that runs sample sizes and does repeated tests of the first two. I recommend uncommenting my unit test skip decorators for the  test functions that you aren't
16:28:37 <suraeNoether> investigating immediately... This will save an enormous amount of time every time you execute testingSimulator
16:29:04 <suraeNoether> And I also recommend just uncommenting the skip decorators for all of the repetition tests
16:29:28 <atoc> ok will do
16:34:42 <suraeNoether> this is also an open invitation to anyone who knows python: this final bug is ... bugging me. *sunglasses*
19:06:59 <koe> looks like people mining with the gui have 33 bytes in the tx extra, at least currently based on this guy https://www.reddit.com/r/Monero/comments/eq8h2s/i_solo_mined_a_block_using_the_gui_thanks_randomx/
19:10:31 <Isthmus> Here's a zoomed in copy
19:10:32 <Isthmus> https://usercontent.irccloud-cdn.com/file/Nv964qfP/image.png
19:10:46 <Isthmus> Confirming - the bottom row of dots is 33 kB
19:14:19 <Isthmus> Oh, let's get the hashrate of solo miners
19:18:11 <Isthmus> Since we activated RandomX, six blocks have been produced by solo miners using the Core software
19:18:28 <Isthmus> (or, technically, by software that mimics the Core software)
19:20:01 <Isthmus> 6/(2015615-1978433)*1000000000
19:20:11 <Isthmus> 161 kH/s by solo miners
19:20:47 <moneromooo> That will have *huge* error bars and a left wall.
19:21:15 <suraeNoether> yep, that's an estimate from censored data
19:21:26 <Isthmus> Yea, 6 is a hilariously small sample size
19:21:28 <suraeNoether> if we are curious, i can formulate a BLUE
19:21:32 <suraeNoether> but with 6
19:21:35 <suraeNoether> we may as well go blue ourselves
19:22:24 <Isthmus> 'censored'?
19:29:20 <koe> emailed u Isthmus
19:31:27 <koe> seems to imply nearly all blocks are pool mined
19:31:47 <koe> at least 6 implementations too
19:32:16 <Isthmus> 6 implementations with significant hashrate, 4 implementations with a little bit of hashrate, and a few solo mined blocks
19:32:46 <koe> are there any outliers flying high above those?
19:32:52 * Isthmus checks
19:35:34 <Isthmus> No outliers, here's the big picture:
19:35:35 <Isthmus> https://usercontent.irccloud-cdn.com/file/tUyNyFO1/image.png
19:36:15 <koe> didnt you mention something about 60kB tx extras?
19:36:52 <Isthmus> https://xmrchain.net/tx/7dfcc4e5d8bd772e3373e51d4140052121503d9b4f3cb6587251292bf06ced9a
19:37:30 <koe> oh 60 bytes of padding lol
19:41:58 <Isthmus> Here's one with 99 bytes of padding - https://xmrchain.net/tx/6598c15d6557a0759f56495f4e8d82171a0696c05524529d18dd0cedb884456c
19:42:06 <koe> it's an extra nonce with 64 bytes, where only 8 bytes are used
19:42:19 <moneromooo> 60 bytes is quite alright. Was the kB a repeated typo ?
19:43:03 <Isthmus> Yea, I was task switching between block propagation (kB scale) and tx_extra (B scale)
19:43:38 <koe> that one is also 56 bytes of padding
19:43:39 <Isthmus> Although there have been kB scale tx_extras for transactions IIRC
19:43:43 <moneromooo> FWIW, a pool could just regenerate the coinbase output to differentiate between miners.
19:43:47 <koe> did you mean a different tx with 99?
19:43:57 <moneromooo> Gets rid of the extra mangling.
19:44:41 <koe> use subaddress maybe? lol
19:45:03 <moneromooo> IIRC mining to subaddresses required lots of changes to monerod so wasn't merged.
19:45:28 <koe> or yeah could just do different transaction pub keys for each miner
19:45:35 <Isthmus> Could you just assign a different stating point in the nonce range for each miner?
19:45:36 <moneromooo> I don't think it's needed though, pools already keep track of template to user.
19:45:55 <koe> Isthmus doesnt work for >1000 miners
19:45:59 <moneromooo> Yes, you'd have to double check the reply's in the range though.
19:46:24 <moneromooo> Otherwise I can connect to you 1000 times, and every time I find a block, ever miner of mine claims they found the same :)
19:46:57 <moneromooo> Damn, I love git bisect. It turns a slow aggravating slog into a slow ok slog.
19:46:59 <Isthmus> The nonce range is waayyyy bigger than that
19:47:16 <koe> the nonce itself is 4 bytes
19:47:33 <koe> thats why pools use extra nonce to differentiate workers
19:49:57 * Isthmus looks around for the back of an envelope
19:55:24 <Isthmus> Ah yep, it is a bit tight
19:55:39 <Isthmus> A fast RandomX miner with 5,000 H/s searching for a block for *20 minutes* could cover almost 2^23 nonces
19:56:05 <Isthmus> So you could comfortably accommodate about 2^(32-23) ~ 512 miners
19:56:15 <Isthmus> Too small for big pool
19:57:21 <Isthmus> Although if this is the main reason that pools are doing all kinds of weird and trivially fingerprint-able stuff, let's just make the nonce range 8 bytes.
19:57:49 <Isthmus> 2^(64-23) would accommodate an astronomically large number of miners
19:57:59 <Isthmus> > 1,000,000,000,000
19:58:03 <moneromooo> I think the main reason is that it was coded that way at first and everyone just reused the same thing.
19:59:00 <moneromooo> So all it takes is someone to write patches and PR them to whoever maintains the code now.
19:59:22 <moneromooo> And then maybe a bit of arse kicking or public mocking to get pools ops to pull.
20:01:02 <Isthmus> Once we extend the nonce range to 8-bytes, to accommodate 1000000000 workers per pool, is there any valid use case for unvalidated tx_extra in the miner transaction?
20:01:59 <Isthmus> Since the value of the nonce itself proves to the pool who mined the block (if that matters)
20:02:17 <Isthmus> Also, pool-mined blocks would then be indistinguishable from solo-mined blocks
20:03:01 <koe> I think it was hyc who pointed out it's always good to have a little wiggle room. That's why tx extra hasn't had hard scrutiny sofar
20:04:36 <koe> if the nonce was bigger it would still be fingerprintable, since each implementation would divide the nonce into different sizes for their workers
20:04:51 <moneromooo> If the intent of extra is allowing unexpected uses, asking whether there's a valid use case for it is... not helpful.
20:06:35 <koe> I feel like the best route is to enforce TLV in the extra field, enforce type sorting, and roll out that update with best practice guidelines. That way when pools update their software, they can at the same time harmonize with other implementations.
20:06:56 <Isthmus> Nonce range should never be divided up deterministically. Pick random starting points for the miners (and reassign any obvious collisions)
20:07:14 <koe> good point
20:07:49 <Isthmus> (best privacy practice for both pool and solo miners)
20:07:59 <moneromooo> jtgrassie: do you feel like patching existing pool code (I think there are 3) to avoid the extra nonce thing ? :)
20:08:05 <koe> could just pick a random starting point, then deterministically increment for each worker
20:08:48 <Isthmus> Ah yea, I wrote it up in an old articlle
20:08:51 <Isthmus> If you are a miner that is using custom/obscure software that mines with an unusual nonce space search pattern (e.g. down from the maximum, or up from the midway point), then you should switch to random sampling so that your blocks blend into your transaction trees and the overall blockchain. Note that for each new block, you only need to call random() once for the first nonce, then you can iterate
20:08:51 <Isthmus> sequentially in either direction — this will entirely remove statistical links between your blocks’ nonces.
20:09:51 <Isthmus> I like the Zcash idea of having a consensus-enforced fixed-size encrypted memo field is great to allow users to safely innovate applications without damaging privacy on the base layer
20:09:55 <cohcho> koe: explain details how will you split N bytes of nonce in non deterministic way between pool workers?
20:10:34 <Isthmus> Unvalidated plaintext memo field is antithetical to fungibility
20:10:39 <koe> cohcho it can be deterministic. Randomly select the first miner's starting point, then for each other miner increment the starting point by 10mill or something
20:11:03 <koe> only works if nonce rises to 8 bytes
20:12:02 <koe> a typical miner will only do 1mill or less hashes per block
20:12:10 <cohcho> you will have lines with 10mill width at block_nonce/height chart
20:12:23 <koe> nope, since for each block the offset is random
20:14:33 <koe> there would be lines if each miner is always mining from the same starting point, but if the entire pool starts at a random point for each block then nonces will be randomly distributed
20:15:55 <cohcho> 8_bytes_nonce % (2^32) will have 10mill width *
20:16:22 <cohcho> this way current nonce distribution will be transformed into 4 LSB
20:16:34 <cohcho> and tx_extra will be represented by 4 MSB
20:17:25 <cohcho> You can concatenate tx_extra to current nonce and plot it
20:17:35 <koe> I think that makes sense to me. Easy change without updating header format
20:17:37 <cohcho> It should be randomly distributed
20:19:26 <Isthmus> Yeah, let's just randomly distribute across the whole range though. Not 4B + 4B or else I'll have two fields to fingerprint on (individual worker search algorithm based on nonce side, pool assignment algorithm based on the extra side)
20:19:45 <koe> does this sound right: in other words, random select a pool starting nonce, then for each miner increment that nonce by 10mill. For each miner, take the 4MSB of their starting nonce and put in tx_extra, and the 4LSB to put in their header nonce.
20:20:44 <koe> pool starting nonce: 8bytes
20:20:59 <cohcho> Currently 4 bytes of nonce is miner's space to iterate and whole tx_extra is for pool only
20:21:32 <cohcho> It's already 8 bytes (at least in monero-pool) plus 8 bytes reserved for special need
20:22:36 <cohcho> some pools write their domain name into tx_etxtra
20:23:16 <koe> ^ Isthmus see if you can fingerprint the different extra length by running ASCII on some fields
20:23:19 <moneromooo> ew. Marketers :S
20:24:32 <Isthmus> @koe I think I get what you're describing, but I don't quite understand the benefit of splitting it into two fields.
20:24:55 <Isthmus> "...see if you can fingerprint the different extra length by running ASCII on some fields" uh oh, I'll check that out
20:25:31 <koe> the benefit of splitting is no need to update the block header format
20:25:41 <koe> it's still version 1 for all I know
20:26:03 <cohcho> the final aim is to have completely random tx_extra represented as TLV and force (or ask somehow) pool workers to choose starting point randomly
20:27:13 <koe> aren't workers given a blob, which has the miner tx extra field smushed inside it?
20:27:21 <koe> so workers arent constructing the blog
20:27:21 <Isthmus> Oh, I meant theoretical reason. I don't mind updates for improvements
20:27:24 <koe> blob
20:27:37 <cohcho> yes, it's hidden behind tree_root_hash, they can only change nonce
20:28:00 <cohcho> But they can iterate nonce in defferent ways
20:28:19 <koe> ah do pool workers run different software?
20:28:34 <koe> theoretical not that I can see
20:29:24 <Isthmus> Might be easier to force pools to standardize by updating the block format. We see 12 implementations in use today, not sure how many will update if we soft standardize
20:29:43 <cohcho> (no one in case of soft)
20:29:50 <Isthmus> hehe exactly
20:30:05 <Isthmus> pools haven't even updated to random nonces, and we've known about that for over a year
20:30:09 <cohcho> I agree to update at least my own
20:30:18 <cohcho> But don't understand yet how to blend in the crowd in the best way
20:30:40 <Isthmus> :- )
20:31:04 <cohcho> simple question: https://github.com/jtgrassie/monero-pool/blob/master/src/pool.c#L2326
20:31:49 <cohcho> monero-pool has tuple with 4 values: the last are rarely used and for special need to support proxy that will distribute work further
20:32:10 <cohcho> Should it be seet to zero or removed when not need (reduce tx_extra to 8 bytes) ?
20:32:15 <cohcho> be set*
20:32:37 <cohcho> the last 2 *
20:32:56 <koe> pools should store information about workers on their own servers, not on the blockchain
20:34:59 <cohcho> more memory (or starage in this case) allows programmer to think less about efficiency during memory allocation
20:36:08 <cohcho> also second element in tuple is being iterated one by one from zero
20:36:50 <cohcho> Is it better to generate it randomly?
20:37:48 <Isthmus> Ah @koe that's the succinct heart of the matter!
20:38:02 <koe> Lets say pools get 4 bytes in the extra nonce to prevent worker overlap. To start a new block, randomly generate an extra nonce and assign to worker 1 (server side assignment). Increment by one bit for each additional worker. When sending out a job, either randomly assign the worker a header nonce to start mining from, or have the worker randomly
20:38:03 <koe> select their own starting nonce.
20:38:11 <Isthmus> https://usercontent.irccloud-cdn.com/file/zwaiPvw1/image.png
20:38:14 <Isthmus> This is making my brain melt
20:38:29 * Isthmus scrolls through pages and pages of dream text
20:38:59 <Isthmus> I might be hallucinating, but I think a few patterns show up very often
20:39:08 <koe> That should be sufficient to make everything appear random, while limiting pools to just 4 bytes per block instead of using the extra field to store info related to their business.
20:39:53 <cohcho> Should pool check explicitely that 4 bytes random tx_extra are uniq or rely or random generator?
20:39:55 <koe> you're hallucinating lol
20:40:05 <cohcho> because in case of overlap two workers will do the same work
20:41:05 <cohcho> Are there any mapping functions with 4 bytes output that guarantee uniq output for each uniq input?
20:41:07 <koe> only do one random number for tx_extra, then increment it for each additonal worker. Worker 1 = 3145, worker 2 = 3146, etc. New job: worker 1 = 889281 worker2 = 889282. That's in the extra field
20:41:14 <moneromooo> identity
20:41:25 <cohcho> except identitiy :)
20:41:27 * moneromooo smartass
20:42:05 <koe> Header nonce can be random without worry of overlap since the extra nonce is different for each worker
20:42:17 <cohcho> starting point should be generated after each new block only once at pool start?
20:42:23 <koe> yes
20:42:33 <cohcho> yes 1st or yes 2nd?
20:42:46 <koe> each new block
20:42:56 <cohcho> ok, that's easy
20:43:15 <cohcho> What's about workers?
20:43:29 <cohcho> They receive start_x, end_x in my case
20:44:22 <koe> workers receive blob which contains the extra nonce computed by pool and assigned to them; they EITHER: receive random header nonce to start mining from , OR generate their own random nonce to start from (up to implementer)
20:45:01 <cohcho> there is xmrig-proxy that do the following:
20:45:26 <cohcho> receive only block template with 4 bytes nonce and split it into 256 ranges ( 3 bytes each) between at most 256 workers
20:45:37 <cohcho> How workers of this software should start iterating?
20:45:38 <Isthmus> hold up y'all
20:45:40 <Isthmus> 2004429
20:45:44 <Isthmus> not hallucinating
20:46:30 <Isthmus> tx_extra reads:
20:46:31 <Isthmus> 
�PeL�F�`L`,4R��/?�SD���minexmr.com
20:46:37 <cohcho> Yes, I was meaning minexmr
20:46:44 <Isthmus> You were right :)
20:46:57 <cohcho> there were 10 blocks in row today by this pool
20:46:57 <koe> whats the length?
20:47:01 <cohcho> let me find height
20:47:08 <cohcho> 92 or 96 probably
20:47:23 <cohcho> 62 *
20:47:30 <cohcho> it varies probably
20:49:07 <Isthmus> 62
20:49:31 <Isthmus> But now that I'm looking at the decoded data they're only responsible for about 2/3 of those len(tx_extra)==62 blocks
20:49:41 <koe> cocho each miner random selects a starting point RAND(3 bytes). If they roll over, just start at 0 (within the 3 byte range they are allowed)
20:51:11 <cohcho> until the first repeat of nonce ?
20:51:38 <cohcho> and then go to sleep until next job or request new one (that's not supported yet)
20:51:43 <koe> should be no repeat since the 4th byte differentiates each worker
20:51:53 <koe> and 3 bytes is 16mill so IDK what super powered worker you have
20:51:59 <cohcho> I mean 3 bytes can iterated within few hours
20:52:06 <cohcho> by slow worker
20:52:11 <cohcho> in worst case
20:52:18 <cohcho> in practise it will not be
20:52:42 <cohcho> I just need correct general solution to cover all cases
20:53:07 <koe> I guess go to sleep, if acting within the limits of that implementation
20:53:14 <koe> what do they do right now? same thing
20:53:32 <cohcho> IN my case sleep, don't know about other software
20:53:38 <cohcho> probably repeat again
21:05:23 <moneromooo> So why not the new output thing ? It saves memory, removes potential patterns in favour of encrypted info, so seems better, no ? What is the catch >
21:06:54 <koe> new output thing?
21:07:21 <cohcho> new output to pool's wallet with 0 length nonce in tx_extra vs the same output with different >0 length nonce?
21:07:30 <moneromooo> Send every a template with a different output.
21:07:43 <cohcho> computation overhead is low?
21:07:43 <moneromooo> every miner.
21:07:51 <moneromooo> I'm not sure.
21:08:03 <cohcho> lower than set 16 bytes and rerun tree_root_hash?
21:08:19 <moneromooo> Probably not. Is this a bottleneck somehow ?
21:08:45 <koe> might be more intense for pools to generate so many curve points
21:08:57 <moneromooo> Or is this "if I get one extra cent profit, I'll cause somene else millions in losses because I'm a human" situation ?
21:09:00 <cohcho> if someone prefers the best efficiency with anonimity tradeoff they will care
21:10:33 <cohcho> What thing guarantee that every new output will not repeat previously generated?
21:10:49 <cohcho> What's the length of random data in new output?
21:11:02 <koe> it's an ECC curve point, so 32 bytes
21:11:04 <koe> random
21:11:31 <koe> bit less than 255 bits technically
21:11:31 <cohcho> Is it safe to generate 1st one randomly and the next by iterating one by one?
21:11:35 <cohcho> I suppose no :)
21:11:36 <koe> i think..
21:11:44 <cohcho> yes?
21:11:57 <koe> um
21:12:58 <koe> no probably not a great idea, since it can reveal the pool's address
21:13:28 <cohcho> And each new output will consume 32 bytes of precious OS entropy, right?
21:13:37 <cohcho> It's too expensive probably
21:14:13 <koe> ok the random space of a curve point is between 2^253 and 2^252
21:15:13 <koe> err random size, this isn't important. What's important is that it costs a couple curve operations to generate a one time output address.
21:19:37 <cohcho> also in that 4 elements tuple: 1st element 4 bytes generated once at pool start and will be constant until restart/crash
21:19:45 <cohcho> Is it better to replace them with random?
21:20:16 <cohcho> It's called "instance id" in code
21:22:57 <koe> probably better to be random, since if certain workers are better than others their instance ID will appear more often
21:24:02 <cohcho> 1st element will be constant for all tx_extra generated by pool instance*
21:25:22 <koe> can probably just roll a random offset for each block, and add the real instance ID to that offset for sending out a job. That way you only have to store the random offset temporarily while the job is alive
21:25:51 <koe> can add the same offset to all chunks of 256 workers
21:27:10 <koe> or 255 workers I guess
21:27:55 <koe> 256 get real koe
21:28:06 <cohcho> 256 workers is different software; Chain of workers is pool (e.g. monero-pool with 4 elements tuple ( 4 bytes instance id, 4 bytes worker specific nonce, 4 bytes reserved, 4 bytes reserved) -> xmrig-proxy ( represent nonce as 2 elements tuple ( 1 byte worker specific offset, 3 bytes nonce for worker) -> worker that iterates within 3 bytes
21:28:38 <cohcho> (and monerod at the beginning)
21:29:17 <cohcho> It's 256 since there are 256 uniq offsets (0 including)
21:53:02 <jtgrassie> moneromooo: thing is with extra nonce, it's not just different miners. proxies also use the extra nonce. so whilst a pool could give each miner a separate coinbase output, it doesn't solve for pool proxies, or multiple workers.
21:59:12 <cohcho> If the last level in chain generate completely random tx and all previous propagates only the first level address Then it will work
22:00:16 <jtgrassie> but up the chain the pool needs to be able to recreate and validate
22:00:58 <cohcho> it will have overhead to generate tx and the last level should back propogate actually generated block template
22:01:13 <jtgrassie> and nothing down-chain should be able to set the output
22:02:03 <jtgrassie> which is why what you are suggesting is flawed
22:04:24 <moneromooo> OK, I can guess what a proxy does based on that statement. Fair enough.
22:05:47 <jtgrassie> I'm also not quite clear on the goal - to stop using tx extra for miner txs?
22:06:13 <jtgrassie> why is this important?
22:06:38 <koe> mostly the goal is to make miners more uniform
22:06:51 <koe> since there are clearly 6+ separate groups of miners
22:06:51 <jtgrassie> but why?
22:07:35 <koe> same reason anything in Monero - indistinguishability, fungibility, reduced fingerprinting, and so on
22:08:35 <jtgrassie> but the very nature of tx extra is anyone can use it already. so unless you are talking about restricting tx extra wholesale, it seems a mistake to focus on just miner txs
22:09:19 <moneromooo> That sounds a lot like a fallacy.
22:09:38 <koe> I think the point is to improve on the current situation
22:10:08 <koe> which has developed due to lack of common guidelines about how to implement the tx extra field. Everyone just does what feels right to them
22:10:31 <jtgrassie> moneromooo: but tx extra is freeform and anyone can put what they want in it.
22:10:43 <koe> miners are the focus since they have specific needs
22:10:57 <koe> but standard tx are also part of the story
22:11:16 <jtgrassie> IIRC there were arguments against validating tx extra.
22:11:45 <koe> hyc suggesting enforcing TLV format, which retains the open endedness while encouraging uniformity
22:12:23 <jtgrassie> perhaps moneromoo can recall the reason why we didn't want to do tx extra validation?
22:13:03 <jtgrassie> I seem to recall the conversation coming up when discussing payment ids.
22:13:28 <moneromooo> Depends who. Me, because it's meant to be arbitrary.
22:13:40 <moneromooo> Some others want it gone.
22:13:57 <moneromooo> Some want it parsed (a bit).
22:14:21 <moneromooo> I don't think parsing just a bit is good.
22:14:28 <jtgrassie> me neither
22:15:01 <jtgrassie> its either arbitrary or not
22:15:28 <moneromooo> Though some kind of generic parsing so that unknown stuff may be skipped seems like a good idea.
22:15:37 <moneromooo> Just structural though.
22:15:52 <moneromooo> We'd have to do away with the 255 limit. Make it a varint.
22:16:30 <jtgrassie> I don't think there is a 255 limit from a consensus pov ?
22:16:38 <moneromooo> There isn't.
22:18:00 <jtgrassie> it is howver used for a number of different puposes already. Trying to restrict it just from a miners pov seems misguided.
22:18:22 <moneromooo> You're confusing things.
22:18:45 <moneromooo> There is changing how pools do things and restricting the use of extra.
22:19:06 <moneromooo> The above was about the first one. The second may or may not happen, independently.
22:19:13 <jtgrassie> the two are related
22:19:20 <koe> I think it's ok for the tx_extra field to permit opt-out privacy, but it would be better if it encouraged remaining private by default (in terms of implementation). Currently there are no restrictions, no guidelines, and so even implementers who DONT want opt-out privacy, are stuck with no reasonable path.
22:19:28 <moneromooo> I hope you're not claiming that relatedness implies implication, right ?
22:19:56 <jtgrassie> no
22:19:57 <moneromooo> I'll assume not. So yes, they're related.
22:20:08 <koe> which totally contradicts the Monero reason d'etre
22:21:37 <jtgrassie> as it is, tx extra is used for tx pub key, multisig, subaddresses, pools, merged mining implementations...
22:22:09 <koe> from my point of view, I agree with hyc that enforced (sorted) TLV would go a long way to privacy by default, without any real restraint on opt-out privacy
22:22:55 <koe> and along with sorted TLV should be explicit guidelines on how to implement the extra field to be the same/similar as other implementations
22:23:04 <moneromooo> TLV is some sort of chunk based structure, right ? Like PNG ?
22:23:19 <koe> type-length-value, it's similar to how the field is already parsed
22:23:43 <koe> it's just, if you want to add something random you must prefix it with a type and length
22:24:27 <koe> and by sorted, I mean each type is sorted by value. Currently some implementations dont sort, but the main wallet does.. ??
22:25:16 <koe> sorry just sort the types, not the values..
22:26:14 <koe> the implementation guidelines would include a list of common/standard types. Public keys, for example
22:28:13 <koe> to verify the field, read it following TLV expectations (byte 0 is a type, byte 1 is a length, bytes 2-8 are a value, byte 9 is a type,...). If the last byte isn;t the last byte of a value, reject the field.
22:28:52 <koe> or, if the type's arent sorted in ascending order, reject the field
22:29:17 <moneromooo> IIRC I sort based on likelihood of usefuless. So probaby txkey first :)
22:29:28 <koe> yeah
22:30:03 <moneromooo> Though the wallet probably has to parse it all anyway, due to some stupid old bug.
22:30:09 <koe> and maybe reserve some types for core updates (idk if that's a good idea)
22:31:07 <koe> types can easily be larger than 1 byte, tag=255 then interpret next byte as an extension of the tag (by addition)
22:34:06 <koe> jtgrassie what's your perspective on sorted TLV?
22:53:43 <cohcho> "which is why what you are suggesting is flawed" <- Anyone with (sec_view, pub_spend) can verify distination of coinbase tx and create it. So if pool shares it then all the next levels will be able to create/verify  uniq tx locally.
22:54:24 <cohcho> jtgrassie: Can you help me understand what is flawed exactly here or privately?
22:57:08 <cohcho> moneromooo: if pool doesn't support proxies then with each new block you will need  1 uniq tx per connected worker (e.g. 60e+3 connected to minexmr.com now , so 60e+3 per 120 seconds of uniq txs)
22:57:32 <moneromooo> 60k ??
22:57:49 <moneromooo> Damn, decentralization is coming ? :D
22:58:06 <cohcho> yes, 60k workers  ~10k  miners
22:58:20 <cohcho> 60k workers of ~10k miners
22:58:49 <cohcho> It's the worst case now probably for single real world pool
22:58:54 <koe> average miner has 6 workers?
23:00:38 <koe> moneromooo it's at least better than ASICs, since if a pool misbehaves then miners can jump to other pools. Each miner has individual decision making power
23:01:59 <cohcho> "and nothing down-chain should be able to set the output" <- This is required due to lack of ability to verify destination of tx submitted at intermediate level. But with (sec_view, pub_send) this ability will be added
23:02:59 <cohcho> (sec_view, pub_send) is called tracking key in cryptonote whitepaper and allows to track destination. It isn't obsolete info at the moment, right?
23:03:15 <koe> no that's accurate
23:24:59 <jtgrassie> koe: I have nothing against TLV, its not much different to how it is currently.
23:25:37 <jtgrassie> cohcho: a proxy does not have the pools secret keys
23:29:13 <cohcho> (sec_view, pub_send) != (sec_view, sec_send), (sec_view, pub_send) can be sent to proxy instead of address
23:30:10 <cohcho> Do you use see any security problems in it?
23:30:21 <cohcho> you see*
23:30:40 <jtgrassie> a pool may not wish to disclose their private view key
23:31:04 <cohcho> At least one shared
23:31:14 <cohcho> It's required only for those who supports proxies
23:31:21 <cohcho> Others can't keep it private
23:31:32 <cohcho> And proxies don't help in decetralization
23:32:02 <jtgrassie> thats a naive statement
23:32:32 <jtgrassie> they neither help nor hinder
23:33:10 <cohcho> With an assumption that pool will share that pair of keys There is no problem, right?
23:33:48 <jtgrassie> forcing pools to disclose private key is going to be a non-starter, IMO
23:35:50 <jtgrassie> and tx extra moved to a strict TLV scheme, why not simply allow a 4 byte type field pools can use?
23:36:07 <jtgrassie> ^and if tx
23:38:34 <cohcho> I just wanted to know that there are no security problems in that approach with uniq tx without tx_extra
23:39:07 <cohcho> I don't know percent of pool users that uses xmr-node-proxy instead of xmrig-proxy
23:39:16 <cohcho> If it's almost zero then it's not a problem
23:39:43 <jtgrassie> both of those proxies have plenty of users
23:40:22 <jtgrassie> pools encourage using a proxy for miners with lots of machines as it takes load off the pool server.
23:41:00 <cohcho> 256 reduction factor from xmrig-proxy is acceptable too
23:41:11 <jtgrassie> the "security problems" is a design which forces disclosure of a secret key
23:41:46 <cohcho> jtgrassie: are you specifically not adding "view" or it's assumed by default?
23:42:22 <jtgrassie> the fact its a view has no bearing. Its a private key that enables disclosure of private data.
23:43:29 <cohcho> I would be happy if pool shared it's tracking key so that I can recheck all found blocks
23:43:39 <cohcho> It's ok for public pool but not for private
23:44:09 <jtgrassie> whilst arguably a pool has nothing to hide, it feels like a flawed assumption
23:44:53 <koe> is there any reason for a pool to make anything public at all? miners can verify their payout based on hashrate vs global hashrate
23:45:00 <jtgrassie> and if tx extra is going to remain free to use (even if using TLV), there is no need for a change.
23:45:30 <cohcho> with view_key it will precise check not probability based
23:46:14 <koe> the question is if any miners care
23:46:26 <koe> since miners already mine, despite no view key
23:46:35 <jtgrassie> seeing what payments a pool recieves does not enable one to determine they have been paid correctly for thier work.
23:47:03 <koe> true^
23:47:11 <jtgrassie> to know that you need to know every other miners work submitted.
23:47:41 <cohcho> nothing shared < pool's blocks can be tracked < pool's submitted shares can be tracked < ...
23:47:56 <cohcho> It's getting closer to ideal with amount of shared info
23:49:50 <jtgrassie> so you are advocating banning of using tx extra for anything non-standard?
23:50:18 <cohcho> Without huge proxies and tx_extra used for nonce I'll be less distinguishable among other miners.
23:50:35 <jtgrassie> which then prevents people using it for things like merge-mining or HTLC or anything else?
23:50:37 <cohcho> If there are other ways to reach the goal Let's discuss it
23:51:54 <jtgrassie> what I'm saying is, you are trying to suggest a forced ban of pools using tx extra, which means you are saying nobody can use tx extra for anything other than monero defined uses.
23:52:47 <cohcho> This argument looks solid, previous one about problem with sharing sec_view wasn't.
23:53:24 <cohcho> I agree to not implement that approach with uniq tx unless the last  problem exists.
23:54:03 <jtgrassie> forcing pools to disclose private view key is a non-starter IMO. That's an unfair enforced policy.
23:55:27 <jtgrassie> and if the goal is reduced information disclosure, you've achieved the exact opposite.
23:58:02 <cohcho> All pools (except only one) pastes height of their blocks, so fingerprint their coinbase tx via side channel
23:58:20 <cohcho> With published sec_view_key it will be fingerprinted but explicitely on blockchain
23:58:29 <cohcho> I don't know what is worse