hyc: paste.debian.net/hidden/f184fe30 - I don't get bans with this, do you ?

The drops at the end of each ~10k blocks are still there, I don't mind them for now.

Actually, reading it as posted, it seems it should be paste.debian.net/hidden/5f6fca2d - building with that one now to test.

will try it in a little while

but you were able to repro the bans without the patch?

Only at the end of the sync, but that was because the source daemon was ~80k height only, and thus inside the HOH hashes.

Better log settings for this: 0,p2p*:DEBUG,*queue*:DEBUG,net.cn:DEBUG

Also, I'm running something based off latest 0.17, not master.

Should not make a difference I think.

Also, it seems weird but appears right: if I want to encode numbers where 50% are 0, 25% are 1, 12.5% are 2, etc (but random), it looks like writing N zeroes with N being the number plus one is pretty good at compression.

But it's kinda de-encoding the number, so it would be totally shite at normal numbers... I found this interesting.

What's about unresponsive synced node ? Is it reproducible or not ?

Unresponsive via RPC ?

That's a very longstanding problem, it's "random" but apparently fairly well reproducible if you leave a node up for hours and wait.

according to logs from hyc there was no response from both p2p and rpc

Any specific requirements for reproduction ?

pruned/full, fast/slow connection, open/close p2p port ?

Heh. Looks like this is actually Huffman coding...

:D

nothing special on source node - full, not pruned. both on same wifi LAN.

Is it reproducible with remote exclusive node ?

probably hard test without fast internet connection

that's a bit more difficult for me to test. my remote node has an in_peers limit. if the 10k disconnect occurs, may not be able to reconnect because other nodes may take that spot.

There is a bug report on github with logs, user name something like ajeuciu4 or the like.

That's about ssl rpc

We're talking about p2p problems

Hmm, OK. I thought this was predating SSL but maybe not.

There are so many reasons why application can unresponsive. We need more info: better log or some runtime info from gdb

* There are so many reasons why application can be unresponsive. We need more info: better log or some runtime info from gdb

or It should be reproduced locally then I'll find all info by myself

you haven't been able to reproduce this?

not yet, syncing old hdd with full node now

2nd chunk of that paste doesn't apply to master. I can't find anything resembling that section of code

neither the surrounding comment nor the error msg text

moneromooo: so I guess I have to test v0.17

eh no, it's not in my v0.17 branch either. must be one of your own patched branches

Oh really. Sorry -_- I'll build vanilla 0.17 then.

<mj-xmr "utxobr, I wrote my comment. Plea"> thanks mj-xmr, I really appreciate the honesty :)) just wrote a reply there 👍️

Carrying over a conversation from #monero-research-lab:matrix.org on what's needed for the next update

hyc, re: tx_extra phaseout - im more thinking of phase 1... like, make it so the CLI wallet doesn't support it anymore etc

It's somewhat pressing because Thorchain plans to start supporting Monero, and that will lead to a bunch of publicly revealed transactions (by revealed view key and key images)

i.e., don't touch consensus yet, just make it so users aren't inclined to use it

Does the CLI wallet support it still?

dunno.

extra_nonce is still being set in tx_extra. pool mining breaks without that

pooled mining strikes again

I definitely want to limit tx_extra further, but doing so in like 2 months is too short of a time period

"Synced 95252/2061118 (4%, 1965866 left) (0.948245 sec, 105.457978 blocks/sec)," nothing interesting except frequent disconnects on every ~10000 blocks

so for this version: BP+, messaging 32 bytes in BP+, ringsize bump, ArticMine fee changes. Am I missing anything?

that sounds like all that's actually ready

If we have a huge public wallet coming online, I want to get a ringsize bump in there for sure, sooner rather than later

so whats the ringsize

11

most people supported 17. I personally voted for 15, but this pushes me to support 17 I think

sech1: :p

is there any evidence that 17 has a drastic advantage over 15? with increased daily transactions we want to keep verification time as low as possible

Does anyone here have an opinion?

i mean i honestly think anything in this range is quibbling (11 - 15 or 17), and any increase is probably statistically handwavy at best. But i like increasing it regardless

knaccc, ?

also if we hardfork in 3-4 months then we have to consider that Triptych will have to wait another 9 months after BP+ hf

is thorchain actually gonna hold off for 3-4 months so they can work within 32 byte messaging in BP+?

also I don't want to rush a hardfork because whatever thorchain is doing

yeah

will that take 3-4 months to add?

selsta's messages aren't coming through to matrix.... hmm

dunno... moneromooo ?

selsta: there is no "drastic" improvement over 15 imo

knaccc supports 17 because it saves a churn transaction in his model

actually I think it was 16 specifically

knaccc: "the only two clear possibilities, in my opinion, are ring size 11 or 16"

selsta: how far away is triptych though in reality?

we will know once sarang has spent more time on multisig

9 mo seems fine for me then personally

selsta> also if we hardfork in 3-4 months then we have to consider that Triptych will have to wait another 9 months after BP+ hf >>>> indeed, but I'd argue that 1) triptych is still unknown and 2) these large HF gaps are assumptions

i.e., we assume its better for the ecosystem to have large gaps, when in reality it might be better to have them closer together. just a thought

gingeropolous: assumption in what way? we won't hardfork 6 months after the previous hf, the ecosystem is way larger than years ago

I would be open to an exception for triptych if it really does come along faster, but I also think it's weird we keep assuming triptych is ready to go in short order

lol disagree, hardfork are quite exhausting for everyone involved, I don't see how rapid consecutive hardfork are supposed to improve things

there are several parts, we clearly accept some difficulty to have them at all

sure.

triptych major update with major gains = potentially worth more effort

I am however concerned that we'll have a large influx of public transactions in 2 months, for which I'm not super comfortable with ringsize 11 to protect if I'm honest

will they still be public with the message in the BP+ thing?

influx from where?

this thorchain thing sech1

just remove tx_extra

thorchain's model will have them receiving monero transactions in wallets with a publicly shared view key and key images

from everything except miner reward tx

they agreed not to spam tx_extra

well there's nothing that can stop anyone from publicly sharing viewkeys

they will use either short payment ID (technically part of tx_extra so you got me) or the BP+ storage

also talking about network stability, probably not best to remove tx_extra with no notice

I am not sure if a bump in ring size without Triptych is quite productive

Especially with the high tx volume currently, sync time has already been affected quite a lot

if a lot of people use thorchain (which I expect them to do), then I think 11 is insufficient. 15 would let me sleep at night

Have you tried to sync 1-2 months worth of blocks recently?

nah my node has been running 24/7. wallet syncs are still fast but that's with a local node and a fancy phone

I'd rather address this problem by bumping fees, and taking the BP+ efficiency win sooner rather than later

I'd be opposed against a fork in the foreseeable future, just fyi

yeah, something something triptych eventually, maybe :p

I personally also think that BP+ efficiency gains are not worth a standalone HF.

what are we going to do if thorchain starts processing ~25% of monero transactions

what is your plan then

If they use an encrypted pID as identifier, are those transactions even publicly identifiable?

yes because the wallet they end up in reveals the view key and key images publicly

I don't know what thorchain is exactly.

some DeFi stuff

Why would they suddenly be 25% of tx volume?

broadly, it's a DEX that tried to imitate an instant swap model, where users can send native assets to a pool of semi-trusted nodes and receive another native asset back

you can "stake" XMR in these pools, you can swap with no KYC in an AMM liquidity pool

do they have any public stats? Number of users, activity etc.

viewblock.io/thorchain shows 535k transactions total

that's in 1 month

that blockchain only went public last month

April 13 medium.com/thorchain/thorchain-laun…ch-multichain-chaosnet-bb9f60008a03

That's across all coins, do you think Monero will be super popular on that platform?

Monero, the only real privacy coin, trading on a DEX with traditional DeFi benefits that allows "staking" for the first time, and has no KYC? Yeah, I think it will be popular

yeah get 500000000% APR until it collapses and you lose everything :D

that's the hot gamble the rest of the crypto space is playing :p

anyway, hopefully you see why I'm concerned about the possible # of public outputs with this coming online

with current tx volume bp+ will save us 1GB chain growth per year

at this point that's just the cherry though, it wouldn't be the main reason for the update because that alone is too small

as to increased ring size, is there any evidence / calculations that a ring size bump without Triptych has any meaningful impact on plausible deniability?

sgp_old: you used to push for evidence before ring size bump in the past :D

yeah plenty, check out the issue where knaccc and I run through the calculations for the same stuff mentioned in mrl-0001 and mrl-0004 monero-project/research-lab #79

> We should only move to ring size 16 if we are willing to pay the price of 46% higher individual tx verification times (and 21% higher verification times in churn scenarios due to one fewer churn tx being required at ring size 16).

I'm skeptical this is worth it :/ A targeted individual will also have issues with ring size 17 without churning.

it all depends how far away Triptych is

So is there serious interest in range proof data embedding?

i think its a better approach than integrated address / encrypted payment ID for any sorta data storage

sarang, ^

A few things to keep in mind with that

- You get 64 bytes per proof (with some restrictions based on scalar representations), and no more

- Recovering embedded data requires partially verifying the resulting proofs (scalar and hash operations only)

- You likely need a way to differentiate between a wallet that produced a proof with no embedding, and one that produced a proof with embedding

The last one means that a wallet that doesn't know about data embedding would produce proofs using random data, and a recipient would "recover" garbage data unless you had authentication or some other way to signal that a pID is present

Notably, the network _cannot_ detect the difference between a proof with embedding and a proof without embedding

right, thats sorta the goal :)

right?

Sure, but the UX of differentiating these two use cases should be considered

yeah that's a big

*big +

indeed. I imagine ":and a recipient would "recover" garbage data unless you had authentication or some other way to signal that a pID is present" .. could be addressed by some sorta standard or prefix in the embedded data

Yep

.e.g, if the wallet decodes and a tag is present, then its a payment ID. if the wallet decodes and no tag, then its random

Depends on the length of the tag

indeed

A certain fraction of non-embedded proofs would have the tag by chance

i wonder if we could sandwich it

To confirm, there would never be a need for per-recipient pIDs? Or longer pIDs than 64 bytes (accounting for scalar restrictions)?

Because again, you get 64 bytes per proof, and that's it

The `pybullet-plus` branch of my `skunkworks` repository has a demo of this embedding, should anyone find it useful

It would need to be updated to reflect the optimized code in the `monero` repository, which I haven't investigated at all

<sarang> To confirm, there would never be a need for per-recipient pIDs? Or longer pIDs than 64 bytes (accounting for scalar restrictions)? >>> so, in other words, if you make a multi-output transaction (for multiple recipients), there's only 1 pID

As long as each transaction has a single range proof, you get 64 bytes per transaction to do with as you choose

No more

(technically a bit less, due to some scalar representation restrictions)

So you need to make sure that this restriction is acceptable, because you can't play around with it

are you sure it's 64? moo kept saying 32

For BP+ it is 64

Actually hold just one moment... I need to check something about the scalars involved there, to make sure I wasn't implicitly assuming common recipients

Rats, it _is_ 32 bytes... you only get 64 bytes if you can assume a common recipient among all outputs

Sorry for the confusion... I neglected to account for this while re-reading my demo code

You can include any value representable as a scalar element in the prime-order curve subgroup

sure, so the 64 byte case is unrealistic unless someone wants to record a message on-chain for themself or something

It does highlight that this approach in general implies restrictions that can't be avoided

Whereas a separate pID field, while irritating for several reasons, offers flexibility

well, flexibility of 8 bytes instead of 32 bytes

in practice BP+ storage seems more flexible in 99% of cases

I mean you could decide to do pIDs per recipient, or longer values, etc. and have this be a fairly arbitrary consensus decision

Whereas you can't do this with the BP+ approach

are pids per recipient supported?

I thought we only supported 1/tx currently

So the strategy for this would be that if a client detects a transaction for which it is a recipient, it would do the following:

- compute a hash variant of its shared secret

- partially verify the range proof

- use this data to attempt to recover embedded pID data

- use any signals/flags defined by the protocol to identify if data is present

- recover data if present

- ignore if data is not present (or if data is intended for another recipient)

Keeping in mind that it is possible for a sender not to properly embed data, in which case the signals/flags would need to be used to identify the presence of data

False positives would be possible

yeah, but if a recipient is not expecting a pID, then false positives are sorta benign.

but you definitely want to avoid false negatives

?

A recipient can always attempt to recover embedded data

18:24 <sgp_old> I thought we only supported 1/tx currently <-- correct

I want to separate two things

1. the 8 byte payment ID in the way it's currently implemented

2. any other conjuration of a form of payment ID with tx_extra playspace

so one ID/recipient would fall under 2 since that's not how anything currently uses it

did anyone ever see "Invalid decimal point specification 11" when opening a .keys file?

corrupted .keys file?

"Sent ... (3.85 GB) ..., average 253.33 kB/s = 12.37% of the limit of 2.00 MB/s", "Synced 1216600/2061198 (59%, 844598 left) ..." nothing interesting, still syncing

What was the lowest height during some ban event ?

"... sent wrong NOTIFY_RESPONSE_GET_OBJECTS: block with ... wasn't requested, dropping connection" such errors are regular but they lead only to disconnect without ban

Odd. That should only happen for very slow connections. Is that a very slow connection ?

both nodes (87% synced full hdd node and 2% synced pruned tmpfs node) are on the same machine but communicating through 192.7.7.1 assigned to loopback

connection between them is surely not slow but hdd node has long delays due to writes

Slow here means a node asks for data, but doesn't receive a reply within 60 seconds.

I guess it might do that when commiting a large txn to HDD.

im doing the monero book's tutorial in chapter 7, 7.4.2 Tutorial 2 - How to generate a pseudo-random address, utils.hash_to_scalar(secret_spend_key) does not exist in any python library?

import utils, doesnt have alot of functions, such as utils.sc_reduce32, utils.hash_to_scalar, utils.publickey_to_privatekey, etc, so on

"github.com/monero-project/mininero/blob/master/oldMiniNero.py#L317" ?

"Odd. That should only happen for very slow connections.", "... kicking idle peer ..." yes, idle peer kicker isn't correct and triggers that error

but anyway it isn't important now since it doesn't lead to ban

moneromooo: What avg upload speed from synced to fresh node did you have at the moment of ban ?

I did not have a ban. hyc did.

Well, apart from the sync ending at 140k (source daemon was there) but it's unrelated.

then there is no successful reproduction outside of hyc environment

* then there is no successful reproduction outside of hyc's environment

oh shit

look who's makin ga lot of commits

ratthing69[m]: please not here :P

moneromooo: do you have an updated patch? or should I just go ahead with chunk#1 of that one?

paste.debian.net/hidden/fe688136 but that should only trigger when you get past the builtin hashes, so not relevant to your bans AFAICT.

hyc: Can you try to reproduce in simplified environment with both nodes on the same machine or full node without out peers?

er, when you sync off a daemon that's itself synced to a height that ends before the end of the builtin hashes.

wfaressuissia[m]: both nodes on same machine, only got disconnects

that was log1/log2.txt

anyway the serving node was fully sync'd

What's about no other peers (--out-peers, --in-peers 1) ?

so hashes would be well beyond the builtin hashes

yeah I can set it up again and try that

What was the last commit/branch without these bans ?

"v0.17.1.5, v0.17.1.6" it's expected v0.17.1.5 shouldn't have bans but v0.17.1.6 will have

Can you try ?

that will take quite a long time to step thru

I'm now running 2 builds of master on same box

no other peers

seeing the 10k disconnects, no surprise there

"that will take quite a long time to step thru" to try 2 tags or to do bisection between them ?

both, because it takes a long time starting from scratch before a ban occurs

Can you send me your environment I'll try something :D ?

wdy mean? this is ubuntu20 laptop

nothing

"I'm now running 2 builds of master on same box" how many % are already synced without bans ?

it is just starting out, 4% syncd so far

fresh run, ok

since this is over localhost I don't believe it will hit any ban

that was only across wifi LAN

but I'll let it run a couple hours and see

nothing here even with non-loopback address

assigned to `lo`

of course not

you're still using the loopback interface

that means some network delays are required to trigger the problem

probably

also notice that bandwidth limits don't impede sync over loopback at all

sw bandwidth limit can't be reached even with fast CPU and SSD?

I'm seeing ~15Mbps in and out over loopback now

`print_net_stats` the result from monerod ?

sent average 1.39MB/s

69.45% of 2MB/s limit

ok

Received 7478183 bytes (7.13 MB) in 2376 packets in 14.1 minutes, average 8.66 kB/s = 0.11% of the limit of 8.00 MB/s

Sent 1242666684 bytes (1.16 GB) in 38907 packets in 14.1 minutes, average 1.41 MB/s = 70.29% of the limit of 2.00 MB/s

10% syncd now

also - reading and writing to 2 separate SSDs

`ip link add name lo1 type dummy`, "wiki.linuxfoundation.org/networking/netem" it should be enough to simulate wifi connection

42% sync'd here no slowdowns. just the usual disconnects

It's expected that other peers connected to 100% node are more important for reproduction than wifi connection since they add more interesting delays into async code

yeah I get the feeling this is only going to show up with other active peer connections

but I'll restart again soon over wifi

over wifi without other peers ?

right

good

it's 50% now I think that's good enough, going to exit and start again

sad that this macbook "pro" has no wired ethernet port

and no possibility to catch thunder strike via ethernet from router :D

well. it's still plugged in to AC power

and WAN port is fiber, so no threat there

I had one because it wasn't fiber internet :D

bummer. talk about rotten luck!

everything else survived except laptop

and router

path of least resistance

huh.

starting on mac with --add-exclusive-node but it is still talking to other peers

oops

other nodes hitting inbound port, forgot I set a port-forward

16% sync'd. looks like no issues

so the problem only occurs when the fullnode has other connections

42% syncd still going

so yeah, can't repro without other peers

What will try after 50% of current test ?

* What will you try after 50% of current test ?

I suppose, re-enable other peer connections

"downloads.getmonero.org/cli/monero-linux-x64-v0.17.1.6.tar.bz2" Can you re-enable other peers and use this binary ?

ok

as which side, full node or new node?

It's likely enough for fresh node

but it would be simpler to use for both if all other fixed problems since 17.1.6+ will not pop up (out-of-memory, ...)

I would have to compile 17.1.6 for my Mac

which is doable of course. just takes more time

"downloads.getmonero.org/cli/monero-mac-x64-v0.17.1.6.tar.bz2" this one isn't usable ?

I mean an ARM binary

that one should run, because of binary translation

but that seems like a bad idea

How much time does it take to compile only monerod target ?

20m ?

starting right now with master on mac

will try 17.1.6 later

mac is full node for now

linux is new node

about 19% sync'd got a ban

mac seems to have been frozen for a while

highlandsun.com/hyc/log1.txt and log2.txt

log2 is the mac fullnode

log1 is the linux new node, 17.1.6

it just seems to be banning because the mac timed out

these are both with loglevel 0,p2p*:DEBUG,*queue*:DEBUG,net.cn:DEBUG

also, the Mac is not cleaning up connections paste.debian.net/1196474

1.155 is the linux new node, 1.214 is the mac full node

uh, try this paste.debian.net/1196475

mebbe it's just a macos bug

never noticed any freezing

it definitely is not doing clean shutdowns of those incoming sockets

and it definitely is being to slow to respond

is the issue with connections not cleaning up on mac only?

haven't fired up a 2nd linux box yet to see

this is also strange. the connection to the mac has been in close_wait for several minutes but is still showing up in debu output

tcp 116585 0 192.168.1.155:56720 192.168.1.214:18080 CLOSE_WAIT

2021-05-05 23:43:51.665 D [192.168.1.214:18080 OUT] next span in the queue has blocks 2353364-2353383, we need 2353364

2021-05-05 23:48:08.329 D [192.168.1.214:18080 OUT] Block process time (20 blocks, 781 txs): 12118 (7042/5076) ms

or I suppose those are new connections

and that one is just still waiting to be cleaned up

dunno since the output doesn't include local port#

no sign of those conns on the Mac side

phantoms

there are no other connections. this node 1.155 thinks it has a conn to 1.214 but 1.214 doesn't show any and the one on the 1.155 side is in CLOSE_WAIT

ok it finally closed properly

so for several minutes it was ignoring that conn, not reading anything off it, while remote side had already closed it