02:04:23 hrm. could do a series of fast challenges that are somehow chained. Any latencies would be subsequently amplified 03:31:08 -xmr-pr- ghdqh1515 opened issue #6904: Unknown CMake command "monero_crypto_autodetect". 03:31:09 -xmr-pr- > https://github.com/monero-project/monero/issues/6904 09:51:55 Seeing a few reports of people being stuck on the fork block (v14). Any idea what could cause this? 09:52:15 https://www.reddit.com/r/Monero/comments/jdct0z/anyone_got_any_idea_of_when_the_hard_fork_will_be/g975vxp/ 09:52:35 https://www.reddit.com/r/monerosupport/comments/jdckrd/monerod_stuck_on_v14_fork/ 09:58:20 Lots of report in #monero as well. 09:58:41 I just went out, cannot check my node now 10:01:27 what I know so far: block 2210720 has at least 2 MLSAG transactions which shouldn't be there. Nodes that were online at the time allowed it. Nodes that were offline and try to sync, don't allow it. 10:01:44 so they all get stuck at height 2210720 10:02:08 good news: all mining pools are fine, they just shouldn't pop blocks until the fix is issued 10:02:20 bad news: whoever was offline can't use Monero now 10:53:32 What's the fix? Allowing MLSAG transactions in block 2210720? 10:54:20 I think the fix basically needs to make those two transactions seen as valid 10:54:24 And use the MLSAG verification rules 10:54:26 Think that would work 10:54:55 should work 10:55:22 Wonderful hack, but yeah, should work 10:55:57 so are we technically in chain split mode now? 10:56:41 A split with only 1 working chain? 10:56:43 one chain is stuck at 2210720, the other is at 2210941 now 10:57:01 well, makes it easy to choose 10:57:07 :) 10:57:48 things will get worse if some of those stuck nodes mines a block 10:58:26 and we'll have two alternative 2210720 10:59:02 But 24/7 nodes synced properly through the fork 10:59:20 yes, so all miners should be fine, including solo miners 10:59:21 So that would have to take someone syncing 'later' and then starting mining 10:59:49 well, someone might start their GUI wallet with mining on, sync to 2210720 and find a block 11:00:42 stop_daemon 11:00:56 my bad wrong window 11:01:01 sech1: Difficulty would still be high though, so that would take some extraordinary luck 11:01:24 fingers crossed, but we're sitting on a ticking time bomb now 11:01:58 well, they'll just have to pop blocks after the fix then, but the less they have to do the better 11:02:21 Where would the MLSAG transactions come from to mine that block? 11:02:29 Or do I misunderstand? 11:02:32 they came from mempool 11:02:36 The txpool 11:02:45 for some reason MLSAG were allowed to enter mempool and be mined after 2210000 11:02:58 which is a bad oversight 11:03:28 we wouldn't be here disussing it if they were banned from mempool after the 1st of 2 forks 11:03:30 They were allowed due to the grace period 11:03:30 Ah, of course, it's a CLSAG block, not a MLSAG block, so no problems to get the necessary transactions. 11:04:13 But what if the decoys are too high? I think it wouldn't be easy to mine such a block. 11:04:20 grace period or not, but 0.16 wallets kept working all the way up to 2210720 11:20:19 So no node is mining in the right fork then ? 11:21:29 everyone is mining at 2210953 now 11:21:37 it's just 2210720 which is messed up 11:21:47 Looks fairly simple. The output check is done in check_outputs, which is called when adding a tx to the txpool, but not when checking a block that was mined. 11:21:54 So if hte rules change midway, it causes that... 11:23:09 I can add a v15 which adds the output check also when checking a block, and allow MLSAG in v14 I guess. 11:23:23 What's "the right fork" is a philosophical question now 11:23:31 And from v15, check outputs also after txpool. 11:23:39 everyone is on the strange chain now which shouldn't even exist 11:24:46 if you add v15, it should be announced in advance so all miners don't mine v14 blocks past the set height for it 11:24:59 not a quick fix 11:25:17 is it better to stay on v14 with an exception for these 2 tx? 11:25:39 A v15 seems pretty radical to me 11:25:52 I already see headlines "emergency hard fork" 11:26:08 it's not like everything is broken now, just syncing for old nodes 11:29:01 if having these 2 tx doesn't break anything but syncing for monerod <= v0.17.1.0, I don't see a need for v15 11:30:24 Maybe needs logic in quite a few places, to make those 2 tx fully normal, spendable, usable as decoys, etc.? 11:30:39 Well, any miner that adds a another MLSAG tx will cause this again, won't they. 11:30:47 this ^ 11:31:05 so allowing MLSAG in v14 won't work 11:31:39 Can't you allow it for a specific block only? 11:32:08 allowing MLSAG for 2210720 only seems like a good idea 11:32:46 Has anyone checked if there are RCTv4 txes past 2210720 ? 11:32:48 Anyone from the core team has an opinion ? 11:32:48 then v15 is not needed at all 11:33:09 Would personally prefer to avoid a HF 11:33:24 moneromooo you know better, if you can add exceptions for these 2 tx to make them spendable and syncable 11:33:36 binaryfate was in /monero about 10 minutes ago. 11:33:49 Yes, I could do that. It's simple. 11:33:51 * iDunk votes for v0.17.2.0 which should be marked as mandatory. 11:33:54 0.17.2.0 "bugfix" release is better than "hardfork" release 11:34:22 I guess they can be used as decoys already, so it's all about syncing and spending 11:34:58 and double check it's only 2 MLSAG tx in that block, I only did a quick look. 11:35:01 Did somebody check higher blocks for other such problematic tx? 11:35:14 I checked a few next blocks 11:35:39 I guess we'll find out once the fix is in and someone tries to sync with it 11:35:40 bugfix is definitely better than hardfork if possible 11:35:42 I guess it becomes pretty improbable quickly after the hardfork 11:36:00 To have such tx at hand at all to put them into blocks 11:36:07 Only if an asshole miner puts one in :) 11:36:28 mempool was cleared after 2210720 as far as I can see, so it's unlikely more MLSAG were added 11:36:33 Hi, any news on monero php library? 11:37:17 apparently most wallet RPC calls involving arrays fail 11:37:19 an example: 11:37:25 would the fix allow for a possible alternative chain to be mined, where a different block but with similar issue would be mined at 2210720, without again having a fork butween 0.17.1 nodes and the bugfixed ones? 11:37:35 Request have return error: Parse error. Request: {"jsonrpc":"2.0","method":"make_multisig","params":{"multisig_info":[null,null],"threshold":2,"password":"mypassword"},"id":14}; 11:37:50 If not, we're effectively hardcoding that the current chain can't be rolled back past a specific number of blocks which is not great 11:37:52 How about we just allow MLSAG always for now ? 11:37:52 It protects against someone mining one later. 11:38:19 how can someone mine it later if nodes will reject it? 11:38:35 That's what happened. 11:38:57 aubergine: Could you use #monero for a bit? 11:38:58 Ah wait. 11:39:21 It lonly got accepted by nodes that already hjad it in their txpool. So yes, very unlikely to happen again... 11:39:26 dEBRUYNE ok no prob 11:39:30 handle_incoming_block or whatever it's called will reject any block with MLSAG 11:40:02 I've checked blocks up to 2210730 manually and I don't see any MLSAG tx there 11:40:15 a few of these blocks were empty, so mempool cleared 11:40:27 If the assumption of a one-off is correct, there shouldn't be any more after 2210720 11:40:48 Maybe it's not one-off, but "tx still in mempool"? 11:41:07 And if that cleared, it should be over 11:41:27 As I understand from moneromooo's comments, it's about having MLSAG in the mempool by the time height hits 2210720 11:41:40 after that MLSAG won't be allowed into the mempool 11:41:53 That's also my understanding, yes 11:42:42 Anyone has the txids of those txes ? 11:42:47 I've got one. 11:42:50 yes 11:42:56 c5... 11:43:13 6f2f117cde6fbcf8d4a6ef8974fcac744726574ac38cf25d3322c996b21edd4c 11:43:25 c5151944f0583097ba0c88cd0f43e7fabb3881278aa2f73b3b0a007c5d34e910 11:43:33 https://pastebin.com/LZ4Z9bSq 11:43:33 ty 11:43:42 ^ that's the two I identified as well 11:45:32 it'll be easy to verify the fix though - if some txid is missing from the exceptions, it won't sync 11:46:36 I have a stuck daemon and one that's below v14 fork height so I can build and test. 11:47:48 Looks like I'm synced to ...963 now. 11:48:05 yes, that's the current height 11:48:29 I'm here to review your PR if it's coming soon 11:48:58 If would be here if debian.net oculd connect... One min.. 11:49:09 Ah: https://paste.debian.net/hidden/80c30661/ 11:49:13 I'll PR. 11:50:46 looks good for sync fix. No changes needed to make them spendable? 11:50:56 sorry if this is dumb but it seems cleaner to retroactively move fork height to 2210721 if possible 11:51:04 it won't work 11:51:07 got it 11:51:13 2210720 already has v14 version baked in 11:51:17 testing my export/import hacks now 11:52:06 sech1, no extra changes needed. 11:52:43 got it. If they can be used as decoys already, so they should be spendable since it's all decoys to a node 11:53:37 There's nothing special about them wrt spendable of anything else than they're not supposed to be accepted past v14. 11:54:29 no need for v15 hf then. Leaves one more scar in the code, but whatever... At least the fix is all in one place 11:55:34 So for next fork, we might need to check outputs in handle_block_to_main_chain, which might give a speed hit. I'll have to think about this. 11:57:09 Just don't allow old type of tx into the mempool right after the first fork height, but let them be mined until the second fork height? 11:57:19 so what will be required from users now then, either update to a newly-released 0.17.2 or keep their current 0.17.1 software but apply the export/import hack? 11:57:43 0.17.1 with export/import hack should also work 11:57:58 because it worked for me until I stopped my node and popped blocks 11:58:20 0.17.2 is certainly recommended update 11:58:27 v0.17.2.0 will lock Ledger users out, I guess they have to wait a couple day(s) then 11:58:37 * selsta still annoyed at the version lock 11:58:58 Can confirm the fix to work, my daemon syncs to top with the PR cherry-picked 11:59:15 on the positive side, 0.17.0.0/1 which have Dandelion propagation bug, will be replaced quicker 11:59:41 moneromooo: opinion on bumping out-peers a bit or not for this release? 11:59:57 Good idea 12:00:08 +1 more out_peers 12:00:21 +10 more out_peers 12:00:31 So every time we fix a bug, ledger stops working ? 12:00:50 Seems so, yes. If we bump the version number. 12:00:53 or keep their current 0.17.1 software but apply the export/import hack? <= Should work as long as you already synced past the fork height I guess 12:01:00 Or alternatively, the export/import hack 12:01:03 we can bump the patch number, otherwise it is hardcoded 12:01:08 -xmr-pr- xiphon opened pull request #6907: [release-v0.17] wallet2_api: implement stop() to interrupt refresh() l... 12:01:08 -xmr-pr- > https://github.com/monero-project/monero/pull/6907 12:01:09 -xmr-pr- moneromooo-monero opened pull request #6906: blockchain: fix sync at v14 boundary 12:01:09 -xmr-pr- > https://github.com/monero-project/monero/pull/6906 12:01:09 -xmr-pr- moneromooo-monero opened pull request #6905: blockchain: fix sync at v14 boundary 12:01:09 -xmr-pr- > https://github.com/monero-project/monero/pull/6905 12:01:26 if that's fixed, do we need the export/import hack then? 12:01:36 I guess it may still be useful to have regardless 12:01:42 No, would be easier for users to simply upgrade 12:01:54 Also +1 here for bumping out_peers 12:02:31 Well, that import/export hack isn't a hack, just that the tools learned something new, no? Or did you hardcode the block number? 12:02:37 technically not a hard fork but it's still very similar if only option is "you must" update now 12:02:43 rbrunner: nothing hardcoded. 12:02:50 yeah just a new feature 12:02:59 and it would allow us to post incremental .raw files 12:03:07 I doubt bumping out_peers will help much, but yes, +1 from me, too. 12:03:57 binaryFate: HF would be 'messier' though as it requires pools to upgrade as well 12:04:01 isn't the cleaner approach to allow mlsag for a little longer in my opinion hardcoding those 2txs as exception seems kinda dirty 12:04:02 Pools don't necessarily need to upgrade now 12:04:15 what's the import/export commands I should use for this? 12:04:52 true still quite better than hardfork 12:04:58 Criispin57 that would require another hardfork to v15 if we allow MLSAG in v14. We can't just bump it to 2210721 height. 12:05:40 pools don't need to upgrade as long as they don't decide to pop 1000 blocks for some weird reason... 12:07:11 patch is working. All synced up. 12:07:20 sech1 yes i know 12:08:01 the patch is working here too, thx 12:09:07 Now we just inform the owners of those two outputs to spend them to finalize testing ... oh wait :) 12:10:01 Works here on Linux and Windows. 12:10:38 moneromooo no CLI wallet update needed, I hope? 12:11:16 Shouldn't need to. 12:11:56 because I'm so paranoic with CLI wallet updates :D I check signatures/hashes 5 times or more... 12:12:03 and update once a year... 12:12:52 and use a canary wallet with small amount to check if anything gets stolen after I open it with new binary :D 12:14:31 Any of the maintainers online btw? luigi1111, Snipa, fluffypony? 12:16:08 -xmr-pr- 90hacker opened issue #6908: There were 0 blocks in the last 90 minutes 12:16:08 -xmr-pr- > https://github.com/monero-project/monero/issues/6908 12:16:11 did someone write up the workaround with import/export? I'd prefer that than getting a new release 12:17:55 import/export need to be rewritten a bit to make this work 12:18:14 I just realised one of my daemons got eclipsed shortly after the fork. I'd expect the same to happen with, say, 16 out peers. 12:20:23 moneromooo checking the tx version in handle_block_to_main_chain should be easy enough right? Don't have to check validity again, since they are already in the pool. 12:20:34 Feels like these... research... nodes are >50% of the network. 12:21:59 It should be enough for the MLSAG/CLSAG switch, but it'd have to be changed for every fork for whih we change output rules. 12:22:33 so could something like this has happened at every fork? 12:22:42 have* 12:23:06 the way I see it, yes for any fork that changes the tx format. 12:23:56 iDunk: are the "research" nodes the same as "Troll" nodes? 12:24:27 Well, Idk what they've been called elsewhere :) 12:24:38 But probably, yes. 12:24:57 Think so. I also think they are more prevalent in Europe. 12:25:06 But not sure 12:25:09 are they computationally inexpensive to set up? e.g. don't need the blockchain as they just report your own block height? 12:25:39 I am not sure somebody already checked how far they work, whether they are true (but doctored) nodes or something else 12:25:50 maybe nodes need a handshake proving they actually hold the chain - ask each other for a few random blocks e.g. 12:31:08 -xmr-pr- emesik opened issue #6909: Transfer back to same wallet shows up only in outgoing transfers 12:31:09 -xmr-pr- > https://github.com/monero-project/monero/issues/6909 12:31:39 I guess privacy opponents are using, what I'd call, network smothering techniques against Monero. 12:32:06 Inge-: a fake node could always ask a real node for the answer 12:34:21 Alice connects to Bob, sending a tx she wants. Bob replies with a tx he wants. Alice sends that tx. Bob sends the tx Alice had asked for. Handshake done. Then you can't ask another node for a block if you don't have at least that much synced. 12:34:37 s/sending a tx she wants/sending the hash of a tx she wants/ 12:34:46 PR#6910 pushed, export with --block-start=2210718 --block-stopp=2210721 12:35:25 if you've popped back to 2210719 you can import with --dangerous-unverified-import=1 12:35:35 and daemon will then sync past the problem 12:36:20 I can also upload a 3-block .dat file if anyone wants it 12:37:06 -rw-r--r-- 1 hyc hyc 137194 Oct 18 13:27 blockchain.raw 12:37:24 cp blockchain.raw . 12:37:25 e7c0a70b5d0cc87fe2ffab07324d2246f63be98af4305700862d5242b03aa151 blockchain.raw 12:38:34 http://highlandsun.com/hyc/blockchain.raw 12:38:58 hmmm wait, permission problem 12:40:11 ok there it goes 12:40:57 how can someone use this .raw file? 12:42:04 you must recompile monero-blockchain-import with PR#6910 applied 12:43:48 Sup? 12:43:52 dEBRUYNE 12:43:57 ? 12:44:04 That insnaity being fun I assume? 12:44:25 we had a lot of fun this morning 12:44:29 nodes stopped syncing 12:44:40 block 2210720 is technically invalid and requires a code patch 12:44:43 Ye, I saw a bit earlier 12:44:43 Snipa: We basically need to do a release soon and some PRs need to be merged 12:44:49 Would consult with moneromooo for the specifics though 12:45:43 Releases, I've yet to do, merges, I'm gine with in general. 12:46:09 -xmr-pr- selsta opened pull request #6911: build: prepare v0.17.2.0 12:46:09 -xmr-pr- > https://github.com/monero-project/monero/pull/6911 12:46:09 -xmr-pr- hyc opened pull request #6910: Allow setting start block on export 12:46:09 -xmr-pr- > https://github.com/monero-project/monero/pull/6910 12:46:20 Yes, I got a couple patches you can merge, 690[56]. Thanks ^_^ 12:46:44 Gimme a sec, getting logged in, work auctually woke me up. :P 12:56:10 Aha! I didn't bork it up. Delightful. 12:56:20 690[56] merged. 12:56:26 ty 12:56:30 Into their proper branches even. 13:00:04 Any other changes that we want to get checked into this? I see 6911 contains the version and checkpoint update for .17.2 and I've verified the checkpoint block. 13:00:42 I do see Xiphon's got 6907 ready to go in and approved, but less sure as we're sort of just patching this one issue, no? 13:00:49 .merges 13:00:49 -xmr-pr- 6862 6875 6881 6882 6886 6891 13:01:31 Snipa: "Any other changes that we want to get checked into this?" 13:01:31 yep 13:01:33 would be nice if we could include GUI fixes else we will have to do another release soon 13:01:38 will submit one 13:01:39 which is just extra work 13:01:42 in a few 13:01:49 Snipa: will ping you once I have the merge lis 13:01:51 list* 13:01:52 kk. 13:02:00 I'll be here for a bit. 13:02:09 * Snipa gets some coffee. 13:02:55 is there a PR which bumps number of out_peers? 13:03:04 we discussed bumping it like an hour ago here 13:05:08 sech1: "is there a PR which bumps number of out_peers?" -> seems not 13:07:08 Snipa: 6912 also 13:08:34 kk, as soon as we get a proper review on it, will vacuum in. 13:09:42 cool 13:11:47 6912 in the .3 ? 13:12:40 are we on .3 already? 13:13:06 please have a look at https://github.com/monero-project/monero/pull/6914 13:13:09 but yeah I think it should go in whatever is the next point release 13:16:08 -xmr-pr- xiphon opened pull request #6914: [release-v0.17] wallet2: wait for propagation timeout before marking t... 13:16:09 -xmr-pr- > https://github.com/monero-project/monero/pull/6914 13:16:09 -xmr-pr- xiphon opened pull request #6913: wallet2: wait for propagation timeout before marking a tx as failed 13:16:09 -xmr-pr- > https://github.com/monero-project/monero/pull/6913 13:16:10 -xmr-pr- hyc opened pull request #6912: Allow setting start block on export [release-v0.17] 13:16:10 -xmr-pr- > https://github.com/monero-project/monero/pull/6912 13:16:15 Poor bot. :( 13:17:55 Snipa: and would be great if you merge https://github.com/monero-project/monero/pull/6907 so we can use the fix in the GUI 13:18:07 selsta: Would it perhaps be wiser to merge them after the release? 13:18:14 Mitigates the surface for errors for the CLI release 13:18:18 Which is kind of crucial 13:18:27 We can simply set the submodule to the release branch for the GUI instead of the tag 13:18:46 ^ is the only reason I'm considering not merging more, but I'm open to whatever list we want to jam together as long as we get eyes on them to make sure we're all happy :) 13:20:12 dEBRUYNE: Snipa: yep, that's okay to skip the PRs i recently sent for now 13:20:35 so we set the submodule to release-v0.17 for gui? 13:20:44 because I would like to include them in GUI release 13:20:57 Yes 13:21:05 :) Once we get final merges in for the .2 release, assuming proper reviews, I can merge the next ones in after. 13:21:09 So basically merge mooo's fix, hyc's PR, your 'prep for release' PR 13:21:11 Then we tag 13:21:15 Then merge the PRs relevant for the GUI 13:21:20 -> set submodule to release-v0.17 13:21:34 So basically merge mooo's fix, hyc's PR, your 'prep for release' PR <= Plus the 'bump out_peers' PR I guess 13:22:35 'bump out_peers' PR 13:22:41 ^ i don't see one 13:22:47 I'll make one then. 13:22:53 cool 13:23:26 s/8/12/ seems ok ? 13:23:47 ok to me 13:23:58 yes 13:24:04 Although I'm running with 25 now, but it's better not to overload the network 13:24:27 fwiw CLI would also benefit from 6913, multiple people have asked why their tx gets marked as failed 13:24:37 16? 13:25:25 out_peers seems an arbitrary number, but high values will put more pressure on nodes 13:25:41 more bandwidth, more CPU 13:26:07 selsta: I guess we could include that one then 13:26:18 Wrt value, 16 seems appropriate 13:26:53 Done, still building though. 13:30:35 Seems to work fine. 13:30:57 The assholes seem to be reporting height 0 now. 13:31:00 do seed nodes need to be updated with this patch? or is this mainly an IBD / sync bug? 13:31:08 -xmr-pr- moneromooo-monero opened pull request #6916: bump default number of connections from 8 to 12 13:31:09 -xmr-pr- > https://github.com/monero-project/monero/pull/6916 13:31:09 -xmr-pr- moneromooo-monero opened pull request #6915: bump default number of connections from 8 to 12 13:31:09 -xmr-pr- > https://github.com/monero-project/monero/pull/6915 13:31:10 Ah, back to my height. 13:31:29 If they're not on the right chain, yes. 13:31:50 Or I guess... if they're on the right chain, but not at the tip yet :) 13:32:49 out-peers should be fine with 12 for now, would not go to 16 13:33:02 sech1: I'm running in64, out64 on muh public node and it rarely breaks a sweat on the 4core VM it runs on 13:33:20 16 should be fine 13:33:22 bandwidth can be an issue with many peers 13:33:25 not CPU 13:33:42 not everyone is on 1 Gbit connection 13:34:01 we can increase it further in a later point release if necessary 13:34:11 but you can limit bandwith seperately, no? 13:34:35 bandwidth limit is separate already and can be changed in command line, yes 13:35:20 my preferred list for this release would be: 6907 6911 6912 6914 6916 13:35:39 my pub node never broke 3MBps daily avg in the past month lol 13:35:45 maybe I'm doing it wrong 13:36:23 6914 and 6912 need an approval 13:38:29 given that the only effect of raising out peers count is mitigating node complete isolation by *asshole* peers 13:38:41 setting out peers 12 instead of 8 by default 13:38:51 is 16 times better :) 13:39:11 cause you only need at least one good peer 13:39:41 IMO 12 is okay for now 13:39:47 fair enough 13:40:11 xiphon are you assuming 50% of peers are *assholes*? 13:41:00 Ok, I'll increase my node to in/out 64 to be that one good peer :D 13:41:02 i did some calculations with 50-70 and 90% of assholes 13:41:32 but i an't simulate the peer list poisoning effect in the calculations 13:41:41 ^ that's what they actively exploit 13:41:51 and that's what has a huge effect 13:44:19 sech1: in is unlimited by default 13:44:53 Hi. I switched my seed to monerujo, and when I tried to transfer my balance to another wallet, it gave me an error saying that I wasnt connected to the daemon. Then after closing the wallet and opening it again, my balance no longer appeared there and no transactions are shown. The seed is the right one since the address is the same.i switched to different wallets and nodes, but still nothing. 13:45:19 ak77: not a question for this -dev channel 13:45:31 #monero please, but try to set a good remote node, e.g. node.xmr.to 18081 13:45:57 Switched nodes, wallets, etc. Nothing 13:46:02 Ok sorry. 13:46:07 hyc: do you think we should include import / export fix in this point release? 13:46:32 it would not be necessary but you coded it already, so I guess depends on what you prefer 13:47:16 well, if no one else is testing/reviewing it, drop it 13:47:40 i can test it but would prefer if someone else reviews it 13:49:15 ok, here's an "unstuck" node python script I haven't tested yet: https://paste.debian.net/hidden/f31701d6/ 13:49:56 I reviewed. One problem, but it might be an existing one, in which case we don't actually care. 13:50:13 fun 13:50:43 sech1: actually is a good regression test 13:51:17 I put up a PSA btw -> https://www.reddit.com/r/Monero/comments/jdgnb2/psa_a_bug_has_caused_some_nodes_to_get_stuck_on/? 13:51:18 cc M5M400 13:51:19 moneromooo: oki, only thing remaining would be 6914 13:51:31 dEBRUYNE: thx 13:53:06 I can tag in the next 20 minutes if stuff is ready 13:53:57 I can sign & publish on website whenever too 13:54:34 6914 seems plausible at first glance... 13:55:02 it would be a temporary workaround until vtnerd adds something else 13:55:27 why not 17.1.1 ? 13:56:03 v0.17.1.1 would be fine for me too tbh, we would avoid ledger issues, and if someone has sync issues they will have to update anyway 13:57:03 any opinions? 13:57:13 moneromooo: ok, replied to your comment. the existing code already did that... 13:57:18 I'd use m_sent_time instead of m_timestamp. 13:57:27 OK, thnaks 13:57:42 xiphon: ^ 13:57:48 on it 13:58:26 +1 on v0.17.1.1 13:58:32 binaryFate: are you available in the evening? 13:58:37 yes 13:58:39 moneromooo> So for next fork, we might need to check outputs in handle_block_to_main_chain, which might give a speed hit. I'll have to think about this. <= I don't understand why ml/clsag is in the outputs check 13:58:55 okay, will be afk the next hours so can’t get bins ready now 13:59:01 but should have time in evening 13:59:23 That's a good point. I guess I put it there because similar chceks were there for outputs :D 13:59:28 would be great to have someone else build too 13:59:30 let's give it a few more hours to see if any patches should go in 13:59:43 6914 looks fine 13:59:52 we should probably "fix" it at some point besides just relying on network mempool state :P 14:04:34 I will change to v0.17.2.0 to v0.17.1.1 unless someone is against it 14:07:16 selsta: If we want people to upgrade, I think v0.17.2.0 sends a stronger message than v0.17.1.1 14:07:26 Hmm, 0.17.2.0 would be easier to recognize as something new. Remember how some people confused 0.17.0.1 and 0.17.1.0 14:07:28 dEBRUYNE: but people will have to update anyway 14:07:29 On the other hand, it basically locks Ledger Monero users out of their wallet again 14:07:32 if they are stuck 14:07:35 I'm not sure I follow that, people are literally unable to use it right now 14:07:36 But they'll be forced to update anyway 14:07:40 exactly 14:07:47 Yes, good point 14:07:54 I guess rbrunner's point still applies though 14:07:59 anyway, I will message ledger anyway and suggest them to remove the version lock 14:08:12 selsta: I think v0.17.1.1 should be the minimum version for v14 in the table then. 14:08:22 dEBRUYNE: if we go with that you should update your reddit post to say 0.17.1.1 14:08:41 Will repost then, yes 14:08:44 moneromooo: actually, this probably needs fixing. it looks like it won't read the entire struct 14:09:00 I thought it was only reading the length. lemme check again. 14:09:09 it's using bootstrap_serializer tho 14:12:36 Will 17.2.0 break Ledger, while 17.1.1 will not? (assuming current ledger version should still work with todays upcoming release) 14:13:02 iDunk: done 14:13:04 Inge-: yes 14:13:17 ah never mind, the parsing is all good 14:15:15 merge list: 6907 6911 6912 6914 6916 6917 14:15:17 then tag 14:16:08 -xmr-pr- selsta opened pull request #6917: README: update fork table recommended version 14:16:08 -xmr-pr- > https://github.com/monero-project/monero/pull/6917 14:19:26 So we're going with v0.17.1.1? 14:19:33 yes 14:23:06 Ok, will repost then 14:23:18 Selsta will you make a blog post for this? I assume we want to spread this as much as possible. So mailing list message as well? 14:23:20 I'll be back 14:23:31 To my laptop in an hour or so 14:24:20 I think it's ok to wait for release and binaires to be ready before sending message to mailing list. Assuming this all happen today. Opinions? 14:24:28 Please upboot -> https://www.reddit.com/r/Monero/comments/jdh5to/psa_a_bug_has_caused_some_nodes_to_get_stuck_on/ 14:24:41 yes it's less confusing to send the message after the binaries are out 14:24:44 binaryFate: Why not two messages? 14:24:44 prepare v0.17.1.1 needs review / approval https://github.com/monero-project/monero/pull/6911/ 14:24:51 binaryFate: I will not get GUI bins ready today 14:24:53 The first message can simply state that a release is following shortly 14:25:07 binaryFate: yeah sure. Better to let people know when we have the solution 14:25:08 You can copy the text from the Reddit PSA if you want 14:25:16 you can see how large exchanges such as binance and huobi had 0 preparation for this even though it was a very simple upgrade in node terms 14:25:30 selsta: I think the mailing list is mostly for exchanges/services. So as soon as we have the new release of daemon it's fine 14:25:32 I wouldn't want to make them even more confused 14:25:51 and we can just say GUI will follow shortly 14:25:54 If the release is superclose, i would go for only one message 14:25:58 ok 14:27:47 is Snipa or fluffypony around to do merges? 14:27:52 Selsta if you are busy i can take care of that. No worries. 14:28:54 Confirmed, I was able to unstuck my 0.17.1.0 node with this script: https://paste.debian.net/hidden/a94081ea/ 14:32:40 dEBRUYNE: let's see how we're doing on release later but I think it's fine to wait few more hours for mailing list message with release ready. It's not security critical and worse case people are just stuck, but most exchanges and services have deposits and withdrawals disabled anyway 14:35:26 dEBRUYNE is the best. hands up for him! 14:37:39 I'll be afk for an hour or two but available after that for bins or site update 14:42:39 binaryFate: Yeah if we are going to release in a few hours it will be okay 15:00:32 Height: 2211067/2211067 (100.0%) ??? 15:02:27 seems about right 15:02:54 word 15:05:12 nice https://paste.debian.net/hidden/f31701d6/ works for me 15:22:05 luigi1111w: I am 15:22:26 6911 and 6917 require an approval then we can merge and tag 15:22:35 checking 15:22:50 also fluffypony can you check your seed nodes, two seem down 15:23:02 which IP addresses? 15:23:05 but that is less priority now 15:23:51 https://community.xmr.to/xmr-seed-nodes 15:24:31 tks 15:25:22 Can we put out a PSA on Twitter for this? Or do we want to wait for binaries for the official account to tweet it? 15:25:42 ok 6911 and 6917 are merged 15:26:13 wait 15:26:32 merge list: 6907 6911 6912 6914 6916 6917 15:26:39 that's the full list, all should be approved 15:28:49 ok 15:40:26 ok all merged 15:41:00 also fixed seed nodes, the LANG / LC_ALL issue strikes again 15:41:27 ty 15:41:42 will check if everything is good and then ping you if we can tag 15:47:13 fluffypony: ok, looks good you can tag v0.17.1.1 15:52:51 sethsimmons: Probably best to wait until the binaries are available 15:57:56 just a note on release engineering is that it looks like tag annotations are incomplete for 0.16 and 0.17 point releases 15:58:00 also 0.15.0.5 15:58:25 not worth fixing retrospectively, but just a note moving forward 15:58:30 I think relevant to binaryFate 15:58:56 tag is pushed 15:59:04 afaik luigi adds annotations 15:59:18 ok whoever is tagging then :) 15:59:23 but maybe he forgot 15:59:25 a couple times 15:59:42 ok, I've fetched the tag and started a gitian build 15:59:51 thanks 16:00:08 what does incomplete mean 16:00:49 luigi1111: I'll pastebin 16:01:09 thx 16:01:31 https://paste.centos.org/view/raw/920e7ebe 16:01:37 that's from git tag -l -n9 16:01:46 it's just the extra bit in the annotation 16:03:11 ok got it. let me investigate when I'm home 16:03:28 Understood. 16:08:03 at least I didn't release "Wolfram Worptangent" 16:08:17 lol 16:08:30 that ones on me lol 16:08:42 guess who copy-pasted 16:08:51 we should include in the announcement "rumors that this bug was due to nation-state attackers are fake news" 16:09:10 Reverse psychology? :p 16:09:16 :D 16:09:25 LOL 16:09:34 To be honest, wouldn't be surprised if someone sent those txes on purpose to mess with the network 16:09:35 lol. i thought you were serious for a moment 16:09:37 it's due to Hunter Biden handing his Monero node in for repairs 16:10:42 lol 16:11:09 good that you restored LAW & ORDER 16:14:41 When they send their ring sigs, they're not sending their best. They're sending ring sigs that have lots of problems. They're bringing MLSAGs. 16:15:59 Is there an RCA on this? Curious to hear what caused the bug but have been out of the loop the past couple of days 16:24:26 can neither confrim nor deny 16:26:22 sethsimmons: yes the bug was simply that MLSAG txs were validly accepted into the txpool 16:26:44 and were still in the pool after v14 took effect 16:27:33 building binaries as well fwiw 16:28:37 Ah ok, so the grace period bled over via the TX pool and borked nodes 16:28:45 Odd that mine didn’t get stuck at that height and continued normally 16:28:54 Even though it was running during the fork 16:29:05 only nodes that were running during the fork kept going 16:29:21 Ahhh, I had it backwards. 16:29:21 Thanks for the info 16:30:02 or: only nodes that already had the tx in their txpool kept going 16:30:21 (2 txs) 16:48:31 do those 2 transactions actually verify? 16:48:56 Yes, I was able to unstuck my 0.17.1.0 node by manually filling mempool with TXs from block 2210720 in offline mode 16:49:18 those 2 verified fine because it wasn't v14 when they popped up 16:49:19 i see the patch just effectively says "txs with these hashes ok". they are in a block, so presumably the miner did verify, but i know things can get noodly 16:49:48 so they were verified at v13 "era", but included in the first v14 block 16:50:08 I rather it just allow any MLSAG TX in the first v14 block, personally. That's a philosophical stance though. 16:50:20 ok. makes sense. how'd you manually fill? 16:50:27 yeah kayabaNerve i agree 16:50:35 I pasted a python script here before 16:50:44 https://paste.debian.net/hidden/a94081ea/ 16:50:48 gingeropolous: https://web.getmonero.org/resources/developer-guides/daemon-rpc.html#send_raw_transaction 16:51:08 write a different patch yo 16:51:38 Or yeah, what sech1 posted 16:51:50 lets watch the sparkles sparkle 16:52:12 are we having a dev metting today? it would be in about 8 minutes if so 16:52:16 meeting 16:54:57 kayabaNerve: the difference is mostly academic. Unless someone mines a longer chain and causes the current main chain to rollback all that way 16:55:42 otherwise, it's a situation that will never occur again, so makes no difference 16:57:48 off the cuff it feels like "devs say tx ok" vs "previous protocol says tx ok" 16:58:46 I prefer it as is. Mistakes should be transparent. 17:01:08 hyc do you want to kick-off the meeting? 17:03:16 TheCharlatan: We can easily change the existing if statement and leave the same commentary to explain the history behind that exclusion. Then there's no need to have the Monero node specially recognize any transaction. 17:03:25 But yes, it's purely academic/philosophical. 17:04:13 +1 kayabaNerve's suggestion. also, this may be a dumb Q, but if somebody submitted a malformed transaction with the with the excepted hashes would, would they be allowed? 17:04:35 be the current fix that is 17:04:35 No. 17:04:46 cool 17:05:27 But if you can brute force keccak, you might be able to brute force something else that make a bad tx valid. 17:05:34 so we are meeting now? 17:07:02 no meta issue, no agenda 17:07:41 If people have things to say, they should feel free. 17:08:50 I guess the conv about the bug & patch can just continue then 17:10:43 "Unless someone mines a longer chain and causes the current main chain to rollback all that way" <-- then these 2 TXs wouldn't even appear again and the patch would just be dead code 17:11:07 other MLSAG TXs could appear in 2210720 though... 17:11:44 if it's a malicious attacker who has 51% and knows the details of this bug 17:11:58 but no 17:12:11 0.17.1.1 wouldn't accept new 2210720 block with different MLSAG TXs 17:12:20 so all good 17:17:44 btw, thanks everybody for working on the issue so quickly. it's really amazing to see the quick reaction and the amount of people working together to fix this issue.. and on a sunday. You are the engine that makes this boat going. Thank you really much, folks. Keep being awesome 17:18:41 moneromooo I don't understand your hesitation for calling check_tx_ouputs from handle_block_to_main_chain 17:18:51 seems cheap enough to me. 17:19:34 https://paste.debian.net/hidden/27ac436d/ 17:19:56 then we'd be all stuck at 2210720 - miners would just keep adding these 2 TXs and all mined blocks would be rejected 17:21:38 iDunk can you upload to mega? 17:22:21 I hesitate because it mgiht be too slow. 17:22:26 I always understood that the grace period of one day was to allow older version txes that are already in the txpool to get mined, not to still allow them in the txpool. Otherwise, the grace period serves no purpose. 17:22:47 yes, that was an oversight 17:23:05 selsta: I will soon. Can't right now. 17:26:01 I looked now, and it seems fast. I thought it'd do range proofs. 17:26:23 A better way might be to revalidate the txpool on fork switch. 17:26:30 That should catch everything. 17:27:09 yeah that sounds better overall 17:27:24 delete txs from pool if they're obsolete 17:27:40 one-time computation cost instead of on every block 17:28:04 Though that'll be a bit annoying, since if we at the block before a fork, the previous rules apply, but after the block gets mined, the new rules will apply to check it. 17:28:20 So some work will have to be done there. 17:29:32 do we know which pool mined the block? 17:30:36 then expire the txs when block N-1 has been mined 17:31:05 it was a 24hr grace period. those txs shouldn't succeed anyway. 17:32:31 gingeropolous: I don't think the miner/pool was malicious. They just had the txs in their txpool too, right? 17:37:11 probably 17:37:41 so instead of a tx flooding attack, we have a dummy node flooding attack? and that's why number of outpeers is bumping from 8 to 12? 17:40:42 in which case, if I was running the dummy nodes, I increase my node count by 50% 17:45:30 right 17:45:52 so they were verified at v13 "era", but included in the first v14 block <-- following up on gingeropolous's concern, how would future users know these transactions were not blindly whitelisted? 17:46:58 I think the if checking for those hashes is only in the path where it's an MLSAG post-fork, so the only thing it's blindly whitelisting is allowing an MLSAG post-fork 17:48:27 it wouldn't allow a double spend or bad range proof, even if those hashes corresponded to one (which they don't) 17:48:54 Ok I guess I'm not clear on the grandfathering thingy 17:49:49 the "whitelisting" is not actually whitelisting, it only allow MLSAG format for these 2 tx after v14, that's all it does 17:50:02 everything else should be checked as usual 17:51:08 ok 18:02:02 TheCharlatan: you're merging the gitian PR usually right? 18:02:43 yes, but there are only my sigs posted vor v0.17.1.1 right now. 18:03:31 I'll be at my PC in about 20 minutes. 18:06:36 my build just finished 18:08:04 http://paste.debian.net/1167717/ 18:08:38 if those hashes are good, I'll submt sigs shortly 18:10:24 iDunk posted https://paste.debian.net/hidden/27ac436d/ earlier. 18:12:41 ah ok 18:12:45 didn't see it 18:13:13 cool, matched 18:17:31 I thought it was a shit Sunday but then I discovered this repeated Worptangent tag 18:24:26 lol 18:24:39 could've been Worf Tangent 18:51:27 hyc can you PR your sigs soonish? 18:52:14 binaryFate hyc's and iDunk's are merged. I don't merge my own without at least one approval. 18:53:08 nice sorry I missed hyc's one. Please check scoobybejesus's one I think it's just fixed 18:53:16 I will not do reproducible builds today, busy with release notes and GUI later 18:53:35 selsta: ok 18:53:44 please feedback for release notes: https://paste.debian.net/hidden/4e961fb6/ 18:54:03 binaryFate, yes merged his as well. 18:56:32 "Allow setting start block on export" maybe say monero-blockchain-export to avoid confusion 18:56:51 ok 19:00:53 I would add a notice in bold to make people understand the it's important that they upgrade or they could have issues 19:01:12 selsta: I take it you took the hashes from someone and I do not assume you built yourself right? 19:01:32 Not that it changes anything for release, just for clarity 19:02:01 I took them from hyc, though they should match the ones posted on gitian repo 19:02:16 oki 19:04:09 should we also include a more detailed explanation of the bug? 19:04:44 I like sech1's explanation: https://www.reddit.com/r/Monero/comments/jdh5to/psa_a_bug_has_caused_some_nodes_to_get_stuck_on/g98nuwl/ 19:05:10 this is only my understanding, I might miss some details 19:22:16 if someone uploaded the binaries please ping me 19:24:04 I'm on it, takes a while 20:03:20 is this 0.17.2? oh i guess its already tagged. this version is now the only one that will sync right? 20:04:11 well, not like the version numbering is tied to consensus i guess 20:05:23 make sure to use release-v0.17 branch and not master 20:05:27 master is kinda behind 20:05:42 gingeropolous: no it's 0.17.1.1 20:06:28 right. in my head i've always wanted release # 20:06:38 's to somehow be related to consensus compatibility 20:06:46 but thats not the way it is so .... potatoes 20:08:52 what i don't understand is why all my node are fine, but if i were to sync then they wouldn't make it past the block 20:09:44 the only way to pass that block is to be in a state where those 2 tws are in your mempool 20:10:21 ah. so the daemon already verified in the mempool, so when it saw thme in the block it went "ok, i saw these before, all good" ? 20:10:25 if you sync you don't have a mempool with these 2 txs, so you're going to check them. And find out they're "illegal" 20:10:38 yeah exactly 20:11:03 optimization strikes again! 20:11:18 seems like related to fluffy blocks then 20:11:38 yeah 20:11:47 but mebbe it would have happened regardless 20:12:13 could try. can run monerod with --no-fluffy-blocks still i think 20:15:24 mm I think fluffy blocks are about txs (non) propagation, but what the node decides to check/recheck/not check is independent 20:17:35 it's logic not to check a transaction twice when neither the transaction nor the protocol have changed. 20:18:27 But it fails when protocol changes in between 20:20:28 ^ that's the best concise explanation I have heard today 21:18:55 dEBRUYNE: bins will be out in ~10mn if you want to post on reddit 21:22:39 binaryFate: Here now, bins are up right? 21:22:50 no but within few minutes 21:24:32 Oki 21:25:34 dEBRUYNE: though bins should be up on the downloads server 21:31:41 dEBRUYNE: ok website updated 21:31:48 great job everyone 21:32:01 I'll send a message on list 21:32:37 good to update my node then? 21:32:56 Thanks for all the hard work tracking this down and fixing! 21:32:57 what a day 21:33:15 could have been worse like chain split or so lol 21:34:26 some solo miner could've synced to 2210720 and mined another block... 21:36:39 indeed lesson could have been way more painful 21:39:11 selsta, binaryFate: Thanks, will create the thread in a bit 21:50:58 Can anyone connect to the node at 2a01:4f8:c0c:7b27::1 ? You need to have your machine setup for IPv6, which mine isn't. 21:51:18 Or at least I think it isn't :D 21:53:44 (P2P) 21:56:01 wow - getmonero.org downloads are slow as **** 21:56:17 usually there is a CDN 21:56:29 Release thread is up on Reddit -> https://www.reddit.com/r/Monero/comments/jdp770/cli_v01711_oxygen_orion_released_fixes_block/? 21:56:33 moneromooo: was not able to connect to it 21:56:38 Can't Download Nothing ? 21:56:56 And can you connect to IPv6 addresses from the same machine ? 21:56:56 added it as peer with --enable-ipv6, it showed as before_handshake and then vanished 21:57:06 OK, thanks. 21:57:12 Wait, not conclusive. 21:57:19 and with exclusive_peer it did not work 21:57:55 yippie my backup node is syncing 22:07:55 Pretty cool though, I've got both in and our tor p2p connections. 22:07:58 * moneromooo likes 22:10:34 selsta: would you mind trying again please ? 22:14:04 "before_handshake" again 22:14:19 started with ./monerod --p2p-use-ipv6 --add-exclusive-node 2a01:4f8:c0c:7b27::1 22:14:39 Hrm. OK, thanks. 22:15:07 https://test-ipv6.com shows my ipv6 ip 22:15:10 should that be enough? 22:15:49 I don't know anything about IPv6 I'm afraid :/ 22:16:12 The node's showing as listening on the v6 at the right port though. 22:16:33 Oh, there might be a firewall. Let's dig a bit... 22:22:02 moneromooo: any tutorial for setting up tor node with incoming connections? 22:22:39 ah it is in the readme 22:22:40 I did what's in ANONYMITY_NETWORKS.md 22:22:46 random thought: when we get a dandelion time out, could we drop the node that was selected to broadcast the tx as likely an asshole? 22:23:00 (and I managed to miss the part that the new onion address is auto generated in /var/lib/tor) 22:23:13 or maybe it's too harsh because the time out might not be because of the first peer 22:23:17 It might be any node along the way. 22:24:12 damn them 22:34:36 moneromooo: is there a known tor peer? 22:35:33 zbjkbsxc5munw3qusl7j2hpcmikhqocdf4pqhnhtpzw5nt5jrmofptid.onion:18083 22:41:43 cool now I have both i2p and tor on one node 22:45:32 moneromooo: the only issue we noticed with i2p / tor that peers seem to drop after a while, not sure if you noticed yet 22:46:30 I did not, but that's because I do not look. 22:47:18 I have 10 tor peers now (in / out) 22:47:24 will check later if they drop 23:14:47 At what point do Tor / I2P transactions get added to clearnet mempool? 23:15:00 Do they also go through hops? 23:16:05 AFAIK the transaction gets sent to 2 peers but don’t know further.