01:09:04 thanks ill try that 03:07:10 what autodetector? 04:02:42 hyc, if you run 'start_mining xmr_ADDRESS auto' in the daemon, it will cycle through a number of threads until it finds the optimal # to get the highest hashrate 04:02:58 optimal # of threads 04:44:32 I got a quick question about one of the unit tests for bulletproofs 04:45:45 the "invalid_gamma_0" test seems to also use an invalid amount (from the previous test). why? what are we testing? 04:45:56 gingeropolous: ah cool. a bit futile in the daemon though, since "optimal" will be impacted by other daemon activity 04:46:40 every other node and wallet interaction will consume CPU+memory, you really can't expect anything like optimal mining efficiency in an active daemon 05:57:28 hello 05:58:47 we’re considering making an auto-output-splitter in cake wallet - is there any danger to using transactions larger than 1in/2out? 06:00:35 we want to avoid an “early 2018 xwallet” situation where certain transactions stick out, but also don’t want to clog the blockchain with churn transactions 07:34:47 skrech: No default location for CLI config file. I write small shell scripts and batchfiles for such cases to avoid too much typing 07:34:59 Ups, wrong channel 10:03:29 Could somebody give me a screenshot of the CLI making a stagenet/testnet transaction? From transfer... to the message of success. 10:05:05 I tried to do it myself. I made the testnet transaction from the faucet and i mined it , but now i have to wait 58 blocks to unlock the balance and seems like i'm the only one transacting on stagenet atm 10:31:37 https://paste.fedoraproject.org/paste/LTHtA4n7Z-eX14dKzCYO0g 10:32:27 ErCiccione[m]: Like so? 10:34:59 rbrunner: yes, something like that, but with included the messages you get when you open the CLI 10:36:06 Is this for some kind of manual then? 10:36:29 it's for the download page of getmonero 10:36:41 this is what i made, but it's ugly AF: 10:36:42 https://imgur.com/a/3dibAg9 10:37:20 would be cool to have something like that, but with a transfer at the bottom 10:38:10 Hmmm, if it's *that* public maybe put some effort into it ... 10:40:41 Also a simple paste would not be nice because not preserving color 10:41:06 ? That is something i just made to show what i would like to have. 10:41:16 it shouldn't be a paste, but a screenshot 10:41:23 Yeah, that's what I mean 10:41:47 And I meant *me* to put some effort into it ... 10:42:14 ah ok. I misunderstood :) 10:48:16 moneromooo gingeropolous xmrchain.net is stuck and can't sync, this is how it's seen from other nodes: https://pastebin.com/raw/0dZikAZF 10:51:05 How about this: https://i.imgur.com/aeSdsaZ.png 10:52:00 2019-12-01 10:51:55.563 [P2P3] ERROR verify src/cryptonote_core/blockchain.cpp:1849 Block recognized as orphaned and rejected, id = <8f3ca1541334c185029047fc57d710fea4f7c9ba6da3c4512c05ff66246656b1>, height 1979078, parent in alt 0, parent in main 0 (parent <0d9fbbfb4db65c4b8978ce007b3ef5f34cfc28b288d795378902874536e4d1b4>, current top <25ee250bb678088722562c1fdf6f0014e62f8ae891b741769620f1ef7f666e61>, chain height 1979077) 10:52:05 Is the main issue it appears. 10:52:14 That's perfect. Thanks a lot rbrunner 10:52:19 The only detail that is not correct is the exact version number of the daemon, it's not proper release ... 10:52:29 There's a number of nodes with the same error. 10:54:21 rbrunner: not a big deal. IMO the point is to show how the cli and a transaction look like 10:54:38 Ok then. Glad to be of help. 10:56:27 jsut one detail rbrunner. Could you take another screenshot, but square this time? This one is rectangular and i would have to manipulate it a bit to show properly. 10:58:00 To clarfy: The GUI screenshot is 888x878. Your CLI screenshot is 1018x630 11:09:14 Looks like: https://pastebin.com/raw/9T0SyymW is the source of the auctual issue. 11:09:31 Daemon version: 0.15.0.1-6def88ad4 11:10:11 The remote listed as IP:PORT has blocks available out to 1979169, so it's clearly validated in their daemon, but the daemon on xmrchain isn't accepting it. 11:11:20 ErCiccione[m]: Ok will do. Perhaps in half an hour or so 11:11:57 may need to resync. would be nice to know how long ago it went bad 11:12:10 block 77 11:12:22 Well, 1979077 is the dead block. 11:12:22 rbrunner: thanks! 11:12:28 Popping off 500+ blocks hasn't helped so far. 11:12:40 Also, there's almost 2 dozen nodes effected already by whatever the heck this is. 11:13:24 Same block on blox. using the same monero version according to the reporter: https://blox.minexmr.com/search?value=0d9fbbfb4db65c4b8978ce007b3ef5f34cfc28b288d795378902874536e4d1b4 11:14:43 hm.not 15.0.1-release? 11:14:58 Apparently not. 11:16:50 the commit ID is the same as the 15.0.1 tag though 11:17:01 so I can't imagine it's missing any fixes 11:17:41 Unlikely would be my suspicion in that case. It was clearly compiled, I've found the traces from Ginger's build script. 11:17:58 Theres 22 nodes effected at the moment based on the logs I'm seeing. 11:19:37 MDB_BAD_TXN is a runtime / memory error, shouldn't persist after a restart 11:20:01 That's the 7th restart so far. 11:20:10 but it means there was an earlier error that monerod didn't respond to 11:20:36 i.e., there was an earlier error and monerod ought ot have aborted the TXN at that point but kept on using the DB txn instead 11:21:41 xmrchain.net runs 0.15.0.1-6def88ad4, so not a release binary 11:21:47 Wellp, I'll start packaging an export from a known-good node. 11:21:52 maybe something borked during compilation and broke now 11:22:04 Interesting. There's a segfault during block popping. 11:23:35 but 6def88ad4 is the same as the v0.15.0.1 tag, so any difference from -release must be in its dependencies and not monero code itself 11:23:53 Block ID: 1973151 causes the pop-blocks to brick on that version. 11:24:13 can you let me get a copy of the DB to examine? 11:24:21 Yeah, I'll export it here in a sec. 11:24:42 blockchain-export may crash too if it's actually corrupted 11:24:42 Well, once I find somewhere to stick it. 11:24:54 Yeah, I'm just going to gzip the lmdb file 11:25:26 ok 11:48:38 https://i.imgur.com/80putO4.png 11:48:53 ErCiccione[m]: Square, 882x882 11:49:46 Awesome. Thanks again rbrunner 11:50:11 You are welcome :) 11:58:10 Testnet without a new block since 2.5 hours. 10 blocks in the last 10 hours. Looks to me that somebody drove up difficulty (or held it up) and then left, about 16 hours ago. 12:00:00 I mean, people were on testnet testing miners, then the fork happened. 12:00:41 yeah, why waste power on testnet coins after you test successfully and mainnet switched 12:00:57 Snipa: "set_log 1", flush_cache bad-txs", then paste the resulting errors on verification. 12:01:09 I know the sxmr testnet pool had 300+kh/s pre-fork. 12:01:12 it's just everyone used testnet for testing (what a surprise), so it's stuck at 1+ MH/s now 12:01:43 moneromooo - I'll hand it off to ginger with those notes, it's 4 AM here, and I'm already up way too late. 12:01:51 Makes sense. Not nice anyway. Will take a while until becoming usable again 12:02:23 Just throw a couple miners on there and let em chug, not much worse than a normal network fork. 12:03:19 Yeah, my old notebook with 1/1000th of that 1 MH/s :) That will show it. 12:03:31 No real problem of course. 12:04:07 Make that 1/10,000 ... 12:26:22 no problem, diff will adjust in about 10 years :P 12:26:55 tevador: As a side note, do you have a current estimated hashrate for mainnet? 12:27:37 I think sech1 has a script to estimate it, I'd say 800 MH/s 12:27:55 Guess it is going to take a few days before more hash joins in 12:28:02 Because at 800 MH/s it should be quite profitable no? 12:30:37 XMR network hashrate 60 : 672.212 MH/s 12:30:37 XMR network hashrate 120: 661.608 MH/s 12:30:38 XMR network hashrate 360: 678.075 MH/s 12:30:39 right now 12:30:48 so it went down a bit 14:24:22 ok, im here. what can i do? 14:25:18 fix xmrchain.net? 14:25:43 it doesn't use official release binaries and got stuck today 14:27:17 well yeah i compiled the tag. it looks like they were trying to debug it 14:27:30 moneromooo, hyc , did Snipa get you everything you need? 14:31:48 No. Said they'd leave you to it :) 14:31:56 "set_log 1", flush_cache bad-txs", then paste the resulting errors on verification. 14:32:00 ok, well what do you need? :) oh 14:32:08 well data.mdb is packed up in a .gz now.... 14:32:23 i guess i could unzip it and try and run it. 14:32:38 well, lemme get that file to a server for dload 14:34:32 Builds on some systems do not display -release since commit hash in version was expanded to 9 digits, instead they display the 9 digit hash even though it's a tagged build. 14:36:28 ermagerd this things on a spinny. everythings gonna take forever 14:40:44 ok, the sha256sum monerod bin in /bin/ matches the bin in /monero/build/release/bin . So its possible its on master and not release 14:53:27 this is worse than watching paint dry 14:54:47 what are you watching? 14:54:55 is that data.mdb downloadable yet? 14:55:15 moving this file around . 14:57:44 question - is the whole mdb needed for this kind of debugging? or could one slice it up somehow? this sort of slicing could be useful for debugging in the future when data.mdb is 9 gajillion terrabytes 14:59:03 well, if there are corrupted pages in the file then yes, we need to see the entire file to trace the corruption. 14:59:45 if the file was normal/non-corrupted, then yes, it's possible to use mdb_copy -c to only copy the most recent data. 15:00:24 gotcha 15:01:57 so the file is downloadable, but moneromooo wants me to try and run the daemon on it with flush_cache. so, i'm copying the file to a different location on snipas server. hopefully i can prop a temporary http so it canbe downloaded straight from that server. i might need to transfer it to my server if that doesn't work. 15:02:32 ok 15:02:39 you're moving the .gz now? 15:02:46 yeah. copying. 15:35:28 hyc, i pmd you the dload server 15:35:49 moneromooo, u as well. 15:37:03 sech1: looks like it has plateaued for now? 15:42:12 Estimated hashrate is 730-750 MH/s now 16:11:00 well i can't unzip this data.mdb until i get the other copy off this server 16:11:07 damn you limited HD space 16:30:06 anyone downloading that file? ima kill that server 16:44:58 hi - we’re considering making an auto-output-splitter in cake wallet (to mitigate the 20minute lock concern) - is there any danger to using transactions larger than 1in/2out? we want to improve UX while avoiding an “early 2018 xwallet” situation where certain transactions stick out. but we don’t want to clog the blockchain with churn transactions (which could happen if we only use 1/2) 16:56:15 No danger per se. Maybe best to make a x->16 tx to blend in with pools ? 16:56:37 Is "early 2018 xwallet" the people wanting to always have 3 outputs ? 16:57:08 ok if anyone wants that mdb its on http://66.85.74.134:8964/ now 16:57:18 Yes. They paid some wallet usage fee to themselves with the third output, also to gain wallet use statistics 16:57:45 One of the worst confidence-damaging moves I ever saw in Monero wallet space 16:58:11 hyc, ^ 16:58:20 well, about four 3 ^ 16:59:37 ok here goes nothing. all we need is "set_log 1", flush_cache bad-txs" right? 17:00:27 Yes. 17:01:09 well it seems snipa tried to pop blocks? 17:02:18 < Snipa> Popping off 500+ blocks hasn't helped so far. 17:02:19 well set_log 1 is going bonkers. couldn't see output of flush_cache bad-txs. tried to run it from other screen 17:02:41 It won't output anything IIRC. It'll just allow the tx to be verified again. 17:03:13 ok 17:06:17 hi. Sorry for being late. Did someone start a meeting? 17:06:22 No. 17:06:33 ok, ima swith to the 15.0.1 i compiled 17:06:54 Can we have one? It won't be long. :) 17:06:59 Or are you guys in the middle of something? 17:07:07 Sure, go ahead if you want. 17:08:18 Ok. Well just a few things to discuss, I guess. 17:08:22 hey, on the road atm. will check in again soon 17:09:16 1. How'd you think the release went? Anything of importance to discuss about it? 17:10:07 apparently some exchanges didn't update in time... ? 17:10:14 Snipa reported a couple dozen stuck nodes. I assume they're all his. Not sure whether it's related yet. 17:11:27 xmrchain.net node went down, but its syncronizing fine now 17:12:10 well, this is talking about the fork itself. Talking about the release. 17:12:18 But if there are things to note about the fork, we can discuss those too. 17:14:06 Nothing of note. That's good. :D 17:14:17 2. What's planned for the immediate future? 17:16:09 Merging stuff to master again :) 17:16:50 I guess some people have been batting back and forth the idea of moving to yearly releases from here on out. 17:16:53 trying to break rpc-payments is high on my list 17:16:56 But that needs to be discussed with the community at large. 17:17:36 GUI will do a point release in the next days. I guess CLI does not need one? 17:18:47 Not unless gingeropolous finds something that needs fixing with this sync issue. 17:18:49 selsta what does the point release do? 17:18:54 bug fixes 17:19:05 also compile Linux GUI with Trezor support 17:19:42 I'm definitely keen on annual releases 17:19:53 I think it's time 17:20:25 annual forks or releases? 17:22:14 fluffypony? 17:23:09 already? 17:24:05 I think we can aim for annual network upgrades in the future but let’s say we have CLSAG ready in 6 months I’m not sure if we should wait 6 months to deploy it. 17:24:18 But (hopefully) no more PoW changes. 17:24:57 I mean, there are a lot of big upgrades being worked on. 17:25:02 perhaps an "as needed" makes sense? 17:25:06 0.15 actually took 9 months since 0.14 17:25:13 it doesn't HAVE to have a set time frame, yeah? 17:25:15 maybe 9 months until the next release? 17:27:01 ErCiccione[m] what other big upgrades are being worked on? 17:27:08 9 months sounds good as a target IMO. 17:27:25 CLSAG. Sublinear signatures. 17:28:50 Ok. So this is something that can obviously be discussed further, but if there are any other opinions on this topic please feel free to speak up. 17:28:54 If you mean also non fork related: dandelion 17:29:27 Well, my opinion is that yearly feels long, but I'm also not the one doing the work so I'm trying to shut up. 17:29:27 moneromooo is there any way to enforce dandelion on a protocol level? Liek is that even possible? 17:29:55 None that come to mind. 17:31:13 CLSAG's been ready for a while, waiting for reviews. If it misses a release, waiting one more year feels harsh. 17:31:24 Alright. Anything else to discuss surrounding the release/fork? 17:31:33 You guys think everything went pretty smoothly this time, all things considered? 17:33:46 fork went smooth, no last minute binaries and also no issue with slow blocks this time 17:36:04 Yeah. A lot of issues from the past went fine this time around. 17:36:24 Alright, I won't hold you guys any longer. Thanks! We can break here. Just wanted to do an after-fork check in. 17:39:47 (test) 17:40:30 test 17:40:37 -=test=- 17:41:22 sorry. I think my matrix handle was relaying the irc chat in a messy order and late. No idea what's happening. Is the meeting over? 17:41:33 I think so. 17:42:08 Did you need a paste of what got said after your last line ? 17:42:28 Alright, then i lost all the conversation and i have to wait my matrix user to catch up -_- 17:42:34 that would be appreciated, yes 17:48:42 rehrar: the upgrades i was talking about were the ones mentioned by mooo. I too think would be a bit a pain to wait that long before they go live 17:51:53 I think annual fork is good. I guess that coincides with major releases but I don't know that it would have to 17:52:43 It has to, if people have to have code that's ready for the fork. Unless you're planning forks a year ahead, but that seems very unlikely. 17:53:10 luigi1111 not a fan of the 9 mo proporsal? 17:54:12 I was thinking every major release 17:54:25 like there could be non forking ones 17:54:42 9 mo is ok as a stepping stone I suppose 18:07:47 sorry meant annual forks 18:07:53 not annual releases lol 18:08:11 selsta: we definitely SHOULD wait 6 months to deploy it 18:08:20 it needs extensive testing on testnet + an audit or 2 or 3 18:08:51 right that’s not what I meant :P 18:09:32 ready = tested and audited 18:09:51 also some of this stuff can be deployed as soft forks 18:09:52 not CLSAG per se, but other things 18:10:28 selsta: we've traditionally taken a very short route with a lot of this stuff, I'd personally like to encourage a little bit more maturity of something like that before it goes live 18:10:53 I’m not even against 1 year forks 18:11:19 I think the added 1-2 month delay this fork was really important 18:12:30 whats the bang for buck for clsag? 18:14:44 hrm, xmrchain node is stuck on 1979077/1979413 again 18:15:56 ok moneromooo , im trying flush_cache bad-txs again 18:16:13 CLSAG saves maybe 25% space on average transactions, and 15-20% verification time 18:16:55 is the plan to bump up ringsize to effectively nullify, or we stickin with 11? 18:18:30 Depends what the consensus is 18:19:09 Would be a very small increase 18:19:51 is the 15.0.1 bind only available from getmonero.org? 18:20:19 I'd rather bump the ring size only if there's a convincing argument that it increases privacy more than epsilon. And I feel weird saying that given I like privacy all around... 18:21:25 gingeropolous: binary? yes 18:21:36 Perhaps it would be better to bump it in conjunction with a switch to a sublinear ring signature scheme (when that becomes feasible to implement) 18:23:22 ok moneromooo , ima try just using the release bins on the explorer node, unless you can think of something else i should try 18:23:53 Well, I just want to see the verification logs when it verifies the tx (because it's 99% going to be a tx that fails to verify). 18:24:23 well i ran that command. is it in the log files ... yeah.. ok i guess i gotta grep 18:24:59 any what the string would be/? 18:25:02 Yes. "verify" category, most likely. 18:25:22 Might be other interesting stuff a bit before/after too. 18:26:42 Silly question, but you didn't write bad-txs verbatim, right ? 18:26:55 I did. 18:27:13 I am personally in favor of 9-12 month schedule too fwiw 18:27:25 There's only one cache atm, but I left it open to have multiple caches that you'd want to differentiate. 18:27:28 With respect to CLSAG, I think it's better to fit the fork to CLSAG than the other way around 18:28:04 fluffypony: In a scenario where we would have annual HFs, would you still advocate for having a major release every 6 months? 18:28:16 To avoid confusion I guess we shouldn't bump the number then 18:28:39 Ah, I misread flush_cache for flush_txpool. 18:28:41 moneromooo its on xmrchain.net:8964 18:29:39 its 1e6 lines of log 18:29:56 ty 18:31:23 At what time did you run flush_cache ? 18:31:46 13:15:56 according to irc logs 18:31:50 Oh, there's a log showing it, nvm. 18:31:50 around there 18:32:06 u got the file? ima kill server 18:32:42 must've got it 18:32:48 dEBRUYNE: sure, or even more often 18:32:52 Yes, I have it, thanks. 18:33:43 And the tx doesn't get re-verified. Hrm. 18:34:20 selsta: we definitely SHOULD wait 6 months to deploy it <= Transaction format also changes as far as I know, which means that third party wallets also have to perform work to ensure their wallets remain compatible 18:34:26 So we need to accomodate some time for that 18:34:33 yeah 18:34:53 good points :P 18:36:46 and how far out are the super-mega-awesome protocols that'll get us 1200 ringsize at 1 kb transactions? 18:37:30 Both Triptych and RCT 3.0's numbers looked quite feasible (as far as I could see) for ring size 128 18:37:33 which may also require ecosystem-wide tx format updated etc? 18:37:33 release should be whenever there's enough new stuff to make sense. some of the time that would include a fork. at least that's how i see it 18:37:40 Afaik, multisig has not been figured out though 18:38:51 i guess my point is that if triptych/rct3 is gonna happen, is it worth doing this twice in such close succession 18:39:09 * moneromooo agrees with luigi1111 18:39:24 Multisig is indeed unsolved with the sublinear ideas 18:39:33 Due to a key inversion step 18:39:57 moneromooo, any ideas for xmrchain node? or should i just try the release binary? 18:40:43 I don't see a reason for the failure so far. 18:41:07 ok. well the data.mdb is copied for further research, so ima try release bins 18:45:51 well that didn't do anything 18:46:24 Looks like it's missing a block between your current top and the one it's trying to add. 18:47:15 So it's a tx verification problem then. Would a level "2,*throttle*:ERROR" log be possible ? ^_^ 18:47:25 it's *not* a tx verification problem 18:49:35 Fwiw the Triptych preprint is nearly complete 18:49:48 Finishing up the intro narrative and citations 18:50:16 moneromooo, files up on same port 18:51:51 Got it. 18:54:20 It's skipping some block because it was reported as invalid earlier. So I'd need a log with "2,*throttle*:ERROR" from the start. 18:54:45 ie, monerod --log-level "2,*throttle*:ERROR" --whatever-else-you-use 18:55:08 And I'll add a flush_cache for that block cache too :) 18:55:29 ah ok 18:56:35 ok its running with the new log level 18:58:34 i flushed and its up on the same port 18:58:44 moneromooo, ^ 18:58:58 Got it, thanks. 18:59:22 no, thank YOU!!!!!!!!!!!!!!!! 19:00:55 Aha: Failed to query m_blocks: MDB_BAD_TXN: Transaction must abort, has a child, or is invalid 19:01:09 Looks like we get that one a lot these days. 19:01:15 bad txn. bad bad 19:01:26 Yeah, I linked the blockID it appeared to be in last night before I passed out. 19:01:32 As it broke the pop-blocks tool. 19:02:00 Causes a real pretty segfault when you hit it. 19:02:19 Oh, I might want to look at the stack trace for that. 19:02:58 On the MDB file, assuming you snagged a copy, block 1973151 causes the segfault. 19:03:09 So any pop-blocks that tries to get past that block segfaults immedately 19:03:58 so what should i do to get this explorer working for the people again 19:04:04 resync 19:04:24 Or the more sensible answer: I'll copy in a clean MDB from a known good server. 19:04:28 Because that's way faster. 19:04:54 As we can't remove the bad block, we'd be resyncing from scratch. 19:05:08 any more debug etc needed from this mdb moneromooo hyc ? as Snipa mentioned, the data.mdb can be get gotten from my backup 19:05:29 How large ? 19:05:31 I'm still uncompressing the file 19:05:45 ok, u gonna take over Snipa with moving a new data.mdb? 19:06:04 I'm going to start the copy from one of sxmr's nodes in a sec, they're in the same DC so it should be fast. 19:06:17 nice. 19:06:33 i'll stop doin stuff on the server 19:07:10 https://paste.debian.net/hidden/a92477d4/ will tell me where that error is coming from. It'll log on stderr. 19:12:36 moneromooo, u want us to run that on the existing data.mdb? 19:14:00 Yes. 19:15:22 Snipa, u seein this? 19:15:48 Yeah, it's a .patch that needs to be applied to the source I'd assume and monerod recompiled. 19:16:04 I'm not entirely here at the moment, so not super useful, just moving data for the most part. 19:16:20 yea yeah, i just wanna make sure you don't delete data.mdb 19:16:33 Nah, I'm copying to my home dir, then we can move it over again. 19:16:37 cause i don't think the server has enough room for ... ok 19:17:09 There's /just/ enough room for a second copy right now. 19:59:02 ok moneromooo what should i do with this patched bin? 20:01:05 Run it till it fails to add this block, and paste stderr where it should tell you the line at which it returned MDB_BAD_TXN. 20:02:08 ah, no special log level 20:02:50 and ther eit is 20:03:10 https://paste.fedoraproject.org/paste/Syy7M-ztQRPLq10KCtYX4A 20:04:25 moneromooo, ^ 20:06:33 Got it. I'll have another patch to add more info where it reported. 20:15:44 so far not looking like there's any corruption in the DB 20:16:09 https://paste.debian.net/hidden/e44bb798/ (replaces the previous one) 20:38:30 can someone throw an estimate of the removal of integrated payment IDs from monero, if they are going to be removed at all? 20:39:18 i guess i would prefer them not to be removed since i have a project revolving around them but if there is a legit reason for their removal i obviously don't mind 20:39:34 There are no plans set for removal yet. 20:40:47 good enough for me 21:05:15 wow moneromooo , that errord out immediately 21:05:34 https://paste.fedoraproject.org/paste/mFAKzWBkiHMITTp7Zw3yyQ 21:06:54 still scanning the data.mdb file here, but still no errors found 21:08:51 Thanks. Seems unlikely it's any race if it fails at startup. 21:10:40 I'll have to leave it to hyc now, since what sets the error flag needs knowing lmdb internals. 21:12:55 ok lessee where that tripped... 21:13:43 hang on moneromooo , i was running that patched bin on snipas new copied database 21:15:15 here's the patched bin on origina mdb https://paste.fedoraproject.org/paste/S~LNSTohbuKR7aCrFFApTQ 21:16:05 that's a bit weird since it's saying MDB_TXN_ERROR is set 21:16:11 but that only gets set in write txns 21:16:43 and line 10758 is an mdb_stat call, which is most likely in a read txn 21:16:58 probably would be more useful to get a stacktrace when that event occurs 21:17:59 I think it's a height() call. 21:18:17 But the one I put logs around doesn't show so I'm not sure. 21:18:36 might as well just use an assert or set a gdb breakpoint 21:19:03 anyway, that most likely means the MDB_TXN_ERROR flag was set before the height() call 21:19:16 so we're seeing the result of an ignored error, long after the fact 21:19:34 Right. But looking at what sets the error flag in mdb.c, I'm out of my depth :) 21:19:55 Alloc is unlikely, after that it's page juggling. 21:19:59 well, we could set breakpoints on each of the instances, there aren't many 21:20:22 gingeropolous: feeling up for it ? :) 21:20:53 sure! 21:20:58 ok 21:21:03 im always hungry for some copypasta 21:21:33 Run with gdb, but before the run command, put breakpoints. Let me go find the syntax again. 21:22:00 1800gosyntax 21:22:01 hmm, lemme apply that patch so my line numbers match yours 21:23:26 first: "list mdb_page_malloc" 21:23:51 Hrm, Should be: break mdb.c:1111 but it's not liking it. 21:23:59 Even with the path. 21:24:33 do the "list mdb_page_malloc" first to give gdb some file contet 21:24:40 theb "break 2031" 21:25:00 ok, so start gdb ./monerod , then put in list mdb_page_malloc run ? 21:25:09 or r 21:25:21 star gdb 21:25:25 then use list command 21:25:27 then use break command 21:25:31 don't run yet 21:25:36 because we want a few more breakpoints too 21:25:51 then "list mdb_page_loose" 21:25:58 then "break 2149" 21:26:10 then "list mdb_page_spill" 21:26:24 then "break 2369" 21:26:29 not a good start: Function "mdb_page_malloc" not defined. 21:26:44 then "list mdb_page_alloc" 21:26:53 then "break 2617" 21:27:02 and this only works on a debug build 21:27:03 ? 21:27:14 Function "mdb_page_malloc" not defined. Works with mdb_env_sync0 (not static). 21:27:22 oh, well then i gotta recompile with debug 21:33:01 ok. 21:33:20 No symbol table is loaded. Use the "file" command. 21:33:38 It does that before it's loaded the process. 21:33:39 hyc, halp. need multipass 21:33:59 Try "run", then ^C after it's started monerod. 21:34:00 im an idiot 21:34:19 ok, still got Function "mdb_page_malloc" not defined. 21:34:27 but screw it, ima plow forward 21:34:35 how about any of the others? 21:35:10 well, i got No line 2031 in the current file. 21:35:19 ok, mdb_dump -a scanned the entire data file and had no errors 21:35:30 yeah the line number is no good if the list command failed 21:36:36 Alternatively, do the same thing I did for MDB_BAD_TXN but with MDB_ERROR, and add assert(0); after the printf. Then enable core dumps, run, load core post mortem. 21:36:51 yeah that'd work 21:37:03 MDB_TXN_ERROR 21:38:41 but I'm puzzled. my debug monerod has all of the symbols present. 21:38:44 gdb monerod 21:38:48 list mdb_page_malloc 21:38:51 ^ works fine 21:40:35 hyc, do u just want access to the box? 21:40:44 that might be easier 21:40:53 ok, lets move to pm 21:48:00 fyi - becuase this debug build uses shared libs for the internal libs, you have to start the process before the symbols are present 21:48:05 so "break main" 21:48:07 "run" 21:48:12 then everything else shows up 21:49:31 ah 21:49:58 yeah i was surprised it didn't error out in compile. i usually have to edit the cmake.whatever file to get debug to build 21:53:52 ok took a few seconds, but hit a break at 6399 21:55:46 fyi https://paste.debian.net/1118956/ 21:55:52 now to figure out why this happened 21:58:01 and your running off of the old db at --data-dir /home/monerodaemon/.bitmonero.bak 21:58:01 correct hyc ? 21:58:31 yes 22:16:19 we're still getting stderr / stdout directed into the DB file 22:16:20 https://paste.debian.net/1118960/ 22:18:29 wasn’t mine also unbound? I forgot 22:18:35 I think it was, yes 22:20:12 y did this crop up now? i feel like i was running master for awhile 22:20:35 this is happening for over a month now but I think it triggers really unlikely 22:20:42 it happened twice on my DB 22:20:48 didn’t happen to others 22:21:02 snipa was saying this happened on 22 of his nodes? 22:21:33 Yup, it's happened to multiple so far. 22:21:45 I think we'll have to look into libunbound and see what it's doing 22:21:46 I've just been blowing em away and resyncing them. 22:22:11 Are you using your system's libinbound or the embedded one ? 22:22:19 how come poppin blocks doesn't fix it? 22:22:20 libunbound* 22:23:02 i dunno, hyc could tell yah :) 22:23:05 ldd monerod | grep libunbound 22:23:11 info shared says system libunbound 22:23:18 Hmm 22:23:40 selsta which are you using? 22:24:13 > Found libunbound shared library 22:24:16 so I guess system? 22:24:43 so maybe try another build using the bundled one instead? 22:25:12 I think our bundled version is more up to date ? 22:25:39 more up to date? Mine is 1.9.5 22:25:39 v1.7.3, I think. 22:25:46 So not really. 22:25:55 ok 22:25:58 but I think it was 1.9.4 the last time i had problems 22:26:31 it's definitely looking like an unbound bug 22:26:45 it still writes `sendto:` when exiting the daemon 22:28:32 https://github.com/NLnetLabs/unbound/commits/master 22:28:45 lots of fixes recently including Fix Out of Bounds Write 22:28:51 not saying that it’s related :P 22:35:31 dang, that's a ton of out of bounds bugs 22:37:30 The one in Ubuntu 18.04 is v1.6.7, but I'm sure security updates (at least) are backported to it. 23:09:37 ok. hyc , are u still hacking around there? i'll try removing the system libunound and recompiling 23:43:00 and just fyi, i started another monerod process on different port bindings to resync using release bins into the default data dir