01:26:49 gingeropolous: no I'm offline now, done with that for the moment 01:29:10 at this point my only thought is to wrap write() and assert if it's writing into the LMDB fd from outside of liblmdb 01:34:09 If you think it's that again, maybe dup2 the lmdb fd to something like 80 or so. Very unlikely to be reused. 01:34:22 Way less than 2 or 1 anyway. 01:34:24 yeah, that occurred to me too 01:34:56 but when we're running interactively, non-detached, it's not going to be 1 or 2 anyway 01:36:45 the problem is that it occurs so infrequently, using dup2 to assign a larger fd won't ever tell us where the actual bug is 01:36:54 I prefer to detect it and assert() so we can kill the bug 01:37:34 You can do both. 01:38:13 Actually, one thing I don't get. You did close 2 in your previous patch. 1 and 2 are the only things you write to you don't open. 01:38:27 So it seems odd it's the same thing again, no ? 01:38:38 yes, quite odd 01:38:58 btw I did a patch in the same vein very recently in OpenLDAP http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blobdiff;f=servers/slapd/daemon.c;h=3ce7d169207e6a294ff95e312d815f0c0bd4040a;hp=d64756a2ee7e5370c8ab48b8d93b84bb82599dcd;hb=4d7be1c161869daabb268bcfadbccf8fb1d2b9e1;hpb=230b4696691b67a84bc8309775c789a40fdcc4ff 01:39:12 we discovered the bug was a redundant close in Heimdal Kerberos that time 01:39:24 I'm not familiar with the stdio mapping to fds. Maybe reopening the fd is transparent to the stdio layer. 01:39:41 stdio always uses 1=stdout and 2=stderr 01:39:49 you can use freopen() to reassign them 01:40:00 but otherwise, it's always 1 and 2 01:40:36 Right, so closing the fd is ineffective as it can be reopened. Or rather, only effective till the next open. 01:40:42 yes 01:41:09 my previous patch was to *reopen* 2. it was left closed and the next open that came along got #2 01:41:26 Ah. OK, that makes sense. 01:41:43 and the previous code has always been wrong. Every canonical example of fork() has always shown close 1-2, open /dev/null. 01:42:44 anyway, selsta already confirmed before that he wasn't running demonized, so the previous patch was not expected to fix it 01:42:58 we just still had no idea where the writes were coming from 01:44:05 I'm a little suspicious about whether this one is an errant write() though. the text didn't begin on a page boundary, like in the previous bug. 01:50:30 so on my own monerod right now, [012] are /dev/tty, 3 is the logfile, [456] are LMDB fds 01:53:26 what if it's a logfile rotation bug 02:23:46 hyc, well there's a monerod running that we can't control 02:25:00 but its ok, working around it 09:22:58 can anyone help "xie" in #monero? he's the binance dev trying to update their stuff but seems shy to come out. 11:28:18 https://www.reddit.com/r/Monero/comments/e45469/is_somebody_using_a_botnet_to_spam_remote_nodes/ 11:28:28 my ban list is still growing 11:28:32 ~50 now 11:29:54 correction - ~90 http://paste.debian.net/1119033/ 11:46:13 gingeropolous, selsta - try this http://paste.debian.net/1119035/ 11:47:15 note that since it closes the lockfile fd, it probably is unsafe to use more than one process at a time 11:48:33 also, probably whatever is doing the bogus writes isn't checking its return code. just making an unwritable FD may not be enough to detect the culprit. 11:49:52 hm, wait, ignore that one 11:52:55 use this one http://paste.debian.net/1119036/ 12:03:31 so will this patch the issue might not happen anymore? 12:03:51 I’ll run it but it will be difficult to verify I guess 12:06:02 with* 12:13:02 right 12:13:27 I guess I can try working up a patch to assert on the bogus writes instead 12:14:11 but from previous straces we never actually saw any writes to the wrong FDs 12:14:25 so it's hard to figure out what to actually wrap and assert on 12:15:05 another possibility is to replace the FDs with a pipe, and have another thread assert if it ever receives data on it 12:15:43 but that probably won't stop the process soon enough; whichever thread does the write will probably have moved on already 12:17:10 still, could be worth a try. what do you think? 12:28:41 selstat try this one then http://paste.debian.net/1119040/ 12:38:56 run under gdb, or make sure core files are enabled 12:42:25 There are a few PRs on the merge list that need review, would appreciate if people would have a look at them 12:42:27 Needs review: 12:42:27 6040 6048 6050 6053 6061 6094 6095 6096 6102 12:42:29 ^ The list 12:44:34 13:38 run under gdb, or make sure core files are enabled <-- will try once I have time 12:45:21 ok cool 12:45:43 I guess we shoulf get gingeropolous and Snipa too 12:47:50 whats this one gonna do? 12:48:16 kinda don't wanna mess with xmrchain node, but if its for science, then.... FOR SCIENCE! 12:50:01 it should abort if any thread tries to write into the LMDB data files 12:50:25 except that it moves the LMDB filedescriptors so that the file should not get corrupted 12:51:15 Since Snipa had so many affected nodes, we might get the fastest result if he deploys it 12:51:35 but dunno what that would do to whatever depends on his nodes ... 13:40:53 Is the "pop blocks" functionality of "monero-blockchain-import" influenced by any other of the options of that tool? Or, on other words: What's the *fastest* way to drop blocks? I guess just that option, but you never know 13:47:12 If you want to pop a LOT, might be quicker to delete and sync up to what you want. 13:47:46 Popping's now slow due to the 100k median. Adding one is a cached special case that's fast, but removing one isn't. 13:50:38 I had the idea to repeatedly drop 10,000 blocks, from current top down to genesis, and sync 200 blocks at every 10,000 block stop, to get relative sync speeds depending on height. 13:51:32 I thought that's better than just downloading 70 gigs, maybe repeatedly. But dropping 10,000 blocks takes a surprising amount of time, several minutes inside my VM 13:52:19 (Better for my ISP and the Internet in general, I don't like to waste bandwidth.) 13:53:43 If you have the space, you can keep a copy of the chain maybe ? 13:54:10 Also, syncing 200 blocks is likely to not give good timings, too small a number. 13:55:00 Are you planning on including a table in monero that says "this portion takes about that much on my machine", and use that to guess at time remaining ? 13:56:20 More or less, but of course generalized, by using factors: "A post-RingCT block uses x time, a pre-RingCT block uses only x/2 time", "small 2015 blocks use only x/5 time" and so on 13:58:13 So the idea to sample at each 10,000 block, around 200 samples 13:59:09 And yes, I have a copy of the full blockchain, to easily repeat such experiments 14:01:50 By the way, does the following mean that I am heavy CPU-bottle-necked?: Synced 1930181/1980019 (97%, 49838 left) (71.919106 sec, 0.278090 blocks/sec), 100.355591 MB queued in 259 spans, stripe 8 -> 8 14:02:04 That queue gets longer and longer while I sync the 200 blocks 14:02:25 No. It could be disk. 14:02:48 As long as it's not much over 100 MB in the queue, it's OK. 14:02:56 * iDunk shuts up 14:03:13 Anyway, my hope is tha 14:03:41 t my config does not matter that much to catch *relative* block processing speeds 14:04:24 I know that of course a ton of factors influence absolute sync speed, but that does not interest me here, I want speed differences, depending on where I am in the blockchain 14:04:34 "If I sell you my computer, you will sync in x amount of time on it" 14:05:04 Future computer vendor advertising spiel? :) 14:05:18 Woosh 14:08:13 I think I still may be out of luck with this method if one factor strongly dominates. If 95% of time goes to disk writes I won't be able to see CPU speed differences pre/post RingCT, or pre/post Bulletproofs ... 14:08:37 Which may matter on other peoples' systems 14:09:02 Do you have a tmpfs /tmp ? 14:09:33 Though I suppose it'd only help for the start of the chain ^_^ 14:10:08 Yes I have, but only small ones, probably because of default install 14:12:01 compiling easylogging++ for webassembly produces errors [1]. mymonero got around this by replacing easylogging++ with vtlogger [2] in misc_log_ex.h [3]. any reason we can’t do the same or similar in monero-project? [1] https://paste.debian.net/1119053/ [2] https://github.com/mymonero/monero-core-custom/tree/a3da4673741cc02b78a86e2cd716f4e8fd5f3375/vtlogger [3] 14:12:01 https://github.com/mymonero/monero-core-custom/blob/master/epee/include/misc_log_ex.h 14:12:23 Why do you think syncing through 200 blocks gives no good average? What would need more blocks "to settle down" into some kind of long-time average? 14:12:39 woodser yeah this is an example of one of those deps 14:12:51 which must be extracted so others can use that code 14:12:57 i'm excited for your work btw 14:18:16 Would fixing it instead not be much simpler, avoid rewriting a fair amount of monero code (unless this is the secret motive), and generally be more sensible ? 14:19:46 Is there another reason why you want to replace it ? 14:20:14 my only motivation is to run in a webassembly environment 14:20:22 I’ll spend some time trying to get the existing easylogging.cc to compile 14:20:57 Looking at the code, it just needs either (1) finding a getenv replacement or (2) returning an empty string. 14:21:00 seems like a PR to replace it would be contained to adding the vtlogger directory and only modifying misc_log_ex.h 14:21:46 OK, sounds plausible. What are the vtlogger advantages (beyond "it works out of the box with webassembly" ?) 14:22:23 maybe @endogenic knows more about vtlogger? 14:23:37 How do you compile monero in webassembly ? 14:24:03 using emscripten 14:24:47 I was thinking more detailed. I've never done this before. 14:27:11 ie, if I was going to go fix what I can, how would I go about getting those errors myself ? 14:28:22 one creates a CMakeLists file (e.g. https://github.com/monero-ecosystem/monero-javascript/blob/wasm53_after_send/CMakeLists.txt) which defines the linker flags to the emscripten compiler, emcc. My understanding is the emcc compiler basically wraps the normal compilation process to be compatible in webassembly 14:31:36 If I were to try to help fix building monero with this, it'd be of general usefulness, right ? 14:31:51 It would. 14:33:07 Does emscripten have another name ? I don't find that name in fedora. 14:33:36 Oh I found webassembly. Must be that. 14:35:00 afaik emcc isn't in most distro repos because it uses the new webassembly target in llvm which only recently got released 14:36:05 Ah. Scratch VM then. Thanks. 14:46:39 vtlogger is LGPL. Not for us then. 14:47:11 What licence(s) do monero prefer? 14:47:33 BSD. 14:47:51 MIT also fine. 14:48:01 Now I love monero even more. :D 14:48:30 I still need to figure out how to compile it on OpenBSD though, as the readme instructions seems to be outdated/fail. 14:48:30 I dunno, I like GPL. It's like active defense instead of just curling up and taking punches. 14:49:39 Hehe, I believe in the idea in truely free software, as long as you credit the author. Not a lot of clauses where you can't do whatever you want with the code, that's not free. :/ 14:49:45 They were replaced recently. Maybe the previous ones will work better :) 14:50:09 I'll give it a try once I have the time.. and if it needs fixing, I'll submit a PR :) 14:51:36 some of the compile errors are due to the environment not being unix or windows, like this condition: https://github.com/monero-project/monero/blob/be6c5afabb3d5146a46f74ae9fa44f520f380e8b/external/easylogging%2B%2B/easylogging%2B%2B.cc#L1194 14:53:43 :o 14:53:55 In my case, I had some errors about some noexecheap flag 14:54:00 havn't had the time to look into it though 14:56:12 Seems even easier. Upstream's got emscripten support. I'll merge it. 15:16:34 woodser: does https://github.com/moneromooo-monero/bitmonero/tree/emsc work ? If so, I'll PR. 15:16:57 Turned out to be pretty simple :) 15:27:01 you'll still fail when it comes time to compile RandomX 15:27:14 that helps a lot. it is compiling but failing on linking elStorage. 15:27:31 Can still do wallet only. 15:27:38 right, wallet only 15:27:50 Can you paste the error on some pastebin site ? 15:28:31 https://paste.debian.net/1119068/ 15:29:20 wait I think I just need to add the source in the cmakelists 15:30:07 at this point, the wallet's pay-for-RPC feature also requires the PoW code 15:30:52 Heh, good point... 15:31:08 woodser: are you defining AUTO_INITIALIZE_EASYLOGGINGPP in your custom makefile ? 15:31:13 nevermind easylogging.cc is already added as a source file 15:31:22 no 15:31:26 Well, you should. 15:31:39 It's enough to define it just for easylogging++.cc. 15:31:51 (I think) 15:35:11 that did the trick :) 16:11:48 Hey all, I'm looking into replacing epee for serialization in the RPC server (as it is macro-laden madness) and I'm currently looking at nlohmann/json. Would that theoretically be an acceptable replacement? 16:13:00 Pros: header only, C++11 compatible, serialization doesn't have to be in a header file, widely used and well tested 16:15:24 Cons: slightly more verbose than json_dto (which I previously suggested for the job), requires reimplementing the serialization 16:16:14 Is it faster ? Does it fix some bugs in the current one without adding more ? Other advantages ? 16:17:22 I'd also consider nice advantages: substantially decreases memory needed to build wallet2, and substantially speeds up the build. 16:17:23 Faster? Probably similar, but compilation may be faster since the headers will be smaller. 16:17:52 Can you give an indication (with the uderstanding that it's just on one particular machine) ? 16:19:30 You mean an indication of how much faster it would be to compile? Hard to tell without actually implementing it. 16:20:07 how is it both header-only and "serialization doesn't have to be in a header" ? 16:21:01 I guess it means the serialization of actual structures. I don't think epee requires that either, unless they're called from elsewhere. 16:21:12 hyc: the library itself is header only, but for serialization of custom types, you'd need to write code for that. But that part does not have to be in a header. 16:21:18 Or what moneromooo said 16:22:36 if it actually allows a significant amount of work to be moved into source files instead of header files, I'd be inclined to at least investigate it 16:23:13 right now you can't even build a debug windows binary on a 32bit machine any more 16:24:08 Hmm. I just thought of a way to split wallet2 without massive diffs. 16:24:26 bertptrs why that over rapidjson, which we already have checked into the codebase? 16:24:44 Other nice things would be: more idiomatic C++ so it's easier to read for newcomers (and veterans), no macros 16:24:54 macros are not the issue 16:25:04 Have new wallet2_1.cpp and wallet2.cpp that just include wallet2.cpp (which is removed from the build) with #define WALLET2_1 (or WALLET2_2), and exclude half wallet2.cpp based on the define ^_^ 16:25:11 possibly the maze of c++, which can probably be simplified, are the issue 16:25:42 vtnerd: rapidjson is nice, but it using it for serialization is a bit verbose, which is (I think) you now use epee to reduce the repetition somewhat 16:26:17 -repeated it, + why after "which is" 16:27:02 rapidjson is not used within epee, it is completely custom 16:27:04 nlohmann/json could be used to replace both while having a similar interface 16:27:49 replace both? what do you mean? 16:28:52 Take out both epee's serializtion and the rapidjson code in wallet2 I think. 16:28:58 Correct. 16:29:22 rapidjson is not used within epee, it is completely custom < I'm looking something up about this, so I may be a little slow to reply 16:29:49 my first glance at this library is a nack 16:30:04 it should be possible to do stuff without wasting cycles on creating a DOM 16:30:37 sounds like it's aimed at web browsers then 16:30:48 Anyway, my view is "not unless advantages are found". "Header only" doesn't sound like one (and epee also is). "C++11" either, since epee also is. Being able to put serialization in .cc files, could be. Advantages I'd like are: faster exec, faster build, less memory hungry, fixes bugs while not adding more. 16:30:49 it should be possible to do this in the existing structure with some refactoring, but the read side is a bit tricky 16:31:16 Otherwise, it's just churn. 16:31:23 agreed 16:31:49 yeah moo, Ive been looking at all 3 16:32:12 FWIW, serialization load is starting to become noticeable in perf top in some cases. 16:32:30 the issue is that every json library allocates tons of memory, the unordered fields of an object are quite problematic for most interfaces/parsers 16:32:51 yeah I think I have a way to help with that, without much hassle 16:33:10 IIRC, fast to read/write was one of the draws to 0MQ as well. 16:33:20 Shame it's so different though. 16:33:20 you'll have to endue the "slice" object going into the reader so that many of the string copies become ref count changes 16:33:48 I'd be fine with that if it gives us a substantial speed boost. 16:34:06 it should, and it will be dead simple to measure at least 16:34:09 In fact, I did it (but I got snagged at needing C++14 for it to make a difference in practice) 16:34:27 hmm. you tried this in epee? 16:34:39 Yes. You NAKed that patch even :D 16:34:51 Because it ended up mostly churn since a copy was needed at the very end :( 16:35:00 (unless C++14) 16:35:16 vtnerd: It appears I was mistake I thought epees json serialization went through RapidJSON objects 16:35:17 That was in the std::map find IIRC 16:35:36 its merged with the custom binary format of the p2p 16:35:58 (and the NAK was a good decision, I might add) 16:36:10 ah right, the C++14 map lookup. I'll have to find that NAK then so I don't make the same mistake 16:36:38 were you using the non-owning "span" in the map? 16:36:53 no it had to be something else, span is an easy lookup 16:36:55 IIRC I made my own object which had optional memory backing. 16:37:03 Since it sometimes needed to. I'll find it. 16:37:24 yeah whats weird is that we are allocating temporary strings anyway sometimes 16:37:38 for those `get_value` calls 16:37:57 does any of this have any impact on the binary p2p serialization? 16:38:07 yes unfortunately 16:38:20 I mean it would improve it, but it touches the code sadly 16:38:25 ah 16:38:45 Ive tried to figure out a way around this, and you can improve it without touching p2p side 16:39:02 but you only reduce the temporary strings for the json values, not the keys themselves 16:43:00 https://github.com/monero-project/monero/pull/5348 16:43:32 See the last comment from you there about that last std::string ctor. 16:44:57 yeah, that interface gotcha is a pain 16:45:16 because its dead simple on the map implementation side 16:46:12 Anyway, I got to doing that because I found it in a profile IIRC. So speeding this up substantially would be an advantage. 16:46:24 this was merged though? did you revert? 16:46:49 There were two patches, the second one got merged only. 16:47:20 BTW I try to often add "substantially" to avoid "I can gain 0.1%, so let's rewrite it all" ^_^ 16:49:17 hmm, I have an idea that will help a small bit (probably not substantial), but should also be an easy review 16:49:55 theres a few key places that should be moves instead of copies - the original code seems to be strictly C++03 17:29:09 142 malicious apps found in Apple's app store https://spectrum.ieee.org/tech-talk/computing/software/new-screening-technique-reveals-142-malicious-apple-apps 17:29:26 so much for app stores being a trusted code provider 17:36:07 uh oh, they are called "Chameleon apps" 17:50:01 yeah, so much for that name :P 20:09:23 hyc, should I run debug for that patch? 20:09:29 why am i even asking. 20:12:29 Yes please. 20:12:53 The assert will generate no code if not a debug build. 21:43:06 sorry looks like my connection dropped for a bit. gingeropolous did I miss anything? 21:59:06 hyc: ok running debug monerod in lldb now with your patch 21:59:09 nope, just me asking stupid questions 21:59:09 I guess now we have to wait :P 21:59:22 yeah, now we wait 21:59:28 woodser: I have a sinking feeling I need to build all the deps myself. Boost, openssl, etc. Am I correct ? 21:59:29 and there's no guarantee that it will trip 21:59:44 If so, where does it stop ? syscalls ? 22:05:55 in an effort to increase the number of stderror writes im using set_log 4. is this logical? 22:06:33 gingeropolous: probably will have no effect 22:06:46 pretty sure it's only error logs directly from libunbound 22:06:54 ah 22:06:59 but ... all guesswork 22:12:00 wallet is claiming two inputs are from the same transaction even though none of the blocks are the same right above that 22:15:11 Details ? 22:50:13 Warning: Some input keys being spent are from the same transaction, which can break the anonymity of ring signature. Make sure this is intentional! 22:50:15 that warning 22:50:56 you want txid or what details? 22:52:23 Let me look at the code for that message first. 22:53:48 Yes, please paste me all the info you have on that. 22:54:45 it comes up on every transaction so far 22:58:41 Quite possibly fixed by: https://paste.debian.net/hidden/9b903c14/ 23:00:08 hmm 23:00:56 I have another funky thing: it looks like one of the random outs in another tx was pulled from at least the same block (if not the same tx); both are market with * as being the real out 23:01:05 marked* 23:01:26 The * is based on height IIRC. 23:01:51 Guess having two * increases the uncertainty about the real input ^_^ 23:01:52 makes sense; should be a rare side effect 23:01:54 haha 23:02:01 even the sender doesn't know! 23:03:54 Indeed based on height, I found the code.