gingeropolous: no I'm offline now, done with that for the moment

at this point my only thought is to wrap write() and assert if it's writing into the LMDB fd from outside of liblmdb

If you think it's that again, maybe dup2 the lmdb fd to something like 80 or so. Very unlikely to be reused.

Way less than 2 or 1 anyway.

yeah, that occurred to me too

but when we're running interactively, non-detached, it's not going to be 1 or 2 anyway

the problem is that it occurs so infrequently, using dup2 to assign a larger fd won't ever tell us where the actual bug is

I prefer to detect it and assert() so we can kill the bug

You can do both.

Actually, one thing I don't get. You did close 2 in your previous patch. 1 and 2 are the only things you write to you don't open.

So it seems odd it's the same thing again, no ?

yes, quite odd

btw I did a patch in the same vein very recently in OpenLDAP openldap.org/devel/gitweb.cgi?p=ope…696691b67a84bc8309775c789a40fdcc4ff

we discovered the bug was a redundant close in Heimdal Kerberos that time

I'm not familiar with the stdio mapping to fds. Maybe reopening the fd is transparent to the stdio layer.

stdio always uses 1=stdout and 2=stderr

you can use freopen() to reassign them

but otherwise, it's always 1 and 2

Right, so closing the fd is ineffective as it can be reopened. Or rather, only effective till the next open.

yes

my previous patch was to *reopen* 2. it was left closed and the next open that came along got #2

Ah. OK, that makes sense.

and the previous code has always been wrong. Every canonical example of fork() has always shown close 1-2, open /dev/null.

anyway, selsta already confirmed before that he wasn't running demonized, so the previous patch was not expected to fix it

we just still had no idea where the writes were coming from

I'm a little suspicious about whether this one is an errant write() though. the text didn't begin on a page boundary, like in the previous bug.

so on my own monerod right now, [012] are /dev/tty, 3 is the logfile, [456] are LMDB fds

what if it's a logfile rotation bug

hyc, well there's a monerod running that we can't control

but its ok, working around it

can anyone help "xie" in #monero? he's the binance dev trying to update their stuff but seems shy to come out.

my ban list is still growing

~50 now

correction - ~90 paste.debian.net/1119033

gingeropolous, selsta - try this paste.debian.net/1119035

note that since it closes the lockfile fd, it probably is unsafe to use more than one process at a time

also, probably whatever is doing the bogus writes isn't checking its return code. just making an unwritable FD may not be enough to detect the culprit.

hm, wait, ignore that one

use this one paste.debian.net/1119036

so will this patch the issue might not happen anymore?

I’ll run it but it will be difficult to verify I guess

with*

right

I guess I can try working up a patch to assert on the bogus writes instead

but from previous straces we never actually saw any writes to the wrong FDs

so it's hard to figure out what to actually wrap and assert on

another possibility is to replace the FDs with a pipe, and have another thread assert if it ever receives data on it

but that probably won't stop the process soon enough; whichever thread does the write will probably have moved on already

still, could be worth a try. what do you think?

selstat try this one then paste.debian.net/1119040

run under gdb, or make sure core files are enabled

There are a few PRs on the merge list that need review, would appreciate if people would have a look at them

Needs review:

6040 6048 6050 6053 6061 6094 6095 6096 6102

^ The list

13:38 <hyc> run under gdb, or make sure core files are enabled <-- will try once I have time

ok cool

I guess we shoulf get gingeropolous and Snipa too

whats this one gonna do?

kinda don't wanna mess with xmrchain node, but if its for science, then.... FOR SCIENCE!

it should abort if any thread tries to write into the LMDB data files

except that it moves the LMDB filedescriptors so that the file should not get corrupted

Since Snipa had so many affected nodes, we might get the fastest result if he deploys it

but dunno what that would do to whatever depends on his nodes ...

Is the "pop blocks" functionality of "monero-blockchain-import" influenced by any other of the options of that tool? Or, on other words: What's the *fastest* way to drop blocks? I guess just that option, but you never know

If you want to pop a LOT, might be quicker to delete and sync up to what you want.

Popping's now slow due to the 100k median. Adding one is a cached special case that's fast, but removing one isn't.

I had the idea to repeatedly drop 10,000 blocks, from current top down to genesis, and sync 200 blocks at every 10,000 block stop, to get relative sync speeds depending on height.

I thought that's better than just downloading 70 gigs, maybe repeatedly. But dropping 10,000 blocks takes a surprising amount of time, several minutes inside my VM

(Better for my ISP and the Internet in general, I don't like to waste bandwidth.)

If you have the space, you can keep a copy of the chain maybe ?

Also, syncing 200 blocks is likely to not give good timings, too small a number.

Are you planning on including a table in monero that says "this portion takes about that much on my machine", and use that to guess at time remaining ?

More or less, but of course generalized, by using factors: "A post-RingCT block uses x time, a pre-RingCT block uses only x/2 time", "small 2015 blocks use only x/5 time" and so on

So the idea to sample at each 10,000 block, around 200 samples

And yes, I have a copy of the full blockchain, to easily repeat such experiments

By the way, does the following mean that I am heavy CPU-bottle-necked?: Synced 1930181/1980019 (97%, 49838 left) (71.919106 sec, 0.278090 blocks/sec), 100.355591 MB queued in 259 spans, stripe 8 -> 8

That queue gets longer and longer while I sync the 200 blocks

No. It could be disk.

As long as it's not much over 100 MB in the queue, it's OK.

Anyway, my hope is tha

t my config does not matter that much to catch *relative* block processing speeds

I know that of course a ton of factors influence absolute sync speed, but that does not interest me here, I want speed differences, depending on where I am in the blockchain

"If I sell you my computer, you will sync in x amount of time on it"

Future computer vendor advertising spiel? :)

Woosh

I think I still may be out of luck with this method if one factor strongly dominates. If 95% of time goes to disk writes I won't be able to see CPU speed differences pre/post RingCT, or pre/post Bulletproofs ...

Which may matter on other peoples' systems

Do you have a tmpfs /tmp ?

Though I suppose it'd only help for the start of the chain ^_^

Yes I have, but only small ones, probably because of default install

<woodser> compiling easylogging++ for webassembly produces errors [1]. mymonero got around this by replacing easylogging++ with vtlogger [2] in misc_log_ex.h [3]. any reason we can’t do the same or similar in monero-project? [1] paste.debian.net/1119053 [2] github.com/mymonero/monero-core-cus…2b78a86e2cd716f4e8fd5f3375/vtlogger [3]

Why do you think syncing through 200 blocks gives no good average? What would need more blocks "to settle down" into some kind of long-time average?

woodser yeah this is an example of one of those deps

which must be extracted so others can use that code

i'm excited for your work btw

Would fixing it instead not be much simpler, avoid rewriting a fair amount of monero code (unless this is the secret motive), and generally be more sensible ?

Is there another reason why you want to replace it ?

<woodser> my only motivation is to run in a webassembly environment

<woodser> I’ll spend some time trying to get the existing easylogging.cc to compile

Looking at the code, it just needs either (1) finding a getenv replacement or (2) returning an empty string.

<woodser> seems like a PR to replace it would be contained to adding the vtlogger directory and only modifying misc_log_ex.h

OK, sounds plausible. What are the vtlogger advantages (beyond "it works out of the box with webassembly" ?)

<woodser> maybe @endogenic knows more about vtlogger?

How do you compile monero in webassembly ?

<woodser> using emscripten

I was thinking more detailed. I've never done this before.

ie, if I was going to go fix what I can, how would I go about getting those errors myself ?

<woodser> one creates a CMakeLists file (e.g. github.com/monero-ecosystem/monero-…ob/wasm53_after_send/CMakeLists.txt) which defines the linker flags to the emscripten compiler, emcc. My understanding is the emcc compiler basically wraps the normal compilation process to be compatible in webassembly

If I were to try to help fix building monero with this, it'd be of general usefulness, right ?

<woodser> It would.

Does emscripten have another name ? I don't find that name in fedora.

Oh I found webassembly. Must be that.

afaik emcc isn't in most distro repos because it uses the new webassembly target in llvm which only recently got released

Ah. Scratch VM then. Thanks.

vtlogger is LGPL. Not for us then.

What licence(s) do monero prefer?

BSD.

MIT also fine.

Now I love monero even more. :D

I still need to figure out how to compile it on OpenBSD though, as the readme instructions seems to be outdated/fail.

I dunno, I like GPL. It's like active defense instead of just curling up and taking punches.

Hehe, I believe in the idea in truely free software, as long as you credit the author. Not a lot of clauses where you can't do whatever you want with the code, that's not free. :/

They were replaced recently. Maybe the previous ones will work better :)

I'll give it a try once I have the time.. and if it needs fixing, I'll submit a PR :)

<woodser> some of the compile errors are due to the environment not being unix or windows, like this condition: github.com/monero-project/monero/bl…ng%2B%2B/easylogging%2B%2B.cc#L1194

:o

In my case, I had some errors about some noexecheap flag

havn't had the time to look into it though

Seems even easier. Upstream's got emscripten support. I'll merge it.

woodser: does github.com/moneromooo-monero/bitmonero/tree/emsc work ? If so, I'll PR.

Turned out to be pretty simple :)

you'll still fail when it comes time to compile RandomX

<woodser> that helps a lot. it is compiling but failing on linking elStorage.

Can still do wallet only.

<woodser> right, wallet only

Can you paste the error on some pastebin site ?

<woodser> paste.debian.net/1119068

<woodser> wait I think I just need to add the source in the cmakelists

at this point, the wallet's pay-for-RPC feature also requires the PoW code

Heh, good point...

woodser: are you defining AUTO_INITIALIZE_EASYLOGGINGPP in your custom makefile ?

<woodser> nevermind easylogging.cc is already added as a source file

<woodser> no

Well, you should.

It's enough to define it just for easylogging++.cc.

(I think)

<woodser> that did the trick :)

Hey all, I'm looking into replacing epee for serialization in the RPC server (as it is macro-laden madness) and I'm currently looking at nlohmann/json. Would that theoretically be an acceptable replacement?

Pros: header only, C++11 compatible, serialization doesn't have to be in a header file, widely used and well tested

Cons: slightly more verbose than json_dto (which I previously suggested for the job), requires reimplementing the serialization

Is it faster ? Does it fix some bugs in the current one without adding more ? Other advantages ?

I'd also consider nice advantages: substantially decreases memory needed to build wallet2, and substantially speeds up the build.

Faster? Probably similar, but compilation may be faster since the headers will be smaller.

Can you give an indication (with the uderstanding that it's just on one particular machine) ?

You mean an indication of how much faster it would be to compile? Hard to tell without actually implementing it.

how is it both header-only and "serialization doesn't have to be in a header" ?

I guess it means the serialization of actual structures. I don't think epee requires that either, unless they're called from elsewhere.

hyc: the library itself is header only, but for serialization of custom types, you'd need to write code for that. But that part does not have to be in a header.

Or what moneromooo said

if it actually allows a significant amount of work to be moved into source files instead of header files, I'd be inclined to at least investigate it

right now you can't even build a debug windows binary on a 32bit machine any more

Hmm. I just thought of a way to split wallet2 without massive diffs.

bertptrs why that over rapidjson, which we already have checked into the codebase?

Other nice things would be: more idiomatic C++ so it's easier to read for newcomers (and veterans), no macros

macros are not the issue

Have new wallet2_1.cpp and wallet2.cpp that just include wallet2.cpp (which is removed from the build) with #define WALLET2_1 (or WALLET2_2), and exclude half wallet2.cpp based on the define ^_^

possibly the maze of c++, which can probably be simplified, are the issue

vtnerd: rapidjson is nice, but it using it for serialization is a bit verbose, which is (I think) you now use epee to reduce the repetition somewhat

-repeated it, + why after "which is"

rapidjson is not used within epee, it is completely custom

nlohmann/json could be used to replace both while having a similar interface

replace both? what do you mean?

Take out both epee's serializtion and the rapidjson code in wallet2 I think.

Correct.

rapidjson is not used within epee, it is completely custom < I'm looking something up about this, so I may be a little slow to reply

my first glance at this library is a nack

it should be possible to do stuff without wasting cycles on creating a DOM

sounds like it's aimed at web browsers then

Anyway, my view is "not unless advantages are found". "Header only" doesn't sound like one (and epee also is). "C++11" either, since epee also is. Being able to put serialization in .cc files, could be. Advantages I'd like are: faster exec, faster build, less memory hungry, fixes bugs while not adding more.

it should be possible to do this in the existing structure with some refactoring, but the read side is a bit tricky

Otherwise, it's just churn.

agreed

yeah moo, Ive been looking at all 3

FWIW, serialization load is starting to become noticeable in perf top in some cases.

the issue is that every json library allocates tons of memory, the unordered fields of an object are quite problematic for most interfaces/parsers

yeah I think I have a way to help with that, without much hassle

IIRC, fast to read/write was one of the draws to 0MQ as well.

Shame it's so different though.

you'll have to endue the "slice" object going into the reader so that many of the string copies become ref count changes

I'd be fine with that if it gives us a substantial speed boost.

it should, and it will be dead simple to measure at least

In fact, I did it (but I got snagged at needing C++14 for it to make a difference in practice)

hmm. you tried this in epee?

Yes. You NAKed that patch even :D

Because it ended up mostly churn since a copy was needed at the very end :(

(unless C++14)

vtnerd: It appears I was mistake I thought epees json serialization went through RapidJSON objects

That was in the std::map find IIRC

its merged with the custom binary format of the p2p

(and the NAK was a good decision, I might add)

ah right, the C++14 map lookup. I'll have to find that NAK then so I don't make the same mistake

were you using the non-owning "span" in the map?

no it had to be something else, span is an easy lookup

IIRC I made my own object which had optional memory backing.

Since it sometimes needed to. I'll find it.

yeah whats weird is that we are allocating temporary strings anyway sometimes

for those `get_value` calls

does any of this have any impact on the binary p2p serialization?

yes unfortunately

I mean it would improve it, but it touches the code sadly

ah

Ive tried to figure out a way around this, and you can improve it without touching p2p side

but you only reduce the temporary strings for the json values, not the keys themselves

See the last comment from you there about that last std::string ctor.

yeah, that interface gotcha is a pain

because its dead simple on the map implementation side

Anyway, I got to doing that because I found it in a profile IIRC. So speeding this up substantially would be an advantage.

this was merged though? did you revert?

There were two patches, the second one got merged only.

BTW I try to often add "substantially" to avoid "I can gain 0.1%, so let's rewrite it all" ^_^

hmm, I have an idea that will help a small bit (probably not substantial), but should also be an easy review

theres a few key places that should be moves instead of copies - the original code seems to be strictly C++03

142 malicious apps found in Apple's app store spectrum.ieee.org/tech-talk/computi…ue-reveals-142-malicious-apple-apps

so much for app stores being a trusted code provider

uh oh, they are called "Chameleon apps"

yeah, so much for that name :P

hyc, should I run debug for that patch?

why am i even asking.

Yes please.

The assert will generate no code if not a debug build.

sorry looks like my connection dropped for a bit. gingeropolous did I miss anything?

hyc: ok running debug monerod in lldb now with your patch

nope, just me asking stupid questions

I guess now we have to wait :P

yeah, now we wait

woodser: I have a sinking feeling I need to build all the deps myself. Boost, openssl, etc. Am I correct ?

and there's no guarantee that it will trip

If so, where does it stop ? syscalls ?

in an effort to increase the number of stderror writes im using set_log 4. is this logical?

gingeropolous: probably will have no effect

pretty sure it's only error logs directly from libunbound

ah

but ... all guesswork

wallet is claiming two inputs are from the same transaction even though none of the blocks are the same right above that

Details ?

Warning: Some input keys being spent are from the same transaction, which can break the anonymity of ring signature. Make sure this is intentional!

that warning

you want txid or what details?

Let me look at the code for that message first.

Yes, please paste me all the info you have on that.

it comes up on every transaction so far

Quite possibly fixed by: paste.debian.net/hidden/9b903c14

hmm

I have another funky thing: it looks like one of the random outs in another tx was pulled from at least the same block (if not the same tx); both are market with * as being the real out

marked*

The * is based on height IIRC.

Guess having two * increases the uncertainty about the real input ^_^

makes sense; should be a rare side effect

haha

even the sender doesn't know!

Indeed based on height, I found the code.