.merges

.merges.

.merges

Alright, stuffing my face to wake up some, then will merge.

Sorry, I was 3/4ths brain dead last time I checked, so was not in a good head-space for attempting to merge.

oki thanks

Merged, I think.

thanks

Of course, was just a bit brain-dead earlier from NYE celebrations.

.merges

Merge queue empty

.murders

I guess the murder queue is empty too

Shame.

I just started a fresh IBD btw two boxes on my wifi. most of the time network is less than 2Mbps

I've seen a spike up to 11Mbps

very brief

I also set limit-up on the serving node to 8MB/s

hm, the client node just lost its connection. was using --add-exclusive-node too

dropped after 87552

client node seemed to be deadlocked.

restarted it. client node is a quadcore laptop but I never see more than 80% CPU use

would expect to see more since block verification is threaded

oo, a peak of 25Mbps

seems low for local net

I get the impression that the speed is really throttled by blocks/sec. bits/sec only goes up when there are full blocks.

and yes, seems slow

if speed increases with multiple incoming connections one would assume it's a bottleneck on the providing node

not disk limited either

the providing node is relatively idle

disk shows 7% busy, hundreds of KB/sec reads

monerod using ~5% CPU there

hmm

sounds like artificially throttled then

dunno where tho

client node is running --db-sync-mode fast:async:10000

it is not waiting for any disk writes

how's memory activity?

10G free

on client node

4G cached

provider node 2.4G free, 17G cached

that's utilization. how about activity? probably hard to monitor

I just caught a few seconds using perf

seems to be bottlenecked in LMDB. how embarrassing

man, and I deleted the "let's replace LMDB with oracle just to be sure it's not the culprit" joke that popped in my head earler :P

LOL

6.7% time in mdb_txn_safe, which is a serialization point. so we're losing concurrency there

like ACID compliance?

not quite. this is in the DB driver for its own consistency control, not internal to LMDB

that function exists to wrap LMDB transactions because higher level callers don't know about them

since the higher levels were all written for the non-transactional in-memory blockchain

I suspect that's why we're not getting concurrency in verification

seems like a good optimization target

yeah. lots of work to remove that wrapper layer though, and make the higher level code smarter about DB transactions

is there a RPC call for print_net_stats?

there has to be

every monerod command uses RPC

(or can use it, to talk to itself)

here is not listed: getmonero.org/resources/developer-guides/daemon-rpc.html

better to just read github.com/monero-project/monero/bl…rpc/core_rpc_server_commands_defs.h

COMMAND_RPC_GET_NET_STATS

ty

curl 127.0.0.1:18081/get_net_stats -H 'Content-Type: application/json'

binaryFate: yes, thank you .. already integrated in my queries

bah. got a segfault, systemd stored the corefile compressed

zstd compressor dies decompressing the core file

what kind of idiot designs so much fragility into critical system facilities

hyc one doesn't need to spend any effort to have fragility in their systems :P

yes but the systemd folks have gone out of their way to do so

M5M400: anyway, this perf report still isn't telling the whole story. If those two LMDB functions were the actual bottleneck, CPU use would be at least 100%

since it's less, and there's no I/O wait, and network is fairly idle, then the rest of the time is probably mutex locking

has anyone else done a full sync from scratch lately? i'm getting lots of failures on v0.17.1.8, but restarting monerod gets past them

for a couple hundred thousand blocks till it happens again

Paste a level 1 log ?

ok next time it hits

was twice in the first 250,000 blocks

of course now I'm at 600k and it hasn't hit again

there it goes weuse.cash/2020/09/11/bitcoin-mixing-a-catch-22

doh

moneromooo: ^

ty

It's syncing off a single peer on the local network ?

yes

exclusive node

and after I exit and restart it just resumes syncing

hyc: there are two "Consistency failure in m_blocks_hash_check construction" messages in src/cryptonote_core/blockchain.cpp, is this the first or the second one, based on line numbers in the log ?

looking...

line 4827

1st one

ty

now up to 1151100 and it hasn't happened again yet

What was the first height it did this ?

first time it happened at block 282112, next at 535040

again at 544768

then 661504

Is pruning involved on any of these daemons ?

nope

hasn't happened again since 661504

seemed to run into a problem at 130572 but it disconnected and reconnected automatically and continued

1303572

Known, assuming no other error.

well. looks like my node is frozen again, even though I am no longer updating the ban list with the ban command or with the dns list

ok, what does frozen mean

gdb, thread apply all bt

same as before. Systemd says it's running but it won't respond to RPC. Even doing a simple 'monerod status' just results in the system printing the name of the release version, then hanging rpetty much forever until I hit ctrl+C

(yeah there was no error message on that disconnect/reconnect)

Lyza: ok, did not happen to any of my nodes yet, so you will have to run it in gdb so that we can find out where it freezes

if it's still hung, attach gdb to the process and get the backtrace like mooo said

this time it seems to have happened while I was interacting with the node with a wallet. but yeah I'm installing gdb rn, if y'all could give me a specific command to run it with that would be helpful

if the process is still there, find out its PID

gdb /path/to/monerod `pidof monerod`

hyc unfortunately already restarted, didn't even know that was a thing you could do

yeah that ^

oh well

usually also "gdb - <pid>" works, gdb can figure out the pathname when it reads the process memory

so I don't need to do anything special when launching, I can just do that if it hangs again?

this person also reported freezing: monero-project/monero #7224#issuecomment-753482686

otherwise I did not read any reports about freezing yet

alright well I'll keep an eye out and try to upload some useful info if it happens again

Lyza: yes, then after it attaches, "thread apply all bt"

got it ty

if monero-project/monero #7254 gets a review we can include it this release, else next one

reviewing it

reviewed

thanks

hyc: I can't repro. If I paste you a log patch and you can repro it, can you paste the logs ?

paste.debian.net/hidden/3eeee02e is the patch.

Actually...

The 1500 bit was to try and exercise it more.

I've restarted the sync

may have hardware issues here, this laptop hasn't been used in a while

.merge+ 7254 7255

Added

monero-project/monero #7264 needs an approval

done

.merge+ 7264 7263 7248 7261 7262

Added

thanks

ok ^ would be release

we still have an issue that daemon sync bans false positives, but we had this issue for a couple releases now

Which case exactly ?

just starting a chain sync from height 0 and I get a lot of false positive bans, currently at 40% synced and 24 peers banned

some of those 24 are in the block list but not all

Did I ask for logs ?

I don’t remember

Well, send them again, assuming you had, necause I don't remember either :)

Though it seems common enough I should be able to repro when I get to it

Didn't repro hyc's though so send logs if yo uhave them.

restarted sync with log 1

hmm is there a different log level I can use? 1 is super spammy during sync

You can always rm ~/.bitmonero/bitmonero.log-* from time to time if it fills up too much.

100 MB each by default.

1,blockchain:ERROR might do. I don't *think* a ban would have an interesting message in that category, and it's what adds the "blclk added" lines

First block hash is unknown, dropping connection

can send you full logs

Oh. Right, I know what that is. It's when you start downloading a lot higher than the current.

I've seen that today, I'd forgot already. Adding to the list..

but I assume is harmless apart from blocking some false positives during sync?

If you're asking if it can corrupt your db or the like, it can't.

paste.debian.net/hidden/5c71d1a8 is a likely fix, seems to be running good here.

ok will test

.merge- 7254 7255

Removed

19:57 <+moneromooo> paste.debian.net/hidden/5c71d1a8 is a likely fix, seems to be running good here. <-- seems better with this patch

after 10% synced only 1 banned and the one banned is in the block list

still 0 false positives after 20%

hyc: I have reproed it once. The logs show the two values are the same, so it looks like a race. But you had a single peer, so it doesn't make sense...

My logs timings do show a race though.

Maybe two bugs.

hyc: paste.debian.net/1179470

Hmm, not enough. Somwthing else locking too.

Has to be the main lock: paste.debian.net/1179474 (not run yet).

"protocol: handle receiving a block hash we've not added yet" <-- do you want to PR this?

done

thanks, after 60% synced I have 2 false positives, but I assume they are banned for other reasons

idle or so

grep "dropping connection$", it'll tell you why.

restarted with log 1 now to see what the remaining false positives are

is it normal that I get so many of these? paste.debian.net/hidden/ad96263d

Is your node pruned ?

I’m not syncing a pruned node

Then it's not expected.

Oh, wait, not quite the same message I thought.

Then it'll depend why it ends up there, more logs needed.

net.cn:DEBUG

And block_queue:DEBUG

And cn.block_queue:DEBUG

together with log 0 or log 1?

Either.

ok glad you repro'd it. I've been running self tests on my ssd cuz I thought it was flaking out on me

Well, as I said, the one I got was a race between two threads getting data. You would have had just one.

hmmm

It'd also race with some other code, but that code only reads, so would not get you an assert there.

in the log file, some were e.g. P2P3 vs P2P1 so maybe there's more than one active at once

You'd need two sets of hashes at once from the excluive peer. I don't think that can happen.

ok. then I'll keep looking for flaky hardware here

But merge the lock and see if it happens again I guess :) If you can repro fairly easily, we'll know.

could*

I was also getting some DB PAGE_NOT_FOUND errors, which is why I'm testing my ssd now

but I suppose a concurrency issue could have corrupted a write too

ok I'll rebuild release branch with the patches

That was not doing anything with the db but reads.

But someone reported monero-project/monero #7259, might be a new db related bug.

I saw that, one of the backtraces may have been in the DB, but the other was not