07:57:56 hyc: https://paste.debian.net/hidden/f184fe30/ - I don't get bans with this, do you ? 07:58:23 The drops at the end of each ~10k blocks are still there, I don't mind them for now. 08:00:14 Actually, reading it as posted, it seems it should be https://paste.debian.net/hidden/5f6fca2d/ - building with that one now to test. 11:33:55 will try it in a little while 11:34:27 but you were able to repro the bans without the patch? 11:40:50 Only at the end of the sync, but that was because the source daemon was ~80k height only, and thus inside the HOH hashes. 11:41:18 Better log settings for this: 0,p2p*:DEBUG,*queue*:DEBUG,net.cn:DEBUG 11:42:17 Also, I'm running something based off latest 0.17, not master. 11:42:35 Should not make a difference I think. 11:44:14 Also, it seems weird but appears right: if I want to encode numbers where 50% are 0, 25% are 1, 12.5% are 2, etc (but random), it looks like writing N zeroes with N being the number plus one is pretty good at compression. 11:44:48 But it's kinda de-encoding the number, so it would be totally shite at normal numbers... I found this interesting. 11:46:18 What's about unresponsive synced node ? Is it reproducible or not ? 11:47:24 Unresponsive via RPC ? 11:47:53 That's a very longstanding problem, it's "random" but apparently fairly well reproducible if you leave a node up for hours and wait. 11:48:31 according to logs from hyc there was no response from both p2p and rpc 11:52:52 Any specific requirements for reproduction ? 11:53:13 pruned/full, fast/slow connection, open/close p2p port ? 11:58:22 Heh. Looks like this is actually Huffman coding... 11:59:41 :D 12:10:14 nothing special on source node - full, not pruned. both on same wifi LAN. 12:11:51 Is it reproducible with remote exclusive node ? 12:12:47 probably hard test without fast internet connection 12:13:13 that's a bit more difficult for me to test. my remote node has an in_peers limit. if the 10k disconnect occurs, may not be able to reconnect because other nodes may take that spot. 12:13:22 There is a bug report on github with logs, user name something like ajeuciu4 or the like. 12:13:48 That's about ssl rpc 12:13:56 We're talking about p2p problems 12:14:30 Hmm, OK. I thought this was predating SSL but maybe not. 12:15:19 There are so many reasons why application can unresponsive. We need more info: better log or some runtime info from gdb 12:15:29 * There are so many reasons why application can be unresponsive. We need more info: better log or some runtime info from gdb 12:18:35 or It should be reproduced locally then I'll find all info by myself 12:24:26 you haven't been able to reproduce this? 12:25:10 not yet, syncing old hdd with full node now 12:28:05 2nd chunk of that paste doesn't apply to master. I can't find anything resembling that section of code 12:28:22 neither the surrounding comment nor the error msg text 12:29:33 moneromooo: so I guess I have to test v0.17 12:30:56 eh no, it's not in my v0.17 branch either. must be one of your own patched branches 12:32:09 Oh really. Sorry -_- I'll build vanilla 0.17 then. 12:49:34 thanks mj-xmr, I really appreciate the honesty :)) just wrote a reply there 👍️ 14:33:14 Carrying over a conversation from #monero-research-lab:matrix.org on what's needed for the next update 14:33:15 hyc, re: tx_extra phaseout - im more thinking of phase 1... like, make it so the CLI wallet doesn't support it anymore etc 14:34:00 It's somewhat pressing because Thorchain plans to start supporting Monero, and that will lead to a bunch of publicly revealed transactions (by revealed view key and key images) 14:34:12 i.e., don't touch consensus yet, just make it so users aren't inclined to use it 14:34:38 Does the CLI wallet support it still? 14:34:49 dunno. 14:35:02 extra_nonce is still being set in tx_extra. pool mining breaks without that 14:35:29 pooled mining strikes again 14:35:57 I definitely want to limit tx_extra further, but doing so in like 2 months is too short of a time period 14:36:27 "Synced 95252/2061118 (4%, 1965866 left) (0.948245 sec, 105.457978 blocks/sec)," nothing interesting except frequent disconnects on every ~10000 blocks 14:36:54 so for this version: BP+, messaging 32 bytes in BP+, ringsize bump, ArticMine fee changes. Am I missing anything? 14:37:10 that sounds like all that's actually ready 14:38:11 If we have a huge public wallet coming online, I want to get a ringsize bump in there for sure, sooner rather than later 14:39:43 so whats the ringsize 14:40:45 11 14:40:58 most people supported 17. I personally voted for 15, but this pushes me to support 17 I think 14:41:04 sech1: :p 14:41:33 https://github.com/monero-project/research-lab/issues/79 14:41:44 is there any evidence that 17 has a drastic advantage over 15? with increased daily transactions we want to keep verification time as low as possible 14:42:28 Does anyone here have an opinion? 14:43:24 i mean i honestly think anything in this range is quibbling (11 - 15 or 17), and any increase is probably statistically handwavy at best. But i like increasing it regardless 14:43:45 knaccc, ? 14:44:12 also if we hardfork in 3-4 months then we have to consider that Triptych will have to wait another 9 months after BP+ hf 14:45:06 is thorchain actually gonna hold off for 3-4 months so they can work within 32 byte messaging in BP+? 14:45:27 also I don't want to rush a hardfork because whatever thorchain is doing 14:45:38 yeah 14:46:10 will that take 3-4 months to add? 14:46:55 selsta's messages aren't coming through to matrix.... hmm 14:47:06 dunno... moneromooo ? 14:47:30 selsta: there is no "drastic" improvement over 15 imo 14:47:50 knaccc supports 17 because it saves a churn transaction in his model 14:48:03 actually I think it was 16 specifically 14:48:45 knaccc: "the only two clear possibilities, in my opinion, are ring size 11 or 16" 14:49:48 selsta: how far away is triptych though in reality? 14:50:15 we will know once sarang has spent more time on multisig 14:50:37 9 mo seems fine for me then personally 14:51:00 selsta> also if we hardfork in 3-4 months then we have to consider that Triptych will have to wait another 9 months after BP+ hf >>>> indeed, but I'd argue that 1) triptych is still unknown and 2) these large HF gaps are assumptions 14:51:45 i.e., we assume its better for the ecosystem to have large gaps, when in reality it might be better to have them closer together. just a thought 14:51:47 gingeropolous: assumption in what way? we won't hardfork 6 months after the previous hf, the ecosystem is way larger than years ago 14:52:39 I would be open to an exception for triptych if it really does come along faster, but I also think it's weird we keep assuming triptych is ready to go in short order 14:52:48 lol disagree, hardfork are quite exhausting for everyone involved, I don't see how rapid consecutive hardfork are supposed to improve things 14:53:20 there are several parts, we clearly accept some difficulty to have them at all 14:53:47 sure. 14:53:51 triptych major update with major gains = potentially worth more effort 14:55:56 I am however concerned that we'll have a large influx of public transactions in 2 months, for which I'm not super comfortable with ringsize 11 to protect if I'm honest 14:56:18 will they still be public with the message in the BP+ thing? 14:56:19 influx from where? 14:56:39 this thorchain thing sech1 14:56:54 just remove tx_extra 14:56:55 thorchain's model will have them receiving monero transactions in wallets with a publicly shared view key and key images 14:57:08 from everything except miner reward tx 14:57:14 they agreed not to spam tx_extra 14:57:48 well there's nothing that can stop anyone from publicly sharing viewkeys 14:57:51 they will use either short payment ID (technically part of tx_extra so you got me) or the BP+ storage 15:00:20 also talking about network stability, probably not best to remove tx_extra with no notice 15:03:48 I am not sure if a bump in ring size without Triptych is quite productive 15:03:59 Especially with the high tx volume currently, sync time has already been affected quite a lot 15:08:00 if a lot of people use thorchain (which I expect them to do), then I think 11 is insufficient. 15 would let me sleep at night 15:09:43 Have you tried to sync 1-2 months worth of blocks recently? 15:10:20 nah my node has been running 24/7. wallet syncs are still fast but that's with a local node and a fancy phone 15:11:44 I'd rather address this problem by bumping fees, and taking the BP+ efficiency win sooner rather than later 15:12:34 I'd be opposed against a fork in the foreseeable future, just fyi 15:13:21 yeah, something something triptych eventually, maybe :p 15:13:54 I personally also think that BP+ efficiency gains are not worth a standalone HF. 15:15:24 what are we going to do if thorchain starts processing ~25% of monero transactions 15:15:27 what is your plan then 15:16:23 If they use an encrypted pID as identifier, are those transactions even publicly identifiable? 15:16:48 yes because the wallet they end up in reveals the view key and key images publicly 15:17:14 I don't know what thorchain is exactly. 15:17:45 some DeFi stuff 15:18:01 Why would they suddenly be 25% of tx volume? 15:18:16 broadly, it's a DEX that tried to imitate an instant swap model, where users can send native assets to a pool of semi-trusted nodes and receive another native asset back 15:18:50 you can "stake" XMR in these pools, you can swap with no KYC in an AMM liquidity pool 15:19:25 do they have any public stats? Number of users, activity etc. 15:20:07 https://viewblock.io/thorchain shows 535k transactions total 15:21:45 that's in 1 month 15:21:58 that blockchain only went public last month 15:22:54 April 13 https://medium.com/thorchain/thorchain-launch-multichain-chaosnet-bb9f60008a03 15:24:25 That's across all coins, do you think Monero will be super popular on that platform? 15:25:17 Monero, the only real privacy coin, trading on a DEX with traditional DeFi benefits that allows "staking" for the first time, and has no KYC? Yeah, I think it will be popular 15:25:38 yeah get 500000000% APR until it collapses and you lose everything :D 15:25:58 that's the hot gamble the rest of the crypto space is playing :p 15:29:24 anyway, hopefully you see why I'm concerned about the possible # of public outputs with this coming online 15:30:18 with current tx volume bp+ will save us 1GB chain growth per year 15:31:50 at this point that's just the cherry though, it wouldn't be the main reason for the update because that alone is too small 15:32:04 as to increased ring size, is there any evidence / calculations that a ring size bump without Triptych has any meaningful impact on plausible deniability? 15:32:29 sgp_old: you used to push for evidence before ring size bump in the past :D 15:32:46 yeah plenty, check out the issue where knaccc and I run through the calculations for the same stuff mentioned in mrl-0001 and mrl-0004 https://github.com/monero-project/research-lab/issues/79 15:39:36 > We should only move to ring size 16 if we are willing to pay the price of 46% higher individual tx verification times (and 21% higher verification times in churn scenarios due to one fewer churn tx being required at ring size 16). 15:40:34 I'm skeptical this is worth it :/ A targeted individual will also have issues with ring size 17 without churning. 15:46:43 it all depends how far away Triptych is 15:48:06 So is there serious interest in range proof data embedding? 15:55:49 i think its a better approach than integrated address / encrypted payment ID for any sorta data storage 15:56:17 sarang, ^ 15:57:08 A few things to keep in mind with that 15:57:47 - You get 64 bytes per proof (with some restrictions based on scalar representations), and no more 15:58:06 - Recovering embedded data requires partially verifying the resulting proofs (scalar and hash operations only) 15:58:33 - You likely need a way to differentiate between a wallet that produced a proof with no embedding, and one that produced a proof with embedding 15:59:17 The last one means that a wallet that doesn't know about data embedding would produce proofs using random data, and a recipient would "recover" garbage data unless you had authentication or some other way to signal that a pID is present 16:00:01 Notably, the network _cannot_ detect the difference between a proof with embedding and a proof without embedding 16:00:13 right, thats sorta the goal :) 16:00:20 right? 16:00:59 Sure, but the UX of differentiating these two use cases should be considered 16:01:04 yeah that's a big 16:01:08 *big + 16:01:36 indeed. I imagine ":and a recipient would "recover" garbage data unless you had authentication or some other way to signal that a pID is present" .. could be addressed by some sorta standard or prefix in the embedded data 16:01:46 Yep 16:01:54 .e.g, if the wallet decodes and a tag is present, then its a payment ID. if the wallet decodes and no tag, then its random 16:02:06 Depends on the length of the tag 16:02:10 indeed 16:02:19 A certain fraction of non-embedded proofs would have the tag by chance 16:02:50 i wonder if we could sandwich it 16:02:52 To confirm, there would never be a need for per-recipient pIDs? Or longer pIDs than 64 bytes (accounting for scalar restrictions)? 16:03:15 Because again, you get 64 bytes per proof, and that's it 16:04:52 The `pybullet-plus` branch of my `skunkworks` repository has a demo of this embedding, should anyone find it useful 16:05:12 It would need to be updated to reflect the optimized code in the `monero` repository, which I haven't investigated at all 16:06:00 To confirm, there would never be a need for per-recipient pIDs? Or longer pIDs than 64 bytes (accounting for scalar restrictions)? >>> so, in other words, if you make a multi-output transaction (for multiple recipients), there's only 1 pID 16:06:31 As long as each transaction has a single range proof, you get 64 bytes per transaction to do with as you choose 16:06:34 No more 16:06:52 (technically a bit less, due to some scalar representation restrictions) 16:07:16 So you need to make sure that this restriction is acceptable, because you can't play around with it 16:08:39 are you sure it's 64? moo kept saying 32 16:09:04 For BP+ it is 64 16:10:00 Actually hold just one moment... I need to check something about the scalars involved there, to make sure I wasn't implicitly assuming common recipients 16:11:12 Rats, it _is_ 32 bytes... you only get 64 bytes if you can assume a common recipient among all outputs 16:11:32 Sorry for the confusion... I neglected to account for this while re-reading my demo code 16:13:31 You can include any value representable as a scalar element in the prime-order curve subgroup 16:14:51 sure, so the 64 byte case is unrealistic unless someone wants to record a message on-chain for themself or something 16:15:28 It does highlight that this approach in general implies restrictions that can't be avoided 16:15:41 Whereas a separate pID field, while irritating for several reasons, offers flexibility 16:20:14 well, flexibility of 8 bytes instead of 32 bytes 16:20:59 in practice BP+ storage seems more flexible in 99% of cases 16:22:11 I mean you could decide to do pIDs per recipient, or longer values, etc. and have this be a fairly arbitrary consensus decision 16:22:25 Whereas you can't do this with the BP+ approach 16:24:06 are pids per recipient supported? 16:24:21 I thought we only supported 1/tx currently 16:24:26 So the strategy for this would be that if a client detects a transaction for which it is a recipient, it would do the following: 16:24:36 - compute a hash variant of its shared secret 16:24:45 - partially verify the range proof 16:24:56 - use this data to attempt to recover embedded pID data 16:25:10 - use any signals/flags defined by the protocol to identify if data is present 16:25:19 - recover data if present 16:25:43 - ignore if data is not present (or if data is intended for another recipient) 16:28:13 Keeping in mind that it is possible for a sender not to properly embed data, in which case the signals/flags would need to be used to identify the presence of data 16:28:23 False positives would be possible 16:31:23 yeah, but if a recipient is not expecting a pID, then false positives are sorta benign. 16:32:17 but you definitely want to avoid false negatives 16:32:49 ? 16:33:07 A recipient can always attempt to recover embedded data 16:33:23 18:24 I thought we only supported 1/tx currently <-- correct 16:33:43 I want to separate two things 16:34:01 1. the 8 byte payment ID in the way it's currently implemented 16:34:31 2. any other conjuration of a form of payment ID with tx_extra playspace 16:35:51 so one ID/recipient would fall under 2 since that's not how anything currently uses it 16:50:42 did anyone ever see "Invalid decimal point specification 11" when opening a .keys file? 16:50:46 corrupted .keys file? 18:17:43 "Sent ... (3.85 GB) ..., average 253.33 kB/s = 12.37% of the limit of 2.00 MB/s", "Synced 1216600/2061198 (59%, 844598 left) ..." nothing interesting, still syncing 18:18:12 What was the lowest height during some ban event ? 18:35:53 "... sent wrong NOTIFY_RESPONSE_GET_OBJECTS: block with ... wasn't requested, dropping connection" such errors are regular but they lead only to disconnect without ban 18:36:58 Odd. That should only happen for very slow connections. Is that a very slow connection ? 18:39:19 both nodes (87% synced full hdd node and 2% synced pruned tmpfs node) are on the same machine but communicating through 192.7.7.1 assigned to loopback 18:39:49 connection between them is surely not slow but hdd node has long delays due to writes 18:43:53 Slow here means a node asks for data, but doesn't receive a reply within 60 seconds. 18:44:32 I guess it might do that when commiting a large txn to HDD. 18:49:17 im doing the monero book's tutorial in chapter 7, 7.4.2 Tutorial 2 - How to generate a pseudo-random address, utils.hash_to_scalar(secret_spend_key) does not exist in any python library? 18:50:31 import utils, doesnt have alot of functions, such as utils.sc_reduce32, utils.hash_to_scalar, utils.publickey_to_privatekey, etc, so on 18:52:04 "https://github.com/monero-project/mininero/blob/master/oldMiniNero.py#L317" ? 18:54:37 "Odd. That should only happen for very slow connections.", "... kicking idle peer ..." yes, idle peer kicker isn't correct and triggers that error 18:54:46 but anyway it isn't important now since it doesn't lead to ban 19:02:34 moneromooo: What avg upload speed from synced to fresh node did you have at the moment of ban ? 19:03:29 I did not have a ban. hyc did. 19:04:13 Well, apart from the sync ending at 140k (source daemon was there) but it's unrelated. 19:05:35 then there is no successful reproduction outside of hyc environment 19:05:50 * then there is no successful reproduction outside of hyc's environment 19:06:14 https://github.com/fireice-uk 19:06:15 oh shit 19:06:22 look who's makin ga lot of commits 19:06:39 ratthing69[m]: please not here :P 19:07:08 moneromooo: do you have an updated patch? or should I just go ahead with chunk#1 of that one? 19:08:22 https://paste.debian.net/hidden/fe688136/ but that should only trigger when you get past the builtin hashes, so not relevant to your bans AFAICT. 19:10:15 hyc: Can you try to reproduce in simplified environment with both nodes on the same machine or full node without out peers? 19:11:23 er, when you sync off a daemon that's itself synced to a height that ends before the end of the builtin hashes. 19:15:02 wfaressuissia[m]: both nodes on same machine, only got disconnects 19:15:16 that was log1/log2.txt 19:15:41 anyway the serving node was fully sync'd 19:15:44 What's about no other peers (--out-peers, --in-peers 1) ? 19:16:03 so hashes would be well beyond the builtin hashes 19:16:40 yeah I can set it up again and try that 19:17:48 What was the last commit/branch without these bans ? 19:20:14 "v0.17.1.5, v0.17.1.6" it's expected v0.17.1.5 shouldn't have bans but v0.17.1.6 will have 19:20:22 Can you try ? 19:21:14 that will take quite a long time to step thru 19:21:45 I'm now running 2 builds of master on same box 19:22:03 no other peers 19:22:17 seeing the 10k disconnects, no surprise there 19:23:07 "that will take quite a long time to step thru" to try 2 tags or to do bisection between them ? 19:23:34 both, because it takes a long time starting from scratch before a ban occurs 19:24:09 Can you send me your environment I'll try something :D ? 19:24:23 wdy mean? this is ubuntu20 laptop 19:25:20 nothing 19:25:44 "I'm now running 2 builds of master on same box" how many % are already synced without bans ? 19:25:58 it is just starting out, 4% syncd so far 19:26:08 fresh run, ok 19:26:37 since this is over localhost I don't believe it will hit any ban 19:26:41 that was only across wifi LAN 19:26:52 but I'll let it run a couple hours and see 19:27:40 nothing here even with non-loopback address 19:27:51 assigned to `lo` 19:27:58 of course not 19:28:05 you're still using the loopback interface 19:28:34 that means some network delays are required to trigger the problem 19:28:42 probably 19:28:55 also notice that bandwidth limits don't impede sync over loopback at all 19:29:45 sw bandwidth limit can't be reached even with fast CPU and SSD? 19:30:22 I'm seeing ~15Mbps in and out over loopback now 19:30:38 `print_net_stats` the result from monerod ? 19:31:05 sent average 1.39MB/s 19:31:12 69.45% of 2MB/s limit 19:31:24 ok 19:32:35 Received 7478183 bytes (7.13 MB) in 2376 packets in 14.1 minutes, average 8.66 kB/s = 0.11% of the limit of 8.00 MB/s 19:32:35 Sent 1242666684 bytes (1.16 GB) in 38907 packets in 14.1 minutes, average 1.41 MB/s = 70.29% of the limit of 2.00 MB/s 19:33:17 10% syncd now 19:33:57 also - reading and writing to 2 separate SSDs 19:51:17 `ip link add name lo1 type dummy`, "https://wiki.linuxfoundation.org/networking/netem" it should be enough to simulate wifi connection 19:56:38 42% sync'd here no slowdowns. just the usual disconnects 19:58:05 It's expected that other peers connected to 100% node are more important for reproduction than wifi connection since they add more interesting delays into async code 20:04:00 yeah I get the feeling this is only going to show up with other active peer connections 20:04:11 but I'll restart again soon over wifi 20:04:51 over wifi without other peers ? 20:05:08 right 20:05:15 good 20:05:33 it's 50% now I think that's good enough, going to exit and start again 20:07:08 sad that this macbook "pro" has no wired ethernet port 20:09:14 and no possibility to catch thunder strike via ethernet from router :D 20:09:49 well. it's still plugged in to AC power 20:10:11 and WAN port is fiber, so no threat there 20:11:04 I had one because it wasn't fiber internet :D 20:11:25 bummer. talk about rotten luck! 20:11:55 everything else survived except laptop 20:12:06 and router 20:12:07 path of least resistance 20:16:11 huh. 20:16:24 starting on mac with --add-exclusive-node but it is still talking to other peers 20:16:55 oops 20:18:01 other nodes hitting inbound port, forgot I set a port-forward 20:33:42 16% sync'd. looks like no issues 20:34:01 so the problem only occurs when the fullnode has other connections 21:00:20 42% syncd still going 21:00:51 so yeah, can't repro without other peers 21:01:15 What will try after 50% of current test ? 21:01:22 * What will you try after 50% of current test ? 21:01:48 I suppose, re-enable other peer connections 21:02:06 "https://downloads.getmonero.org/cli/monero-linux-x64-v0.17.1.6.tar.bz2" Can you re-enable other peers and use this binary ? 21:02:42 ok 21:07:00 as which side, full node or new node? 21:12:30 It's likely enough for fresh node 21:14:19 but it would be simpler to use for both if all other fixed problems since 17.1.6+ will not pop up (out-of-memory, ...) 21:14:45 I would have to compile 17.1.6 for my Mac 21:15:18 which is doable of course. just takes more time 21:15:37 "https://downloads.getmonero.org/cli/monero-mac-x64-v0.17.1.6.tar.bz2" this one isn't usable ? 21:15:48 I mean an ARM binary 21:16:05 that one should run, because of binary translation 21:16:08 but that seems like a bad idea 21:18:58 How much time does it take to compile only monerod target ? 21:19:00 20m ? 21:23:34 starting right now with master on mac 21:23:41 will try 17.1.6 later 21:23:52 mac is full node for now 21:23:55 linux is new node 21:45:10 about 19% sync'd got a ban 21:46:12 mac seems to have been frozen for a while 21:52:03 http://highlandsun.com/hyc/log1.txt and log2.txt 21:52:08 log2 is the mac fullnode 21:52:18 log1 is the linux new node, 17.1.6 21:52:39 it just seems to be banning because the mac timed out 21:53:13 these are both with loglevel 0,p2p*:DEBUG,*queue*:DEBUG,net.cn:DEBUG 21:57:31 also, the Mac is not cleaning up connections https://paste.debian.net/1196474/ 21:57:48 1.155 is the linux new node, 1.214 is the mac full node 21:59:18 uh, try this https://paste.debian.net/1196475/ 22:08:56 mebbe it's just a macos bug 22:39:15 never noticed any freezing 23:04:03 it definitely is not doing clean shutdowns of those incoming sockets 23:04:16 and it definitely is being to slow to respond 23:08:06 is the issue with connections not cleaning up on mac only? 23:13:24 haven't fired up a 2nd linux box yet to see 23:46:58 this is also strange. the connection to the mac has been in close_wait for several minutes but is still showing up in debu output 23:47:10 tcp 116585 0 192.168.1.155:56720 192.168.1.214:18080 CLOSE_WAIT 23:48:03 2021-05-05 23:43:51.665 D [192.168.1.214:18080 OUT] next span in the queue has blocks 2353364-2353383, we need 2353364 23:48:56 2021-05-05 23:48:08.329 D [192.168.1.214:18080 OUT] Block process time (20 blocks, 781 txs): 12118 (7042/5076) ms 23:49:02 or I suppose those are new connections 23:49:13 and that one is just still waiting to be cleaned up 23:49:24 dunno since the output doesn't include local port# 23:52:10 no sign of those conns on the Mac side 23:52:13 phantoms 23:58:07 there are no other connections. this node 1.155 thinks it has a conn to 1.214 but 1.214 doesn't show any and the one on the 1.155 side is in CLOSE_WAIT 23:59:24 ok it finally closed properly 23:59:50 so for several minutes it was ignoring that conn, not reading anything off it, while remote side had already closed it