00:06:09 <moneromooo> hyc: if I call a loop of m_db->have_block(H); sleep(1) in a thread, without an overarching db txn, and at some point another thread adds the block H, which did not exist prior to the addition, will have_block switch from returning false to true at roughly the right time ?
00:06:26 <moneromooo> If sounds "yes, duh", but it's unclear when a read txn "resets".
00:12:14 <hyc> if there's no overarching txn, then yes, that call gets a new txn each time
00:12:32 <hyc> and a new txn always sees the current state of the DB at its time of creation
00:13:05 <moneromooo> ty
00:13:05 <hyc> so, yes, duh.  :P
00:23:08 <FROMSOUTHWEST> i first started monero as a gui and clicked on remote node. but now I started monerod to run a fullnode myself. will my gui recognize it when restarting or do I have to reconfigure it?
00:23:34 <selsta> FROMSOUTHWEST: select local node inside Settings -> Node
00:35:14 <FROMSOUTHWEST> ok, thx
01:18:38 <selsta> .merge+ 7234 7235 7220 7249 7246 7247 7248
01:18:38 <xmr-pr> Added
01:36:52 <selsta> .merge+ 7250
01:36:52 <xmr-pr> Added
02:17:09 <selsta> vtnerd: please also 7251 against master so that we don’t go out of sync
02:17:25 <selsta> .merge+ 7251
02:17:26 <xmr-pr> Added
02:49:11 <one-horse-wagon> Is there a limit on number of i.p. addresses you can have in the ban-list for monerod?
02:52:20 <Mmmmmmmmmm> Based on my limited experience working with the JsonRPC, I'd say No. `std::map<std::string, time_t> blocked_hosts = m_p2p.get_blocked_hosts();` Blocked hosts are stored in a map that can get as large as you'd like it to be
02:59:28 <one-horse-wagon> I've got 2833 addresses blocked, including all TOR exit nodes on my two full nodes to see if it stops the nonsense.
03:03:16 <selsta> my node did not crash once with https://gui.xmr.pm/files/block_tor_new.txt
03:04:45 <Mmmmmmmmmm> I'm not accepting incoming connections :D
03:05:07 <selsta> that works too lol
03:45:38 <one-horse-wagon> There are a number of i.p. addresses on your list that are not on mine.  I'm going to combine both lists, get rid of the duplicates and use it.
03:46:09 <selsta> mine is tor exit nodes + relay nodes with 18080 exit policy
03:56:03 <Mmmmmmmmmm> What's the ethos on allowing people to make mistakes? For example, in my API I can theoretically allow anyone to send an unlock_time of up to uint64_t. But that unlock_time (the largest unsigned 64 bit integer) is literally an eternity. Should I cap the unlock_time value at 1 year's worth of blocks? Say, throw an exception? In C# there is no
03:56:04 <Mmmmmmmmmm> overflow checking, so if someone does 0 - 1, they'll get ulong.MaxValue (18,446,744,073,709,551,615), which is a long time to be waiting for your funds to unlock :)
03:56:55 <one-horse-wagon> My list is a complete listing of all tor exit nodes.  I'm missing your relay nodes . Combining and getting rid of duplicates will add them in.
04:00:13 <selsta> Mmmmmmmmmm: https://github.com/monero-project/monero#blockchain-based
04:00:19 <selsta> I don’t think other APIs limit this
04:01:29 <Mmmmmmmmmm> If someone accidentally misuses the API and sends funds with an unlock time of 18 quadrillion blocks, I'm getting a hitman called on me.
04:02:17 <selsta> lol
04:08:08 <Mmmmmmmmmm> Maybe I'll add a Safe and Unsafe transfer mode
04:08:22 <Mmmmmmmmmm> Imagine Hodling for 18 Quadrillion blocks...
05:01:22 <ndorf> diamond hands
10:21:37 <MarkoSK90> jess: I can tell you why you can't defeat the 'spam'. You are thinking in cult doctrine. If it was real spam, and I was selling Viagra for example - you could easily ban keywords and urls. Instead, stop being a sheep, think like a cult leader. Recoginse that this 'spam' is just some bullshit that you tell to the sheep.
14:09:47 <FROMSOUTHWEST> how can the miner retrieve the fee when the amount of the outputs being send is hidden?
14:14:37 <moneromooo> Go through all the txes included in a block, sum up the fees, add that to the subsidy.
14:15:11 <moneromooo> Output amounts being hidden is irrelevant.
14:20:25 <FROMSOUTHWEST> does the miner calculate the fee of every transaction himself?
14:20:30 <FROMSOUTHWEST> sry im a bit confused
14:20:48 <TheCharlatan> FROMSOUTHWEST the tx fee is in plaintext in the transaction. The output commitments only commit to 0 with the fee taken into account.
14:26:13 <FROMSOUTHWEST> ah ok. so i just subtract the fee amount from my change?
14:28:29 <moneromooo> If you send X, using Y, and the fee is F, you will get back Y-X-F as change.
14:31:00 <FROMSOUTHWEST> ok, thx
18:00:42 <FROMSOUTHWEST> is the blockreward penalty only when the block size is bigger than 300kb or also when the block the block size is bigger than the average of the last 100 blocks?
18:03:14 <moneromooo> Bigger than the median of the last N blocks, possibly with a delay.
18:10:11 <mj-xmr> Hi and welcome in the New Year.
18:10:58 <mj-xmr> My browser crashed. Was there any feedback on my initial ML attack detection proposal? If yes, please kindly copy-paste it. (I need to install that bouncer today)
18:18:46 <Lyza> my node is currently unresponsive to RPC, but isn't using more ram or cpu than normal. is there like, a new fun thing going on?
18:20:03 <dgoddard> @Lyza is monerod process in a running state? Is the machine as a whole otherwise responsive?
18:20:37 <Lyza> <dgoddard> yes
18:22:22 <Lyza> just restarted and it looks like I'm about 700 blocks /24 hours behind
18:22:30 <selsta> Lyza: which version are you running?
18:22:37 <Lyza> 0.17.1.8
18:22:46 <selsta> dns blocklist enabled?
18:22:57 <Lyza> yes and also using block_tor_new.txt
18:23:10 <selsta> ok, DNS blocklist has a deadlock
18:23:16 <Lyza> which Ik is redundant but I was still getting oom errors without the tor stuff blocked
18:23:19 <selsta> that can trigger if you are unlucky
18:23:27 <Lyza> oh ew ok will disable for now then
18:23:34 <selsta> it is fixed in code already
18:23:50 <Lyza> cool
18:23:52 <selsta> will be included in the next release
18:24:20 <Lyza> perfect, looking forward to being able to use that and stopblocking tor exit nodes, even though people shouldn't be using exit nodes to connect to me =P
18:34:53 <dgoddard> @selta was this the fix? https://github.com/monero-project/monero/pull/7221
18:35:15 <selsta> for what?
18:35:36 <selsta> deadlock? https://github.com/monero-project/monero/pull/7239
18:35:52 <dgoddard> ty
18:43:47 <FROMSOUTHWEST> can someone explain me what deadlock banning is
18:44:39 <selsta> it means the daemon freezes if it tries to ban peers while it is currently updating the peer list
18:44:56 <selsta> just a bug, not an attack
18:45:33 <selsta> it also happens only rarely
18:51:05 <sech1> .9 tag when?
18:56:17 <selsta> soon^tm, still trying to set the object limit so that it does not break wallet synv
18:56:21 <selsta> xync
18:59:09 <selsta> .merges
18:59:09 -xmr-pr- 7220 7234 7235 7237 7239 7243 7244 7246 7247 7248 7249 7250 7251
19:02:18 <Lyza> calling monerod ban with a ban list spams the absolute hell out of the log file if it's a long list like block_tor>new.txt -- just saying
19:04:31 <selsta> spams 1000 entries?
19:04:40 <selsta> or more than 1 per ip?
19:06:22 <sech1> 1 per ip
19:06:46 <sech1> but I don't care, my bitmonero.log is already 45 MB :D
19:07:40 <selsta> not sure how else to do this apart from printing the IPs that are getting banned
19:07:42 <Lyza> yeah is only 1 per entry but still, would be nicer I think to just say like 'loading file whatever.txt into ban list' or w/e
19:08:30 <Lyza> just makes it annoying to look for anything else in the logs, my logs are like 98% so and so is blocked because I update the ban list once an hour
19:09:16 <moneromooo> man 1 grep
19:11:08 <Lyza> lol thanks yeah I can use grep, could probably get better at it. still though.
19:11:19 <Lyza> *shrug* I ain't arguin it was just a thought. seems a bit much to me
19:11:41 <selsta> which log level do you use?
19:11:47 <Lyza> the default, zero
19:18:51 <Lyza> honestly it's ban enough that I'm going to go back to downloading the new list, diffing them, and banning the new entries one at a time. the way @ban works now, it takes a full ~20 seconds to run as it appears to load the every entry in the list one at a time, every time
19:19:38 <selsta> you mean the ban @filename command?
19:19:46 <Lyza> yes
19:19:46 <selsta> ok, weird
19:20:17 <selsta> it takes less than 1 second on my system
19:20:24 <Lyza> weird
19:20:41 <selsta> ssd?
19:21:03 <Lyza> nah it's a spinning raid5
19:21:18 <selsta> maybe that’s why
19:21:30 <Lyza> not sure why it would cause that much disk io
19:24:52 <Lyza> ok looking at older logs, it did load faster when I wasn't actively syncing, I was still finishing up from being a day behind
19:25:30 <selsta> with .9 hopefully the ban list will be less important again
19:26:05 <Lyza> hopefully :)
19:28:42 <Lyza> selsta do you know if the monerod freezing issue also happens with the ban command, or just the dns based bans?
19:28:54 <selsta> both
19:28:56 <Lyza> cause my RPC seems frozen again, even though I'm synced with the network
19:28:57 <Lyza> ah shit
19:29:00 <Lyza> well damn
19:29:06 <selsta> hmm
19:29:21 <selsta> if you only apply the ban list on start
19:29:25 <selsta> it should not happen
19:29:38 <selsta> with the next version it can't freeze anymore
19:30:01 <Lyza> roger that I'll just restart monerod to update the ban list for now, and only update it like once a day
19:30:32 <selsta> the longer the ban list the higher the chance for it to freeze
19:30:50 <Lyza> makes sense, unfortunately I was still crashing from OOM without blocking the tor nodes
19:31:08 <selsta> .merge- 7248
19:31:08 <xmr-pr> Removed
19:31:18 <selsta> ^ readded once confirmed
19:31:23 <selsta> .merges
19:31:23 -xmr-pr- 7220 7234 7235 7237 7239 7243 7244 7246 7247 7249 7250 7251
19:32:22 <selsta> Snipa luigi1111w? :)
19:58:28 <selsta> moneromooo: I assume https://github.com/monero-project/monero/pull/7254 will result in less bans, right?
19:58:49 <selsta> due to them being deprioritized after kicking once
19:59:05 <moneromooo> Likely.
20:06:40 <science> why I got so little connected incoming peers after switching to 0.17.1.8, maybe i messed up my firewall settings or new nodes cant connect to old nodes and vice versa?
20:07:54 <selsta> how many do you have?
20:08:07 <selsta> did you just freshly start your node?
20:08:48 <science> yeah i think im getting more, but I used to get more new connection quicker after freshly starting my node, but its ok i guess
20:09:44 <selsta> mine have between 40-70 incoming connections which seems normal on v0.17.1.8
20:10:31 <science> I got 9 incoming peers after 15mins FYI
20:10:49 <selsta> that sounds normal
20:31:17 <mj-xmr_> I'll ask my question in a different way: If I spend time on properly analysing the banning problem from a ML perspective, create a proper CCS proposal, and in the end provide an elegant self-adapting solution, would you be fine accepting it or are you in a way ML averse? Or have some questions that need clarifying?
20:34:02 <moneromooo> How good it is, whether it can adapt well (ie, no overfitting to current weather), whether it needs gobs of CPU, how much new code it needs...
20:34:09 <moneromooo> Who knows in advance without more info ?
20:34:42 <moneromooo> Oh, and how much type 1/2 errors.
20:34:53 <moneromooo> Guess it's how good it is.
20:34:54 <mj-xmr_> OK. You're not ML averse, just skeptical. This is OK for me.
20:36:47 <mj-xmr_> So these questions I will be able to answer once I'm nearly ready with this. So just to mention that if I go in this direction (which is a large investment), and I provide satisfying results, I understand that you'd accept it.
20:37:08 <mj-xmr_> as far as I understand.
20:37:57 <moneromooo> I'll refer you to my previous line :)
20:38:15 <mj-xmr_> There are methods to make it good. It requires time.
20:39:01 <mj-xmr_> And imagine a situation that I work like a dog, I'm satisfied and can prove that it's good and somebody tells me - not acceptable, because Skynet, I'd be quite sad :)
20:41:46 <mj-xmr_> Overfitting can be mitigated by simulating various random time series processes. I imagine, that I'd define a few bad agents, which after a period of silence, would suddenly start spamming. The quality would be measured by the amount of 1/2 errors.
20:42:10 <mj-xmr_> *few bad agents and many good agents of course.
20:43:17 <mj-xmr_> And by saying "time series process" I mean for instance the distribution of traffic over time.
20:43:30 <moneromooo> That kinda screams for a proxy, does it not.
20:43:44 <moneromooo> Then you can go wild.
20:44:53 <mj-xmr_> Proxy in what meaning? I imagine that each node would have a pre-trained model and would in parallel do an on-line training, and perhaps make an ensemble method of both.
20:45:02 <moneromooo> Proxy being MITM software.
20:46:43 <mj-xmr_> I don't understand where MITM software fits to what I'm saying.
20:47:00 <mj-xmr_> I come from a bit different field :)
20:47:26 <moneromooo> Software that runs between monerod and the network.
20:47:41 <moneromooo> Which then decides whether allow some traffic or not.
20:47:51 <mj-xmr_> OK I'd say that the monerod would just have a class that does it.
20:48:14 <mj-xmr_> No need to making an another binary, from my perspective.
20:48:55 <mj-xmr_> I just need to hook some filter into the part, which has access to realtime traffic flow, nothing more.
20:49:17 <luigi1111w> .merges
20:49:17 -xmr-pr- 7220 7234 7235 7237 7239 7243 7244 7246 7247 7249 7250 7251
20:49:20 <luigi1111w> .soon
20:49:38 <mj-xmr_> I think this makes an easier architecture, but I'm open for other opinions.
20:52:46 <mj-xmr_> Can somebody tell me how you extend the ban list currently? By hand? Is there a class that tries to do it automatically?
20:53:27 <moneromooo> Call block_host on the p2p object.
20:54:21 <mj-xmr_> I shall have a look. Thanks.
20:59:39 <mj-xmr_> And to make it clear - the fitness will strongly depend on the quality of the simulation. I just think, that it's quite easy to define the objective function here by answering the question "what hurts me most?". I currently imagine that it's sustained high traffic from a given host or from a set of hosts, incomparable to the traffic of other connected hosts (even if they're in minority)
21:02:47 <moneromooo> I think the objective is wrong. That can be left to existing software. If you want to do some ML based analyzer, I think you should concentrate on monero specific things, not things that any existing proxy can handle.
21:04:36 <moneromooo> (I assume there's already lots of existing proxies to detect/shed high traffic, but I don't know that for a fact)
21:04:57 <mj-xmr_> I'm not pushing a specific solution.
21:05:14 <mj-xmr_> It's just my idea how it could be handled.
21:05:37 <mj-xmr_> My current idea.
21:06:52 <mj-xmr_> Or in general I'm trying to address the problems, currently plaguing the development, whatever solution we choose.
21:06:59 <mj-xmr_> I'm just reading up on the proxies.
21:08:45 <moneromooo> It just means a separate binary, without having to plug it in another.
21:09:38 <mj-xmr_> I understand now. I'm just looking for a proxy solution, being able to filter out spam.
21:10:53 <selsta> I think it would make more sense for someone who has a let of network knowledge to look at our current code and search for issues / improvements, a ML solution sounds significantly more risky, especially if the underlying code still has issues.
21:14:42 <mj-xmr_> I'm not a network specialist. You only have to take into account, that for Linux you'll find filtering tools, but Monero is multiplatform, so no IPTables under Windows.
21:19:22 <moneromooo> do you mean no iptables, or nothing like it ? The former, maybe, but I have trouble believing you if the latter.
21:20:46 <moneromooo> Then again I really don't care to discuss windows firewalls ^_^
21:23:46 <mj-xmr_> I mean no iptables. The reason why I'm saying this, is that you don't want to have to maintain a different proxy for each OS. I'm just looking for something multiplatform.
21:25:25 <mj-xmr_> I already figured out that for the testing part (not production), one could use several traffic generators.
21:25:57 <selsta> luigi1111w: how soon is soon
21:25:59 <mj-xmr_> Not sure about the actual filtering.
21:29:31 <mj-xmr_> I see, that there are some GUI tools, but neither they are multiplatform, nor free / open source/
21:29:47 <mj-xmr_> For limiting bandwidth.
21:31:48 <moneromooo> tbh I'm not sure there'll be many people who (1) run monerod on more than one OS and (2) don't already have their preferred firewall.
21:32:24 <moneromooo> So making yet another firewall just for this is likely not very productive.
21:32:58 <mj-xmr_> Do you suggest then rather creating a proper firewall configuration?
21:33:09 <moneromooo> No.
21:33:11 <mj-xmr_> and distributing it? Or at least documenting?
21:36:15 <mj-xmr_> And if a node of a Windows user doesn't actively block the bad traffic, but Linux do, because we've setup a firewall (somehow) under Linux, you'll be effectively locking out Windows users, since they will be spammed.
21:38:10 <luigi1111w> tonight probably
21:40:14 <mj-xmr_> It might be maybe 500 lines of portable C++ code. And by saying ML, I'm not necessarily pointing at Deep Learning stuff. Sometimes it's enough just to use a moving average and a moving standard deviations.
21:40:27 <mj-xmr_> But OK. If you change your mind, you know whom to ping.
21:52:10 <moneromooo> That seems ok to have in monerod. I was assuming machine learning since you mentioned it.
21:53:17 <mj-xmr_> Even the linear regression is a machine learning method. So sorry for confusion.
21:56:46 <mj-xmr_> Here's what I propose - I'll try to create a very rough proof of concept, if *anything* works under a poor-man's simulation of an attack, I'll create a proper CCS proposal with the initial results, and we shall talk later. Deal?
21:57:02 <mj-xmr_> (and the simulation will be done via nmap)
22:00:08 <moneromooo> 500 lines of C++ for things like linear regression for monerod specific stuff seems ok. The same for traffic volume/endpoints doesn't IMHO. But I don't get to decide so don't feel like you have to get a deal with me.
22:01:03 <moneromooo> Once a traffic volume DoS hits monerod, it's a bit late to do anything anyway.
22:01:04 <mj-xmr_> Sure, I'm waiting for feedback from others.
22:01:11 <moneromooo> iptables or wahtever else will be a lot faster to drop.
22:02:02 <moneromooo> monerod has access to info iptables (or similar) does not, so that's what it should be using to determine whether some traffic is bad.
22:02:31 <mj-xmr_> Couldn't monerod expose an ip to iptables or other FW right after detection?
22:02:51 <moneromooo> Sure. Sounds good.
22:03:45 <moneromooo> I'd suggest using the same scheme as currently ("dropping connection$" on some particular category).
22:07:32 <mj-xmr_> Noted down.
22:09:03 <mj-xmr_> nmap has got it all... https://null-byte.wonderhowto.com/how-to/use-nmap-7-discover-vulnerabilities-launch-dos-attacks-and-more-0168788/
22:21:51 <selsta> Snipa: do you have a bit earlier time for merges than luigi?
22:22:07 <selsta> Something is conflicting so we need some merges.