Review requested: monero-project/monero #6451

What is the preferred amount of lockable memory to dispel the warning? Is this some established value?

nice compact improvement in 6451 :)

Thanks TheCharlatan!

ah, love that 6451 sarang

Hello

I'd like to contribute to the project, but would like to learn, if you are willing to accept my change, before I even start working. References:

The POC contribution can be easily done unfunded.

mjxmr: i think that some parts go a bit crazy with templates (-ftemplate-depth needs to be increased). do you think it would be worth going through with clang's profiler seeing if there's anything that should be changed?

asymptotically: I think, that using profilers is always a better idea, than using a gut feeling (same applies for speed optimisations). Otherwise a programmer falls into a dogmatic way of thinking, while I prefer to work evidence based.

However in this case, boost headers, I'm absolutely sure, that this is the major slowdown. Abuse of templates create the need of having to include the boost headers in a parent header. This can be avoided with some effort.

I am so sure, because I ate my teeth on this many times already.

also what does the ccache integration with cmake do? i thought you just had to set CC and CXX

Yes, you can do this of course, but not everybody is proficient enough to do this. My goal would be to set the CC / CXX automatically if ccache is detected in path. This is minor thing, yeah.

At our job we disable ccache for special final builds, to minimize the risk of ccache affecting the binaries that go to the customer, but we have it on by default.

... during the development, I mean, so 90% of time.

asymptotically: "Abuse of templates create the need of having to include the boost headers in a parent header. " A poor man's solution to this is using precompiled headers of the boost headers collection. This helps to reduce the compilation time by up to 50% per header, but a lot more this can be achieved via wrappers.

Perhaps it wouldn't be a bad idea to start off with this, as the least intrusive measure?

I’d wait for feedback by moneromooo / vtnerd

Thanks for the tip. While I wait, I want to give mad props for cutting off the ASICs. This was a giant leap towards decentralisation.

A speed boost would be nice. Whether it's worth churn that can introduce bugs depends on how much is gained.

Last I tried, precompiled headers did not gain much, the patch end up rejected. Maybe your method might gain more though.

Someone else started working on moving some serialization away from headers. Can't recall the nick now though.

ccache works fine with monero, I use it all the time.

I did not know CLANG had a compile time profiler. Sounds nifty.

not sure if chromium is the only tool that can open the output :/ aras-p.info/blog/2019/01/16/time-tr…line-flame-chart-profiler-for-Clang

moneromooo: What would you propose doing now? It might be faster to just start moving the serialization away, since I have a lot of free time until the mid-May. I don't know how I would find the contributor with over 2.200 forks currently.

But I see, that there is interest. I think I'll just start then :)

How about doing something else, since nobody's doing precmpiled headers AFAIK ? :)

IIRC when I tried it, the reason why it did not gain much at all was we had several directories, and the precompiled thing was being done one per directory.

Anyone remembers who was asking about changing serialization ? IIRC we talked about the speedup vs churn pros/cons too :)

This is also cool. But when was the last time, that you heard from this person, who offered moving the serialization away from headers?

Very roughly, I'd say... a couple months maybe ?

because I can work full time on this now, until mid-May, meaning I could quickly bring it into life. Then just an hour or two after work.

Well, your pick really.

I think I'll give it a shot.

Historically, I've been pushing against churn (ie, rewriting stuff for no good reason (aka some code standard says so)). I'm trying to do it less now :)

So build times are a good reason, if the speedup is substantial. I'm curious to see how much you get.

The point here is obviously not to create any differences in functionality.

but I know from my experience, that developers usually are careless, and need a janitor :) So if you assume no churn, the mess never gets cleaned up.

was it woodser ?

I don't think so.

A small finding:

enjo@tp:~/devel/github/monero-build$ make test

Running tests...

Test project /home/enjo/devel/github/monero-build

No tests were found!!!

make release-test

To run one particular tests, make -C $BUILDDIR/subtree test

eg: make -C build/release/foo/tests/unit_tests test

make release-test

make: *** No rule to make target 'release-test'. Stop.

building out of source.

You messed up your Makefile most likely. Check the README.md file.

See git diff

Ah ok. This can't be executed from an out of source directory.

It can if you know what you're doing. You hinted you know cmake, so look at the contents of Makefile.

it's been a couple of months since i last tried the tests but iirc it just worked when i did `cmake ..` instead of trying the makefile

Let's say, that I will address this at a later time. For now I will be happy if the tests execute and keep me from screwing things up.

moneromooo: For how long do the core_tests normally run? I feel like they hung since like an hour.

Sounds plausible. If you run them often, there's a (closed) patch from me on github that speeds them up a lot.

Takes ~ 10 min with the patch or so.

Thank you.

I will run them for the night then.

Could you give me a hint, which tests cover the class tx_source_entry? I think I will start from this one as a sample small enough.

Transaction tests.

Which is going to be most of them.

Great. Thanks!

mjxmr moneromooo selsta : I have a github issue related to serialization, the technique woud reduce compile times as things can be split into a cpp

this one?

other things can be done to reduce compile times, but doing so without re-organizing large parts of the code at once coudl be tricky

yes

someone could reduce it further if the macro stuff was a bit much, but the general idea of splitting header/cpp is somewhat straightforward

the biggest compilation issues are the epee server stuff, and its going to be tricky to refactor that without it being a full rewrite (it may have to be to achieve the goa)

I started some small stuff with hex encoding, trying to remove boost::lexical_cast from being included so often, its fairly heavy to pull in the inculdes

gcc and clang both have a flag for reporting internal time spent compiling which is useful too

the flag gives breakdowns on much time spent doing macro expansion, etc., so you can get an interna look at time spent

ah found the proposal was in the other tab

yeah moving them into a cpp should help, but the bulk is _probably_ epee, but its hard to profile for certain due to the ways things are compiled

the boost serialization stuff was separate than what I was working on, and has nearly the same problem+solution

vtnerd: Shall we coordinate our efforts? My concrete proposal is to completely wrap the boost::archive. Putting the serialization templates can't be achieved, as it turns out, but there's no problem in wrapping the boost:archive.

It will be however hard to prove improvements, since even if I wrap the b::archive in one header, it is still included by some other headers, yielding no improvement, until they are sorted out.

In other words, the project needs *months* of maintenance, just be sure, that you realize it.

We have to do it one by header type by one.

EDIT: "Putting the serialization templates INTO .CPP FILES can't be achieved, as it turns out"

vtnerd: Wrapping the serialization will make it easier to integrate different serialization techniques later on.