.merge+ #1178

Added

.merge+ #1179

Added

The last piece of my CCS is about adding and activating gitlab's CI. I took a look and should be straightforward enough. We need to install a runner on the machine hosting giltab (or another one). Can be easily done with Docker.

i made a couple of tests and the gitlab-ci.yml file to build the website is very simple.

moneromooo: could you take care of installing the runner? docs.gitlab.com/runner

I read somewhere that snipa has experience with gitlab's CI settings, we could ask him to take a look

Does it make sense to self host CI?

what's your concern selsta? The runner would be self hosted. AFAIK there is no other solution with self hosted gitlab

(gitlab non self-hosted and github have integrated runners)

The concern is server costs.

We don’t want the server running Gitlab also be busy with building monero / GUI / etc

see also monero-project/meta #420

CircleCI, Github actions, Azure pipelines, ... all have free tiers for open source projects

The problem is that things complicate once we are self-hosted. I looked up jenkins some time ago, but i don't remember why i didn't look further. I checked a couple more and they all give you the pipelines but they needed a gitlab runner

btw i agree would be better to not self-host if possible. I'm gonna look to other possibilites today and report back

using gitlab ci for the website would probably be ok

but not for other things

unless we want to waste money on servers

well, we have been having buildbots on our servers for years

though an argument can be made that if we only use one linux server the costs will be fine

12:28 <ErCiccione[m]> well, we have been having buildbots on our servers for years <-- fluffy wants to turn them off

AFAIK they are separate severs

yeah and i totally agree with that, i wanted to do that proposal time ago. My point is that we already have a server running the buildbots, so adding runners for gitlab wouldn't add more bloating

having runners on a different server is totally fine according to gitlab's docs

We could substitute the buildbots with runners. Would be a huge improvement. This is worth doing if we are moving to gitlab with GUI and CLI as well tho

anyway the discussion about moving on gitlab or not is still not rsolved, so i'm gonna look for something for monero-site only for now

Why would you want to run more build bots if we're shutting down existing ones ?

moneromooo: Because the buildbots can only be configured and maintained by the people who have access to them. What i'm suggesting is to have something like do now with github's actions ond travis on github, where the config file is in the repo. To do that on a self-hosted repo you need a runner or to use a third party service like selsta was suggesting.

fluffypony: are you ok with that ?

I'll install it anyway, can't hurt

btw as i was saying early, third party services (the ones i saw at least) often require a runner (gitlab.com and github.com have shared runners integrated), so ideally we could just replace the present buildbots with gitlab's runners

Gah. #%!&*! thing says "Install it on a server separate than where GitLab is installed on" *after* the install instructions...

So... I guess I'm stuck at that step.

Not sure why the "separate" part...

afaik can be on the same server

i mean, i remeber i read it in their docs

k, I can do that, but I'm not sure I'd be able to undo it since I've no idea where it stores its config.

I expect it's monero-site you want it on ?

for now yes, at least until we don't know the fate of the other main repos

Hmm. I know why they'd want a separate machine. Because random assholes would be able to run their changes on the machine...

So we can use docker. And hope noone is motivated enough to bundle a docker escape in a makefile.

ErCiccione[m]: it might be ready now.

Not sure how to tell :)

yeah maybe i was wrong, but i thought the docker instance could run on the same host.

great thanks!

I've set it up on the same.

pigeons: if you think this is a mistake security wise, say so.

ErCiccione[m]: tell me when you're going to try using it, I'm sure I'm going to have hunt stuff that doesn't work.

alright. If pipelines are activated, should be enough to PR the gitlab-ci.yml file. After that is merged there should be pipelines for each PR (at least from my tests with shared runners on gitlab.com)

moneromooo: challenge accepted /s Anyway there should be no exploit available publicly for docker... At the same time, there are several PoC on the web that might be cool to try. given example: exploit-db.com/exploits/47147

I PRd the config file repo.getmonero.org/monero-project/monero-site/merge_requests/1182

ErCiccione: invited Snipa for CI discussion

good idea, thanks selsta :)

Morning, nothing like good ole CI.

Do you have experience with the Gitlab self hosted runner?

Does it make sense to use that one or should we use non self hosted services. AFAIK they have good open source tiers.

Yup, I maintain our entire build process at work using a quad of gitlab runners.

The integrated gitlab runner is fine for about 90% of use cases, there's a few corner cases, usually around things like secret deployment keys/etc that are a bit harder to manage, but it's certianly not impossible to manage.

Integrated = non self hosted?

We self-host.

The newer versions of gitlab have better secret control, but we deploy our deploy keys to a dedicated runner that /only/ deals with deployments for our own sanity, as I control the deployment server.

Is the comment about the integrated runner is fine for about 90% use cases about the self hosted version?

Yes.

ok

Sorry, still waking up over here, so not fully coherent. :)

All of my commentary will be to self-hosted, I've not used the GL hosted services at this point.

My dream would be building reproducible builds on version tag push, not sure how difficult that will be :D

Probally pretty trivial. The hardest part is that if you want to do multi-OS builds, it'll probally take some work.

You can do a regex sub-match for a build stage in the CI file.

Snipa the situation right now is: moneromooo installed a runner running on docker and we will be using that for monero-site, it's not a shared runner yet, but in my mind that's the next goal if we move all core repos on repo.getmonero

The Github actions thing has been really helpful IMO

I want a replacement for that if we switch of Github :P

eyah but it's not much different from using travis as we did before, no?

there was no travis for GUI

yeah i'm talkign about CLI. GUI was more messed up to integrate because of all those build scripts i guess

18:40 <Snipa> Probally pretty trivial.  The hardest part is that if you want to do multi-OS builds, it'll probally take some work. <-- AFAIK we can cross compile for all OSs

Sounds like a pretty normal build process, we usually use shell executors, but we're building docker containers in several places, so it's a bit easier than doing docker in docker, the processes are all about the same though.

If cross-compile is fine, it's pretty easy to get setup.

gui will be different because we can’t cross compile yet

Good ole mac.

great. I already PRd a very minimal gitlab-ci.yml file for now. Waiting for it to be merged, then we can test things more accurately

Does Gitlab CI supports caches?

support*

In terms of storing build artifacts, or storing mid-process artifacts? The latter, not so much as an explicit thing.

Mid process artifacts.

You can sort of jank around it with scripting/etc, but I usually prefer to let it just build everything from scratch every time. It's just CPU time in the end.

The depends scripts compile every single dependency before compiling monero

would probably take ages

but let’s see later :D

Eh, just increase the job count. :)

Honestly, that's where I tend to abuse docker a fair bit, because docker will do the caching of layers sensibly, and you can get some nice boosts through that, particularly if you're using a docker based env for your builds to make the builders portable.