09:26:56 <ErCiccione[m]> .merge+ #1178
09:26:57 <xmr-pr> Added
09:28:17 <ErCiccione[m]> .merge+ #1179
09:28:17 <xmr-pr> Added
10:41:39 <ErCiccione[m]> The last piece of my CCS is about adding and activating gitlab's CI. I took a look and should be straightforward enough. We need to install a runner on the machine hosting giltab (or another one). Can be easily done with Docker.
10:42:14 <ErCiccione[m]> i made a couple of tests and the gitlab-ci.yml file to build the website is very simple.
10:43:29 <ErCiccione[m]> moneromooo: could you take care of installing the runner? https://docs.gitlab.com/runner/
10:43:50 <ErCiccione[m]> I read somewhere that snipa has experience with gitlab's CI settings, we could ask him to take a look
11:13:02 <selsta> Does it make sense to self host CI?
11:18:17 <ErCiccione[m]> what's your concern selsta? The runner would be self hosted. AFAIK there is no other solution with self hosted gitlab
11:18:43 <ErCiccione[m]> (gitlab non self-hosted and github have integrated runners)
11:19:00 <selsta> The concern is server costs.
11:19:22 <selsta> We don’t want the server running Gitlab also be busy with building monero / GUI / etc
11:20:00 <selsta> see also https://github.com/monero-project/meta/issues/420
11:20:24 <selsta> CircleCI, Github actions, Azure pipelines, ... all have free tiers for open source projects
11:24:37 <ErCiccione[m]> The problem is that things complicate once we are self-hosted. I looked up jenkins some time ago, but i don't remember why i didn't look further. I checked a couple more and they all give you the pipelines but they needed a gitlab runner
11:25:35 <ErCiccione[m]> btw i agree would be better to not self-host if possible. I'm gonna look to other possibilites today and report back
11:27:21 <selsta> using gitlab ci for the website would probably be ok
11:27:26 <selsta> but not for other things
11:27:43 <selsta> unless we want to waste money on servers
11:28:51 <ErCiccione[m]> well, we have been having buildbots on our servers for years
11:28:53 <selsta> though an argument can be made that if we only use one linux server the costs will be fine
11:29:27 <selsta> 12:28 <ErCiccione[m]> well, we have been having buildbots on our servers for years <-- fluffy wants to turn them off
11:29:50 <selsta> AFAIK they are separate severs
11:30:35 <ErCiccione[m]> yeah and i totally agree with that, i wanted to do that proposal time ago. My point is that we already have a server running the buildbots, so adding runners for gitlab wouldn't add more bloating
11:30:59 <ErCiccione[m]> having runners on a different server is totally fine according to gitlab's docs
11:34:10 <ErCiccione[m]> We could substitute the buildbots with runners. Would be a huge improvement. This is worth doing if we are moving to gitlab with GUI and CLI as well tho
11:35:44 <ErCiccione[m]> anyway the discussion about moving on gitlab or not is still not rsolved, so i'm gonna look for something for monero-site only for now
13:22:17 <moneromooo> Why would you want to run more build bots if we're shutting down existing ones ?
13:54:43 <ErCiccione[m]> moneromooo: Because the buildbots can only be configured and maintained by the people who have access to them. What i'm suggesting is to have something like do now with github's actions ond travis on github, where the config file is in the repo. To do that on a self-hosted repo you need a runner or to use a third party service like selsta was suggesting.
13:58:00 <moneromooo> fluffypony: are you ok with that ?
14:04:27 <moneromooo> I'll install it anyway, can't hurt
14:06:42 <ErCiccione[m]> btw as i was saying early, third party services (the ones i saw at least) often require a runner (gitlab.com and github.com have shared runners integrated), so ideally we could just replace the present buildbots with gitlab's runners
14:10:51 <moneromooo> Gah. #%!&*! thing says "Install it on a server separate than where GitLab is installed on" *after* the install instructions...
14:11:21 <moneromooo> So... I guess I'm stuck at that step.
14:11:51 <moneromooo> Not sure why the "separate" part...
14:25:06 <ErCiccione[m]> afaik can be on the same server
14:25:29 <ErCiccione[m]> i mean, i remeber i read it in their docs
14:26:03 <moneromooo> k, I can do that, but I'm not sure I'd be able to undo it since I've no idea where it stores its config.
14:31:36 <moneromooo> I expect it's monero-site you want it on ?
14:33:05 <ErCiccione[m]> for now yes, at least until we don't know the fate of the other main repos
14:38:27 <moneromooo> Hmm. I know why they'd want a separate machine. Because random assholes would be able to run their changes on the machine...
14:42:14 <moneromooo> So we can use docker. And hope noone is motivated enough to bundle a docker escape in a makefile.
14:57:47 <moneromooo> ErCiccione[m]: it might be ready now.
14:57:52 <moneromooo> Not sure how to tell :)
14:57:56 <ErCiccione[m]> yeah maybe i was wrong, but i thought the docker instance could run on the same host.
14:58:02 <ErCiccione[m]> great thanks!
14:58:05 <moneromooo> I've set it up on the same.
14:58:27 <moneromooo> pigeons: if you think this is a mistake security wise, say so.
14:58:57 <moneromooo> ErCiccione[m]: tell me when you're going to try using it, I'm sure I'm going to have hunt stuff that doesn't work.
15:00:57 <ErCiccione[m]> alright. If pipelines are activated, should be enough to PR the gitlab-ci.yml file. After that is merged there should be pipelines for each PR (at least from my tests with shared runners on gitlab.com)
15:01:35 <SerHack> moneromooo: challenge accepted /s Anyway there should be no exploit available publicly for docker... At the same time, there are several PoC on the web that might be cool to try. given example: https://www.exploit-db.com/exploits/47147
16:13:14 <ErCiccione[m]> I PRd the config file https://repo.getmonero.org/monero-project/monero-site/merge_requests/1182
17:30:39 <selsta> ErCiccione: invited Snipa for CI discussion
17:32:34 <ErCiccione> good idea, thanks selsta :)
17:32:46 <Snipa> Morning, nothing like good ole CI.
17:33:32 <selsta> Do you have experience with the Gitlab self hosted runner?
17:33:59 <selsta> Does it make sense to use that one or should we use non self hosted services. AFAIK they have good open source tiers.
17:34:00 <Snipa> Yup, I maintain our entire build process at work using a quad of gitlab runners.
17:35:19 <Snipa> The integrated gitlab runner is fine for about 90% of use cases, there's a few corner cases, usually around things like secret deployment keys/etc that are a bit harder to manage, but it's certianly not impossible to manage.
17:35:44 <selsta> Integrated = non self hosted?
17:35:57 <Snipa> We self-host.
17:36:18 <Snipa> The newer versions of gitlab have better secret control, but we deploy our deploy keys to a dedicated runner that /only/ deals with deployments for our own sanity, as I control the deployment server.
17:38:42 <selsta> Is the comment about the integrated runner is fine for about 90% use cases about the self hosted version?
17:38:47 <Snipa> Yes.
17:38:53 <selsta> ok
17:39:11 <Snipa> Sorry, still waking up over here, so not fully coherent. :)
17:39:23 <Snipa> All of my commentary will be to self-hosted, I've not used the GL hosted services at this point.
17:39:43 <selsta> My dream would be building reproducible builds on version tag push, not sure how difficult that will be :D
17:40:12 <Snipa> Probally pretty trivial.  The hardest part is that if you want to do multi-OS builds, it'll probally take some work.
17:40:28 <Snipa> You can do a regex sub-match for a build stage in the CI file.
17:40:33 <ErCiccione> Snipa the situation right now is: moneromooo installed a runner running on docker and we will be using that for monero-site, it's not a shared runner yet, but in my mind that's the next goal if we move all core repos on repo.getmonero
17:41:05 <selsta> The Github actions thing has been really helpful IMO
17:41:20 <selsta> I want a replacement for that if we switch of Github :P
17:41:22 <ErCiccione> eyah but it's not much different from using travis as we did before, no?
17:41:39 <selsta> there was no travis for GUI
17:42:18 <ErCiccione> yeah i'm talkign about CLI. GUI was more messed up to integrate because of all those build scripts i guess
17:43:26 <selsta> 18:40 <Snipa> Probally pretty trivial.  The hardest part is that if you want to do multi-OS builds, it'll probally take some work. <-- AFAIK we can cross compile for all OSs
17:43:31 <Snipa> Sounds like a pretty normal build process, we usually use shell executors, but we're building docker containers in several places, so it's a bit easier than doing docker in docker, the processes are all about the same though.
17:43:59 <Snipa> If cross-compile is fine, it's pretty easy to get setup.
17:44:16 <selsta> https://github.com/monero-project/monero-gui/blob/master/.github/workflows/build.yml
17:44:33 <selsta> gui will be different because we can’t cross compile yet
17:44:47 <Snipa> Good ole mac.
17:44:50 <ErCiccione> great. I already PRd a very minimal gitlab-ci.yml file for now. Waiting for it to be merged, then we can test things more accurately
17:45:46 <selsta> Does Gitlab CI supports caches?
17:45:52 <selsta> support*
17:46:31 <Snipa> In terms of storing build artifacts, or storing mid-process artifacts?  The latter, not so much as an explicit thing.
17:46:51 <selsta> Mid process artifacts.
17:47:42 <Snipa> You can sort of jank around it with scripting/etc, but I usually prefer to let it just build everything from scratch every time.  It's just CPU time in the end.
17:49:01 <selsta> The depends scripts compile every single dependency before compiling monero
17:49:17 <selsta> would probably take ages
17:49:29 <selsta> but let’s see later :D
17:49:46 <Snipa> Eh, just increase the job count. :)
17:51:27 <Snipa> Honestly, that's where I tend to abuse docker a fair bit, because docker will do the caching of layers sensibly, and you can get some nice boosts through that, particularly if you're using a docker based env for your builds to make the builders portable.