Add allinone (clean/vendor/build) default target to Makefile, Remove gui/providers/providers.json from git repo, Don't use unlink on *nix in Makefile

Fixes #867 (closed)

Makes developing a bit easier.

I removed the auto-generated providers.json from the repo. It seems like git also removed the upper directory. Seems like there was nothing else inside.... Nevermind: I had to change the gen-providersjson python script to create the directory....

changed milestone to %2024.10 LEAP VPN Release

assigned to @peanut2

changed title from *Add ollinone (clean/vendor/build) default target to Makefile, Remove gui/providers/providers.json from git repo, Don't use unlin on nix to *Add ollinone (clean/vendor/build) default target to Makefile, Remove gui/providers/providers.json from git repo, Don't use unlink on nix in Makefile

added 30m of time spent at 2024-07-22

Ah. I removed the install_go target. Ubuntu 22.04 needs go from ppa. We need to update the README and/or the Makefile

resolved all threads

added 2 commits

• 156f7157 - [Makefile] Add default target allinone (clean/vendor/build)
• 37a697f0 - [Makefile] Remove install_go target

Compare with previous version

added 6 commits

• 8bc5fc06 - 1 commit from branch main
• 7d1c9d19 - Create working dir in gen-providers-json helper script
• 6f9e737f - Remove gui/providers/providers.json from git repo
• 0c7f6eef - [Makefile] Use unlink only on Windows for make clean
• 28a1dc43 - [Makefile] Add default target allinone (clean/vendor/build)
• 14c523e1 - [Makefile] Remove install_go target

Compare with previous version

added 17 commits

• 14c523e1...1256c39b - 13 commits from branch main
• 08365033 - Remove gui/providers/providers.json from git repo
• 21577654 - [Makefile] Use unlink only on Windows for make clean
• 60ea4b93 - [Makefile] Add default target allinone (clean/vendor/build)
• 9e539d6b - [Makefile] Remove install_go target

Compare with previous version

resolved all threads

added 15m of time spent at 2024-08-01

I have some really strange behaviour:

If I run PROVIDER=riseup QMAKE=qmake6 make allinone, eveything works.
If I run it with -j $(nproc), it fails!

pea@peabox:bitmask-vpn PROVIDER=riseup QMAKE=qmake6 make allinone -j $(nproc)
RUNNING CLEAN
RUNNING vendor
[+] Generating providers.json...
output: gui/providers/providers.json
[+] Got provider riseup from environment
[+] Configured provider: riseup
git archive HEAD | tar -x -C ./build/bitmask-vpn_0.24.8-rc.2-7-g9e539d6b-src
GEN PKG SNAP
generate.py  hooks  local  snapcraft-template.yaml
[+] Got provider riseup from environment
[+] Configured provider: riseup
['branding/scripts/generate-snap', 'build/riseup/snap/data.json']
Traceback (most recent call last):
  File "/home/pea/projects/leap/bitmask-vpn/branding/scripts/generate-snap", line 39, in <module>
    writeOutput(data, outputf)
  File "/home/pea/projects/leap/bitmask-vpn/branding/scripts/generate-snap", line 19, in writeOutput
    with open(outfile, 'w') as outf:
         ^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'build/riseup/snap/data.json'
make: *** [Makefile:405: gen_pkg_snap] Error 1
make: *** Waiting for unfinished jobs....
pea@peabox:bitmask-vpn 

This is the exact error we have in #887 (closed). But I can't reproduce on current main (that's the issue others have...). Super spooky.

The problem is that build/riseup directory does not exist.
Normally, it gets created in the prepare_templates target.

  prepare_templates: tgz 
+     @echo "RUNNING prepare_templates"
      @mkdir -p build/${PROVIDER}/bin/ deploy

I also added an echo here. The strange thing here is: If I run with -j $(nproc) the prepare_templates target does not get called. We could just fix it (without understanding it):

• create the build/$PROVIDER directory like described here: https://aur.archlinux.org/packages/riseup-vpn-git#comment-983938
• remove the -j $(nproc) and add it into the Makefile

Nevertheless I would like to understand. @jkito can you reproduce?

changed title from *Add ollinone (clean/vendor/build) default target to Makefile, Remove gui/providers/providers.json from git repo, Don't use unlink on nix in Makefile to *Add allinone (clean/vendor/build) default target to Makefile, Remove gui/providers/providers.json from git repo, Don't use unlink on nix in Makefile

I found the issue: Calling make with -j calls the targets at the same time. Also the dependencies. Our Makefile expects the dependency targets to be run in the order they are specified:
vendor: gen_providers_json prepare_templates gen_pkg_snap gen_pkg_deb

That means:

• Calling make -j runs random targets. That's why errors are that random
• We currently can't run make -j. We could fix it by creating the build/$PRIVDER directory in other targets, but I don't know if that solves all race condition errors
• still strange: I don't have this issue on main (others have - some? Arch Linux people). We don't have the issue creating the build in the CI. And you @jkito - can you reproduce ?

If I run it without -j, it only uses a single core on my computer (mpstat -P ALL 1). That's the issue we should solve here by adding this into the Makefile. Then we can update the PKGBUILD in the CI/AUR.

added 15m of time spent at 2024-08-01

added 22 commits

• 9e539d6b...bd2b8dcd - 15 commits from branch main
• 7e9dfbe4 - Remove gui/providers/providers.json from git repo
• bfcecdaa - [Makefile] Use unlink only on Windows for make clean
• 5c05a2f5 - [Makefile] Add default target allinone (clean/vendor/build)
• ac5fd432 - [Makefile] Remove install_go target
• 4c0a8b2d - [makefile] Fix racecondition in Makefile
• 5694fa8b - [makefile] Use qmake6 instead of qmake6
• 6f47fa44 - [makefile] Always build with -j $(nproc)

Compare with previous version

I added some changes. Now

• we can call it with -j or without - the directories will be created in each target
• use qmake6 as default - Today I forgot to set QMAKE=qmak6 and got random compilation errors. That's super confusing
• always use -j $(nproc)

requested review from @jkito

added 2 commits

• bb434081 - [Makefile] Use qmake6 instead of qmake if available
• 3c999225 - [makefile] Always build with -j $(nproc)

Compare with previous version

added 1h of time spent at 2024-08-18

added 15m of time spent at 2024-08-19

added 14 commits

• 3c999225...4b200a2b - 8 commits from branch main
• d04fd4d3 - Remove gui/providers/providers.json from git repo
• acd8dc18 - [Makefile] Use unlink only on Windows for make clean
• 32152325 - [Makefile] Add default target allinone (clean/vendor/build)
• a0543ecc - [Makefile] Remove install_go target
• e4607f5c - [makefile] Fix racecondition in Makefile
• fa6ca683 - [makefile] Don't overwrite QMAKE env when calling build.sh

Compare with previous version

added 15m of time spent at 2024-09-02

approved this merge request

resolved all threads

added 2 commits

• e65d5d0c - [Makefile] Fix racecondition in Makefile
• 89e85ee2 - [Makefile] Don't overwrite QMAKE env when calling build.sh

Compare with previous version

enabled an automatic merge when the pipeline for 89e85ee2 succeeds

added 15m of time spent at 2024-09-04

merged

mentioned in issue #887 (closed)

By the way: make allinone has still the issue that if you use it with -j <number> the execution of the dependencies is random. But only sometimes - really rare. Like when vendor is called and then clean.

.PHONY: allinone
allinone: clean vendor build