Verified Commit 59985587 authored by aguestuser's avatar aguestuser Committed by kwadronaut

Update README, Dockerfile to build, smoothing on-boarding

Closes leap/bitmask_android!1

README:
add detailed instructions on installing and compiling
provide workarounds for gotchas setting up emulator on 64-bit debian
provide updated dockerfile & offer instructions for building in docker
Docker:
move dockerfiles to registry contained w/in bitmask_android repo
provide separate dockerfiles for sdk (base), ndk, and emulator images
resolve issues causing containerized build to fail, including:
upstream ics-openvpn conflict with clang (:. transitively with ndk >
r12b)
(subtle!) missing dependency of ndk r12b on file (message: "missing
file" meant a missing program called "file" not an actual missing file)
missing locales causing errors in copyIcsOpenVPNFiles build step
(resolved by generating all locales in dockerfile)
silent failure in sdknmanager's isntallation of platforms;android-23
during image build causing install failure in remote (but not local!)
containers
Signed-off-by: kwadronaut's avatarkwadronaut <kwadronaut@leap.se>
parents 0688c9e8 4333fd93
image: "0xacab.org:4567/leap/gitlab-buildpackage:android"
#image: "gfx2015/android"
image: "0xacab.org:4567/leap/bitmask_android/android-ndk:latest"
stages:
- build
......
# Bitmask Android App
# Bitmask Android Client
This repository contains the source code for the [Bitmask](https://bitmask.net/) Android client. Bitmask Android offers one-click free VPN service from trusted providers of the LEAP stack.
To lean about the stack, visit [leap.se](https://leap.se).
Please see the [issues](https://0xacab.org/leap/bitmask_android/issues) section to report any bugs or feature requests, and to see the list of known issues.
# Table Of Contents
* [License](#license)
* [Installing](#installing)
* [JDK](#jdk)
* [C Libraries](#c-libraries)
* [Android SDK](#android-sdk)
* [With Android Studio](#with-android-studio)
* [With Bash](#with-bash)
* [Updating Your PATH](#updating-your-path)
* [With Docker](#with-docker)
* [Submodules](#submodules)
* [Compiling](#compiling)
* [Just Build It!](#just-build-it)
* [Debug APKs](#debug-apks)
* [Release APKs](#release-apks)
* [Signed Release APKs](#signed-release-apks)
* [Running Tests](#running-tests)
* [Debugging in an Emulator](#debugging-in-an-emulator)
* [From Android Studio](#from-android-studio)
* [From The Shell](#from-the-shell)
* [Debian Gotchas](#debian-gotchas)
* [Virtualization Not Enabled](#virtualization-not-enabled)
* [Unpatched Filepaths Bug](#unpatched-filepaths-bug)
* [Outdated GL Libraries](#outdated-gl-libraries)
* [Updating Submodules](#updating-submodules)
* [Acknowledgments](#acknowledgments)
* [Contributing](#contributing)
## License <a name="license"></a>
This repository contains the source code for the [Bitmask](https://bitmask.net/) Android app.
* [See LICENSE file](https://github.com/leapcode/bitmask_android/blob/master/LICENSE.txt)
Please see the [issues](https://github.com/leapcode/bitmask_android/issues) section to
report any bugs or feature requests and to see the list of known issues.
## License
## Installing <a name="installing"></a>
* [See LICENSE file](https://github.com/leapcode/bitmask_android/blob/master/LICENSE.txt)
We will assume for convenience that you are installing on a Debian- or Ubuntu-based GNU/Linux machine. (Patches welcome with instructions for Mac, Windows, or other GNU/Linux distributions!)
The Bitmask Android Client has the following system-level dependencies:
* JDK v. 1.8
* Assorted 32-bit C libraries
* Android SDK Tools, v. 25.2.5, with these packages:
* Platform-Tools, v. 25.0.2
* Build-Tools, API v. 23-25
* Platforms 23-25
* Android Support Repository
* Google Support Repository
* NDK v. r12b (enables C code in Android)
* For running the app in an emulator, you will also need these packages:
* Android Emulator
* System Images for Android APIs 23-25
* The ICS-OpenVpn submodule
You can install them as follows:
### JDK <a name="jdk"></a>
Install with:
```bash
sudo apt install default-jdk
```
### C Libraries <a name="c-libraries"></a>
These are necessary to make sure the program cross-compiles to 32-bit architectures successfully from 64-bit GNU/Linux machines:
```
sudo apt make gcc file install lib32stdc++ lib32z1
```
### Android SDK <a name="android-sdk"></a>
#### With Android Studio <a name="with-android-studio"></a>
All of the Android SDK and NDK packages are downloadable through Android Studio, which (sadly) is probably the most hassle-free way to go about things.
You can download Android studio here:
https://developer.android.com/studio/index.html
Once you've got it installed, use the `SDK Manager` tool (Android figure Icon with blue arrow second from the right in the tool pane) to download all the Android SDK and NDK depencencies listed above.
#### With Bash <a name="with-bash"></a>
Alternatley (eg: for build machines), you may download and unzip the `android-sdk` bundle from Google as follows (assuming an install location of `/opt/android-sdk-linux`:
```
curl -L https://dl.google.com/android/android-sdk_r25.2.5-linux.zip -o sdk-tools.zip \
&& unzip -q sdk-tools.zip -d /opt/android-sdk-linux \
&& rm -f sdk-tools.zip
```
To download the NDK (for cross-compiling and running the C code used in `ics-openvpn`), use:
```
curl -L http://dl.google.com/android/repository/android-ndk-r12b-linux-x86_64.zip -o ndk.zip \
&& unzip ndk.zip -d /opt/android-sdk-linux/android-ndk-r12b \
&& rm -rf ndk.zip
```
After updating your PATH (see next step), you may now use the `sdkmanager` tool bundled with `android-sdk` to browse and install new sdk packages from Google.
To browse all available packages, run:
```shell
sdkmanager --list
```
## Build Requirements
To search for available packages of a certain type (eg: `tools`), run:
Install from developer.android.com:
```shell
sdkmanager --list | grep tools
```
* Android SDK, API 23: http://developer.android.com/sdk/index.html
* Android NDK, r10e: http://developer.android.com/tools/sdk/ndk/index.html
To install all of the dependencies listed above (targetting SDK versions 23 - 25), run:
Make sure add the necessary android tools to your bin path. For example, assuming you installed
the SDK and NDK to `~/dev` on a linux machine, you would add this to your path:
```shell
sdkmanager tools
sdkmanager platform-tools
sdkmanager extras;android;m2repository
sdkmanager extras;google;m2repository
sdkmanager build-tools;25.0.2
sdkmanager build-tools;24.0.3
sdkmanager build-tools;23.0.3
sdkmanager platforms;android-25
sdkmanager platforms;android-24
sdkmanager platforms;android-23
```
~/dev/android-sdk-linux/tools/
~/dev/android-sdk-linux/platform-tools/
~/dev/android-sdk-linux/ndk-bundle/
#### Updating Your Path <a name="updating-your-path"></a>
Installable via `android` command (SDK Manager):
Once you've installed Android SDK & NDK packages, you need to modify your PATH so you can invoke all the programs you just installed. You can do that with something like the following in your `~/.shellrc` or `~/.bash_profile`:
* Android SDK Build-tools, 23.0.3
* Android Support Repository, 4+
```shell
export ANDROID_HOME=<path/where/you/installed/android/sdk>
export ANDROID_NDK_HOME=$ANDROID_HOME/ndk-bundle
export PATH=$ANDROID_NDK_HOME:$PATH
export PATH=$ANDROID_HOME/platform-tools:$PATH
export PATH=$ANDROID_HOME/tools/bin:$PATH
```
Finally, install a java compiler. For example:
NOTE: On GNU/Linux machines, Android Studio installs the Android SDK in `~/Android/Sdk/`. Our dockerfile installs it in `/opt/android-sdk-linux`. You can install it wherever you want! Just be sure to remember where so you can add it to your PATH! :)
sudo apt-get install default-jdk
#### With Docker <a name="with-docker"></a>
If you are using a 64-bit machine, you will need to install some libraries too:
Geesh! If all that above seems like a lot, it is!
sudo apt-get install lib32stdc++6 lib32z1
To keep ourselves from messing it up all the time everyone someone new joins the project, we made a Dockerfile that creates the above environment with one line. You can pull the image and run builds from inside it, or consult the [Dockerfile](/docker/android-sdk.dockerfile) itself for requirements that your system might need but be missing.
## Update git submodules
Assuming you've already [installed docker](https://docs.docker.com/engine/installation/), you can pull the image with:
We build upon ics-openvpn, which meets a submodule in our project structure.
``` shell
docker pull 0xacab.org:4567/aguestuser/bitmask_android:android-ndk:latest
```
For that reason, it is necessary to initialize and update them before being able to build Bitmask Android.
Run the image with:
git submodule init
git submodule update
cd ics-openvpn
git submodule init
git submodule update
``` shell
docker run --rm -it 0xacab.org:4567/aguestuser/bitmask_android:android-ndk:latest
```
More likely than not, you'll want to run the image with the source code mounted. You can do that with:
## Build native sources
``` shell
cd <path/to/bitmask_android>
docker run --rm -it -v`pwd`:/bitmask_android -t 0xacab.org:4567/aguestuser/bitmask_android:android-ndk:latest
```
To build NDK sources, you need to issue these commands:
cd ics-openvpn/main
misc/build-native.sh
cd .. (to get back to the project directory)
### Submodules <a name="submodules"></a>
### Compiling from the command line
We depend on [ics-openvpn](https://github.com/schwabe/ics-openvpn) as an interface to Android's OpenVPN implementation. We include it as a [git submodule](https://git-scm.com/book/en/v2/Git-Tools-Submodules) in our project, which requires that we initialize and update it (and its respective upstream submodule dependencies) in order to compile and run Bitmask Android.
#### Signed APK
We do so with:
If you want to release a signed APK, you'll have to create a gradle.properties file in the project root with the following structure:
```bash
cd <path/to/bitmask_android>
git submodule init
git submodule update --init --recursive
```
storeFileProperty=fullPath
storePasswordProperty=store password without quotation marks
keyAliasProperty=key alias without quotation marks
keyPasswordProperty=key password without quotation marks
## Compiling <a name="compiling"></a>
#### Actual command
You have lots of options for compiling, all of which will output Android-executable `apk` packages to `/bitmask_android/app/build/outputs/apk/`.
./gradlew build
### Just Build It! <a name="just-build-it"></a>
The resulting apk(s) will be in `app/build/apk`.
You are welcome to run:
### Using Android Studio
```
./gradlew build
```
* `Import project` => select bitmask_android top folder
This will compile the code and run the tests, but not output any `apk` packages. As such, it's not all that useful. :)
## Running tests
### Debug APKs <a name="debug-apks"></a>
To assemble debug packages for running locally or testing in CI, run:
```bash
./gradlew assembleDebug
```
This will output `app-insecure-debug.apk` and `app-production-debug.apk` to `/bitmask_android/app/build/outputs/apk/`.
### Release APKs <a name="release-apks"></a>
To assemble release packages, run:
```bash
./gradlew assembleRelease
```
This will output `app-insecure-release.apk` and `app-production-release.apk` to `/bitmask_android/app/build/outputs/apk/`.
### Signed Release APKs <a name="signed-release-apks"></a>
If you want to release a signed APK (which you *must* do to publish the app to the Google Play store), you'll have to create a gradle.properties file in the project root with the following structure:
```properties
storeFileProperty=<fullPath>
storePasswordProperty=<store password without quotation marks>
keyAliasProperty=<key alias without quotation marks>
keyPasswordProperty=<key password without quotation marks>
```
### Building In Docker <a name="building-in-docker"></a>
If you want to make sure the environment you use to build APKs matches exactly the environment that Gitlab will use to build and publish artifacts, you can run any of the above build commands from inside Docker. To assemble a release build this way, run the following commands:
``` shell
$ cd <path/to/bitmask_android>
$ sudo docker run --rm -it -v `pwd`:/bitmask_android 0xacab.org/leap/bitmask_android/android-ndk:latest
# cd /bitmask_android
# ./gradlew assembleRelease
```
## Running Tests <a name="running-tests"></a>
To run the automated tests:
......@@ -92,33 +259,211 @@ Due to the nature of some tests, adb will lose its connectivity and you won't re
2. adb logcat | less
3. Look for: "failed: test"
We'll polish this process soon, but right now that's what we're doing (well, in fact, we run "adb logcat" in Emacs and then search "failed: test" in the corresponding buffer ;) ).
We'll polish this process soon, but right now that's what we're doing.
## Debugging in an Emulator <a name="debugging-in-an-emulator"></a>
You can run the app in an emulator running any version of Android and simulating (almost) any device. To run it you'll have to create an emulator, run the emulator, and then load an assembled APK of the app onto the emulator. (You can then use all sort of nifty tools in [Anroid Debug Bridge](https://developer.android.com/studio/command-line/adb.html) to tail the logs and debug the app.)
Assuming you've already tackled (or don't need to tackle) the [Debian Gotchas](#debian-gotchas) listed below, you can do that using either Android Studio or a bash shell as follows:
### From Android Studio <a name="from-android-studio"></a>
To create an emulator:
* Select `Tools/Android/AVD Manager` from the application menu
* Follow the instructions
To run a pre-existing emulator:
* Open the `AVD Manager` as above
* Press the "Play" button next to the emulator you want to run
To run the app:
* Ensure you have an emulator running
* Open the left-hand project pane (Meta-1 or Cmd-1, depending on your keybindings)
* Navigate to `bitmask_android/app/src/main/java/se/leap/bitmaskclient/Dashboard`
* Right-click over the `Dashboard` filename and click the `Run 'Dashboard'` option (or use Shift-Ctl-F10 or Shift-Ctl-R, depending on your keybindings)
* After you have done this once, you should be able to simply select `Dashboard` from the dropdown menu next to the big green arrow in the toolbar, then click the green arrow to run the app.
### From the Shell <a name="from-the-shell"></a>
## Updating ics-openvpn
To list the available avd images for creating an emulator:
cd ics-openvpn
git remote add upstream https://github.com/schwabe/ics-openvpn.git
git pull --rebase upstream master
``` shell
avdmanager list
```
To create an emulator:
``` shell
avdmanager create avd
```
To list the emulators you have already created:
``` shell
avdmanager list avd
```
To run a pre-existing emulator called `Nexus_5_API_25`:
``` shell
emulator @Nexus_5_API_15
```
Verify the device is running with:
``` shell
adb devices
```
You should see something like:
``` shell
List of devices attached
emulator-5554 device
```
Install APK with:
``` shell
abd install <path/to/your>.apk
```
Uninstall with:
``` shell
abd uninstall se.leap.bitmaskclient
```
Install with option to reinstall:
``` shell
abd install -r <path/to/your/apk>
```
### Debian Gotchas <a name="debian-gotchas"></a>
If you are running Debian on a 64-bit machine, your emulator will likely not work out of the gate. Test to see if this is the case by:
* first creating an emulator in Android Studio (with name, eg, `Nexus_5_API_25`)
* then running:
```shell
cd ~/
emulator @<name_of_your_emulator>
```
If you can launch an emulator, HUZZAH! If not, you likely have one of 3 problems:
#### 1. Virtualization Not Enabled <a name="virtualization-not-enabled"></a>
Boo! Try turning it on. The second half of [this article](https://docs.fedoraproject.org/en-US/Fedora/13/html/Virtualization_Guide/sect-Virtualization-Troubleshooting-Enabling_Intel_VT_and_AMD_V_virtualization_hardware_extensions_in_BIOS.html) is a decent enough guide.
#### 2. Unpatched Filepaths Bug <a name="unpatched-filepaths-bug"></a>
**Symptoms:** If you have this bug, you will see something like the following when you try to spin up an emulator:
``` shell
[140500439390016]:ERROR:./android/qt/qt_setup.cpp:28:Qt library not found at ../emulator/lib64/qt/lib
Could not launch '../emulator/qemu/linux-x86_64/qemu-system-i386': No such file or directory
```
As [documented here](https://stackoverflow.com/questions/42554337/cannot-launch-avd-in-emulatorqt-library-not-found), there is a standing bug in the version of `emulator` packaged for emulator that assumes it always runs from within the `$ANDROID_HOME/emulator` directory, and can thus safely use relative filepaths, when in fact this is almost never the case. (Cool bug!)
**Fixes:**
You have a couple options. The second is more robust:
1. Always run emulator from within its own directory (clunky!):
``` shell
cd "$(dirname "$(which emulator)")"
emulator <name_of_your_emulator>
```
2. Insert a line in your `~/.bashrc` to automatically navigate to the correct directory (and back) whenever you invoke `emulator`:
```shell
function emulator { pushd `pwd`; cd "$(dirname "$(which emulator)")" && ./emulator "$@"; popd;}
```
#### 3. Outdated GL Libraries <a name="outdated-gl-libraries"></a>
**Symptoms:** If you have this bug, you will see something like the following:
``` shell
libGL error: failed to load driver: swrast
X Error of failed request: BadValue (integer parameter out of range for operation)
# redacted incredibly long stack trace
```
As documented [here](http://stackoverflow.com/questions/36554322/cannot-start-emulator-in-android-studio-2-0), the current emulator package ships without outdated versions of LibGL libraries. To work around this:
1. Install modern GL libriaries with:
``` shell
sudo apt-get install mesa-utils
```
2. Ensure that `emulator` always uses the correct libraries by either:
a. always calling `emulator` with the `-use-system-libs` flag, like so:
``` shell
emulator -use-system-libs -avd Nexus_5_API_25
```
b. adding the following line to your ~/.bashrc or ~/.bash_profile:
```shell
export ANDROID_EMULATOR_USE_SYSTEM_LIBS=1
```
**Special Android Studio Debian Bonus Gotcha:**
Assuming you have made all the above fixes (great job!), to be able to launch emulators from Android Studio, you must either:
1. Use the environment variable solution above (option a), then *always* launch Android Studio from a bash shell with:
``` shell
studio
```
This means never using the desktop launcher. :(
2. If you want to use the desktop launcher:
* You must *always* launch emulators from the terminal. :(
* But: you can quickly access a terminal inside of Android Studio with `OPTION-F12`
## Updating Submodules <a name="updating-submodules"></a>
If you need to refresh of our upstream dependency on ics-openvpn, you may do so with:
``` shell
cd <path/to/bitmask_android>
./gradlew updateIcsOpenVpn
```
Alternately:
```shell
cd <path/to/bitmask_android>
cd ics-openvpn
git remote add upstream https://github.com/schwabe/ics-openvpn.git
git pull --rebase upstream master
```
A bunch of conflicts may arise. The guidelines are:
1. Methods in HEAD (upstream) completely removed from Bitmask should be removed again (e.g. askPW)
2. Sometimes, Dashboard.class is in Bitmask while in ics-openvpn it is replaced by MainActivity.class and other classes. Keep removing them to keep Dashboard.class in there.
3. Some resources files are stripped from several entries. Remove them if possible (check the code we compile is not using anything else new).
./gradlew updateIcsOpenVpn
## Acknowledgements
## Acknowledgments <a name="acknowledgments"></a>
This project bases its work in [ics-openvpn project](https://code.google.com/p/ics-openvpn/).
## Contributing
## Contributing <a name="contributing"></a>
Please fork this repository and contribute back using
[pull requests](https://github.com/leapcode/leap_android/pulls).
Please fork this repository and contribute back using [pull requests](https://github.com/leapcode/leap_android/pulls).
Our preferred method for receiving translations is our [Transifex project](https://www.transifex.com/projects/p/bitmask-android).
Any contributions, large or small, major features, bug fixes, additional
language translations, unit/integration tests are welcomed and appreciated
but will be thoroughly reviewed and discussed.
Any contributions, large or small, major features, bug fixes, additional language translations, unit/integration tests are welcomed and appreciated but will be thoroughly reviewed and discussed.
......@@ -2,8 +2,8 @@ apply plugin: 'com.android.application'
android {
compileSdkVersion 23
buildToolsVersion "23.0.3"
buildToolsVersion '25.0.2'
;
signingConfigs {
release {
storeFile project.hasProperty('storeFileProperty') ? file(storeFileProperty) : null
......@@ -13,26 +13,26 @@ android {
}
}
productFlavors {
production {
productFlavors {
production {
}
insecure {
}
insecure {
}
}
}
buildTypes {
release {
//runProguard true
if(signingConfigs.contains(release))
signingConfig signingConfigs.release.isSigningReady() ? signingConfigs.release : signingConfigs.debug
signingConfig signingConfigs.release.isSigningReady() ? signingConfigs.release : signingConfigs.debug
}
}
lintOptions {
abortOnError false
}
lintOptions {
abortOnError false
}
sourceSets {
main {
......@@ -48,6 +48,7 @@ android {
dependencies {
androidTestCompile 'com.jayway.android.robotium:robotium-solo:5.5.4'
testCompile 'junit:junit:4.12'
compile 'com.jakewharton:butterknife:6.1.0'
provided 'com.squareup.dagger:dagger-compiler:1.2.2'
compile 'com.github.pedrovgs:renderers:1.5'
......@@ -55,7 +56,7 @@ dependencies {
compile 'com.google.code.gson:gson:2.4'
compile 'org.thoughtcrime.ssl.pinning:AndroidPinning:1.0.0'
compile 'mbanje.kurt:fabbutton:1.1.4'
compile 'com.android.support:support-annotations:23.2.1'
compile 'com.android.support:support-annotations:23.2.1'
}
def processFileInplace(file, Closure processText) {
......@@ -82,17 +83,17 @@ task copyIcsOpenVPNClasses( type: Copy ) {
include '**/dimens.xml'
include '**/logmenu.xml'
include '**/core/**.java'
include '**/activities/BaseActivity.java'
include '**/activities/BaseActivity.java'
includeEmptyDirs = false
filter {
line -> line.replaceAll('de.blinkt.openvpn.R', 'se.leap.bitmaskclient.R')
}
filter {
line -> line.replaceAll('de.blinkt.openvpn.BuildConfig', 'se.leap.bitmaskclient.BuildConfig')
}
filter {
filter {
line -> line.replace('package de.blinkt.openvpn;', 'package de.blinkt.openvpn;\n\nimport se.leap.bitmaskclient.R;')
}
} into '.'
......@@ -107,11 +108,11 @@ task copyIcsOpenVPNXml( type: Copy ) {
include '**/styles.xml'
include '**/dimens.xml'
include '**/refs.xml'
include '**/colours.xml'
include '**/colours.xml'
include '**/logmenu.xml'
include '**/white_rect.xml'
includeEmptyDirs = false
rename 'strings.xml', 'strings-icsopenvpn.xml'
filter {
line -> line.replaceAll('.*name="app".*', '')
......@@ -128,8 +129,8 @@ task copyIcsOpenVPNImages( type: Copy ) {
include '**/ic_close*.png'
include '**/ic_edit*.png'
include '**/ic_check*.png'
include '**/ic_pause*.png'
include '**/ic_play*.png'
include '**/ic_pause*.png'
include '**/ic_play*.png'
includeEmptyDirs = false
} into '.'
}
......@@ -141,14 +142,14 @@ task removeDuplicatedStrings() {
if(it.name.equals('strings.xml')) {
def ics_openvpn_file = file(it.absolutePath.replace('strings.xml', 'strings-icsopenvpn.xml'))
if(ics_openvpn_file.exists()) {
def ics_openvpn_strings_names = (new XmlParser()).parse(ics_openvpn_file)
def current_file = it
ics_openvpn_strings_names.string.each {
processFileInplace(current_file) { text ->
text.replaceAll('.*name=\"' + it.attribute('name') + '\".*(\n)*.*string>.*\n+', '')