.. _development: Development =========== This chapter will get you started with MSS development. MSS is written in Python. Once a stable release is published we do only bug fixes in stable and release regulary new minor versions. If a fix needs a API change or it is likly more a new feature you have to make a pull request to the develop branch. Documentation of changes is done by using our `issue tracker `_. When it is ready the developer version becomes the next stable. The stable version of MSS is tracked on `BLACK DUCK Open Hub `_ Using our Issue Tracker on github --------------------------------- How to Report Bugs .................. Please open a new issue in the appropriate GitHub repository `https://github.com/Open-MSS/MSS/issues/new `_ with steps to reproduce the problem you're experiencing. Be sure to include as much information including screenshots, text output, and both your expected and actual results. How to Request Enhancements ........................... First, please refer to the applicable `GitHub repository `_ and search `the repository's GitHub issues `_ to make sure your idea has not been (or is not still) considered. Then, please `create a new issue `_ in the GitHub repository describing your enhancement. Be sure to include as much detail as possible including step-by-step descriptions, specific examples, screenshots or mockups, and reasoning for why the enhancement might be worthwhile. Forking the Repo ---------------- 1. Firstly you have to make your own copy of project. For that you have to fork the repository. You can find the fork button on the top-right side of the browser window. 2. Kindly wait till it gets forked. 3. After that copy will look like */MSS* forked from *Open-MSS/MSS*. Cloning the Repo ................ 1. Now you have your own copy of project. Here you have to start your work. 2. Go to desired location on your computer where you want to set-up the project. 3. Right click there and click on git bash. A terminal window will pop up 4. Click The big green button which says "Code". Copy the URL. `Like this `_ 5. Now Type the command ``git clone .git`` and hit enter. Also, you could use ``gh repo clone /``, If you don't have ``gh``, see this for installation `here `_. 6. Wait for few seconds till the project gets copied or simply head over here for `cloning a repository `_ 7. Add the path of your local cloned mss directory to $PYTHONPATH. Setting up a git remote ....................... 1. Now you have to set up remote repositories 2. Type ``git remote -v`` in terminal to list remote connections to your repo. 3. It will show something like this:: origin git@github.com:/MSS.git (fetch) origin git@github.com:/MSS.git (push) 4. Add an upstream remote by the command :: git remote add upstream git@github.com:Open-MSS/MSS.git 5. Again type in command ``git remote -v`` to check if remote has been set up correctly 6. It should show something like this :: origin git@github.com:/MSS.git (fetch) origin git@github.com:/MSS.git (push) upstream git@github.com:Open-MSS/MSS.git (fetch) upstream git@github.com:Open-MSS/MSS.git (push) Update local stable branch .......................... If you don't have a stable branch, create one first or change to that branch:: git checkout [-b] stable git pull git@github.com:Open-MSS/MSS.git stable git push Setting Up a Local Environment ------------------------------ In the description we added as example to setup access to the mslib an export of the PYTHONPATH in your environment :: cd workspace/MSS export PYTHONPATH=`pwd` When you don’t want to enter this you can add the PYTHONPATH to mslib to your .bashrc If you don’t want the PYTHONPATH by export changed you can start modules differently:: cd workspace/MSS PYTHONPATH=. python mslib/msui/msui.py Requirements ............ 1. System requirements | Any system with basic configuration. | Operating System : Any (Windows / Linux / Mac). 2. Software requirement | Python | `Mambaforge `_ | `Additional Requirements `_ 3. Skill set | Knowledge of git & github | Python Using predefined docker images instead of installing all requirements ..................................................................... You can easily use our testing docker images which have all libraries pre installed. These are based on mambaforgen. We provide two images. In openmss/testing-stable we have mss-stable-env and in openmss/testing-develop we have mss-develop-env defined. In the further course of the documentation we speak of the environment mssdev, this corresponds to one of these evironments. You can either mount your MSS workdir in the container or use the environment from the container as environment on your machine. Running pytest inside the docker container ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We mount the MSS workdir into the docker container and use an env var to access the directory for running pytest on that dir. :: ~/workspace/MSS$ docker pull openmss/testing-stable # get recent version ~/workspace/MSS$ docker run -it --mount src=`pwd`,target=`pwd`,type=bind -e MSSDIR=`pwd` openmss/testing-stable # mount dir into container, create env var MSSDIR with dir (base) root@78f42ac9ded7:/# cd $MSSDIR # change directory to the mounted dir (base) root@78f42ac9ded7:/# conda activate mss-stable-env # activate env (mss-stable-env) root@78f42ac9ded7:/# pytest tests # run pytest Use the docker env on your computer, initial setup ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This example shows by using mss-stable-env how to set it up for testing and development of stable branch. The images gets updates when we have to add new dependencies or have do pinning of existing modules. On an updated image you need to redo these steps :: rm -rf $HOME/mambaforge/envs/mss-stable-env # cleanup the existing env mkdir $HOME/mambaforge/envs/mss-stable-env # create the dir to bind to xhost +local:docker # may be needed docker run -it --rm --mount type=volume,dst=/opt/conda/envs/mss-stable-env,volume-driver=local,volume-opt=type=none,volume-opt=o=bind,volume-opt=device=$HOME/mambaforge/envs/mss-stable-env --network host openmss/testing-stable # do the volume bind exit # we are in the container, escape :) sudo ln -s $HOME/mambaforge/envs/mss-stable-env /opt/conda/envs/mss-stable-env # we need the origin location linked because hashbangs interpreters are with that path. (only once needed) conda activate mss-stable-env # activate env cd workspace/MSS # go to your workspace MSS dir export PYTHONPATH=`pwd` # add it to the PYTHONPATH python mslib/msui/msui.py # test if the UI starts pytest tests # run pytest After the image was configured you can use it like a self installed env :: xhost +local:docker # may be needed conda activate mss-stable-env # activate env cd workspace/MSS # go to your workspace MSS dir export PYTHONPATH=`pwd` # add it to the PYTHONPATH pytest tests # run pytest Manual Installing dependencies .............................. MSS is based on the software of the conda-forge channel located. The channel is predefined in Mambaforge. Create an environment and install the dependencies needed for the mss package:: $ mamba create -n mssdev $ mamba activate mssdev $ mamba install mss=8.3.4 --only-deps Compare versions used in the meta.yaml between stable and develop branch and apply needed changes.:: $ git diff stable develop -- localbuild/meta.yaml Install requirements for local testing ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With sending a Pull Request our defined CIs do run all tests on github. You can do run tests own system too. For developers we provide additional packages for running tests, activate your env and run:: $ mamba install --file requirements.d/development.txt On linux install the `conda-forge package pyvirtualdisplay` and `xvfb` from your linux package manager. This is used to run tests on a virtual display. If you don't want tests redirected to the xvfb display just setup an environment variable:: $ export TESTS_VISIBLE=TRUE We have implemented demodata as data base for testing. On first call of pytest a set of demodata becomes stored in a /tmp/mss* folder. If you have installed gitpython a postfix of the revision head is added. Setup msui_settings.json for special tests .......................................... On default all tests use default configuration defined in mslib.msui.MissionSupportSystemDefaultConfig. If you want to overwrite this setup and try out a special configuration add an msui_settings.json file to the testings base dir in your tmp directory. You call it by the custom `--msui_settings` option Setup MSWMS server ------------------ In the MSS package is some demodata included. The default where this is stored is $HOME/mss. Your clone of the MSS repository needs a different folder, e.g. workspace/MSS. Avoid to mix data and source. :ref:`demodata ` is provided by executing:: $(mssdev) python mslib/mswms/demodata.py --seed To use this data add the mswms_settings.py in your python path:: $(mssdev) cd $HOME/workspace/MSS $(mssdev) export PYTHONPATH="`pwd`:$HOME/mss" $(mssdev) python mslib/mswms/mswms.py Setup MSColab server -------------------- The MSColab server is built using the Flask rest framework which communicates with the PyQt5 frontend of MSS. You can view the default configuration of MSColab in the file `mslib/mscolab/conf.py`. If you want to change any values of the configuration, please take a look at the "Configuring Your MSColab Server" section in :ref:`mscolab` When using for the first time you need to initialise your database. Use the command :code:`python mslib/mscolab/mscolab.py db --init` to initialise it. The default database is a sqlite3 database. You can add some dummy data to your database by using the command :code:`python mslib/mscolab/mscolab.py db --seed`. The content of the dummy data can be found in the file `mslib/mscolab/seed.py`. To start your server use the command :code:`python mslib/mscolab/mscolab.py start`. This would start the MSColab server on port 8083. Going to http://localhost:8083/status should now show "MSColab server". This means your server has started successfully. Now you can use the MSS desktop application to connect to it using the MSColab window of the application. Code Style ---------- We generally follow `PEP8 `_, with 120 columns instead of 79. Output and Logging ------------------ When writing logger calls, always use correct log level (debug only for debugging, info for informative messages, warning for warnings, error for errors, critical for critical errors/states). Building the docs with Sphinx ----------------------------- The documentation (in reStructuredText format, .rst) is in docs/. Usually building the docs also includes creating the images and pages for the gallery feature. This can be ommitted by setting an environment variable :: export GALLERY=False To build the html version of it, you need to have sphinx installed:: cd docs/ make html Then point a web browser at docs/_build/html/index.html. For heading hierarchy we use :: H1 == H2 -- H3 .. H4 ~~ Run Tests --------- After you installed the dependencies for testing you could invoke the tests by `pytest` with various options. Our tests are using the pytest framework. You could run tests serial and parallel :: $ pytest tests or parallel :: $ pytest -n auto --dist loadscope --max-worker-restart 0 tests Use the -v option to get a verbose result. By the -k option you could select one test to execute only. Verify Code Style ................. A flake8 only test is done by `py.test --flake8 -m flake8` or `pytest --flake8 -m flake8` Instead of running a ibrary module as a script by the -m option you may also use the pytest command. Coverage ........ :: $ pytest --cov mslib tests This plugin produces a coverage report, example:: ----------- coverage: platform linux, python 3.7.3-final-0 ----------- Name Stmts Miss Branch BrPart Cover -------------------------------------------------------------------------- mslib/__init__.py 2 0 0 0 100% mslib/msui/__init__.py 23 0 0 0 100% mslib/msui/aircrafts.py 52 1 8 1 97% mslib/msui/constants.py 12 2 4 2 75% mslib/msui/flighttrack.py 383 117 141 16 66% Profiling ......... Profiling can be done by e.g.:: $ python -m cProfile -s time ./mslib/mswms/demodata.py --seed > profile.txt example:: /!\ existing server config: "mswms_settings.py" for demodata not overwritten! /!\ existing server auth config: "mswms_auth.py" for demodata not overwritten! To use this setup you need the mswms_settings.py in your python path e.g. export PYTHONPATH=~/mss 557395 function calls (543762 primitive calls) in 0.980 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 23 0.177 0.008 0.607 0.026 demodata.py:1089(generate_file) 631 0.113 0.000 0.230 0.000 demodata.py:769(_generate_3d_data) 179 0.077 0.000 0.081 0.000 {method 'createVariable' of 'netCDF4._netCDF4.Dataset' objects} Pushing your changes -------------------- 1. Now you have made the changes, tested them and built them. So now it's time to push them. 2. Goto your terminal and type git status and hit enter, this will show your changes from the files 3. Then type in git add and hit enter, this will add all the files to staging area 4. Commit the changes by ``git commit -m ""`` and hit enter. 5. Now push your branch to your fork by ``git push origin `` and hit enter. Creating a pull request ----------------------- By this time you can see a message on your github fork as your fork is ahead of Open-MSS:develop by of commits and also you can see a button called Compare and pull request. Click on Compare and pull request button. You will see a template. Fill out the template completely by describing your change, cause of change, issue getting fixed etc. After filling the template completely click on Pull request Merging stable into develop --------------------------- Bug fixes we have done in stable we need to merge regulary into develop too:: git checkout stable git pull git@github.com:Open-MSS/MSS.git stable git checkout develop git pull git@github.com:Open-MSS/MSS.git develop git checkout -b merge_stable_to_develop git merge stable git push git@github.com:Open-MSS/MSS.git merge_stable_to_develop Then create the proposed merge request. The merge request must *not* be squashed or rebased. To allow the merging, the requirement for a linear-history must be disabled *temporarily* for the develop branch and one needs to ensure that the merge request is accepted with a regular merge with merge commit. Remove the merge_stable_to_develop branch if still present. Testing local build ------------------- We provide in the dir localbuild the setup which will be used as a base on conda-forge to build mss. As developer you should copy this directory and adjust the source path, build number. using a local meta.yaml recipe:: $ cd yourlocalbuild $ conda build . $ conda create -n mssbuildtest mamba $ conda activate mssbuildtest $ mamba install --use-local mss Take care on removing alpha builds, or increase the build number for a new version. Creating a new release ---------------------- * make sure all issues for this milestone are closed or moved to the next milestone * update CHANGES.rst, based on git log * check version number of upcoming release in CHANGES.rst * verify that version.py, meta.yaml, MANIFEST.in and setup.py are complete * for a new stable release merge from develop to stable * tag the release:: git tag -s -m "tagged/signed release X.Y.Z" X.Y.Z git push origin X.Y.Z * write a release information on https://github.com/Open-MSS/MSS/releases * create a release on anaconda conda-forge * announce on: * Mailing list * Twitter (follow @TheMSSystem for these tweets) Publish on Conda Forge ---------------------- * update a fork of the `mss-feedstock `_ - set version string - set sha256 checksum of the tagged release - update dependencies * rerender the feedstock by conda smithy * send a pull request * maintainer will merge if there is no error Google Summer of Code(TM) ------------------------- MSS takes part in Google Summer of Code as a sub-organization of Python Software Foundation(PSF). GSoC'22 Projects ................ - `Sreelakshmi Jayarajan: Automated Command Line Plotting Tool : GSoC 2022 `_ - `Jatin Jain: UI and server improvements GSOC 2022 `_ GSoC'21 Projects ................ - `Hrithik Kumar Verma: Generating a tool chain tutorial for the MSUI user interface by automation operations : GSoC 2021 `_ - `Aravind Murali: MSUI: UI Redesign GSOC 2021 `_ GSoC'20 Projects ................ - `Aryan Gupta: Mission Support System : Enhance KML Support `_ - `Tanish Grover: Mission Support System: Mission Support Collaboration Improvements `_ GSoC'19 Projects ................ - `Anveshan Lal: Updating Geographical Plotting Routines `_ - `Shivashis Padhi: Collaborative editing of flight path in real-time `_