From b3d6d6da08cae2f3b1e145d8935bc7a1345478c6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexis=20M=C3=A9taireau?= Date: Tue, 28 Nov 2023 15:54:11 +0100 Subject: [PATCH 1/8] [docs] Update the documentation --- docs/architecture/frontend.md | 73 ++++++++++++ docs/contributing.md | 91 +++------------ docs/custom.md | 1 - docs/{ => deploy}/docker.md | 57 ++++----- docs/{ => deploy}/ubuntu.md | 14 ++- docs/index.md | 17 ++- docs/install.md | 213 +++++++++++++++++++++++----------- docs/install_windows.md | 144 ----------------------- docs/settings.md | 2 +- mkdocs.yml | 37 +++++- pyproject.toml | 1 + 11 files changed, 311 insertions(+), 339 deletions(-) create mode 100644 docs/architecture/frontend.md rename docs/{ => deploy}/docker.md (64%) rename docs/{ => deploy}/ubuntu.md (97%) delete mode 100644 docs/install_windows.md diff --git a/docs/architecture/frontend.md b/docs/architecture/frontend.md new file mode 100644 index 00000000..5ebd9243 --- /dev/null +++ b/docs/architecture/frontend.md @@ -0,0 +1,73 @@ +# The frontend + +uMap is using Leaflet on the frontend. It's actually extending it: each uMap object extends the ones from Leaflet. + +```mermaid +erDiagram + Map ||--o{ DataLayer : datalayers + Map{ + JSON permissions + JSON metadata + } + DataLayer ||--o{Feature : data + DataLayer { + JSON metadata + JSON permissions + } +``` + +Here are the important concepts and files: + +- `umap.js` contains `Map`, the map object. It's the main entry point, which references all the related properties. It contains metadata, permissions, and data layers. +- `umap.layer.js` contains `DataLayer`, which contains metadata, permissions, and data. +- `umap.permissions.js` handles the permissions of the map. There is a different file handling the permissions of the datalayer: +- `umap.datalayer.permissions.js`. + +## Map (`L.U.Map`) + +`L.U.Map` is the class that's called by the server templates (in `map_init.html` and `map_fragment.html` used when we display lists of maps, like the homepage). + +It contains references to datalayers, and to the controls (the buttons that appears on the map) + +To be able to change the way the client behaves, querystring parameters can be used to overload the settings stored in the database. This is useful for instance when using iframes to display the map on a different website. + +uMap has an edit mode. If you don't have the rights you cannot save nor edit, you can't edit the permissions as well. + +A map contains references to: + +- controls +- datalayers + +## DataLayers (`L.U.Datalayer`) + +The datalayers contains data, and a layer (a way to represent them). + +Each data layer contains a "layer", to know what type of layer it is. It's one of: + +- Choropleth (`L.U.Layer.Choropleth`) +- Cluster (`L.U.Layer.Cluster`) +- Heat (`L.U.Layer.Heat`) + +When the data layers are initialized, they can have two states: +- `loaded`: the object is loaded in memory. At this stage we have access to all the datalayer's metada (name, type, id) +- `dataLoaded` : the data is available to be used, so we can for instance compute the center of the map (when it's dynamic). + +- `backupOptions` is here to make it possible to cancel what have been done (using undo). It stores the old settings for the data-layer. + +## Storage + +To mark what needs to be synced with the server, uMap currently mark objects as "dirty". Something marked as "dirty" has changed on the client, but is not yet saved on the server. + +Each map, datalayer and permission objects can be marked as "dirty". When a change is made on an object, it will mark its parent as "dirty" as well, so it can be updated accordingly. + +### Saving data to the server with `umap.save()` + +Here is what's being done when you call `map.save()`: + +1. `map.saveSelf()`, posting `name`, `center` and `settings` to the server, and then +2. calls `permission.save()`, which will post the permissions to the server, and then call back +3. `map.continueSaving()`, which will take the first dirtyLayer and call +4. `datalayer.save()` on it. It does the following: + 1. Post the data (`name`, `displayOnLoad`, `rank`, `settings`, and `geojson`) + 2. Calls `permission.save()`, posting `edit_status` to the server, and then calling `map.continue_saving()` and remove the datalayer from `dirtyDatalayers`. +5. When the `dirtyDatalayers` list is empty, we are done. diff --git a/docs/contributing.md b/docs/contributing.md index 05f7f8da..abf37703 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -1,92 +1,31 @@ # Contributing +So you want to contribute to uMap? Great news 🙌 + +We've put together this document so you have a brief overview of how things work. +You can help on different areas: translation, bug triage, documentation and development. + ## Translating -Translation is managed through [Transifex](https://www.transifex.com/openstreetmap/umap/). +uMap is translated to more than 50 languages! The translation is managed through [Transifex](https://www.transifex.com/openstreetmap/umap/). You will need an account to get started, and then you'll be able to translate easily. -## Bug Triaging +## Bug Triage -You are very welcome to help us triaging [uMap issues](https://github.com/umap-project/umap/issues). +You are very welcome to help us triage [uMap issues](https://github.com/umap-project/umap/issues). Don't hesitate to help other users by answering questions, give your point of view in discussions and just report bugs! -* Help other users by answering questions -* Give your point of view in discussions -* And so on... +## Reporting a bug -## Development on Ubuntu +If you've encountered a bug, don't hesitate to tell us about it. The best way to do this is by [opening a ticket on the bug tracker](https://github.com/umap-project/umap/issues/new/choose). But please, first, [have a look around](https://github.com/umap-project/umap/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc) to see if other users already reported something 😅 -### Environment setup +## Hacking on the code -Choose one of the following two config: +Following the [installation instructions](install.md) should get you started to hack on the code. -#### Config global to your desktop +## Merging rules -Follow the procedure [Ubuntu from scratch](ubuntu.md) +Pull requests -But instead using folders /etc/umap, you can create a ~/.umap folder. -This folder will contain the umap.conf file. - -And for folder /srv/umap, you can create a ~/umap folder (We will remove this folder later) - -You will have to set an env var, we will set it in your .bashrc: - - nano ~/.bashrc - -Add the following at the end of file: - -``` -# uMap -export UMAP_SETTINGS=~/.umap/umap.conf -``` - -Then refresh your terminal - - source ~/.bashrc - -Run your local uMap and check that it is working properly. - -#### Config inside your local git repo - -Follow the procedure [Ubuntu from scratch](ubuntu.md) - -You can use the local.py.sample in the git repo and copy it to your local git repo to umap/settings/local.py - -See [Installation](install.md) - -### Hacking on the code - -Create a workspace folder `~/wk` and go into it. - -"git clone" the main repository and go in the `umap/` folder. - -Once you are in the `umap/` folder, create a Python virtual environment: - - python3 -m venv venv - source venv/bin/activate - -Now, the `umap` command will be available. - -*Note: if you close your terminal, you will need to re-run that command from `~/wk/umap`:* - - source venv/bin/activate - -To test your code, you will add to install umap from your git folder. Go to `~/wk/umap` and run: - - make install - -This command will check dependencies and install uMap from sources inside folder. - -When installing from the git repository, do not forget to run `make installjs` and `make vendors`, before running `umap collectstatic` (as mentioned in [Ubuntu from scratch](ubuntu.md)). - -Create a PostgreSQL database and apply migrations to setup your database: - - createdb umap - umap migrate - -You should now be able to start your local uMap instance: - - umap runserver 0.0.0.0:8000 - -### Update translations +## Update translations Install needed tools: diff --git a/docs/custom.md b/docs/custom.md index 50d8204a..7c094a20 100644 --- a/docs/custom.md +++ b/docs/custom.md @@ -1,6 +1,5 @@ # Customize your uMap installation - When running your own uMap, you may want to changed its appearance, for example you want your own logo on the home page, or you want to apply some design, or you want to add some tracking (but anonymous!) script… diff --git a/docs/docker.md b/docs/deploy/docker.md similarity index 64% rename from docs/docker.md rename to docs/deploy/docker.md index 08f47e04..ff42bfc3 100644 --- a/docs/docker.md +++ b/docs/deploy/docker.md @@ -1,11 +1,14 @@ # Docker -There is now an official [uMap](https://hub.docker.com/r/umap/umap) image. +## Docker Hub -To run it with docker compose, create a `docker-compose.yml` like this: +An official [uMap docker image](https://hub.docker.com/r/umap/umap) is available on the docker hub. -_Sample docker-compose.yml_ -```yaml +## Docker compose + +If you prefer to run it with docker compose, here is the configuration file: + +```yaml title="docker-compose.yml" version: '3' services: db: @@ -47,16 +50,17 @@ You can use this example below and it will run, but you may want to look at the Make sure the settings in the docker-compose don't conflict with the sample config and vice-versa. In particular, remove the DATABASES section from the config file if using the docker-compose file, or it will override the DATABASE_URL setting and things won't work. -_Sample umap.conf_ -```python -""" -Example settings for docker quickstart: -Lots of stuff has been removed for simplicity - -See https://umap-project.readthedocs.io/en/master/settings/ - -Things YOU need to change before launching: -- modify SECRET_KEY +```python title="umap.conf" +""" +Example settings for docker quickstart: lots of stuff has been removed for simplicity. + +You can get the whole list of settings at: + + https://umap-project.readthedocs.io/en/master/settings/ + +Here are the settings YOU HAVE TO CHANGE before launching: + +- SECRET_KEY """ from umap.settings.base import * # pylint: disable=W0614,W0401 @@ -88,32 +92,15 @@ Some basic settings are available through env vars (see https://github.com/umap- but if you need more custom ones (like custom OAuth configuration), the easiest way is to put them in a [settings file](settings.md) and mount it to `/etc/umap/umap.conf`. ---- +### Getting started with docker compose -#### Quick sidebar about docker and docker-compose -A multi-platform tutorial on [installing docker](https://docs.docker.com/get-docker/) and docker-compose is outside the scope of this howto, but if you are on a recent Ubuntu, this is all you need to install both: +With docker installed on your machine, start the server with -```bash -sudo groupadd docker -sudo snap install docker -sudo usermod -aG docker $USER -# EXIT and log back in to pickup the docker group membership, then test with -docker run hello-world -``` - -Current versions of docker compose are integrated into the docker command and launch with `docker compose` (two words) -- older versions use a separate binary named `docker-compose`. Either one will work for this. - ---- - -### Initial launch - -Start the server with ```bash docker compose up ``` -... and let it run some initial setup until the output quiesces with a message about spawning uWSGI workers. Because there is a race between the time the app tries to connect to the DB and when the DB is actually ready, you might see a few exceptions/errors about 'psycopg' being unable to connect. This should sort itself out as the app retries. -### Create superuser +... and let it run some initial setup until the output quiesces with a message about spawning uWSGI workers. Because there is a race between the time the app tries to connect to the DB and when the DB is actually ready, you might see a few exceptions/errors about 'psycopg' being unable to connect. This should sort itself out as the app retries. Now you need to create your site superuser. Stop the server (Ctrl-C) and then type: ```bash @@ -122,8 +109,6 @@ docker-compose run app /venv/bin/umap createsuperuser Once that's done, you can relaunch your server with `docker compose up` -### Try It! - You should now be able to browse to your uMap instance from a browser on your local system, by pointing your browser to `https://localhost:8001/` (equivalent to `${SITE_URL}` in the docker-compose file, above). ### Administration diff --git a/docs/ubuntu.md b/docs/deploy/ubuntu.md similarity index 97% rename from docs/ubuntu.md rename to docs/deploy/ubuntu.md index cd6de65b..21298b64 100644 --- a/docs/ubuntu.md +++ b/docs/deploy/ubuntu.md @@ -6,14 +6,20 @@ You need sudo grants on this server, and it must be connected to Internet. ## Install system dependencies - sudo apt update - sudo apt install python3 python3-dev python3-venv virtualenv wget nginx uwsgi uwsgi-plugin-python3 postgresql gcc postgis libpq-dev +``` bash +sudo apt update +sudo apt install python3 python3-dev python3-venv virtualenv wget nginx uwsgi uwsgi-plugin-python3 postgresql gcc postgis libpq-dev +``` -*Note: nginx and uwsgi are not required for local development environment* +!! note + + Nginx and uwsgi are not required for local development environment. ## Create deployment directories: - sudo mkdir -p /etc/umap +```bash +sudo mkdir -p /etc/umap +``` *You can change this path, but then remember to adapt the other steps accordingly.* diff --git a/docs/index.md b/docs/index.md index 48b53902..0efef186 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,8 +1,17 @@ # uMap developer documentation -*This documentation is intended for people aiming to install and configure uMap. -If you are looking for user documentation, have a look at [the OSM wiki page](http://wiki.openstreetmap.org/wiki/UMap#Tutorials).* +!!! info "Developer docs" -uMap is a Django project, so in case of doubt always refer to the [Django documentation](https://docs.djangoproject.com). + This documentation is intended for people aiming to install and configure uMap. -Now let's start by [installing uMap](install.md)! + If you are looking for user documentation, have a look at [the OSM wiki page](http://wiki.openstreetmap.org/wiki/UMap#Tutorials). + +uMap lets you create maps with OpenStreetMap layers in a minute, and embed them in your site. + +If you want to get started, follow the [installation](install.md) guide. + +You might want to: + +- See [the developer docs](developer.md) +- Read [the contributing guidelines](contributing.md) +- Read through the [deployment tips](deploy/tips.md) diff --git a/docs/install.md b/docs/install.md index fb087f2c..a527bf6d 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1,119 +1,192 @@ # Installation -*Note: for Ubuntu, follow procedure [Ubuntu from scratch](ubuntu.md)* +!!! info "Specific guides" -*Note: for Windows, follow procedure [Installing on Windows](install_windows.md)* + This page covers how to get started with uMap. If you plan on deploying the service on a server for other people to consume, you can follow different tutorials: -Create a geo aware database. See [Geodjango doc](https://docs.djangoproject.com/en/dev/ref/contrib/gis/install/) for backend installation. + - [Deploying uMap on Ubuntu](deploy/ubuntu.md) (from scratch) + - [Deploying uMap using Docker](deploy/docker.md) + - Recent Windows distribution provide [WSL 2](https://learn.microsoft.com/en-us/windows/wsl/install), which makes it possible to run a Linux distribution of your choice. -Create a virtual environment +## System dependencies - python -m venv venv - source venv/bin/activate +uMap is built with the [Python](https://python.org) language, and the [Django](https://djangoproject.com) framework. It needs a [PostgreSQL](https://www.postgresql.org/) database, with the [Postgis](https://postgis.net/) extension enabled. -Install dependencies and project +Here are the commands to install the required system dependencies. - pip install umap-project +=== "Debian" + + ```bash + sudo apt update + sudo apt install python3 python3-dev python3-venv virtualenv postgresql gcc postgis libpq-dev + ``` -Create a default `local_settings` file +=== "Arch Linux" + ```bash + yay postgis extra/postgresql-libs + ``` - wget https://raw.githubusercontent.com/umap-project/umap/master/umap/settings/local.py.sample -O local_settings.py +=== "OS X (with brew)" + ```bash + brew install postgis + ``` + +=== "Fedora" + + ```bash + sudo dnf install postgis libpq-devel make gcc python3-devel + ``` + +### PostgreSQL + +Depending on your system, you might need to create a postgres user, the database, and initialize postgres. Here's how: + +```bash +createuser umap -U postgres +createdb umap -O umap -U postgres +psql umap -c "CREATE EXTENSION postgis" -Upostgres +``` + +## Getting started + +Create a geo aware database. See the [GeoDjango docs](https://docs.djangoproject.com/en/dev/ref/contrib/gis/install/) for backend installation. + +### Creating a virtual environment + +It is recommended to install python projects in a virtual environment to avoid mixing the project installation with your system dependencies. But it's not a requirement and it is up to you 🫣 + +```bash +python -m venv venv +source venv/bin/activate +``` + +### Installing the dependencies + +You can get all the project dependencies installed with the following command: + +```bash +pip install umap-project +``` + +### Configuration + +Create a default `local_settings.py` file, that you will modify with your setting. + +```bash +wget https://raw.githubusercontent.com/umap-project/umap/master/umap/settings/local.py.sample -O local_settings.py +``` Reference it as env var: - export UMAP_SETTINGS=`pwd`/local_settings.py - +```bash +export UMAP_SETTINGS=`pwd`/local_settings.py +``` Add database connection information in `local_settings.py`, for example - DATABASES = { - 'default': { - 'ENGINE': 'django.contrib.gis.db.backends.postgis', - 'NAME': 'umap', - } +```python +DATABASES = { + 'default': { + 'ENGINE': 'django.contrib.gis.db.backends.postgis', + 'NAME': 'umap', } +} +``` -Depending on your installation, you might need to change the USER that connects the database. +Depending on your installation, you might need to change the user that connects the database. It should look like this: - DATABASES = { - "default": { - "ENGINE": "django.contrib.gis.db.backends.postgis", - "NAME": "umap", - "USER": "postgres", - } +```python +DATABASES = { + "default": { + "ENGINE": "django.contrib.gis.db.backends.postgis", + "NAME": "umap", + "USER": "postgres", } - +} +``` Add a `SECRET_KEY` in `local_settings.py` with a long random secret key - SECRET_KEY = "a long and random secret key that must not be shared" +```python +SECRET_KEY = "a long and random secret key that must not be shared" +``` -You can easily generate one with openssl: +You can easily generate one with [openssl](https://www.openssl.org/): - openssl rand -base64 32 +```bash +openssl rand -base64 32 +``` -uMap uses [python-social-auth](http://python-social-auth.readthedocs.org/) for user authentication. So you will need to configure it according to your -needs. For example +uMap uses [python-social-auth](http://python-social-auth.readthedocs.org/) for user authentication. So you will need to configure it according to your needs. For example - AUTHENTICATION_BACKENDS = ( - 'social_auth.backends.contrib.github.GithubBackend', - 'social_auth.backends.contrib.bitbucket.BitbucketBackend', - 'social_auth.backends.twitter.TwitterBackend', - 'django.contrib.auth.backends.ModelBackend', - ) - GITHUB_APP_ID = 'xxx' - GITHUB_API_SECRET = 'zzz' - BITBUCKET_CONSUMER_KEY = 'xxx' - BITBUCKET_CONSUMER_SECRET = 'zzz' - TWITTER_CONSUMER_KEY = "xxx" - TWITTER_CONSUMER_SECRET = "yyy" +```python +AUTHENTICATION_BACKENDS = ( + 'social_auth.backends.contrib.github.GithubBackend', + 'social_auth.backends.contrib.bitbucket.BitbucketBackend', + 'social_auth.backends.twitter.TwitterBackend', + 'django.contrib.auth.backends.ModelBackend', +) +GITHUB_APP_ID = 'xxx' +GITHUB_API_SECRET = 'zzz' +BITBUCKET_CONSUMER_KEY = 'xxx' +BITBUCKET_CONSUMER_SECRET = 'zzz' +TWITTER_CONSUMER_KEY = "xxx" +TWITTER_CONSUMER_SECRET = "yyy" +``` Example of callback URL to use for setting up OAuth apps - http://umap.foo.bar/complete/github/ +http://umap.foo.bar/complete/github/ Adapt the `STATIC_ROOT` and `MEDIA_ROOT` to your local environment. -Create the tables +## Bootstrapping the database - umap migrate +Here are the commands you'll need to run to create the tables, collect the static files, etc. -Collect and compress the statics +```bash +# Create the database tables +umap migrate - umap collectstatic - umap compress +# Collect and compress static files +umap collectstatic +umap compress -Create a superuser +# Create a super user +umap createsuperuser - umap createsuperuser +# Finally start the server +umap runserver 0.0.0.0:8000 +``` -Start the server - - umap runserver 0.0.0.0:8000 - -## Search +## Configuring PostgreSQL search UMap uses PostgreSQL tsvector for searching. In case your database is big, you -may want to add an index. For that, you should do so: +may want to add an index. For that, here are the SQL commands to run: - # Create a basic search configuration - CREATE TEXT SEARCH CONFIGURATION umapdict (COPY=simple); +```SQL +# Create a basic search configuration +CREATE TEXT SEARCH CONFIGURATION umapdict (COPY=simple); - # If you also want to deal with accents and case, add this before creating the index - CREATE EXTENSION unaccent; - CREATE EXTENSION btree_gin; - ALTER TEXT SEARCH CONFIGURATION umapdict ALTER MAPPING FOR hword, hword_part, word WITH unaccent, simple; +# If you also want to deal with accents and case, add this before creating the index +CREATE EXTENSION unaccent; +CREATE EXTENSION btree_gin; +ALTER TEXT SEARCH CONFIGURATION umapdict ALTER MAPPING FOR hword, hword_part, word WITH unaccent, simple; - # Now create the index - CREATE INDEX IF NOT EXISTS search_idx ON umap_map USING GIN(to_tsvector('umapdict', COALESCE(name, ''::character varying)::text), share_status); +# Now create the index +CREATE INDEX IF NOT EXISTS search_idx ON umap_map USING GIN(to_tsvector('umapdict', COALESCE(name, ''::character varying)::text), share_status); +``` -And change `UMAP_SEARCH_CONFIGURATION = "umapdict"` in your settings. +And change your settings: +```python +UMAP_SEARCH_CONFIGURATION = "umapdict" +``` -## Emails +## Configuring emails UMap can send the anonymous edit link by email. For this to work, you need to add email specific settings. See [Django](https://docs.djangoproject.com/en/4.2/topics/email/#smtp-backend) @@ -121,7 +194,7 @@ documentation. In general, you'll need to add something like this in your local settings: -``` +```python FROM_EMAIL = "youradmin@email.org" EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend" EMAIL_HOST = "smtp.provider.org" @@ -133,11 +206,11 @@ EMAIL_USE_TLS = True EMAIL_USE_SSL = True ``` -## Upgrading +## Upgrading your installation Usually, for upgrading, you need those steps: -``` +```bash pip install umap-project --upgrade umap migrate umap collectstatic @@ -146,6 +219,6 @@ umap compress Then you need to restart your python server, for example: -``` +```bash sudo systemctl restart uwsgi # or gunicorn, or… ``` diff --git a/docs/install_windows.md b/docs/install_windows.md deleted file mode 100644 index 5ae27a1b..00000000 --- a/docs/install_windows.md +++ /dev/null @@ -1,144 +0,0 @@ -# Installing uMap on Windows - -The **good news** is that it is possible to run uMap server on Windows. However, it is recommended using uMap on a -Linux distribution as it will be easier to install, modify, and deploy. While the following steps have been tested on -Windows 7, they may work for other versions of Windows. - - -## 1. Prepare the Database - -This assumes you've installed PostgreSQL. -- Create a database called "umap" -- Install PostGIS extension in it - -##2. Create a directory and a Python virtual environment - -This assumes you've installed Python (version 3.8+ 64-bit is a good choice) and virtualenv. - -Open a Windows command window, and cd to a directory of your choice. You need to create a sub-directory but the name is -up to you (it doesn't need to be called "production"): -``` -mkdir production -cd production -virtualenv venv -venv\Scripts\activate.bat -``` - -##3. Install GDAL for Python - -It is really difficult to install GDAL the "standard" way since it requires compiling GDAL. Instead download an already -compiled pip-compatible wheel package file from -[Unofficial Windows Binaries for Python Extension Packages](https://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal). Note -that cp38 refers to the Python version you are using, so make sure you select the one that matches your Python version -for download. - -In the command window, install the downloaded wheel package: -`pip install GDAL-3.0.4-cp38-cp38-win_amd64.whl` - -You can test the install from the Python command line. From the Windows command window invoke Python: -``` -python -``` -then enter some Python commands: -```python ->>> import gdal ->>> print(int(gdal.VersionInfo('VERSION_NUM'))) ->>> exit() -``` - -##4. Install uMap - -In the Windows command window: -``` -mkdir static -mkdir data -pip install umap-project -``` -***Windows Work-Around 1*** - -Setting the UMAP_SETTINGS environment variable doesn't seem to work on Windows, so put the file in umap's fall-back -location of \etc\umap\umap.conf : -``` -mkdir \etc\umap -wget https://raw.githubusercontent.com/umap-project/umap/master/umap/settings/local.py.sample -O \etc\umap\umap.conf -``` -Edit the umap.conf file: - -***Windows Work-Around 2*** - -It might be possible to modify django's libgdal.py (umap installed django as one of its dependencies) to detect the -installed GDAL, but until then you can explicitly state the required paths. - -Add the GDAL paths somewhere near the top of the umap.conf file (make sure the last part, "gdal300", is the name of the -GDAL DLL in its package dir): -```python -GDAL_LIBRARY_PATH = r'C:\temp\production\venv\Lib\site-packages\osgeo\gdal300' -GEOS_LIBRARY_PATH = r'C:\temp\production\venv\Lib\site-packages\osgeo\geos_c' -PROJ_LIB = r'C:\temp\production\venv\Lib\site-packages\osgeo\data\proj' -``` -And while you're editing umap.conf, add the needed parameters to the DATABASES default object : -```python -DATABASES = { - 'default': { - 'ENGINE': 'django.contrib.gis.db.backends.postgis', - 'NAME': 'umap', - 'USER': 'postgres', - 'PASSWORD': 'postgres', - 'HOST': 'localhost', - 'PORT': '5432' - } -} -``` -And set umap's paths to where you've created the directories: -```python -STATIC_ROOT = '/temp/production/static' -MEDIA_ROOT = '/temp/production/data' -``` -Now that the minimal configuration is done, you can do the django-ish portion of the umap install. In the Windows -command window: -``` -umap migrate -umap collectstatic -umap createsuperuser -``` - -***Windows Work-Around 3*** - -Strangely, having the installed `umap.exe` is not enough. Some script tries to execute "umap" without the ".exe" -extension, so here's a hack to make that work: -``` -copy venv\scripts\umap.exe venv\scripts\umap -``` - -##5. Run umap server -In the Windows command window: -``` -umap runserver 127.0.0.1:8000 -``` -You should now be able to open a browser and go to http://127.0.0.1:8000 - -If you add some features to a new map and try to save them, you will likely see an error in the Windows command window -running the umap server. This error is a Python error related to doing -`os.remove(name)` on Windows: -``` - File "c:\temp\test\venv\lib\site-packages\django\core\files\storage.py", line 303, in delete - os.remove(name) -PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: -'C:\\temp\\production\\data\\datalayer\\1\\1\\layer-1.geojson' -``` - -***Windows Work-Around 4*** - -Edit `test\venv\lib\site-packages\django\core\files\storage.py`, and comment out lines 302 and 303: -```python -# else: -# os.remove(name) -``` -Now adding features and saving should work. _Now here's the weird part._ Edit `storage.py` to restore it to it's -original state by removing the comment characters you put in. Save the changes, do some more feature editing and -saving in your browser. It still works! This may be due to file/directory locking by Windows. - -##6. Installing for development - -The previous sections describe the install procedure for running the released version of uMap "as-is". If you want to -modify uMap (and possibly contribute your changes back to the uMap team), have a look at [Contributing](contributing.md) \ No newline at end of file diff --git a/docs/settings.md b/docs/settings.md index 08b6a90f..70a8073f 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -5,7 +5,7 @@ are doing. The Django settings reference is here: https://docs.djangoproject.com/en/4.2/ref/settings/ -Here are a few relevent settings for uMap. +Here are a few relevant settings for uMap. ## Usage diff --git a/mkdocs.yml b/mkdocs.yml index 2afda197..d6555821 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,15 +1,46 @@ site_name: uMap +site_description: uMap lets you create maps with OpenStreetMap layers in a minute and embed them in your site +repo_name: umap-project/umap +repo_url: https://github.com/umap-project/umap nav: - Home: index.md - Installation: install.md + - Architecture: + - Frontend (JS): architecture/frontend.md + - Deployment: + - Ubuntu Server Howto: deploy/ubuntu.md + - Docker: deploy/docker.md + - Windows: deploy/windows.md - Configuration: settings.md - Administration: administration.md - Contributing: contributing.md - how-tos: - - Ubuntu from scratch: ubuntu.md - Customize your uMap style: custom.md - - Install with Docker: docker.md - Changelog: changelog.md -theme: readthedocs +theme: + name: material + palette: + - scheme: 'default' + media: '(prefers-color-scheme: light)' + toggle: + icon: 'material/lightbulb' + name: "Switch to dark mode" + - scheme: 'slate' + media: '(prefers-color-scheme: dark)' + primary: 'blue' + toggle: + icon: 'material/lightbulb-outline' + name: 'Switch to light mode' + features: + - navigation.sections markdown_extensions: - pymdownx.magiclink + - admonition + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format + - pymdownx.tabbed: + alternate_style: true + combine_header_slug: true diff --git a/pyproject.toml b/pyproject.toml index a7ca170a..b1042a20 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -48,6 +48,7 @@ dev = [ "ruff==0.1.6", "djlint==1.31.0", "mkdocs==1.5.2", + "mkdocs-material>=9.4.14,<10", "vermin==1.5.2", "pymdown-extensions==10.4", "isort==5.12", From 8e94c9e4707d0b00e0d835eeeb24223397dc174b Mon Sep 17 00:00:00 2001 From: David Larlet Date: Tue, 28 Nov 2023 21:39:24 -0500 Subject: [PATCH 2/8] [docs] Bump mkdocs to 1.5.3 This way mkdocs-material is happy: > mkdocs-material 9.4.14 depends on mkdocs>=1.5.3 and ~=1.5 --- pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index b1042a20..a4309fca 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -47,7 +47,7 @@ dev = [ "hatch==1.7.0", "ruff==0.1.6", "djlint==1.31.0", - "mkdocs==1.5.2", + "mkdocs==1.5.3", "mkdocs-material>=9.4.14,<10", "vermin==1.5.2", "pymdown-extensions==10.4", From 8804a953ee8e04f903e1043dcb0c2c1a18355e2a Mon Sep 17 00:00:00 2001 From: David Larlet Date: Tue, 28 Nov 2023 21:40:09 -0500 Subject: [PATCH 3/8] [docs] Adapt primary color to match uMap one --- docs/stylesheets/extra.css | 3 +++ mkdocs.yml | 17 ++++++++++------- 2 files changed, 13 insertions(+), 7 deletions(-) create mode 100644 docs/stylesheets/extra.css diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css new file mode 100644 index 00000000..1434324f --- /dev/null +++ b/docs/stylesheets/extra.css @@ -0,0 +1,3 @@ +:root { + --md-primary-fg-color: #323e56; +} diff --git a/mkdocs.yml b/mkdocs.yml index d6555821..89df7ac0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -6,11 +6,11 @@ nav: - Home: index.md - Installation: install.md - Architecture: - - Frontend (JS): architecture/frontend.md + - Frontend (JS): architecture/frontend.md - Deployment: - - Ubuntu Server Howto: deploy/ubuntu.md - - Docker: deploy/docker.md - - Windows: deploy/windows.md + - Ubuntu Server Howto: deploy/ubuntu.md + - Docker: deploy/docker.md + - Windows: deploy/windows.md - Configuration: settings.md - Administration: administration.md - Contributing: contributing.md @@ -22,17 +22,20 @@ theme: palette: - scheme: 'default' media: '(prefers-color-scheme: light)' + primary: 'custom' toggle: icon: 'material/lightbulb' - name: "Switch to dark mode" + name: 'Switch to dark mode' - scheme: 'slate' media: '(prefers-color-scheme: dark)' - primary: 'blue' + primary: 'custom' toggle: icon: 'material/lightbulb-outline' name: 'Switch to light mode' features: - - navigation.sections + - navigation.sections +extra_css: + - stylesheets/extra.css markdown_extensions: - pymdownx.magiclink - admonition From c058db8ae21e0a19d35b0aeb87adff9d5754071b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexis=20M=C3=A9taireau?= Date: Tue, 5 Dec 2023 16:18:10 +0100 Subject: [PATCH 4/8] [docs] Rework the documentation --- docs/architecture/frontend.md | 6 +- docs/architecture/overview.md | 11 + docs/assets/logo.svg | 4 + docs/{custom.md => config/customize.md} | 6 +- docs/{administration.md => config/icons.md} | 18 +- docs/{ => config}/settings.md | 69 +++- docs/contributing.md | 30 +- docs/deploy/docker.md | 84 +--- docs/deploy/nginx.md | 121 ++++++ docs/deploy/ubuntu.md | 412 -------------------- docs/index.md | 14 +- docs/install.md | 65 +-- docs/stylesheets/extra.css | 3 + mkdocs.yml | 25 +- 14 files changed, 272 insertions(+), 596 deletions(-) create mode 100644 docs/architecture/overview.md create mode 100644 docs/assets/logo.svg rename docs/{custom.md => config/customize.md} (94%) rename docs/{administration.md => config/icons.md} (73%) rename docs/{ => config}/settings.md (69%) create mode 100644 docs/deploy/nginx.md delete mode 100644 docs/deploy/ubuntu.md diff --git a/docs/architecture/frontend.md b/docs/architecture/frontend.md index 5ebd9243..7a5b99a9 100644 --- a/docs/architecture/frontend.md +++ b/docs/architecture/frontend.md @@ -1,6 +1,10 @@ # The frontend -uMap is using Leaflet on the frontend. It's actually extending it: each uMap object extends the ones from Leaflet. +!!! info "These are notes" + + Consider this page as notes taken while exploring uMap. It's not covering everything but hopefully will help you get a basic understanding of how things work. + +uMap uses Leaflet on the frontend. A lot of uMap objects extends the ones from Leaflet with additional abilities. ```mermaid erDiagram diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md new file mode 100644 index 00000000..fd8ac0ec --- /dev/null +++ b/docs/architecture/overview.md @@ -0,0 +1,11 @@ +# Overview of uMap + +uMap is a server and a client. The server is build with the Django framework, and the client uses (vanilla) JavaScript, on top of Leaflet. + +Basically, here is how it works: + +- Most of the data is stored as [geoJSON files](https://datatracker.ietf.org/doc/html/rfc7946), on the server. +- Some other parts of the data is stored in the database (users, permissions, etc) +- PostGIS is used for some of its geo features, but for the most part, the computation is done on the frontend with Leaflet. + +The server is meant to be a simple layer to do the storage and serve the JavaScript. \ No newline at end of file diff --git a/docs/assets/logo.svg b/docs/assets/logo.svg new file mode 100644 index 00000000..fa32dc37 --- /dev/null +++ b/docs/assets/logo.svg @@ -0,0 +1,4 @@ + + + + diff --git a/docs/custom.md b/docs/config/customize.md similarity index 94% rename from docs/custom.md rename to docs/config/customize.md index 7c094a20..0d12ae6b 100644 --- a/docs/custom.md +++ b/docs/config/customize.md @@ -1,6 +1,6 @@ # Customize your uMap installation -When running your own uMap, you may want to changed its appearance, for example +When running your own uMap, you may want to change its appearance, for example you want your own logo on the home page, or you want to apply some design, or you want to add some tracking (but anonymous!) script… @@ -9,7 +9,7 @@ that. So basically you'll have your own templates and/or statics directories where you will put the templates or statics you want to control (and only those). -Inside thore directory, you need to respect the exact relative path of the +Inside those directory, you need to respect the exact relative path of the templates or statics you are adding, relatively to the [templates](https://github.com/umap-project/umap/tree/master/umap/templates) and @@ -99,4 +99,4 @@ There are three settings you can play with to control that: # Which field to use in the URL, may also be for example "pk" to use the # primary key and not expose the username (which may be private or may change too # often for URL persistance) - USER_URL_FIELD = "username" + USER_URL_FIELD = "username" \ No newline at end of file diff --git a/docs/administration.md b/docs/config/icons.md similarity index 73% rename from docs/administration.md rename to docs/config/icons.md index 84488613..6ffcc260 100644 --- a/docs/administration.md +++ b/docs/config/icons.md @@ -1,10 +1,4 @@ -# Administration - -You can access uMap administration page by navigating to `https://your.server.org/admin` - -You will have to connect with the admin account created during installation. Default admin username is "umap". - -## Icons +# Icons Icons (aka pictograms in uMap sources) can be used in your map markers. @@ -22,11 +16,11 @@ Example of icons libraries you may want to use: - [SJJB Icons](http://www.sjjb.co.uk/mapicons/contactsheet) - [Remix](https://remixicon.com/) -### Import icons manually +## Import icons manually You can import icons manually by going to your uMap admin page: `https://your.server.org/admin` -### Import icons automatically +## Import icons automatically To import icons on your uMap server, you will need to use the command `umap import_pictograms`. @@ -34,9 +28,11 @@ Note: you can get help with `umap import_pictograms -h` Basic usage: - umap import_pictograms --attribution "Maki Icons by Mapbox" path/to/icons/directory/ +```bash +umap import_pictograms --attribution "Maki Icons by Mapbox" path/to/icons/directory/ +``` -### Categories +## Categories uMap can render icons grouped into categories. When using the import script, any subfolder will be used as category. diff --git a/docs/settings.md b/docs/config/settings.md similarity index 69% rename from docs/settings.md rename to docs/config/settings.md index 70a8073f..181d6ac0 100644 --- a/docs/settings.md +++ b/docs/config/settings.md @@ -16,6 +16,7 @@ Those settings should either: `UMAP_SETTINGS` env var - be declared as env vars directly, for simple ones (string/boolean/list) + #### ALLOWED_HOSTS The hosts that uMap expects. @@ -23,15 +24,46 @@ The hosts that uMap expects. Can be set through env var too: `ALLOWED_HOSTS=umap.mydomain.org,u.mydomain.org` +#### COMPRESS_ENABLED +#### COMPRESS_STORAGE + +To activate the compression of the static files, you can set this flag to `True`. + +You can then run the following command to compress the assets: + +```bash +umap compress +``` + +Optionally add `COMPRESS_STORAGE = "compressor.storage.GzipCompressorFileStorage"` +and add `gzip_static on` directive to Nginx `/static` location, so Nginx will +serve pregenerated files instead of compressing them on the fly. + #### DEBUG -Set it to True for easier debugging in case of error. +Set it to `True`` for easier debugging in case of error. #### EMAIL_BACKEND Must be configured if you want uMap to send emails to anonymous users. -See [Emails](install.md#emails) for more details. +UMap can send the anonymous edit link by email. For this to work, you need to +add email specific settings. See [Django](https://docs.djangoproject.com/en/4.2/topics/email/#smtp-backend) +documentation. + +In general, you'll need to add something like this in your local settings: + +```python title="local_settings.py" +FROM_EMAIL = "youradmin@email.org" +EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend" +EMAIL_HOST = "smtp.provider.org" +EMAIL_PORT = 456 +EMAIL_HOST_USER = "username" +EMAIL_HOST_PASSWORD = "xxxx" +EMAIL_USE_TLS = True +# or +EMAIL_USE_SSL = True +``` #### ENABLE_ACCOUNT_LOGIN @@ -48,6 +80,7 @@ See `EMAIL_BACKEND`. Set it to the default language you want. `LANGUAGE_CODE = "it"` +#### Default map center #### LEAFLET_LONGITUDE, LEAFLET_LATITUDE, LEAFLET_ZOOM Default longitude, latitude and zoom for the map @@ -128,14 +161,14 @@ To be used when you want to override some HTML templates: UMAP_CUSTOM_TEMPLATES = "/path/to/custom/templates" -See [customization](custom.md) for details. +See [customization](customize.md) for details. #### UMAP_CUSTOM_STATICS To be used when you want to override some CSS or images: UMAP_CUSTOM_STATICS = "/path/to/custom/static" -See [customization](custom.md) for details. +See [customization](customize.md) for details. #### UMAP_EXTRA_URLS @@ -198,7 +231,27 @@ How many maps to show in the user "my maps" page. Use it if you take control over the search configuration. -See [search](install.md#search) for details. +UMap uses PostgreSQL tsvector for searching. In case your database is big, you +may want to add an index. For that, here are the SQL commands to run: + +```SQL +# Create a basic search configuration +CREATE TEXT SEARCH CONFIGURATION umapdict (COPY=simple); + +# If you also want to deal with accents and case, add this before creating the index +CREATE EXTENSION unaccent; +CREATE EXTENSION btree_gin; +ALTER TEXT SEARCH CONFIGURATION umapdict ALTER MAPPING FOR hword, hword_part, word WITH unaccent, simple; + +# Now create the index +CREATE INDEX IF NOT EXISTS search_idx ON umap_map USING GIN(to_tsvector('umapdict', COALESCE(name, ''::character varying)::text), share_status); +``` + +Then set: + +```python title="settings.py" +UMAP_SEARCH_CONFIGURATION = "umapdict" +``` #### UMAP_READONLY @@ -208,6 +261,12 @@ Is your instance readonly? Useful for server maintenance. Should uMap gzip datalayers geojson. +#### UMAP_XSENDFILE_HEADER + +Can be set to `X-Accel-Redirect` to enable the [NGINX X-Accel](https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/) feature. + +See the NGINX documentation in addition. + #### SOCIAL_AUTH_OPENSTREETMAP_KEY, SOCIAL_AUTH_OPENSTREETMAP_SECRET If you use OpenStreetMap as OAuth provider, use those settings. diff --git a/docs/contributing.md b/docs/contributing.md index abf37703..52cb2945 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -21,11 +21,35 @@ If you've encountered a bug, don't hesitate to tell us about it. The best way to Following the [installation instructions](install.md) should get you started to hack on the code. -## Merging rules +### Installing dev dependencies -Pull requests +To be sure to install all the dev dependencies, and have everything working locally. -## Update translations +```bash +make develop +``` + +### Hack! + +You can now do your changes in a specific branch, and when you're ready you can open a pull-request for us to review. + +### Run tests + +Multiple tests suites are in use in the project. + +| Test suite | Location | Command | +| ---------- | -------- | ------- | +| Python unit tests | `umap/tests/integration` | `pytest . --ignore umap/tests/integration` | +| Javascript unit tests | `umap/static/test` | `make testjs`| +| Integration tests | `umap/tests/integration` | `pytest umap/tests/integration` | + +All the tests are run when you're creating a pull request, to avoid regressions. + +### Merging rules + +Pull requests need to be accepted by one maintainer of the project. Please be patient, we try to do our best, but it sometimes takes time. + +## Update the translations Install needed tools: diff --git a/docs/deploy/docker.md b/docs/deploy/docker.md index ff42bfc3..1ae44c33 100644 --- a/docs/deploy/docker.md +++ b/docs/deploy/docker.md @@ -1,12 +1,6 @@ # Docker -## Docker Hub - -An official [uMap docker image](https://hub.docker.com/r/umap/umap) is available on the docker hub. - -## Docker compose - -If you prefer to run it with docker compose, here is the configuration file: +An official uMap docker image is [available on the docker hub](https://hub.docker.com/r/umap/umap). But, if you prefer to run it with docker compose, here is the configuration file: ```yaml title="docker-compose.yml" version: '3' @@ -41,78 +35,4 @@ services: volumes: umap_userdata: umap_db: - -``` - -Next, create a basic settings file, named `umap.conf` in the same directory. - -You can use this example below and it will run, but you may want to look at the project sample config, using `wget https://raw.githubusercontent.com/umap-project/umap/master/umap/settings/local.py.sample -O /etc/umap/umap.conf` and modify as needed. - -Make sure the settings in the docker-compose don't conflict with the sample config and vice-versa. In particular, remove the DATABASES section from the config file if using the docker-compose file, or it will override the DATABASE_URL setting and things won't work. - -```python title="umap.conf" -""" -Example settings for docker quickstart: lots of stuff has been removed for simplicity. - -You can get the whole list of settings at: - - https://umap-project.readthedocs.io/en/master/settings/ - -Here are the settings YOU HAVE TO CHANGE before launching: - -- SECRET_KEY -""" - -from umap.settings.base import * # pylint: disable=W0614,W0401 - -SECRET_KEY = '!!secretsecret!!' -INTERNAL_IPS = ('127.0.0.1', ) -ALLOWED_HOSTS = ['*', ] - -DEBUG = True -COMPRESS_ENABLED = True -COMPRESS_OFFLINE = True -LANGUAGE_CODE = 'en' - -# Set to False if login into django account should not be possible. You can -# administer accounts in the admin interface. -ENABLE_ACCOUNT_LOGIN = True -AUTHENTICATION_BACKENDS = ( - 'django.contrib.auth.backends.ModelBackend', -) - -# Enables a banner letting users know this site is not for production use -UMAP_DEMO_SITE = True - -# Whether to allow non authenticated people to create maps. -UMAP_ALLOW_ANONYMOUS = True -``` - -Some basic settings are available through env vars (see https://github.com/umap-project/umap/blob/master/umap/settings/base.py) and can be defined right in the docker-compose file, -but if you need more custom ones (like custom OAuth configuration), the easiest -way is to put them in a [settings file](settings.md) and mount it to `/etc/umap/umap.conf`. - -### Getting started with docker compose - -With docker installed on your machine, start the server with - -```bash -docker compose up -``` - -... and let it run some initial setup until the output quiesces with a message about spawning uWSGI workers. Because there is a race between the time the app tries to connect to the DB and when the DB is actually ready, you might see a few exceptions/errors about 'psycopg' being unable to connect. This should sort itself out as the app retries. - -Now you need to create your site superuser. Stop the server (Ctrl-C) and then type: -```bash -docker-compose run app /venv/bin/umap createsuperuser -``` - -Once that's done, you can relaunch your server with `docker compose up` - -You should now be able to browse to your uMap instance from a browser on your local system, by pointing your browser to `https://localhost:8001/` (equivalent to `${SITE_URL}` in the docker-compose file, above). - -### Administration - -To administer the site (add users, change map tiles, other customizations) log in as the root user you just created, then go to `${SITE_URL}/admin`. - - +``` \ No newline at end of file diff --git a/docs/deploy/nginx.md b/docs/deploy/nginx.md new file mode 100644 index 00000000..eb97b725 --- /dev/null +++ b/docs/deploy/nginx.md @@ -0,0 +1,121 @@ +# Configuring Nginx + +Here are some configuration files to use umap with nginx and [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/), a server for python, which will handle your processes for you. + +```nginx title="nginx.conf" +upstream umap { + server unix:///srv/umap/uwsgi.sock; +} + +server { + # the port your site will be served on + listen 80; + listen [::]:80; + listen 443 ssl; + listen [::]:443 ssl; + # the domain name it will serve for + server_name your-domain.org; + charset utf-8; + + # max upload size + client_max_body_size 5M; # adjust to taste + + # Finally, send all non-media requests to the Django server. + location / { + uwsgi_pass umap; + include /srv/umap/uwsgi_params; + } +} +``` + +## uWSGI + + +```nginx title="uwsgi_params" +uwsgi_param QUERY_STRING $query_string; +uwsgi_param REQUEST_METHOD $request_method; +uwsgi_param CONTENT_TYPE $content_type; +uwsgi_param CONTENT_LENGTH $content_length; + +uwsgi_param REQUEST_URI $request_uri; +uwsgi_param PATH_INFO $document_uri; +uwsgi_param DOCUMENT_ROOT $document_root; +uwsgi_param SERVER_PROTOCOL $server_protocol; +uwsgi_param REQUEST_SCHEME $scheme; +uwsgi_param HTTPS $https if_not_empty; + +uwsgi_param REMOTE_ADDR $remote_addr; +uwsgi_param REMOTE_PORT $remote_port; +uwsgi_param SERVER_PORT $server_port; +uwsgi_param SERVER_NAME $server_name; +``` + + +```ini title="uwsgi.ini" +[uwsgi] +uid = umap +gid = users +# Python related settings +# the base directory (full path) +chdir = /srv/umap/ +# umap's wsgi module +module = umap.wsgi +# the virtualenv (full path) +home = /srv/umap/venv + +# process-related settings +# master +master = true +# maximum number of worker processes +processes = 4 +# the socket (use the full path to be safe) +socket = /srv/umap/uwsgi.sock +# ... with appropriate permissions - may be needed +chmod-socket = 666 +stats = /srv/umap/stats.sock +# clear environment on exit +vacuum = true +plugins = python3 +``` + +## Static files + +```nginx title="nginx.conf" +location /static { + autoindex off; + access_log off; + log_not_found off; + sendfile on; + gzip on; + gzip_vary on; + alias /path/to/umap/var/static/; +} + +location /uploads { + autoindex off; + sendfile on; + gzip on; + gzip_vary on; + alias /path/to/umap/var/data/; + # Exclude direct acces to geojson, as permissions must be + # checked py django. + location /uploads/datalayer/ { return 404; } +} +``` + +## X-Accel-Redirect + +With this configuration, NGINX will directly serve the geoJSON layers, but uMap will check the permissions. + +```title="umap.conf" +UMAP_XSENDFILE_HEADER = 'X-Accel-Redirect' +``` + +```title="nginx.conf" + location /internal/ { + internal; + gzip_vary on; + gzip_static on; + alias /path/to/umap/var/data/; + } +``` diff --git a/docs/deploy/ubuntu.md b/docs/deploy/ubuntu.md deleted file mode 100644 index 21298b64..00000000 --- a/docs/deploy/ubuntu.md +++ /dev/null @@ -1,412 +0,0 @@ -# Tutorial - -This tutorial will cover an installation from scratch of a uMap instance in an Ubuntu server. - -You need sudo grants on this server, and it must be connected to Internet. - -## Install system dependencies - -``` bash -sudo apt update -sudo apt install python3 python3-dev python3-venv virtualenv wget nginx uwsgi uwsgi-plugin-python3 postgresql gcc postgis libpq-dev -``` - -!! note - - Nginx and uwsgi are not required for local development environment. - -## Create deployment directories: - -```bash -sudo mkdir -p /etc/umap -``` - -*You can change this path, but then remember to adapt the other steps accordingly.* - - -## Create a Unix user - - sudo useradd -N umap -m -d /srv/umap/ - -*Here we use the name `umap`, but this name is up to you. Remember to change it -on the various commands and configuration files if you go with your own.* - - -## Give umap user access to the config folder - - sudo chown umap:users /etc/umap - sudo chown umap:users /srv/umap - - -## Create a postgresql user - - sudo -u postgres -D ~postgres createuser umap - - -## Create a postgresql database - - sudo -u postgres -D ~postgres createdb umap -O umap - - -## Activate PostGIS extension - - sudo -u postgres -D ~postgres psql umap -c "CREATE EXTENSION postgis" - - -## Login as umap Unix user - - sudo -u umap -i - -From now on, unless we say differently, the commands are run as `umap` user. - - -## Create a virtualenv and activate it - - virtualenv /srv/umap/venv --python=/usr/bin/python3.10 - . /srv/umap/venv/bin/activate - -*Note: this activation is not persistent, so if you open a new terminal window, -you will need to run again this last line.* - - -## Install umap - - pip install umap-project - - -## Create a local configuration file - - wget https://raw.githubusercontent.com/umap-project/umap/master/umap/settings/local.py.sample -O /etc/umap/umap.conf - -## Customize umap.conf - - nano /etc/umap/umap.conf - -* update the [SECRET_KEY](settings.md#secret_key) -* update the ADMINS list - -## Create the tables - - umap migrate - -## Collect the statics - - umap collectstatic - -## Create a superuser - - umap createsuperuser - -## Start the demo server - - umap runserver 0.0.0.0:8000 - -You can now go to [http://localhost:8000/](http://localhost:8000/) and try to create a map for testing. - -When you're done with testing, quit the demo server (type Ctrl+C). - - -## Configure the HTTP API - -Now let's configure a proper HTTP server. - -### uWSGI - -Create a file named `/srv/umap/uwsgi_params`, with this content -(without making any change on it): - -``` -uwsgi_param QUERY_STRING $query_string; -uwsgi_param REQUEST_METHOD $request_method; -uwsgi_param CONTENT_TYPE $content_type; -uwsgi_param CONTENT_LENGTH $content_length; - -uwsgi_param REQUEST_URI $request_uri; -uwsgi_param PATH_INFO $document_uri; -uwsgi_param DOCUMENT_ROOT $document_root; -uwsgi_param SERVER_PROTOCOL $server_protocol; -uwsgi_param REQUEST_SCHEME $scheme; -uwsgi_param HTTPS $https if_not_empty; - -uwsgi_param REMOTE_ADDR $remote_addr; -uwsgi_param REMOTE_PORT $remote_port; -uwsgi_param SERVER_PORT $server_port; -uwsgi_param SERVER_NAME $server_name; -``` - -Then create a configuration file for uWSGI: - - nano /srv/umap/uwsgi.ini - -And paste this content. Double check paths and user name in case you -have customized some of them during this tutorial. If you followed all the bits of the -tutorial without making any change, you can use it as is: - -``` -[uwsgi] -uid = umap -gid = users -# Python related settings -# the base directory (full path) -chdir = /srv/umap/ -# umap's wsgi module -module = umap.wsgi -# the virtualenv (full path) -home = /srv/umap/venv - -# process-related settings -# master -master = true -# maximum number of worker processes -processes = 4 -# the socket (use the full path to be safe -socket = /srv/umap/uwsgi.sock -# ... with appropriate permissions - may be needed -chmod-socket = 666 -stats = /srv/umap/stats.sock -# clear environment on exit -vacuum = true -plugins = python3 - -``` - -### Nginx - -Create a new file: - - nano /srv/umap/nginx.conf - -with this content: - -``` -# the upstream component nginx needs to connect to -upstream umap { - server unix:///srv/umap/uwsgi.sock; -} - -# configuration of the server -server { - # the port your site will be served on - listen 80; - listen [::]:80; - listen 443 ssl; - listen [::]:443 ssl; - # the domain name it will serve for - server_name your-domain.org; - charset utf-8; - - # max upload size - client_max_body_size 5M; # adjust to taste - - # Finally, send all non-media requests to the Django server. - location / { - uwsgi_pass umap; - include /srv/umap/uwsgi_params; - } -} -``` - -Remember to adapt the domain name. - -### Activate and restart the services - -Now quit the `umap` session, simply by typing Ctrl+D. - -You should now be logged in as your normal user, which is sudoer. - -- Activate the Nginx configuration file: - - sudo ln -s /srv/umap/nginx.conf /etc/nginx/sites-enabled/umap - -- Activate the uWSGI configuration file: - - sudo ln -s /srv/umap/uwsgi.ini /etc/uwsgi/apps-enabled/umap.ini - -- Restart both services: - - sudo systemctl restart uwsgi nginx - - -Now you should access your server through your url and create maps: - - http://yourdomain.org/ - - -Congratulations! - -- - - - -## Troubleshooting - -- Nginx logs are in /var/log/nginx/: - - sudo tail -f /var/log/nginx/error.log - sudo tail -f /var/log/nginx/access.log - -- uWSGI logs are in /var/log/uwsgi: - - sudo tail -f /var/log/uwsgi/umap.log - - -## Before going live - -### Add a real SECRET_KEY - -In your local.py file, add a real secret and unique `SECRET_KEY`, and do -not share it. - -### Remove DEMO flag - -In your local.py: - - UMAP_DEMO_SITE = False - DEBUG = False - -### Configure Nginx to serve statics and uploaded files: - -In your nginx config: - - location /static { - autoindex off; - access_log off; - log_not_found off; - sendfile on; - gzip on; - gzip_vary on; - alias /path/to/umap/var/static/; - } - - location /uploads { - autoindex off; - sendfile on; - gzip on; - gzip_vary on; - alias /path/to/umap/var/data/; - # Exclude direct acces to geojson, as permissions must be - # checked py django. - location /uploads/datalayer/ { return 404; } - } - -### Configure social auth - -Now you can login with your superuser, but you may allow users to user social -authentication. - -### Configure default map center - -In your local.py change those settings: - - LEAFLET_LONGITUDE = 2 - LEAFLET_LATITUDE = 51 - LEAFLET_ZOOM = 6 - -### Activate statics compression - -In your local.py, set `COMPRESS_ENABLED = True`, and then run the following command - - umap compress - -Optionally add `COMPRESS_STORAGE = "compressor.storage.GzipCompressorFileStorage"` -and add `gzip_static on` directive to Nginx `/static` location, so Nginx will -serve pregenerated files instead of compressing them on the fly. - -### Configure the site URL and short URL - -In your local.py: - - SITE_URL = "http://localhost:8019" - SHORT_SITE_URL = "http://s.hort" - -Also adapt `ALLOWED_HOSTS` accordingly. - -### Configure X-Accel-Redirect - -In order to let Nginx serve the layer geojsons but uMap still check the permissions, -you can add this settings: - - UMAP_XSENDFILE_HEADER = 'X-Accel-Redirect' - -And then add this new location in your nginx config (before the `/` location): - - location /internal/ { - internal; - gzip_vary on; - gzip_static on; - alias /path/to/umap/var/data/; - } - - -### Configure ajax proxy cache - -uMap allows to use remote URL as data sources, but those URLs are not always -CORS open, so this is why there is this "ajax-proxy" feature, where the URL is -passed to the backend. - -Additionally, there is a caching feature, which duration is configurable through -frontend settings. Valid values are: disabled, 5 min, 1 hour, 1 day. - -This configuration provides a mix option, where python deals with validating the -URL and parsing the TTL parameter, and then it passes the hand to nginx which -will serve the remote content. - -So, roughly: - -- the client calls `/ajax-proxy/?url=xxx&ttl=300` -- python will validate the URL (not internal calls…) -- if `UMAP_XSENDFILE_HEADER` is set, then the python returns an empty response - with the path `/proxy/http://url` plus it will set the cache TTL through the - header `X-Accel-Expires` -- this `/proxy/` location is then handled by nginx - - -In Nginx: - -- add the proxy cache - - proxy_cache_path /tmp/nginx_ajax_proxy_cache levels=1:2 keys_zone=ajax_proxy:10m inactive=60m; - proxy_cache_key "$uri$is_args$args"; - -- add those locations (before the `/` location): - - location ~ ^/proxy/(.*) { - internal; - add_header X-Proxy-Cache $upstream_cache_status always; - proxy_cache ajax_proxy; - proxy_cache_valid 1m; # Default. Umap will override using X-Accel-Expires - set $target_url $1; - # URL is encoded, so we need a few hack to clean it back. - if ( $target_url ~ (.+)%3A%2F%2F(.+) ){ # fix :// between scheme and destination - set $target_url $1://$2; - } - if ( $target_url ~ (.+?)%3A(.*) ){ # fix : between destination and port - set $target_url $1:$2; - } - if ( $target_url ~ (.+?)%2F(.*) ){ # fix / after port, the rest will be decoded by proxy_pass - set $target_url $1/$2; - } - resolver 8.8.8.8; - add_header X-Proxy-Target $target_url; # For debugging - proxy_pass_request_headers off; - proxy_set_header Content-Type $http_content_type; - proxy_set_header Content-Encoding $http_content_encoding; - proxy_set_header Content-Length $http_content_length; - proxy_read_timeout 10s; - proxy_connect_timeout 5s; - proxy_pass $target_url; - proxy_intercept_errors on; - error_page 301 302 307 = @handle_proxy_redirect; - } - location @handle_proxy_redirect { - resolver 8.8.8.8; - set $saved_redirect_location '$upstream_http_location'; - proxy_pass $saved_redirect_location; - } - - - - -## Add more tilelayers, pictograms… - -Go to the admin: [http://localhost:8020/admin/](http://localhost:8020/admin/), -and create the objects you want. - diff --git a/docs/index.md b/docs/index.md index 0efef186..8bc5dfd3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -2,16 +2,12 @@ !!! info "Developer docs" - This documentation is intended for people aiming to install and configure uMap. - + This documentation is intended for developers or system administrators. + If you are looking for user documentation, have a look at [the OSM wiki page](http://wiki.openstreetmap.org/wiki/UMap#Tutorials). uMap lets you create maps with OpenStreetMap layers in a minute, and embed them in your site. -If you want to get started, follow the [installation](install.md) guide. - -You might want to: - -- See [the developer docs](developer.md) -- Read [the contributing guidelines](contributing.md) -- Read through the [deployment tips](deploy/tips.md) +- 🤗 Get started and [follow the installation guide](install.md). +- 🤩 Want to contribute? [Read the contributing guidelines](contributing.md). +- 🤔 Curious about umap internals? [Check the architecture overview](architecture/overview.md). \ No newline at end of file diff --git a/docs/install.md b/docs/install.md index a527bf6d..70b60cd6 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1,13 +1,5 @@ # Installation -!!! info "Specific guides" - - This page covers how to get started with uMap. If you plan on deploying the service on a server for other people to consume, you can follow different tutorials: - - - [Deploying uMap on Ubuntu](deploy/ubuntu.md) (from scratch) - - [Deploying uMap using Docker](deploy/docker.md) - - Recent Windows distribution provide [WSL 2](https://learn.microsoft.com/en-us/windows/wsl/install), which makes it possible to run a Linux distribution of your choice. - ## System dependencies uMap is built with the [Python](https://python.org) language, and the [Django](https://djangoproject.com) framework. It needs a [PostgreSQL](https://www.postgresql.org/) database, with the [Postgis](https://postgis.net/) extension enabled. @@ -85,7 +77,7 @@ export UMAP_SETTINGS=`pwd`/local_settings.py Add database connection information in `local_settings.py`, for example -```python +```python title="local_settings.py" DATABASES = { 'default': { 'ENGINE': 'django.contrib.gis.db.backends.postgis', @@ -98,7 +90,7 @@ Depending on your installation, you might need to change the user that connects It should look like this: -```python +```python title="local_settings.py" DATABASES = { "default": { "ENGINE": "django.contrib.gis.db.backends.postgis", @@ -110,7 +102,7 @@ DATABASES = { Add a `SECRET_KEY` in `local_settings.py` with a long random secret key -```python +```title="local_settings.py" SECRET_KEY = "a long and random secret key that must not be shared" ``` @@ -122,7 +114,7 @@ openssl rand -base64 32 uMap uses [python-social-auth](http://python-social-auth.readthedocs.org/) for user authentication. So you will need to configure it according to your needs. For example -```python +```title="local_settings.py" AUTHENTICATION_BACKENDS = ( 'social_auth.backends.contrib.github.GithubBackend', 'social_auth.backends.contrib.bitbucket.BitbucketBackend', @@ -162,49 +154,10 @@ umap createsuperuser umap runserver 0.0.0.0:8000 ``` -## Configuring PostgreSQL search - -UMap uses PostgreSQL tsvector for searching. In case your database is big, you -may want to add an index. For that, here are the SQL commands to run: - -```SQL -# Create a basic search configuration -CREATE TEXT SEARCH CONFIGURATION umapdict (COPY=simple); - -# If you also want to deal with accents and case, add this before creating the index -CREATE EXTENSION unaccent; -CREATE EXTENSION btree_gin; -ALTER TEXT SEARCH CONFIGURATION umapdict ALTER MAPPING FOR hword, hword_part, word WITH unaccent, simple; - -# Now create the index -CREATE INDEX IF NOT EXISTS search_idx ON umap_map USING GIN(to_tsvector('umapdict', COALESCE(name, ''::character varying)::text), share_status); -``` - -And change your settings: - -```python -UMAP_SEARCH_CONFIGURATION = "umapdict" -``` - ## Configuring emails UMap can send the anonymous edit link by email. For this to work, you need to -add email specific settings. See [Django](https://docs.djangoproject.com/en/4.2/topics/email/#smtp-backend) -documentation. - -In general, you'll need to add something like this in your local settings: - -```python -FROM_EMAIL = "youradmin@email.org" -EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend" -EMAIL_HOST = "smtp.provider.org" -EMAIL_PORT = 456 -EMAIL_HOST_USER = "username" -EMAIL_HOST_PASSWORD = "xxxx" -EMAIL_USE_TLS = True -# or -EMAIL_USE_SSL = True -``` +add email specific settings. See [the related settings](config/settings.md#email_backend) for more info. ## Upgrading your installation @@ -215,10 +168,4 @@ pip install umap-project --upgrade umap migrate umap collectstatic umap compress -``` - -Then you need to restart your python server, for example: - -```bash -sudo systemctl restart uwsgi # or gunicorn, or… -``` +``` \ No newline at end of file diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css index 1434324f..ded19ade 100644 --- a/docs/stylesheets/extra.css +++ b/docs/stylesheets/extra.css @@ -1,3 +1,6 @@ :root { --md-primary-fg-color: #323e56; } +.md-typeset a { + color: #7990be !important; +} \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 89df7ac0..7788d235 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,24 +1,26 @@ -site_name: uMap +site_name: uMap docs site_description: uMap lets you create maps with OpenStreetMap layers in a minute and embed them in your site repo_name: umap-project/umap repo_url: https://github.com/umap-project/umap nav: - - Home: index.md - - Installation: install.md + - How-tos: + - Getting started: install.md + - Contribute: contributing.md - Architecture: - - Frontend (JS): architecture/frontend.md + - Overview: architecture/overview.md + - Frontend: architecture/frontend.md + - Configuration: + - Settings: config/settings.md + - Customize: config/customize.md + - Icon packs: config/icons.md - Deployment: - - Ubuntu Server Howto: deploy/ubuntu.md - Docker: deploy/docker.md - - Windows: deploy/windows.md - - Configuration: settings.md - - Administration: administration.md - - Contributing: contributing.md - - how-tos: - - Customize your uMap style: custom.md + - Nginx: deploy/nginx.md - Changelog: changelog.md theme: name: material + logo: assets/logo.svg + homepage: https://umap-project.org palette: - scheme: 'default' media: '(prefers-color-scheme: light)' @@ -34,6 +36,7 @@ theme: name: 'Switch to light mode' features: - navigation.sections + - navigation.footer extra_css: - stylesheets/extra.css markdown_extensions: From 4676fd09e67ec529b9157e2a013cb08e385e6758 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexis=20M=C3=A9taireau?= Date: Tue, 5 Dec 2023 23:12:09 +0100 Subject: [PATCH 5/8] Update docs/architecture/overview.md Co-authored-by: Yohan Boniface --- docs/architecture/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index fd8ac0ec..b6c264dc 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -4,7 +4,7 @@ uMap is a server and a client. The server is build with the Django framework, an Basically, here is how it works: -- Most of the data is stored as [geoJSON files](https://datatracker.ietf.org/doc/html/rfc7946), on the server. +- Most of the data is stored as [geoJSON files](https://geojson.org/), on the server. - Some other parts of the data is stored in the database (users, permissions, etc) - PostGIS is used for some of its geo features, but for the most part, the computation is done on the frontend with Leaflet. From 18afcad918562f4d9e7f3cf96f73fd1388d342ba Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexis=20M=C3=A9taireau?= Date: Tue, 5 Dec 2023 23:12:28 +0100 Subject: [PATCH 6/8] Update docs/config/customize.md Co-authored-by: Yohan Boniface --- docs/config/customize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/config/customize.md b/docs/config/customize.md index 0d12ae6b..9c2eb202 100644 --- a/docs/config/customize.md +++ b/docs/config/customize.md @@ -9,7 +9,7 @@ that. So basically you'll have your own templates and/or statics directories where you will put the templates or statics you want to control (and only those). -Inside those directory, you need to respect the exact relative path of the +Inside those directories, you need to respect the exact relative path of the templates or statics you are adding, relatively to the [templates](https://github.com/umap-project/umap/tree/master/umap/templates) and From b04ad692764288ae03917e842d14f2520b1972e8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexis=20M=C3=A9taireau?= Date: Tue, 5 Dec 2023 23:12:40 +0100 Subject: [PATCH 7/8] Update docs/config/settings.md Co-authored-by: Yohan Boniface --- docs/config/settings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/config/settings.md b/docs/config/settings.md index 181d6ac0..6d211959 100644 --- a/docs/config/settings.md +++ b/docs/config/settings.md @@ -41,7 +41,7 @@ serve pregenerated files instead of compressing them on the fly. #### DEBUG -Set it to `True`` for easier debugging in case of error. +Set it to `True` for easier debugging in case of error. #### EMAIL_BACKEND From ce4b411cec3167eb80b588cc6e542b0440f60481 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexis=20M=C3=A9taireau?= Date: Tue, 5 Dec 2023 23:13:27 +0100 Subject: [PATCH 8/8] Update docs/contributing.md Co-authored-by: Yohan Boniface --- docs/contributing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing.md b/docs/contributing.md index 52cb2945..c96991a8 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -39,7 +39,7 @@ Multiple tests suites are in use in the project. | Test suite | Location | Command | | ---------- | -------- | ------- | -| Python unit tests | `umap/tests/integration` | `pytest . --ignore umap/tests/integration` | +| Python unit tests | `umap/tests/` | `pytest . --ignore umap/tests/integration` | | Javascript unit tests | `umap/static/test` | `make testjs`| | Integration tests | `umap/tests/integration` | `pytest umap/tests/integration` |