From 2e71acbbdb846e1ef8a90f960e2a3984c7efe7f2 Mon Sep 17 00:00:00 2001 From: sparky8512 <76499194+sparky8512@users.noreply.github.com> Date: Sat, 16 Jan 2021 10:17:32 -0800 Subject: [PATCH 1/3] Changes to work better with Docker containers Handle SIGTERM to enable graceful script shutdown when a container is stopped. This currently only matters for the InfluxDB scripts, and only when they run in a loop, since if the script is hard-terminated, it won't flush out any queued data points to the InfluxDB server. This also required changing the entrypoint script to exec python instead of running it as a child process of the shell running entrypoint.sh, since Docker will only deliver SIGTERM to the parent process it started directly. Also, add -t 30 to the default Docker command to match the script default behavior prior to the changes in 46f65a62144b783af1c1857ae6b4380adbebf80d --- Dockerfile | 4 ++-- README.md | 14 ++++++++++---- dishHistoryInflux.py | 13 +++++++++++++ dishStatusInflux.py | 13 +++++++++++++ entrypoint.sh | 4 ++-- 5 files changed, 40 insertions(+), 8 deletions(-) diff --git a/Dockerfile b/Dockerfile index b909990..900ee20 100644 --- a/Dockerfile +++ b/Dockerfile @@ -19,7 +19,7 @@ WORKDIR /app # run crond as main process of container ENTRYPOINT ["/bin/sh", "/app/entrypoint.sh"] -CMD ["dishStatusInflux.py"] +CMD ["dishStatusInflux.py", "-t", "30"] # docker run -d --name='starlink-grpc-tools' -e INFLUXDB_HOST=192.168.1.34 -e INFLUXDB_PORT=8086 -e INFLUXDB_DB=starlink -# --net='br0' --ip='192.168.1.39' neurocis/starlink-grpc-tools dishStatusInflux.py +# --net='br0' --ip='192.168.1.39' neurocis/starlink-grpc-tools dishStatusInflux.py -t 30 diff --git a/README.md b/README.md index 6cee089..a1dffb1 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,8 @@ For more information on what Starlink is, see [starlink.com](https://www.starlin ## Prerequisites +Most of the scripts here are [Python](https://www.python.org/) scripts. To use them, you will either need Python installed on your system or you can use the Docker image. If you use the Docker image, you can skip the rest of the prerequisites other than Docker itself. For Linux systems, the python package from your distribution should be fine, as long as it is Python 3. The JSON script should actually work with Python 2.7, but the grpc scripts all require Python 3 (and Python 2.7 is past end-of-life, so is not recommended anyway). + `parseJsonHistory.py` operates on a JSON format data representation of the protocol buffer messages, such as that output by [gRPCurl](https://github.com/fullstorydev/grpcurl). The command lines below assume `grpcurl` is installed in the runtime PATH. If that's not the case, just substitute in the full path to the command. All the tools that pull data from the dish expect to be able to reach it at the dish's fixed IP address of 192.168.100.1, as do the Starlink [Android app](https://play.google.com/store/apps/details?id=com.starlink.mobile), [iOS app](https://apps.apple.com/us/app/starlink/id1537177988), and the browser app you can run directly from http://192.168.100.1. When using a router other than the one included with the Starlink installation kit, this usually requires some additional router configuration to make it work. That configuration is beyond the scope of this document, but if the Starlink app doesn't work on your home network, then neither will these scripts. That being said, you do not need the Starlink app installed to make use of these scripts. @@ -15,6 +17,8 @@ The scripts that use [MQTT](https://mqtt.org/) for output require the `paho-mqtt The scripts that use [InfluxDB](https://www.influxdata.com/products/influxdb/) for output require the `influxdb` Python package. Information about how to install that can be found at https://github.com/influxdata/influxdb-python. Note that this is the (slightly) older version of the InfluxDB client Python module, not the InfluxDB 2.0 client. It can still be made to work with an InfluxDB 2.0 server, but doing so requires using `influx v1` [CLI commands](https://docs.influxdata.com/influxdb/v2.0/reference/cli/influx/v1/) on the server to map the 1.x username, password, and database names to their 2.0 equivalents. +Running the scripts within a [Docker](https://www.docker.com/) container requires Docker to be installed. Information about how to install that can be found at https://docs.docker.com/engine/install/ + ## Usage Of the 3 groups below, the grpc scripts are really the only ones being actively developed. The others are mostly by way of example of what could be done with the underlying data. @@ -69,7 +73,7 @@ To collect and record summary stats at the top of every hour, you could put some By default, all of these scripts will pull data once, send it off to the specified data backend, and then exit. They can instead be made to run in a periodic loop by passing a `-t` option to specify loop interval, in seconds. For example, to capture status information to a InfluxDB server every 30 seconds, you could do something like this: ``` -python3 dishStatusInflux.py -t 30 [... probably other args to specifiy server options ...] +python3 dishStatusInflux.py -t 30 [... probably other args to specify server options ...] ``` Some of the scripts (currently only the InfluxDB ones) also support specifying options through environment variables. See details in the scripts for the environment variables that map to options. @@ -101,14 +105,16 @@ The Starlink router also exposes a gRPC service, on ports 9000 (HTTP/2.0) and 90 Initialization of the container can be performed with the following command: ``` -docker run -d --name='starlink-grpc-tools' -e INFLUXDB_HOST={InfluxDB Hostname} \ +docker run -d -t --name='starlink-grpc-tools' -e INFLUXDB_HOST={InfluxDB Hostname} \ -e INFLUXDB_PORT={Port, 8086 usually} \ -e INFLUXDB_USER={Optional, InfluxDB Username} \ -e INFLUXDB_PWD={Optional, InfluxDB Password} \ -e INFLUXDB_DB={Pre-created DB name, starlinkstats works well} \ - neurocis/starlink-grpc-tools dishStatusInflux.py -v + neurocis/starlink-grpc-tools dishStatusInflux.py -v -t 30 ``` -`dishStatusInflux.py -v` is optional and will run same but not -verbose, or you can replace it with one of the other scripts if you wish to run that instead. There is also an `GrafanaDashboard - Starlink Statistics.json` which can be imported to get some charts like: +The `-t` option to `docker run` will prevent Python from buffering the script's standard output and can be omitted if you don't care about seeing the verbose output in the container logs as soon as it is printed. + +The `dishStatusInflux.py -v -t 30` is optional and omitting it will run same but not verbose, or you can replace it with one of the other scripts if you wish to run that instead, or use other command line options. There is also an `GrafanaDashboard - Starlink Statistics.json` which can be imported to get some charts like: ![image](https://user-images.githubusercontent.com/945191/104257179-ae570000-5431-11eb-986e-3fedd04bfcfb.png) diff --git a/dishHistoryInflux.py b/dishHistoryInflux.py index 07e43f7..fcecaec 100644 --- a/dishHistoryInflux.py +++ b/dishHistoryInflux.py @@ -14,6 +14,7 @@ import getopt import datetime import logging import os +import signal import sys import time import warnings @@ -23,6 +24,15 @@ from influxdb import InfluxDBClient import starlink_grpc +class Terminated(Exception): + pass + + +def handle_sigterm(signum, frame): + # Turn SIGTERM into an exception so main loop can clean up + raise Terminated() + + def main(): arg_error = False @@ -220,6 +230,7 @@ def main(): # user has explicitly said be insecure, so don't warn about it warnings.filterwarnings("ignore", message="Unverified HTTPS request") + signal.signal(signal.SIGTERM, handle_sigterm) influx_client = InfluxDBClient(**icargs) try: next_loop = time.monotonic() @@ -231,6 +242,8 @@ def main(): time.sleep(next_loop - now) else: break + except Terminated: + pass finally: if gstate.points: rc = flush_points(influx_client) diff --git a/dishStatusInflux.py b/dishStatusInflux.py index 6a708f3..7fa1bd7 100644 --- a/dishStatusInflux.py +++ b/dishStatusInflux.py @@ -11,6 +11,7 @@ import getopt import logging import os +import signal import sys import time import warnings @@ -23,6 +24,15 @@ import spacex.api.device.device_pb2 import spacex.api.device.device_pb2_grpc +class Terminated(Exception): + pass + + +def handle_sigterm(signum, frame): + # Turn SIGTERM into an exception so main loop can clean up + raise Terminated() + + def main(): arg_error = False @@ -244,6 +254,7 @@ def main(): # user has explicitly said be insecure, so don't warn about it warnings.filterwarnings("ignore", message="Unverified HTTPS request") + signal.signal(signal.SIGTERM, handle_sigterm) influx_client = InfluxDBClient(**icargs) try: next_loop = time.monotonic() @@ -255,6 +266,8 @@ def main(): time.sleep(next_loop - now) else: break + except Terminated: + pass finally: # Flush on error/exit if gstate.pending: diff --git a/entrypoint.sh b/entrypoint.sh index 5cce7fe..cd88da5 100644 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -2,7 +2,7 @@ printenv >> /etc/environment ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone -grpcurl -plaintext -protoset-out dish.protoset 192.168.100.1:9200 describe SpaceX.API.Device.Device +grpcurl -plaintext -protoset-out dish.protoset 192.168.100.1:9200 describe SpaceX.API.Device.Device > /dev/null python3 -m grpc_tools.protoc --descriptor_set_in=dish.protoset --python_out=. --grpc_python_out=. spacex/api/device/device.proto python3 -m grpc_tools.protoc --descriptor_set_in=dish.protoset --python_out=. --grpc_python_out=. spacex/api/common/status/status.proto python3 -m grpc_tools.protoc --descriptor_set_in=dish.protoset --python_out=. --grpc_python_out=. spacex/api/device/command.proto @@ -10,4 +10,4 @@ python3 -m grpc_tools.protoc --descriptor_set_in=dish.protoset --python_out=. -- python3 -m grpc_tools.protoc --descriptor_set_in=dish.protoset --python_out=. --grpc_python_out=. spacex/api/device/dish.proto python3 -m grpc_tools.protoc --descriptor_set_in=dish.protoset --python_out=. --grpc_python_out=. spacex/api/device/wifi.proto python3 -m grpc_tools.protoc --descriptor_set_in=dish.protoset --python_out=. --grpc_python_out=. spacex/api/device/wifi_config.proto -/usr/local/bin/python3 $@ +exec /usr/local/bin/python3 $@ From 9b04e8387ce48961e2c7f7cd3e9794ed5fd5be36 Mon Sep 17 00:00:00 2001 From: sparky8512 <76499194+sparky8512@users.noreply.github.com> Date: Sat, 16 Jan 2021 10:33:22 -0800 Subject: [PATCH 2/3] Minor changes based on thorough proof read --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a1dffb1..5bea7dc 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ For more information on what Starlink is, see [starlink.com](https://www.starlin ## Prerequisites -Most of the scripts here are [Python](https://www.python.org/) scripts. To use them, you will either need Python installed on your system or you can use the Docker image. If you use the Docker image, you can skip the rest of the prerequisites other than Docker itself. For Linux systems, the python package from your distribution should be fine, as long as it is Python 3. The JSON script should actually work with Python 2.7, but the grpc scripts all require Python 3 (and Python 2.7 is past end-of-life, so is not recommended anyway). +Most of the scripts here are [Python](https://www.python.org/) scripts. To use them, you will either need Python installed on your system or you can use the Docker image. If you use the Docker image, you can skip the rest of the prerequisites other than making sure the dish IP is reachable and Docker itself. For Linux systems, the python package from your distribution should be fine, as long as it is Python 3. The JSON script should actually work with Python 2.7, but the grpc scripts all require Python 3 (and Python 2.7 is past end-of-life, so is not recommended anyway). `parseJsonHistory.py` operates on a JSON format data representation of the protocol buffer messages, such as that output by [gRPCurl](https://github.com/fullstorydev/grpcurl). The command lines below assume `grpcurl` is installed in the runtime PATH. If that's not the case, just substitute in the full path to the command. @@ -53,7 +53,7 @@ python3 -m grpc_tools.protoc --descriptor_set_in=../dish.protoset --python_out=. python3 -m grpc_tools.protoc --descriptor_set_in=../dish.protoset --python_out=. --grpc_python_out=. spacex/api/device/wifi.proto python3 -m grpc_tools.protoc --descriptor_set_in=../dish.protoset --python_out=. --grpc_python_out=. spacex/api/device/wifi_config.proto ``` -Then move the resulting files to where the Python scripts can find them in its import path, such as in the same directory as the scripts themselves. +Then move the resulting files to where the Python scripts can find them in the import path, such as in the same directory as the scripts themselves. Once those are available, the `dishHistoryStats.py` script can be used in place of the `grpcurl | parseJsonHistory.py` pipeline, with most of the same command line options. For example: ``` @@ -115,6 +115,6 @@ docker run -d -t --name='starlink-grpc-tools' -e INFLUXDB_HOST={InfluxDB Hostnam The `-t` option to `docker run` will prevent Python from buffering the script's standard output and can be omitted if you don't care about seeing the verbose output in the container logs as soon as it is printed. -The `dishStatusInflux.py -v -t 30` is optional and omitting it will run same but not verbose, or you can replace it with one of the other scripts if you wish to run that instead, or use other command line options. There is also an `GrafanaDashboard - Starlink Statistics.json` which can be imported to get some charts like: +The `dishStatusInflux.py -v -t 30` is optional and omitting it will run same but not verbose, or you can replace it with one of the other scripts if you wish to run that instead, or use other command line options. There is also a `GrafanaDashboard - Starlink Statistics.json` which can be imported to get some charts like: ![image](https://user-images.githubusercontent.com/945191/104257179-ae570000-5431-11eb-986e-3fedd04bfcfb.png) From 9a57c93d738e603b8ab7e58fb3f534ad04e40b6f Mon Sep 17 00:00:00 2001 From: sparky8512 <76499194+sparky8512@users.noreply.github.com> Date: Sat, 16 Jan 2021 20:12:22 -0800 Subject: [PATCH 3/3] Back out changing the default command options Per review feedback, this could have interfered with the ability to set this option via environment variable. It was a bit messy, anyway. --- Dockerfile | 4 ++-- README.md | 6 ++++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Dockerfile b/Dockerfile index 900ee20..b909990 100644 --- a/Dockerfile +++ b/Dockerfile @@ -19,7 +19,7 @@ WORKDIR /app # run crond as main process of container ENTRYPOINT ["/bin/sh", "/app/entrypoint.sh"] -CMD ["dishStatusInflux.py", "-t", "30"] +CMD ["dishStatusInflux.py"] # docker run -d --name='starlink-grpc-tools' -e INFLUXDB_HOST=192.168.1.34 -e INFLUXDB_PORT=8086 -e INFLUXDB_DB=starlink -# --net='br0' --ip='192.168.1.39' neurocis/starlink-grpc-tools dishStatusInflux.py -t 30 +# --net='br0' --ip='192.168.1.39' neurocis/starlink-grpc-tools dishStatusInflux.py diff --git a/README.md b/README.md index 5bea7dc..78960f3 100644 --- a/README.md +++ b/README.md @@ -110,11 +110,13 @@ docker run -d -t --name='starlink-grpc-tools' -e INFLUXDB_HOST={InfluxDB Hostnam -e INFLUXDB_USER={Optional, InfluxDB Username} \ -e INFLUXDB_PWD={Optional, InfluxDB Password} \ -e INFLUXDB_DB={Pre-created DB name, starlinkstats works well} \ - neurocis/starlink-grpc-tools dishStatusInflux.py -v -t 30 + neurocis/starlink-grpc-tools dishStatusInflux.py -v ``` The `-t` option to `docker run` will prevent Python from buffering the script's standard output and can be omitted if you don't care about seeing the verbose output in the container logs as soon as it is printed. -The `dishStatusInflux.py -v -t 30` is optional and omitting it will run same but not verbose, or you can replace it with one of the other scripts if you wish to run that instead, or use other command line options. There is also a `GrafanaDashboard - Starlink Statistics.json` which can be imported to get some charts like: +The `dishStatusInflux.py -v` is optional and omitting it will run same but not verbose, or you can replace it with one of the other scripts if you wish to run that instead, or use other command line options. There is also a `GrafanaDashboard - Starlink Statistics.json` which can be imported to get some charts like: ![image](https://user-images.githubusercontent.com/945191/104257179-ae570000-5431-11eb-986e-3fedd04bfcfb.png) + +You'll probably want to run with the `-t` option to `dishStatusInflux.py` to collect status information periodically for this to be meaningful.