Commit graph

142 commits

Author SHA1 Message Date
Dean Cording
323fc26754 Fix for NaN values on startup that upset Javascript JSON decoders 2022-03-02 21:22:01 +10:00
Dean Cording
26437e00ea Typo fixed 2022-03-02 16:15:05 +10:00
Dean Cording
16bdf3df9d Added systemd support for dish_grpc_mqtt 2022-03-02 15:15:37 +10:00
sparky8512
3977f0b422
Merge pull request #39 from DeanCording/main
Added option to send data as a single MQTT message in JSON format.
2022-02-27 14:07:24 -08:00
Dean Cording
357ea6bcc7 Added option to send data as a single MQTT message in JSON format. 2022-02-27 22:03:31 +10:00
sparky8512
09b8717131 Tweaks/fixes mostly related to --poll-loops
Adjust how the --poll-loops option handles the first set of polled loops for history stats, to make that set of data run against a number of samples (and therefore total time interval) that more closely matches subsequent sets of polled loops. This especially applies to the case where the stats are resuming from a prior counter, where without this logic, the first set had an unintuitively large number of samples.

Fix timestamp reporting for history stats to better reflect the actual accumulated data, for cases where later polls of the history data had failed. This introduced the potential for the status data and the history stats data to have inconsistent timestamps, and required the timestamp collection to be moved from the individual output scripts into dish_common. Rather than deal with the complication this would create for CSV output, where there is only 1 timestamp field, just disallow the combination of options that could result in different timestamps (For CSV output only).

Fix an error case where poll loop counting was not being reset correctly.

Fix explicit --samples option to be honored when resuming from counter for history stats.
2022-02-20 13:39:07 -08:00
sparky8512
c0f7dd5096 Remove samples option default
Stop pretending 3600 is a meaningful default for the --samples command line option. That hasn't been the case since the history buffer was reduced to 900 samples.

The default behavior if loop interval not set is now to process all samples, which was what effectively what it was doing, anyway.
2022-02-20 10:36:09 -08:00
sparky8512
9ca1a8a51c Remove valid check from obstruction status data
Remove the validity check that was gating the reporting of the obstruction_duration and obstruction_interval data items.

When that got added to the grpc protocol, I assumed that the valid flag meant whether the other 2 fields were useful data. That may actually have been the intent, for all I know, but I've never seen that flag report true. However, I also never saw the banner in the app telling me I would get obstructions every X minutes, either. That is, until a few weeks ago, probably due to a dish firmware change. Now I see the app reporting this data, so it must be ignoring that valid flag, or at least not using it for that purpose.

Probably nobody cares about this data (including me), given that nobody ever complained about it always reporting empty values, but I don't want to remove it and mess up db schemas, so it might as well be populated.
2022-02-19 16:16:54 -08:00
sparky8512
32567059b8 Resume from counter for history stats CSV output
Add an option to output to a specified file instead of standard output (which is still the default), and if set, attempt to read prior end counter for use in resuming history stats computation at that point. This behavior can be disabled using the --skip-query (-k) option.

Resuming will only work for CSV files that start with a header line that matches the last line in the file, and is currently only enabled for history stats, not bulk history, because the file read operation is not at all optimized for large files. (And because I don't think anyone is really using CSV for recording bulk history data, I only implemented that because it was easy to do so and helps with testing.)

While testing this, I realized that the implementation of the --poll-loops option has an awkward interaction with resuming from prior counter value, but that impacts all the scripts that support resuming from counter, so I will address that in a subsequent change.
2022-02-19 15:51:46 -08:00
sparky8512
0225a9783c Various fixes to the InfluxDB 2.x script
This should address all the comments I dropped onto pull request #37.

The only significant fix here is that the SSL command line options should all work now, although slightly differently than how they work in the InfluxDB 1.x script. Whether or not SSL is enabled is controlled by the URL, specifically whether it is an https: URL or an http: one, and the options just control the CA cert parameters.
2022-02-19 15:33:58 -08:00
sparky8512
52dc5ec79d Document new alert added in most recent firmware 2022-01-20 14:07:40 -08:00
sparky8512
8655f75bab Record stats from polled data on script shutdown
If there was history data collected via the --poll-loops option, but not yet used to compute history stats, do so on script shutdown. This should reduce the amount of data lost when the script restarts, for example to reboot the system on which it runs, since --poll-loops allows for collection of history data beyond the 15-minute buffer size the dish (currently) holds.

This will only work if the script is shut down via SIGTERM or SIGINT (for example, by interrupting with Control-C).
2022-01-20 14:07:09 -08:00
sparky8512
7dbb47ab40 New utility script to record dish protocol data
Not really Starlink specific, and pretty much redundant with similar functionality in grpcurl, but more appropriate for running periodically, as it will use a filename that should be specific to the protocol data content, so old data will not be overwritten when there is new data available.
2022-01-20 14:04:46 -08:00
sparky8512
0ace643acc A few more updates related to the new script
Add the  Python packages required for the InfluxDB 2.x client to Dockerfile, at the specific versions that I tested with. Most of the package dependencies overlap with the InfluxDB 1.x client package, so there was not much added.

Note the new script name in the relevant parts of the README.
2021-11-26 15:34:51 -08:00
sparky8512
abd63c2225 Correct a few cosmetic copy/paste errors
Noticed while reviewing the new InfluxDB 2.x script, which was derived from this one.

Also, denote this script as being for InfluxDB 1.x, since there is now a separate script for 2.x.
2021-11-26 14:58:50 -08:00
sparky8512
5b6521a51c
Merge pull request #37 from derpeter/main
add native support for influxDB 2.0 / Dashboard for influxDB 2.0
2021-11-26 14:36:24 -08:00
Peter Hasse
acca52ce75 added grafana dashboard for influx2 2021-11-26 11:02:51 +01:00
Peter Hasse
6fc08ec277 updates systemD unit 2021-11-26 11:02:33 +01:00
Peter Hasse
d8cf8c6a7f updated readme for inlfuxdDB 2.x 2021-11-26 11:02:16 +01:00
Peter Hasse
5d47274cdc small changes on systemd unit 2021-11-24 18:14:51 +01:00
Peter Hasse
f303f28209 added simple systemd unit 2021-11-24 17:26:57 +01:00
Peter Hasse
08ebefcb6e revert from utc to local time 2021-11-24 17:26:40 +01:00
Peter Hasse
0471a2d6e1 updates requirements for influx2 2021-11-24 15:09:03 +01:00
Peter Hasse
5905fb00f9 initial commit influxDB 2.0 client 2021-11-24 15:06:36 +01:00
sparky8512
859dc84b88 Documentation updates
A bunch of content from the README and get_history_notes.txt has been moved to the Wiki, as it is not critical to understanding how to install or use the scripts.

Move the checked in Grafana dashboard into a subdirectory in a feeble attempt to encourage other people to submit more of them.

Change the officially supported Docker image to the one published to GitHub Package repository by this project's workflow task.
2021-11-08 20:10:39 -08:00
sparky8512
46c8604dfc Update the yagrc package version for Docker image
This should have gone in with change c35588d01f, but I forgot that the Docker config has a separate list of package versions so they can be pinned to specific version numbers, instead of just specifying a minimum.
2021-11-08 19:56:37 -08:00
sparky8512
f11b826684 Port recent changes from starlink_grpc.py
This brings the fix for the protocol changes that removed some of the history fields into the JSON version of this module.

I'm really only bothering with this because the script that uses it is mentioned in the project README, which I am in the process of updating.
2021-11-08 19:52:21 -08:00
sparky8512
c35588d01f Fixes related to failed grpc network connection
Add timeouts to all gRPC remote calls and bump yagrc package requirement to a version that does same for the reflection service, as well as fixing a state issue around failed lazy import resolution.

This should address the script hang symptom on issue #36.
2021-11-06 20:16:50 -07:00
sparky8512
833f82f575
Rename action to disambiguate
Was previously just the default file name and the copy/pasted action name from the example in the docs.
2021-10-25 14:01:56 -07:00
sparky8512
b0f40c7dd7 Remove usage of another obsolete field
last_24h_obstructed_s has been removed from the grpc service protocol, along with all the other deprecated fields. This renders the seconds_obstructed item in the status mode group useless.

This is the down side of using reflection at run time to pull the protocol definitions. If something gets removed from the protocol that is still being used, it will break the run time instead of just returning default values. Then again, if it gets removed from the protocol, it's no longer useful, anyway.

This should address issue #35.
2021-10-25 07:39:48 -07:00
sparky8512
0ba1de9c66
Manual action for publishing Docker image 2021-10-19 15:41:59 -07:00
sparky8512
3dddd95ff3 Bring back the state data item in status group
Derive connectivity state information from the "outage" field of the get_status response, which I hadn't noticed before because it only populates when the dish is not in a connected state. This restores the state data item in the status mode group, which had been rendered useless due to a grpc service change.

In addition to the previous possible state names, this adds a few more that pertain to outages while otherwise connected, which I think were just previously reported as "CONNECTED", as well as some special cases of offline.
2021-10-19 15:34:45 -07:00
sparky8512
79839c0b19 Remove usage of obsolete grpc fields
A recent firmware change has stopped populating a number of result message fields in the grpc service that had previously been marked as deprecated. This caused the script to start crashing in some use cases.

While those fields are still in the protocol definition for now, this change removes usage of them entirely, in case they get removed. As things are now, they are useless, anyway, since they will always just return default values.

This renders useless the state, snr, scheduled, obstructed, and unscheduled items in the status, bulk_history, and ping_drop mode groups. Those items now mostly return empty values.

See issue #32 for more detail.
2021-10-16 08:47:18 -07:00
sparky8512
9f726e71af Further tweak to how -o loop polling works
The prior 2 changes made the handling of the first set of polled loops inconsistent with subsequent sets with respect to maintaining data across dish reboot. This change makes them both work the same (and correctly).
2021-10-15 09:57:10 -07:00
sparky8512
afab7553e3 Fix -o option breakage from recent change
This restores the ability of -o to keep polled data across a reboot. It broke due to a simple issue in concatenate_history, but I realized that counter tracking for -o accumulated data was also a bit broken, so I fixed that, too.

Fix #31
2021-10-07 12:23:48 -07:00
sparky8512
a0366b4526
Merge pull request #30 from sparky8512/obstruction-map
Obstruction map support

Fixes #27
2021-09-08 13:49:36 -07:00
sparky8512
1a9af6ad5d Interval loop support for obstruction maps
Tracked on issue #27
2021-09-08 13:45:36 -07:00
sparky8512
e1070965f2 Initial cut of obstruction map support
Add a new command line script, dish_obstruction_map.py, that writes a PNG image based on the obstruction map data queried from the dish.

Supports color or greyscale output and either with or without alpha channel.

Does not yet support running in an interval loop, mostly because that will require templatizing the output filename in order to be useful.

Tracked on issue #27
2021-09-07 17:29:56 -07:00
sparky8512
af940a9727 Improvements to how the -o option works
Change the loop polling function (-o) to aggregate the history data each polling loop instead of just keeping the last polled history so it can be logged when reboot is detected. This allows for computing statistics across a longer period than the size of the dish's history buffer, which has been reduced to 15 minutes recently.

This change also makes it so data is not logged right away when dish reboot is detected, so the logging always happens at the specified interval whether there was a reboot or not.

Finally, change the poll loop counting so data is not emitted on the first loop when polling is configured. That made sense to do when the history buffer was large enough to have the entire period's worth of data, but now it just results in a short period in the log output every time the script is restarted.

Fixes #29
2021-09-07 12:02:14 -07:00
sparky8512
41caa76962 Add a few more fields to the status group
Add dish direction and "prolonged" obstruction info to the status mode group.

These were added to the grpc service at some point over the last several months.

Only lightly tested, given that my dish no longer reports significant periods of obstruction.

This is related to discussion in issue #27, although it doesn't address that issue in the slightest.
2021-09-05 17:41:13 -07:00
sparky8512
0f540a4b96
Add note about reduction of history buffer data size 2021-08-19 13:38:36 -07:00
sparky8512
74b0a98ffa Write empty string instead of None in text output
Not sure how I failed to notice this in testing, but for the cases where there is no data to output, the common layers pass up a None python object, and the text output was sometimes turning that into "None", whereas for CVS output, at least, just omitting any value in the field would be more appropriate.

Oddly, the bulk data path did have logic for turning None into empty string, but not the status or history stats code paths. This makes it so all text output has the same transformation logic.
2021-05-23 18:54:27 -07:00
sparky8512
d603272d90 Add option to emit booleans as numeric values
New command line option, -N or --numeric, that will cause all boolean values, including those in sequences/arrays, to be written as 1 or 0 instead of True or False.

Per request in issue 26.

WARNING: Use or non-use of this option with the database output scripts will change the schema of the data. sqlite doesn't care about that, because it stores booleans as integers, anyway, but InfluxDB will trip an error if you try to record data points with this option to a database that has data point recorded without it, or vice versa.
2021-05-23 18:45:18 -07:00
sparky8512
23b54c1344
Merge pull request #24 from sparky8512/docker-reflection
Switch docker reflection from grpcurl to yagrc
2021-03-27 08:21:03 -07:00
sparky8512
77e4046ba9 Switch docker reflection from grpcurl to yagrc
Remove grpcurl and grpcio-tools from container configuration and add yagrc, so that the direct reflection support in the Python scripts can be used. Also, pin all Python packages, including package dependencies, to specific version numbers, since that was already the case by happenstance due to the way Docker caches its build images and an undesirable version of the protobuf package was being cached.

Addresses #23, which was directly about this change.
Expected to also address #22, as a result of pinning the protobuf package version.
Should also prevent a recurrence of #18, since yagrc will automatically get any new dependant protocol files via reflection.
2021-03-23 18:32:00 -07:00
sparky8512
07389cb0d9 Remove dependence on Python 3.8 or later
statistics.quantiles was not present in Python 3.7 or earlier, which is a problem on Windows if you want to run a binary optimized version of the protobuf package, since those are not currently being posted for Python 3.8 or later.

This change switches to use the weighted median function just with equal weights. It's a bit of overkill, but it also cuts out the mess that was working around deficiencies of the statistics.quantiles implementation.
2021-03-16 13:19:06 -07:00
sparky8512
55ba411db8 Go back to using message number for alert bits
SpaceX has been using inconsistent field ordering when adding alerts, so field index cannot be used to consistently identify the specific alerts. Message number is more appropriate for that, anyway, but is not guaranteed to be a low enough number to fit into a bit field. Oh well, in the unlikely event that SpaceX switches to larger message numbers, they just won't show up in the alerts bit field (but will still show up in alert_detail).

This does make the bit ordering in alerts inconsistent with prior versions of these tools, but I've never actually seen one of these alerts report true, so hopefully this doesn't impact anyone.

The alerts are still sorted by index number in the alert_detail text output, which is a problem for CSV output, but I think ordering by message number instead would be pointlessly complex. alert_detail is not a great fit for CSV output anyway, due to its variable length, so just added a warning about that in the text script module doc.
2021-03-07 09:22:52 -08:00
sparky8512
203efaf84d Note detail of where data is recorded per backend
It came up in the course of discussion in issue #20 that I hadn't actually documented this anywhere.
2021-02-27 16:39:28 -08:00
sparky8512
be776cde1c Resume from last counter for history stats
Currently only implemented for sqlite, since with InfluxDB, there is the complication that the InfluxDB server may not be available to query at script start time. Also, it only applies when polling all samples, which is not the default, and even then can be disabled with either --skip-query or --no-counter options.

Remove the crontab instructions from the README, since the periodic loop functionality is probably now a better approach for periodic recording of stats data.
2021-02-27 15:57:35 -08:00
sparky8512
206bbbf919
Switch two more references to new script name 2021-02-21 14:11:19 -08:00