Merge pull request #96 from Tkd-Alex/analytics

Show analytics of your points mining
This commit is contained in:
Alessandro Maggio
2021-03-12 22:50:22 +01:00
committed by GitHub
12 changed files with 448 additions and 13 deletions

3
.gitignore vendored
View File

@@ -148,4 +148,5 @@ chromedriver*
cookies/*
logs/*
screenshots/*
htmls/*
htmls/*
analytics/*

View File

@@ -34,10 +34,11 @@ Read more about channels point [here](https://help.twitch.tv/s/article/channel-p
- [Bet strategy](#bet-strategy)
- [FilterCondition](#filtercondition)
- [Example](#example)
6. 🍪 [Migrating from old repository (the original one)](#migrating-from-old-repository-the-original-one)
7. 🪟 [Windows](#windows)
8. 📱 [Termux](#termux)
9. ⚠️ [Disclaimer](#disclaimer)
6. 📈 [Analytics](#analytics)
7. 🍪 [Migrating from old repository (the original one)](#migrating-from-old-repository-the-original-one)
8. 🪟 [Windows](#windows)
9. 📱 [Termux](#termux)
10. ⚠️ [Disclaimer](#disclaimer)
## Community
@@ -61,6 +62,7 @@ If you have any issues or you want to contribute, you are welcome! But please be
- Auto claim game drops from Twitch inventory [#21](https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/issues/21) Read more about game drops [here](https://help.twitch.tv/s/article/mission-based-drops)
- Place the bet / make a prediction and win or lose (🍀) your channel points!
No browser needed. [#41](https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/issues/41) ([@lay295](https://github.com/lay295))
- Analytics chart
## Logs feature
### Full logs
@@ -259,7 +261,7 @@ If you follow so many streamers on Twitch, but you don't want to mine points for
```python
from TwitchChannelPointsMiner import TwitchChannelPointsMiner
twitch_miner = TwitchChannelPointsMiner("your-twitch-username")
twitch_miner.mine(followers=True, blacklist=["user1", "user2"]) # Automatic use the followers list OR
twitch_miner.mine(followers=True, blacklist=["user1", "user2"]) # Blacklist example
```
4. Start mining! `python run.py`
@@ -389,6 +391,25 @@ Allowed values for `where` are: `GT, LT, GTE, LTE`
- If you want to place the bet ONLY if the highest bet is lower than 2000
`FilterCondition(by=OutcomeKeys.TOP_POINTS, where=Condition.LT, value=2000)`
## Analytics
We have recently introduced a little frontend where you can show with a chart you points trend. The script will spawn a Flask web-server on your machine where you can select binding address and port.
The chart provides some annotation to handle the prediction and watch strike events. Usually annotation are used to notice big increase / decrease of points. If you want to can disable annotations.
On each (x, y) points Its present a tooltip that show points, date time and reason of points gained / lost. This web page was just a funny idea, and it is not intended to use for a professional usage.
If you want you can toggle the dark theme with the dedicated checkbox.
| Light theme | Dark theme |
| ----------- | ---------- |
| ![Light theme](./assets/chart-analytics-light.png) | ![Dark theme](./assets/chart-analytics-dark.png) |
For use this feature just call the `analytics` method before start mining. Read more at: [#96](https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/issues/96)
The chart will be autofreshed each `refresh` minutes. If you want to connect from one to second machine that have that webpanel you have to use `0.0.0.0` instead of `127.0.0.1`.
```python
from TwitchChannelPointsMiner import TwitchChannelPointsMiner
twitch_miner = TwitchChannelPointsMiner("your-twitch-username")
twitch_miner.analytics(host="127.0.0.1", port=5000, refresh=5) # Analytics web-server
twitch_miner.mine(followers=True, blacklist=["user1", "user2"])
```
## Migrating from an old repository (the original one):
If you already have a `twitch-cookies.pkl` and you don't want to log in again, please create a `cookies/` folder in the current directory and then copy the .pkl file with a new name `your-twitch-username.pkl`
```

View File

@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-
import copy
import logging
import os
import random
import signal
import sys
@@ -10,7 +10,9 @@ import time
import uuid
from collections import OrderedDict
from datetime import datetime
from pathlib import Path
from TwitchChannelPointsMiner.classes.AnalyticsServer import AnalyticsServer
from TwitchChannelPointsMiner.classes.entities.PubsubTopic import PubsubTopic
from TwitchChannelPointsMiner.classes.entities.Streamer import (
Streamer,
@@ -33,8 +35,10 @@ from TwitchChannelPointsMiner.utils import (
# - chardet.charsetprober - [feed]
# - chardet.charsetprober - [get_confidence]
# - requests - [Starting new HTTPS connection (1)]
# - Flask (werkzeug) logs
logging.getLogger("chardet.charsetprober").setLevel(logging.ERROR)
logging.getLogger("requests").setLevel(logging.ERROR)
logging.getLogger("werkzeug").setLevel(logging.ERROR)
logger = logging.getLogger(__name__)
@@ -62,12 +66,16 @@ class TwitchChannelPointsMiner:
username: str,
password: str = None,
claim_drops_startup: bool = False,
# Settings for logging and selenium as you can see.
priority: list = [Priority.STREAK, Priority.DROPS, Priority.ORDER],
# This settings will be global shared trought Settings class
logger_settings: LoggerSettings = LoggerSettings(),
# Default values for all streamers
streamer_settings: StreamerSettings = StreamerSettings(),
):
Settings.analytics_path = os.path.join(Path().absolute(), "analytics", username)
Path(Settings.analytics_path).mkdir(parents=True, exist_ok=True)
self.username = username
# Set as global config
@@ -100,6 +108,12 @@ class TwitchChannelPointsMiner:
for sign in [signal.SIGINT, signal.SIGSEGV, signal.SIGTERM]:
signal.signal(sign, self.end)
def analytics(self, host: str = "127.0.0.1", port: int = 5000, refresh: int = 5):
http_server = AnalyticsServer(host=host, port=port, refresh=refresh)
http_server.daemon = True
http_server.name = "Analytics Thread"
http_server.start()
def mine(self, streamers: list = [], blacklist: list = [], followers=False):
self.run(streamers=streamers, blacklist=blacklist, followers=followers)
@@ -185,7 +199,9 @@ class TwitchChannelPointsMiner:
if streamer.viewer_is_mod is True:
streamer.settings.make_predictions = False
self.original_streamers = copy.deepcopy(self.streamers)
self.original_streamers = [
streamer.channel_points for streamer in self.streamers
]
# If we have at least one streamer with settings = make_predictions True
make_predictions = at_least_one_value_in_settings_is(
@@ -276,7 +292,13 @@ class TwitchChannelPointsMiner:
if self.sync_campaigns_thread is not None:
self.sync_campaigns_thread.join()
time.sleep(1)
# Check if all the mutex are unlocked.
# Prevent breaks of .json file
for streamer in self.streamers:
if streamer.mutex.locked():
streamer.mutex.acquire()
streamer.mutex.release()
self.__print_report()
sys.exit(0)
@@ -322,7 +344,7 @@ class TwitchChannelPointsMiner:
if self.streamers[streamer_index].history != {}:
gained = (
self.streamers[streamer_index].channel_points
- self.original_streamers[streamer_index].channel_points
- self.original_streamers[streamer_index]
)
logger.info(
f"{repr(self.streamers[streamer_index])}, Total Points Gained (after farming - before farming): {_millify(gained)}",

View File

@@ -0,0 +1,62 @@
import logging
import os
from pathlib import Path
from threading import Thread
from flask import Flask, Response, cli, render_template
from TwitchChannelPointsMiner.classes.Settings import Settings
cli.show_server_banner = lambda *_: None
logger = logging.getLogger(__name__)
def streamers_available():
path = Settings.analytics_path
return [
f
for f in os.listdir(path)
if os.path.isfile(os.path.join(path, f)) and f.endswith(".json")
]
def read_json(streamer):
path = Settings.analytics_path
streamer = streamer if streamer.endswith(".json") else f"{streamer}.json"
return Response(
open(os.path.join(path, streamer)) if streamer in streamers_available() else [],
status=200,
mimetype="application/json",
)
def index(refresh=5):
return render_template(
"charts.html",
refresh=(refresh * 60 * 1000),
streamers=",".join(streamers_available()),
)
class AnalyticsServer(Thread):
def __init__(self, host: str = "127.0.0.1", port: int = 5000, refresh: int = 5):
super(AnalyticsServer, self).__init__()
self.host = host
self.port = port
self.refresh = refresh
self.app = Flask(
__name__,
template_folder=os.path.join(Path().absolute(), "assets"),
static_folder=os.path.join(Path().absolute(), "assets"),
)
self.app.add_url_rule("/", "index", index, defaults={"refresh": refresh})
self.app.add_url_rule("/json/<string:streamer>", "json", read_json)
def run(self):
logger.info(
f"Analytics running on http://{self.host}:{self.port}/",
extra={"emoji": ":globe_with_meridians:"},
)
self.app.run(host=self.host, port=self.port, threaded=True)

View File

@@ -176,11 +176,18 @@ class WebSocketsPool:
if streamer_index != -1:
try:
if message.topic == "community-points-user-v1":
if message.type in ["points-earned", "points-spent"]:
balance = message.data["balance"]["balance"]
ws.streamers[streamer_index].channel_points = balance
ws.streamers[streamer_index].persistent_series(
event_type=message.data["point_gain"]["reason_code"]
if message.type == "points-earned"
else "Spent"
)
if message.type == "points-earned":
earned = message.data["point_gain"]["total_points"]
reason_code = message.data["point_gain"]["reason_code"]
balance = message.data["balance"]["balance"]
ws.streamers[streamer_index].channel_points = balance
logger.info(
f"+{earned}{ws.streamers[streamer_index]} - Reason: {reason_code}.",
extra={
@@ -193,6 +200,9 @@ class WebSocketsPool:
ws.streamers[streamer_index].update_history(
reason_code, earned
)
ws.streamers[streamer_index].persistent_annotations(
reason_code, f"+{earned} - {reason_code}"
)
elif message.type == "claim-available":
ws.twitch.claim_bonus(
ws.streamers[streamer_index],
@@ -333,8 +343,18 @@ class WebSocketsPool:
-points["won"],
counter=-1,
)
if event_prediction.result["type"] != "LOSE":
ws.streamers[streamer_index].persistent_annotations(
event_prediction.result["type"],
f"{ws.events_predictions[event_id].title}",
)
elif message.type == "prediction-made":
event_prediction.bet_confirmed = True
ws.streamers[streamer_index].persistent_annotations(
"PREDICTION_MADE",
f"Decision: {event_prediction.bet.decision['choice']} - {event_prediction.title}",
)
except Exception:
logger.error(
f"Exception raised for topic: {message.topic} and message: {message}",

View File

@@ -58,7 +58,11 @@ class Message(object):
else (
self.data["channel_id"]
if "channel_id" in self.data
else self.topic_user
else (
self.data["balance"]["channel_id"]
if "balance" in self.data
else self.topic_user
)
)
)
)

View File

@@ -1,5 +1,9 @@
import json
import logging
import os
import time
from datetime import datetime
from threading import Lock
from TwitchChannelPointsMiner.classes.entities.Bet import BetSettings
from TwitchChannelPointsMiner.classes.entities.Stream import Stream
@@ -60,6 +64,7 @@ class Streamer(object):
"raid",
"history",
"streamer_url",
"mutex",
]
def __init__(self, username, settings=None):
@@ -81,6 +86,8 @@ class Streamer(object):
self.streamer_url = f"{URL}/{self.username}"
self.mutex = Lock()
def __repr__(self):
return f"Streamer(username={self.username}, channel_id={self.channel_id}, channel_points={_millify(self.channel_points)})"
@@ -146,3 +153,44 @@ class Streamer(object):
and self.stream.drops_tags is True
and self.stream.campaigns_ids != []
)
# === ANALYTICS === #
def persistent_annotations(self, event_type, event_text):
event_type = event_type.upper()
if event_type in ["WATCH_STREAK", "WIN", "PREDICTION_MADE"]:
primary_color = (
"#45c1ff"
if event_type == "WATCH_STREAK"
else ("#ffe045" if event_type == "PREDICTION_MADE" else "#54ff45")
)
data = {
"borderColor": primary_color,
"label": {
"style": {"color": "#000", "background": primary_color},
"text": event_text,
},
}
self.__save_json("annotations", data)
def persistent_series(self, event_type="Watch"):
self.__save_json("series", event_type=event_type)
def __save_json(self, key, data={}, event_type="Watch"):
# https://stackoverflow.com/questions/4676195/why-do-i-need-to-multiply-unix-timestamps-by-1000-in-javascript
# data.update({"x": round(time.time() * 1000)})
now = datetime.now().replace(microsecond=0)
data.update({"x": round(datetime.timestamp(now) * 1000)})
if key == "series":
data.update({"y": self.channel_points})
if event_type is not None:
data.update({"z": event_type.replace("_", " ").title()})
fname = os.path.join(Settings.analytics_path, f"{self.username}.json")
with self.mutex:
json_data = json.load(open(fname, "r")) if os.path.isfile(fname) else {}
if key not in json_data:
json_data[key] = []
json_data[key].append(data)
json.dump(json_data, open(fname, "w"), indent=4)

Binary file not shown.

After

Width:  |  Height:  |  Size: 111 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 107 KiB

227
assets/charts.html Normal file
View File

@@ -0,0 +1,227 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Twitch-Channel-Points-Miner-v2</title>
<script src="https://cdn.jsdelivr.net/npm/apexcharts"></script>
<script src="https://cdn.jsdelivr.net/npm/jquery@3.5.1/dist/jquery.min.js"></script>
<link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.css" rel="stylesheet"/>
<link href="https://cdnjs.cloudflare.com/ajax/libs/bulma/0.6.1/css/bulma.css" rel="stylesheet"/>
<link href="{{url_for('static', filename='dark-theme.css')}}" rel="stylesheet"/>
</head>
<body style="height: 100vh;">
<div class="container">
<div style="text-align: center">
<img style="margin-top: -20px;" width="600px" src="{{url_for('static', filename='banner.png')}}" alt="banner">
<br>
<div style="margin-top: -15px;">
<a style="text-decoration: none;" href="https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/blob/master/LICENSE">
<img alt="License" src="https://img.shields.io/github/license/Tkd-Alex/Twitch-Channel-Points-Miner-v2" />
</a>
<a style="text-decoration: none;" href="https://www.python.org/downloads/release/python-360/">
<img alt="Python3" src="https://img.shields.io/badge/built%20for-Python≥3.6-red.svg?style=flat">
</a>
<a style="text-decoration: none;" href="https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/pulls">
<img alt="PRsWelcome" src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat" />
</a>
<a style="text-decoration: none;" href="https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/stargazers">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/Tkd-Alex/Twitch-Channel-Points-Miner-v2" />
</a>
<a style="text-decoration: none;" href="https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/issues?q=is%3Aissue+is%3Aclosed">
<img alt="GitHub closed issues" src="https://img.shields.io/github/issues-closed/Tkd-Alex/Twitch-Channel-Points-Miner-v2">
</a>
<a style="text-decoration: none;" href="https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2">
<img alt="GitHub last commit" src="https://img.shields.io/github/last-commit/Tkd-Alex/Twitch-Channel-Points-Miner-v2" />
</a>
</div>
</div>
<br>
<div class="columns">
<div class="column is-10">
<div class="tabs">
<ul>
{% for streamer in streamers.split(',')|sort %}
<li onClick="changeStreamer('{{ streamer }}', {{ loop.index }})"><a>{{ streamer.replace(".json", "") }}</a></li>
{% endfor %}
</ul>
</div>
</div>
<div class="column" style="text-align: right">
<label class="checkbox">
Annotations
<input type="checkbox" checked="true" id="annotations">
</label>
<label class="checkbox">
Dark mode
<input type="checkbox" checked="true" id="dark-mode">
</label>
</div>
</div>
<div class="box" id="chart" style="padding: 0.30rem;"></div>
<!-- <hr style="margin: 1.2rem 0;"> -->
<div style="text-align: center">If you want to help on this project, please leave a star 🌟 and share it with your friends! 😎 - A coffee is always a gesture of LOVE ❤️
<a href="https://www.buymeacoffee.com/tkdalex" target="_blank">
<img style="margin-bottom: -8px; margin-left: 10px;" src="https://cdn.buymeacoffee.com/buttons/lato-yellow.png" alt="Buy Me A Coffee" height="5" width="138">
</a>
</div>
<br>
</div>
</body>
<script type="text/javascript">
// https://apexcharts.com/javascript-chart-demos/line-charts/zoomable-timeseries/
var options = {
series: [],
chart: {
type: 'area',
stacked: false,
height: 400,
zoom: {
type: 'x',
enabled: true,
autoScaleYaxis: true
},
// background: '#2B2D3E',
foreColor: '#fff'
},
dataLabels: {
enabled: false
},
stroke: {
curve: 'smooth',
},
markers: {
size: 0,
},
title: {
text: 'Channel points (dates are displayed in UTC)',
align: 'left'
},
colors: ["#f9826c"],
fill: {
type: 'gradient',
gradient: {
shadeIntensity: 1,
inverseColors: false,
opacityFrom: 0.5,
opacityTo: 0,
stops: [0, 90, 100]
},
},
yaxis: {
title: {
text: 'Channel points'
},
},
xaxis: {
type: 'datetime',
labels: {
datetimeUTC: false
}
},
tooltip: {
theme: 'dark',
shared: false,
x: {
show: true,
format: 'HH:mm:ss dd MMM',
},
custom: ({series, seriesIndex, dataPointIndex, w}) => {
return (`<div class="apexcharts-active">
<div class="apexcharts-tooltip-title">${w.globals.seriesNames[seriesIndex]}</div>
<div class="apexcharts-tooltip-series-group apexcharts-active" style="order: 1; display: flex; padding-bottom: 0px !important;">
<div class="apexcharts-tooltip-text">
<div class="apexcharts-tooltip-y-group">
<span class="apexcharts-tooltip-text-label"><b>Points</b>: ${series[seriesIndex][dataPointIndex]}</span><br>
<span class="apexcharts-tooltip-text-label"><b>Reason</b>: ${w.globals.seriesZ[seriesIndex][dataPointIndex] ? w.globals.seriesZ[seriesIndex][dataPointIndex] : ''}</span>
</div>
</div>
</div>
</div>`
)
}
},
noData: {
text: 'Loading...'
}
};
var chart = new ApexCharts(document.querySelector("#chart"), options);
var currentStreamer = null;
var annotations = [];
var refresh = parseInt("{{ refresh }}");
$(document).ready(function() {
chart.render();
$("li").eq(0).click();
});
function changeStreamer(streamer, index) {
$("li").removeClass( "is-active" )
$("li").eq(index - 1).addClass('is-active');
currentStreamer = streamer;
getStreamerData(streamer);
}
function getStreamerData(streamer) {
if(currentStreamer == streamer){
$.getJSON(`./json/${streamer}`, function (response) {
chart.updateSeries([{
name: streamer.replace(".json", ""),
data: response["series"]
}], true)
annotations = response["annotations"];
updateAnnotations()
setTimeout(function(){getStreamerData(streamer);}, 300000); // 5 minutes
});
}
}
function updateAnnotations() {
if($('#annotations').prop("checked") === true){
clearAnnotations()
if(annotations && annotations.length > 0)
annotations.forEach((annotation, index) => {
annotations[index]['id'] = `id-${index}`
chart.addXaxisAnnotation(annotation, true)
})
} else clearAnnotations()
}
function clearAnnotations(){
if(annotations && annotations.length > 0)
annotations.forEach((annotation, index) => {
chart.removeAnnotation(annotation['id'])
})
}
function toggleDarkMode(){
var darkMode = $('#dark-mode').prop("checked")
$("link[href='{{url_for('static', filename='dark-theme.css')}}']").prop("disabled", !darkMode);
chart.updateOptions({
colors: darkMode === true ? ["#f9826c"] : ['#008ffb'],
chart: {
foreColor: darkMode === true ? "#fff" : '#373d3f'
},
tooltip: {
theme: darkMode === true ? "dark" : "light"
}
})
// if (darkMode === true) $("#chart").addClass("box")
// else $("#chart").removeClass("box")
}
$('#annotations').click(() => { updateAnnotations(); });
$('#dark-mode').click(() => { toggleDarkMode(); });
</script>
</html>

29
assets/dark-theme.css Normal file
View File

@@ -0,0 +1,29 @@
body {
background: #343E59;
color: #fff;
}
.box {
background-color: #2B2D3E;
}
.tabs a {
border-bottom-color: #dbdbdb;
color: #fff;
border-bottom-style: none;
}
.tabs li.is-active a {
border-bottom-color: #f9826c;
color: #fff;
border-bottom-style: solid;
}
.tabs a:hover {
border-bottom-color: #dbdbdb;
color: #dbdbdb;
border-bottom-style: solid;
}
.tabs ul{
margin-bottom: 5px;
border-bottom-style: none;
}
.checkbox:hover{
color: #f9826c;
}

View File

@@ -6,4 +6,5 @@ python-dateutil
emoji
millify
pre-commit
flask
colorama