Page MenuHomePhabricator
Create Task
Maniphest T403236

Fix revscoring load tests to match staging deployments
Closed, ResolvedPublic
Actions

Description

The revscoring locust load test failed (see T400350#11127808) because enwiki-goodfaith is not deployed in staging, while zhwiki-goodfaith is.

We could resolve this issue with one of two solutions:

1.Switch to zhwiki-goodfaith: Remove the EnwikiGoodfaith locust class and add a ZhwikiGoodfaith class in models/revscoring/goodfaith/goodfaith.py. Then create a new data/zhwiki_revids.tsv file, since data/enwiki_revids.tsv only contains revision IDs from enwiki.

2.Deploy enwiki-goodfaith in staging: Keep the existing EnwikiGoodfaith class and deploy the enwiki-goodfaith service in staging. This keeps the test aligned with other isvc load tests that already rely on data/enwiki_revids.tsv.

Option 2 is preferable since it's more scalable. Currently, only two revscoring isvc load tests exist (damaging and goodfaith). In this task, we should also add load tests for other revscoring ivcs that are deployed in staging, including:

  • articlequality
  • articletopic
  • draftquality
  • drafttopic
  • reverted

Maintaining consistency with enwiki across all load tests will simplify this task. There will be no need for creating additional language-specific datasets and test configs.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
locust_test: Add locust (load) test for revscoring-editquality-reverted model.machinelearning/liftwing/inference-servicesmain+68 -7
locust_tests: Add locust tests for revscoring models.machinelearning/liftwing/inference-servicesmain+130 -5
ml-services: Remove zhwiki revscoring-editquality-goodfaith from staging.operations/deployment-chartsmaster+0 -5
ml-services: Deploy enwiki-goodfaith on staging.operations/deployment-chartsmaster+5 -0
Customize query in gerrit

Related Objects

Event Timeline

kevinbazira created this task.Aug 29 2025, 6:15 AM
Restricted Application added subscribers: Stang, Aklapper. · View Herald TranscriptAug 29 2025, 6:15 AM
kevinbazira mentioned this in T400350: Upgrade revscoring model servers from debian bullseye to bookworm.Aug 29 2025, 10:06 AM
isarantopoulos moved this task from Unsorted to Ready To Go on the Machine-Learning-Team board.Mon, Sep 8, 7:03 AM
gerritbot added a comment.Fri, Oct 3, 10:35 AM

Change #1193391 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Deploy enwiki-goodfaith on staging.

https://gerrit.wikimedia.org/r/1193391

gerritbot added a project: Patch-For-Review.Fri, Oct 3, 10:35 AM
gerritbot added a comment.Fri, Oct 3, 11:04 AM

Change #1193391 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Deploy enwiki-goodfaith on staging.

https://gerrit.wikimedia.org/r/1193391

gkyziridis subscribed.EditedFri, Oct 3, 11:17 AM

Update

enwiki-goodfaith is deployed on staging.

$ kube_env revscoring-editquality-goodfaith ml-staging-codfw
$ kubectl get pods
NAME                                                              READY   STATUS    RESTARTS   AGE
enwiki-goodfaith-predictor-00001-deployment-59dbbf5966-jxkpb      3/3     Running   0          41s
zhwiki-goodfaith-predictor-default-00029-deployment-79c9fdvdnl4   3/3     Running   0          36d

I will develop locust_tests for each of the models listed in the description in the follow-up patches, and I will run the tests on stat machines.

isarantopoulos subscribed.Fri, Oct 3, 11:25 AM

We could remove the zhwiki deployment and free up some resources. In staging it makes sense to have 1 deployment for each family of models or an additional one if there is something different about it that would be worth testing (e.g. an additional deployment for on wikidata)

Maintenance_bot removed a project: Patch-For-Review.Fri, Oct 3, 11:30 AM
gerritbot added a comment.Fri, Oct 3, 11:37 AM

Change #1193398 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[operations/deployment-charts@master] ml-services: Remove zhwiki revscoring-editquality-goodfaith from staging.

https://gerrit.wikimedia.org/r/1193398

gerritbot added a project: Patch-For-Review.Fri, Oct 3, 11:37 AM
gkyziridis added a comment.EditedFri, Oct 3, 11:52 AM

We could remove the zhwiki deployment and free up some resources. In staging it makes sense to have 1 deployment for each family of models or an additional one if there is something different about it that would be worth testing (e.g. an additional deployment for on wikidata)

I totally agree, I already removed it in the patch above.

$ kube_env revscoring-editquality-goodfaith ml-staging-codfw
$ kubectl get pods
NAME                                                           READY   STATUS    RESTARTS   AGE
enwiki-goodfaith-predictor-00001-deployment-59dbbf5966-jxkpb   3/3     Running   0          71m

Hitting the API on staging:

curl -X POST \
"https://inference-staging.svc.codfw.wmnet:30443/v1/models/enwiki-goodfaith:predict" \
-H  "Host: enwiki-goodfaith.revscoring-editquality-goodfaith.wikimedia.org" \
-H "Content-Type: application/json" \
-d '{"rev_id": 12345}' 

{"enwiki":{"models":{"goodfaith":{"version":"0.5.1"}},"scores":{"12345":{"goodfaith":{"score":{"prediction":true,"probability":{"false":0.07396339218373627,"true":0.9260366078162637}}}}}}}
gerritbot added a comment.Fri, Oct 3, 11:54 AM

Change #1193398 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Remove zhwiki revscoring-editquality-goodfaith from staging.

https://gerrit.wikimedia.org/r/1193398

Maintenance_bot removed a project: Patch-For-Review.Fri, Oct 3, 12:30 PM
gerritbot added a comment.Fri, Oct 3, 1:19 PM

Change #1193415 had a related patch set uploaded (by Gkyziridis; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] locust_tests: Add locust tests for revscoring models.

https://gerrit.wikimedia.org/r/1193415

gerritbot added a project: Patch-For-Review.Fri, Oct 3, 1:19 PM
gkyziridis added a comment.EditedFri, Oct 3, 2:49 PM

Update

I am having some issues while trying to run the locust tests (even for models that we had reran them like edit-check).
It seems that locust does not import the model from the environment variable and tries to run all the tests, although it is facing issues with the data paths as well.
I am not sure if I am missing anything.
Here is the output with some print statements for debugging.

(.venv) gkyziridis@stat1010:~/inference-services/test/locust$ MODEL=edit_check locust
MODEL: edit-check
models.edit-check
Traceback (most recent call last):
  File "/srv/home/gkyziridis/.venv/bin/locust", line 8, in <module>
    sys.exit(main())
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/locust/main.py", line 112, in main
    docstring, _user_classes, shape_classes = load_locustfile(_locustfile)
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/locust/util/load_locustfile.py", line 70, in load_locustfile
    loader.exec_module(imported)
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/srv/home/gkyziridis/inference-services/test/locust/locustfile.py", line 13, in <module>
    _models = importlib.import_module(f"models.{model}")
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/srv/home/gkyziridis/inference-services/test/locust/models/__init__.py", line 1, in <module>
    from .article_country import *  # noqa
  File "/srv/home/gkyziridis/inference-services/test/locust/models/article_country/__init__.py", line 1, in <module>
    from .article_country import ArticleCountry  # noqa
  File "/srv/home/gkyziridis/inference-services/test/locust/models/article_country/article_country.py", line 5, in <module>
    articles = pd.read_csv("data/articles_lang_and_title.csv")
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
    self.handles = get_handle(
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/pandas/io/common.py", line 873, in get_handle
    handle = open(
FileNotFoundError: [Errno 2] No such file or directory: 'data/articles_lang_and_title.csv'
(.venv) gkyziridis@stat1010:~/inference-services/test/locust$ MODEL=article_country locust
MODEL: article_country
models.article_country
Traceback (most recent call last):
  File "/srv/home/gkyziridis/.venv/bin/locust", line 8, in <module>
    sys.exit(main())
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/locust/main.py", line 112, in main
    docstring, _user_classes, shape_classes = load_locustfile(_locustfile)
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/locust/util/load_locustfile.py", line 70, in load_locustfile
    loader.exec_module(imported)
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/srv/home/gkyziridis/inference-services/test/locust/locustfile.py", line 13, in <module>
    _models = importlib.import_module(f"models.{model}")
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/srv/home/gkyziridis/inference-services/test/locust/models/__init__.py", line 1, in <module>
    from .article_country import *  # noqa
  File "/srv/home/gkyziridis/inference-services/test/locust/models/article_country/__init__.py", line 1, in <module>
    from .article_country import ArticleCountry  # noqa
  File "/srv/home/gkyziridis/inference-services/test/locust/models/article_country/article_country.py", line 5, in <module>
    articles = pd.read_csv("data/articles_lang_and_title.csv")
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
    self.handles = get_handle(
  File "/srv/home/gkyziridis/.venv/lib/python3.9/site-packages/pandas/io/common.py", line 873, in get_handle
    handle = open(
FileNotFoundError: [Errno 2] No such file or directory: 'data/articles_lang_and_title.csv'
gkyziridis@stat1010:~/inference-services/test/locust$ MODEL_LOCUST_DIR="logo_detection" make run-locust-test
MODEL=logo_detection my_locust_venv/bin/locust --headless --csv results/logo_detection
MODEL: logo_detection
models.logo_detection
Traceback (most recent call last):
  File "/srv/home/gkyziridis/inference-services/test/locust/my_locust_venv/bin/locust", line 7, in <module>
    sys.exit(main())
  File "/srv/home/gkyziridis/inference-services/test/locust/my_locust_venv/lib/python3.9/site-packages/locust/main.py", line 108, in main
    docstring, _user_classes, shape_classes = load_locustfile(_locustfile)
  File "/srv/home/gkyziridis/inference-services/test/locust/my_locust_venv/lib/python3.9/site-packages/locust/util/load_locustfile.py", line 70, in load_locustfile
    loader.exec_module(imported)
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/srv/home/gkyziridis/inference-services/test/locust/locustfile.py", line 13, in <module>
    _models = importlib.import_module(f"models.{model}")
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/srv/home/gkyziridis/inference-services/test/locust/models/__init__.py", line 1, in <module>
    from .article_country import *  # noqa
  File "/srv/home/gkyziridis/inference-services/test/locust/models/article_country/__init__.py", line 1, in <module>
    from .article_country import ArticleCountry  # noqa
  File "/srv/home/gkyziridis/inference-services/test/locust/models/article_country/article_country.py", line 5, in <module>
    articles = pd.read_csv("data/articles_lang_and_title.csv")
  File "/srv/home/gkyziridis/inference-services/test/locust/my_locust_venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/srv/home/gkyziridis/inference-services/test/locust/my_locust_venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/srv/home/gkyziridis/inference-services/test/locust/my_locust_venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/srv/home/gkyziridis/inference-services/test/locust/my_locust_venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
    self.handles = get_handle(
  File "/srv/home/gkyziridis/inference-services/test/locust/my_locust_venv/lib/python3.9/site-packages/pandas/io/common.py", line 873, in get_handle
    handle = open(
FileNotFoundError: [Errno 2] No such file or directory: 'data/articles_lang_and_title.csv'
make: *** [Makefile:19: run-locust-test] Error 1
isarantopoulos added a comment.EditedFri, Oct 3, 3:09 PM

@gkyziridis the error is quite self explanatory the file is just not there :)

It seems that it has been renamed in https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1192833 from test/locust/data/articles_lang_and_title.csv to test/locust/data/articles_lang_title_id.csv

I'm not sure if all tests are actually being ran or it is just evaluating the imports.

gkyziridis added a comment.Fri, Oct 3, 3:19 PM
In T403236#11241544, @isarantopoulos wrote:

@gkyziridis the error is quite self explanatory the file is just not there :)

It seems that it has been renamed in https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1192833 from test/locust/data/articles_lang_and_title.csv to test/locust/data/articles_lang_title_id.csv

I'm not sure if all tests are actually being ran or it is just evaluating the imports.

I thought that since we are passing a specific model to locust, it should only run the corresponding locust tests.
Indeed the file is renamed, I will update: the tests for article_country.py and article_descriptions.py

gkyziridis added a comment.EditedFri, Oct 3, 3:50 PM

Update

  • Locust tests issue fixed ✅
  • 2nd option: Deploy enwiki-goodfaith on staging and keep the test aligned with other isvc load tests that already rely on data/enwiki_revids.tsv. ✅
  • Locust tests added for the revscoring models which have enwiki deployed on staging:
    • articlequality ✅
    • articletopic ✅
    • draftquality ✅
    • drafttopic ✅
  • Is there enwiki for reverted model? ❌

I see only enwiktionary, so I did not create tests for it, the viwiki is deployed on staging for revscoring-editquality-reverted.

Locust tests results

P83594 Locust (load) tests for revscoring models
1gkyziridis@stat1010:~/inference-services/test/locust$ MODEL_LOCUST_DIR=revscoring make run-locust-test
2# users = 30
3# spawn-rate = 2
4# run-time = 120s
5
6MODEL=revscoring my_locust_venv/bin/locust --headless --csv results/revscoring
7[2025-10-03 15:35:03,793] stat1010/INFO/locust.main: Run time limit set to 120 seconds
8[2025-10-03 15:35:03,793] stat1010/INFO/locust.main: Starting Locust 2.31.5
9[2025-10-03 15:35:03,794] stat1010/INFO/locust.runners: Ramping to 30 users at a rate of 2.00 per second
10[2025-10-03 15:35:17,813] stat1010/INFO/locust.runners: All users spawned: {"EnwikiArticlequality": 5, "EnwikiArticletopic": 5, "EnwikiDamaging": 5, "EnwikiDraftquality": 5, "EnwikiDrafttopic": 5, "EnwikiGoodfaith": 5} (30 total users)
11[2025-10-03 15:37:03,242] stat1010/INFO/locust.main: --run-time limit reached, shutting down
12Load test results are within the threshold
13[2025-10-03 15:37:03,400] stat1010/INFO/locust.main: Shutting down (exit code 0)
14Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
15--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
16POST /v1/models/enwiki-articlequality:predict 144 0(0.00%) | 910 410 4292 660 | 1.20 0.00
17POST /v1/models/enwiki-articletopic:predict 179 0(0.00%) | 189 114 696 150 | 1.50 0.00
18POST /v1/models/enwiki-damaging:predict 130 0(0.00%) | 1373 166 10566 530 | 1.09 0.00
19POST /v1/models/enwiki-draftquality:predict 146 0(0.00%) | 934 166 3963 490 | 1.22 0.00
20POST /v1/models/enwiki-drafttopic:predict 178 0(0.00%) | 199 107 762 150 | 1.49 0.00
21POST /v1/models/enwiki-goodfaith:predict 136 0(0.00%) | 1068 173 5963 450 | 1.14 0.00
22--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
23 Aggregated 913 0(0.00%) | 723 107 10566 330 | 7.64 0.00
24
25Response time percentiles (approximated)
26Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
27--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
28POST /v1/models/enwiki-articlequality:predict 660 810 910 970 1400 3100 4100 4200 4300 4300 4300 144
29POST /v1/models/enwiki-articletopic:predict 150 170 180 210 300 380 680 690 700 700 700 179
30POST /v1/models/enwiki-damaging:predict 530 960 2000 2900 3500 4400 7700 8300 11000 11000 11000 130
31POST /v1/models/enwiki-draftquality:predict 510 910 1200 1600 2400 3000 3400 3600 4000 4000 4000 146
32POST /v1/models/enwiki-drafttopic:predict 150 170 180 210 320 660 720 730 760 760 760 178
33POST /v1/models/enwiki-goodfaith:predict 450 590 1100 1800 3300 4300 4900 5900 6000 6000 6000 136
34--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
35 Aggregated 330 550 710 850 1900 3200 4200 4500 11000 11000 11000 913

gkyziridis claimed this task.Fri, Oct 3, 4:00 PM
gerritbot added a comment.Mon, Oct 6, 8:15 AM

Change #1193415 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] locust_tests: Add locust tests for revscoring models.

https://gerrit.wikimedia.org/r/1193415

gkyziridis updated the task description. (Show Details)Mon, Oct 6, 8:16 AM
gkyziridis mentioned this in rMLIS75e4d192342c: locust_tests: Add locust tests for revscoring models..Mon, Oct 6, 8:21 AM
Maintenance_bot removed a project: Patch-For-Review.Mon, Oct 6, 8:31 AM
gkyziridis moved this task from Ready To Go to In Progress on the Machine-Learning-Team board.Mon, Oct 6, 11:50 AM
kevinbazira added a comment.EditedTue, Oct 7, 11:46 AM
In T403236#11241938, @gkyziridis wrote:

I see only enwiktionary, so I did not create tests for it, the viwiki is deployed on staging for revscoring-editquality-reverted.

Thank you for working on this @gkyziridis. Since there is no enwiki reverted model (and we don't plan to train one), the locust test for this isvc will need to be an exception and follow option 1 from the task description above. We'll have to create a ViwikiReverted locust class and use a new data/viwiki_revids.tsv file, since the viwiki model-server is already deployed in staging.

gerritbot added a comment.Tue, Oct 7, 12:13 PM

Change #1194185 had a related patch set uploaded (by Gkyziridis; author: Gkyziridis):

[machinelearning/liftwing/inference-services@main] locust_test: Add locust (load) test for revscoring-editquality-reverted model.

https://gerrit.wikimedia.org/r/1194185

gerritbot added a project: Patch-For-Review.Tue, Oct 7, 12:13 PM
gkyziridis added a comment.Tue, Oct 7, 12:40 PM

Update

  • Locust tests issue fixed ✅
  • 2nd option: Deploy enwiki-goodfaith on staging and keep the test aligned with other isvc load tests that already rely on data/enwiki_revids.tsv. ✅
  • Locust tests added for the revscoring models which have enwiki deployed on staging:
    • articlequality ✅
    • articletopic ✅
    • draftquality ✅
    • drafttopic ✅
  • reverted ✅ exception uses viwiki deployed on staging
  • Created "data/viwiki_revids.tsv" file with rev_ids using: curl "https://vi.wikipedia.org/w/api.php?action=query&list=recentchanges&rcprop=ids&rcnamespace=0&rclimit=30&format=json" | jq '.query.recentchanges[] | .revid'

Locust tests results for all revscoring models

P83650 Locust test results for all revscoring models.
1gkyziridis@stat1010:~/inference-services/test/locust$ MODEL_LOCUST_DIR=revscoring make run-locust-test
2# users = 35
3# spawn-rate = 2
4# run-time = 120s
5
6MODEL=revscoring my_locust_venv/bin/locust --headless --csv results/revscoring
7[2025-10-07 12:31:35,632] stat1010/INFO/locust.main: Run time limit set to 120 seconds
8[2025-10-07 12:31:35,632] stat1010/INFO/locust.main: Starting Locust 2.31.5
9[2025-10-07 12:31:35,633] stat1010/INFO/locust.runners: Ramping to 35 users at a rate of 2.00 per second
10[2025-10-07 12:31:52,653] stat1010/INFO/locust.runners: All users spawned: {"EnwikiArticlequality": 5, "EnwikiArticletopic": 5, "EnwikiDamaging": 5, "EnwikiDraftquality": 5, "EnwikiDrafttopic": 5, "EnwikiGoodfaith": 5, "ViwikiReverted": 5} (35 total users)
11[2025-10-07 12:33:35,096] stat1010/INFO/locust.main: --run-time limit reached, shutting down
12Load test results are within the threshold
13[2025-10-07 12:33:35,188] stat1010/INFO/locust.main: Shutting down (exit code 0)
14Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
15--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
16POST /v1/models/enwiki-articlequality:predict 134 0(0.00%) | 1032 369 4788 710 | 1.12 0.00
17POST /v1/models/enwiki-articletopic:predict 183 0(0.00%) | 168 114 705 140 | 1.53 0.00
18POST /v1/models/enwiki-damaging:predict 133 0(0.00%) | 1175 154 7984 480 | 1.11 0.00
19POST /v1/models/enwiki-draftquality:predict 140 0(0.00%) | 970 157 4897 460 | 1.17 0.00
20POST /v1/models/enwiki-drafttopic:predict 174 0(0.00%) | 211 111 865 150 | 1.46 0.00
21POST /v1/models/enwiki-goodfaith:predict 139 0(0.00%) | 1015 163 4293 420 | 1.16 0.00
22POST /v1/models/viwiki-reverted:predict 162 0(0.00%) | 279 125 1246 170 | 1.36 0.00
23--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
24 Aggregated 1065 0(0.00%) | 642 111 7984 270 | 8.91 0.00
25
26Response time percentiles (approximated)
27Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
28--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
29POST /v1/models/enwiki-articlequality:predict 720 830 970 1100 2900 3800 4100 4300 4800 4800 4800 134
30POST /v1/models/enwiki-articletopic:predict 140 160 170 180 260 290 370 520 710 710 710 183
31POST /v1/models/enwiki-damaging:predict 480 960 1500 2000 3300 4200 4900 6800 8000 8000 8000 133
32POST /v1/models/enwiki-draftquality:predict 460 960 1300 1800 2400 3600 4000 4600 4900 4900 4900 140
33POST /v1/models/enwiki-drafttopic:predict 150 170 230 250 340 660 700 790 870 870 870 174
34POST /v1/models/enwiki-goodfaith:predict 420 720 1100 1500 3200 3900 4100 4200 4300 4300 4300 139
35POST /v1/models/viwiki-reverted:predict 170 240 310 330 640 760 900 920 1200 1200 1200 162
36--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
37 Aggregated 270 460 660 780 1600 3100 4000 4300 6800 8000 8000 1065

gkyziridis updated the task description. (Show Details)Tue, Oct 7, 12:46 PM
gerritbot added a comment.Tue, Oct 7, 1:43 PM

Change #1194185 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] locust_test: Add locust (load) test for revscoring-editquality-reverted model.

https://gerrit.wikimedia.org/r/1194185

gkyziridis mentioned this in rMLIS8e251d7b0b3a: locust_test: Add locust (load) test for revscoring-editquality-reverted model..Tue, Oct 7, 1:44 PM
gkyziridis closed this task as Resolved.Tue, Oct 7, 1:59 PM
gkyziridis moved this task from In Progress to 2025-2026 Q1 Done on the Machine-Learning-Team board.
Maintenance_bot removed a project: Patch-For-Review.Tue, Oct 7, 2:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits