It does not pollute the output because they go to sqlite-log/gitea.log
and it may provide clues for debugging. When trying to figure out a
problem in production, Debug is likely to be used but in a development
environment a very fine grain log is the most useful. If that's really
too much noise, the lines can be trimmed by removing all those with [T].
(cherry picked from commit 666b02eb9d6e50b3a8d3ec5659fa6fb8451d0d9f)
(cherry picked from commit b69e4d30581258d82d9b2a4daf932f75d7d7171e)
(cherry picked from commit 0c1aecd7ec0d3bf9801c02ae24c89dd5ce074255)
(cherry picked from commit 0304e41345e8ab19537d3686b5487c8cb4b280c2)
(cherry picked from commit 4bcc7e02ce496bea54de39d1059cf39617d2a8d9)
(cherry picked from commit 98364235f2792ce058984c03dde2f609adfbfc88)
(cherry picked from commit ca465e97c1b5d1db684cf9fa1ecbaa14f4bb077c)
(cherry picked from commit 653045ade1e2423ce363ffd2682c1f5cadf8059f)
(cherry picked from commit 9ca2e644ab28c867a92925a6eb83ecbb6805434f)
(cherry picked from commit 47c8a05b37d17201b24bab2d211d672e1fd6c2e0)
(cherry picked from commit 4235d929ed1cb03e96c4f117a33b3120372235e0)
(cherry picked from commit 10541c4b1c61ce6dc8c56481c0999e6ed52dbc80)
(cherry picked from commit c1a0e1eb2f7673de83a57295531675c1ffbcdc8e)
(cherry picked from commit ba5d0193dad6466238afa891166ace513b0c942f)
(cherry picked from commit 58d0e2e138df9cbae514c6bf8d8b9fe0e44d2e93)
(cherry picked from commit d411ab911b31d5e90942ee78658c23d062f160e0)
(cherry picked from commit 454b5d92805153eae89ce8a9b1bd059bce0d05f5)
(cherry picked from commit 1d7ce2a39c841e77492ef08c0e86c3544ecca88d)
(cherry picked from commit 3af8757583ab9bf8fc5eb05ed1e6901a34682512)
(cherry picked from commit 0b818152096d58e402e3c8bbc2415695af0cdfb9)
(cherry picked from commit 7180a3222a00c33ec7dc9bcaa7e5a1e3ae924735)
(cherry picked from commit 069044f4a6dd7457662e5ca296c714275d16becf)
(cherry picked from commit dd1438f38995e397399d8957fca696f82b73707e)
(cherry picked from commit 4ae6b5d64d59902ce003199b5d652d6c4799ffcf)
(cherry picked from commit c572666d78d0f2cf2a33d5a50ea363430ac9a0a9)
(cherry picked from commit de0a4480bc3e40d61aa5342d93a2ef510abe3826)
(cherry picked from commit 60272ef4f73608407ffedd27892376b6c36a7069)
(cherry picked from commit a68fdde4becef625ce4e654ba6ea8ebee55251e5)
(cherry picked from commit ff448cf3a698dd45c713b734fce3e374c99940bf)
(cherry picked from commit 208498caaab339d4543abfb2b77de3c5793a73c8)
(cherry picked from commit 1abd3efad842f276f1c768ab40213200b62a0d3b)
(cherry picked from commit 818c7f4ca9)
(cherry picked from commit 1e552bbe44f79d9a2f5567a0180505b9a41d39d3)
(cherry picked from commit 88dbd6da1d7b4a7faf9dda2d25d68f3202c250a5)
(cherry picked from commit b1c86fb7775dd34511f4d5069db1bd3e5ef68e7a)
(cherry picked from commit 185b1e25b12b56c9ae904adb787854da9b74ddc2)
(cherry picked from commit 0af35605ae8a2594ae0b594278748242bce9d8e0)
(cherry picked from commit 350975b63f12df508e7d8d01845032d32f30035c)
(cherry picked from commit 2788142515ff6bb316a120be60556f39659ee9ea)
(cherry picked from commit b418a53717b247e10d2dda26d1d236400d423a94)
(cherry picked from commit 086cde3f10d786cd3c05bf59e359f08a23183d91)
(cherry picked from commit 3e4818c779bd894b4b86a3672d24623fa45e1824)
(cherry picked from commit f2bf1fe64f9ebf8460a93805289fdba2d5cad6d3)
(cherry picked from commit db4eb288ee464c9a799ff2e059c0a5aa632e633b)
(cherry picked from commit f8db31821b8089d5c9390f902854b7817a0381ef)
(cherry picked from commit 2b13c06f499110ba76d3121d1c2f09df3023564f)
(cherry picked from commit 3fd0092056a4e0f43ff59b2123539ddb1ea0afcc)
(cherry picked from commit 6f7662507f26c34a3de54b3e69ea65ab10921163)
(cherry picked from commit 8a15e41887ee3a72fdb5b7718d2750d48615205b)
(cherry picked from commit 3540ab6d4a215ec7197f1e4907bb33a4a4e97be9)
(cherry picked from commit a207d02ff23b4d6e879c0e06712901f8d4031f30)
(cherry picked from commit 35309db8095fd2aef78a6131f6ac668686f2aa99)
(cherry picked from commit 6ff35732f0570f453f596286fa09747a987d35b3)
(cherry picked from commit e6d922612da7b47462e821cbe2017e34acdac632)
The setting `MAILER_TYPE` is deprecated.
According to the config cheat sheet, it should be `PROTOCOL`.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
* Clean the "tools" directory. The "tools" directory contains only two
files, move them.
* The "external_renderer.go" works like "cat" command to echo Stdin to
Stdout , to help testing.
* The `// gobuild: external_renderer` is incorrect, there should be no
space: `//gobuild: external_renderer`
* The `fmt.Print(os.Args[1])` is not a well-defined behavior, and it's
never used.
* The "watch.sh" is for "make watch", it's somewhat related to "build"
* After this PR, there is no "tools" directory, the project root
directory looks slightly simpler than before.
* Remove the legacy "contrib/autoboot.sh", there is no
"gogs_supervisord.sh"
* Remove the legacy "contrib/mysql.sql", it's never mentioned anywhere.
* Remove the legacy "contrib/pr/checkout.go", it has been broken for
long time, and it introduces unnecessary dependencies of the main code
base.
