(cherry picked from commit 1d7ce2a39c841e77492ef08c0e86c3544ecca88d)
(cherry picked from commit 3af8757583ab9bf8fc5eb05ed1e6901a34682512)
(cherry picked from commit 0b818152096d58e402e3c8bbc2415695af0cdfb9)
(cherry picked from commit 7180a3222a00c33ec7dc9bcaa7e5a1e3ae924735)
(cherry picked from commit 069044f4a6dd7457662e5ca296c714275d16becf)
(cherry picked from commit dd1438f38995e397399d8957fca696f82b73707e)
(cherry picked from commit 4ae6b5d64d59902ce003199b5d652d6c4799ffcf)
(cherry picked from commit c572666d78d0f2cf2a33d5a50ea363430ac9a0a9)
(cherry picked from commit de0a4480bc3e40d61aa5342d93a2ef510abe3826)
(cherry picked from commit 60272ef4f73608407ffedd27892376b6c36a7069)
(cherry picked from commit a68fdde4becef625ce4e654ba6ea8ebee55251e5)
(cherry picked from commit ff448cf3a698dd45c713b734fce3e374c99940bf)
(cherry picked from commit 208498caaab339d4543abfb2b77de3c5793a73c8)
(cherry picked from commit 1abd3efad842f276f1c768ab40213200b62a0d3b)
(cherry picked from commit 818c7f4ca9)
(cherry picked from commit 1e552bbe44f79d9a2f5567a0180505b9a41d39d3)
(cherry picked from commit 88dbd6da1d7b4a7faf9dda2d25d68f3202c250a5)
(cherry picked from commit b1c86fb7775dd34511f4d5069db1bd3e5ef68e7a)
(cherry picked from commit 185b1e25b12b56c9ae904adb787854da9b74ddc2)
(cherry picked from commit 0af35605ae8a2594ae0b594278748242bce9d8e0)
(cherry picked from commit 350975b63f12df508e7d8d01845032d32f30035c)
(cherry picked from commit 2788142515ff6bb316a120be60556f39659ee9ea)
(cherry picked from commit b418a53717b247e10d2dda26d1d236400d423a94)
(cherry picked from commit 086cde3f10d786cd3c05bf59e359f08a23183d91)
(cherry picked from commit 3e4818c779bd894b4b86a3672d24623fa45e1824)
(cherry picked from commit f2bf1fe64f9ebf8460a93805289fdba2d5cad6d3)
(cherry picked from commit db4eb288ee464c9a799ff2e059c0a5aa632e633b)
(cherry picked from commit f8db31821b8089d5c9390f902854b7817a0381ef)
(cherry picked from commit 2b13c06f499110ba76d3121d1c2f09df3023564f)
(cherry picked from commit 3fd0092056a4e0f43ff59b2123539ddb1ea0afcc)
(cherry picked from commit 6f7662507f26c34a3de54b3e69ea65ab10921163)
(cherry picked from commit 8a15e41887ee3a72fdb5b7718d2750d48615205b)
(cherry picked from commit 3540ab6d4a215ec7197f1e4907bb33a4a4e97be9)
(cherry picked from commit a207d02ff23b4d6e879c0e06712901f8d4031f30)
(cherry picked from commit 35309db8095fd2aef78a6131f6ac668686f2aa99)
(cherry picked from commit 6ff35732f0570f453f596286fa09747a987d35b3)
(cherry picked from commit e6d922612da7b47462e821cbe2017e34acdac632)
(cherry picked from commit a8df762405)
The setting `MAILER_TYPE` is deprecated.
According to the config cheat sheet, it should be `PROTOCOL`.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
- This PR attempts to split our various DB tests into separate
pipelines.
- It splits up some of the extra feature-related tests rather than
having most of them in the MySQL test.
- It disables the race detector for some of the pipelines as well, as it
can cause slower runs and is mostly redundant when the pipelines just
swap DBs.
- It builds without SQLite support for any of the non-SQLite pipelines.
- It moves the e2e test to using SQLite rather than PG (partially
because I moved the minio tests to PG and that mucked up the test
config, and partially because it avoids another running service)
- It splits up the `go mod download` task in the Makefile from the tool
installation, as the tools are only needed in the compliance pipeline.
(Arguably even some of the tools aren't needed there, but that could be
a follow-up PR)
- SQLite is now the only arm64 pipeline, moving PG back to amd64 which
can leverage autoscaler
Should resolve#22010 - one thing that wasn't changed here but is
mentioned in that issue, unit tests are needed in the same pipeline as
an integration test in order to form a complete coverage report (at
least as far as I could tell), so for now it remains in a pipeline with
a DB integration test.
Please let me know if I've inadvertently changed something that was how
it was on purpose.
---
I will say sometimes it's hard to pin down the average time, as a
pipeline could be waiting for a runner for X minutes and that brings the
total up by X minutes as well, but overall this does seem to be faster
on average.
---------
Signed-off-by: jolheiser <john.olheiser@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Disable this test for the moment because the used imap container image
seems unstable which results in many failed CI builds.
Co-authored-by: Jason Song <i@wolfogre.com>
closes#13585fixes#9067fixes#2386
ref #6226
ref #6219fixes#745
This PR adds support to process incoming emails to perform actions.
Currently I added handling of replies and unsubscribing from
issues/pulls. In contrast to #13585 the IMAP IDLE command is used
instead of polling which results (in my opinion 😉) in cleaner code.
Procedure:
- When sending an issue/pull reply email, a token is generated which is
present in the Reply-To and References header.
- IMAP IDLE waits until a new email arrives
- The token tells which action should be performed
A possible signature and/or reply gets stripped from the content.
I added a new service to the drone pipeline to test the receiving of
incoming mails. If we keep this in, we may test our outgoing emails too
in future.
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
