- Change the values for the nodeinfo API, to use branded values.
- Resolves https://codeberg.org/forgejo/forgejo/issues/257
(cherry picked from commit 4608c57688d8b12dbc265dd21bfe7cd269efb116)
(cherry picked from commit e837e8a52943f803a40cd0151e24f7fe8edb11ec)
(cherry picked from commit 6601328d3ce9b57dbaa768dd2d41295293ff94f9)
(cherry picked from commit c6be21d4870e6b748a85f0da19bd4b717875b224)
(cherry picked from commit 5adc6ffee2e6f1af72039747df809aa6ebd2198f)
(cherry picked from commit 2ff8d166ac1e56ab7a349d70f875bd2ae9763418)
(cherry picked from commit b6a90e7e5af0e998cbbf1fc1edb901ae31090999)
(cherry picked from commit d1089e706cda009a6a23462adf498fd24a609b0a)
Conflicts:
tests/integration/api_nodeinfo_test.go
(cherry picked from commit 7a29df737d979abed4d37f084e3a92ee788d2c6e)
(cherry picked from commit 3655a30c60229167bc007e139d0461a5648741d5)
(cherry picked from commit c90d61141016ffbbaaa6b4f4657dacf5446f30c0)
(cherry picked from commit 0274bd8860bd00de628fba990e42bf7385ddf5b9)
(cherry picked from commit fdb786b71decd1c968f7c473c94463140f192fc3)
(cherry picked from commit 4f08f100a19886210b89d1cacfd09e6db0e48fb7)
(cherry picked from commit 56a27118227521fae93e052bb77265fb10c6dcf8)
(cherry picked from commit 3b2cfa452df0d2651a75c5e3fcd442c38acff109)
(cherry picked from commit 773ddcf956a897bbaa75aff3087f8a64b254239c)
(cherry picked from commit b3c12e6bccb4ea65f7fa1ab38abaa313013aac9d)
(cherry picked from commit 1315177082a6b32cb04719124d76105ed656d09f)
(cherry picked from commit 3f1007795ce769ce16f2c74454ad4459723378cf)
(cherry picked from commit 352bda0147a02c81f1a59412a0ab9f34993cb127)
(cherry picked from commit 3d8c153ded7547f2343d0da6051b296c9265f286)
(cherry picked from commit 20b56692693e054bb3c04b4ef12b29b0715b4530)
(cherry picked from commit 1574643a6a9634e5b92c033a4bfb69062a86bd05)
Update semantic version according to specification
(cherry picked from commit 22510f41306f9c133a7e99b61f9c38dabfd4b810)
Mise à jour de 'Makefile'
(cherry picked from commit c3d85d8409f1bb18a45659a167cf1ffee057f3b6)
(cherry picked from commit 5ea23098513c068444226af41faf9be9c4c998e6)
(cherry picked from commit ec5217b9d1b94bb04e34ce8c27eecbdc6f3a247a)
(cherry picked from commit 14f08e364b2ad7e3176b45d3ec8da5f5605b4315)
(cherry picked from commit b4465c67b8737aed9fd79a43046aa3795a298e68)
[API] [SEMVER] replace number with version
(cherry picked from commit fba48e64977a75e77c9c6cf8e3f8a679a5e7b733)
(cherry picked from commit 532ec5d8782703b62d7d02312764e50637cd016e)
[API] [SEMVER] [v1.20] less is replaced by css
(cherry picked from commit 01ca3a4f4291cf4c62df42cbd12a00638f3cdc70)
(cherry picked from commit 1d928c3ab2e1fda6082e1400d89ea96d6605ebaa)
(cherry picked from commit a39dc804cd1da86444ed709455857f069475e7db)
Conflicts:
webpack.config.js
(cherry picked from commit adc68578b3eb41215338125edfca91804706f8a1)
(cherry picked from commit 9b8d98475fd7efcc15bf0d91b0af6ef3aff64508)
(cherry picked from commit 25161039745a63282257eded9616390241af8013)
(cherry picked from commit 18e6287963a0e1863cd08fd514a4e732a1904a68)
(cherry picked from commit e9694e67ab93ad74f7eae3152d88d95e8493e739)
(cherry picked from commit a9763edaf002c8009a2cca4115cb8b926f8cdc85)
(cherry picked from commit e2b550f4fba144f9637a212b56bfe30cac1adb3b)
(cherry picked from commit 2edac36701)
[API] Forgejo API /api/forgejo/v1 (squash)
Update semver as v1.20 is entering release candidate mode
(cherry picked from commit 4995098ec3c6eee19a7ad7bc956b71c6e937c04c)
(cherry picked from commit 3dd6f2862e10250e5caefbf46a6c8092ab68eabc)
(cherry picked from commit e8192737ef5773b520f85c6ea405219ad6fce1a5)
(cherry picked from commit 7b1245cc706c95e70c01601c54e0dfabb162ba31)
(cherry picked from commit 4d7cd59703dffbd2ab5bb417afe8ace3d0af7141)
(cherry picked from commit 7a7a293dbd8f5e2acd313d3d96c54a337f717f1f)
(cherry picked from commit 2f95143000e4ccc94ef14332777b58fe778edbd6)
(cherry picked from commit cf77f1e6d96f4456a25d7375012a7181fb1bcce4)
(cherry picked from commit 0983b62ca007a6c270aff366c338a21ce6e105ce)
(cherry picked from commit 6f7905c8ecf17d5f74ac9a71a453d6768c212b6d)
(cherry picked from commit 2a958031a9dc0b489a8fde85278a1d02ffb3169c)
(cherry picked from commit 946d209f4688691e2e030229b1a428338733c086)
(cherry picked from commit 08be2b226e46d9f41e08f66e936b317bcfb4a257)
(cherry picked from commit 9a0f6315da038493dda98875b3a395e21d267693)
(cherry picked from commit 2f264eeeec3ec5a0724d4faf7553d0daad47817b)
(cherry picked from commit 2555e315f7561302484b15576d34c5da0d4cdb12)
(cherry picked from commit 987df92935ecc03e36f1754b7cbe7a81df8f000f)
(cherry picked from commit 78e7eee59ae43b0e38ff5ab4cdf7fbca33cb6789)
f92e0a4018 added minio to the tests/pgsql.ini.tmpl and Forgejo CI does
not run a minio server. It will if there are external storage related
changes at some point but it is not the case now so it is not worth
the burden.
(cherry picked from commit 4cfbf4718d1a7efc68c09e71d7a08553c9ccfe88)
(cherry picked from commit 9bd644d6011bc8fed025a64278ff7321721a6357)
(cherry picked from commit 604636c7c466c0ed7526c4c450d7a7c184f0262f)
(cherry picked from commit 341cdb8540b162601f15014cf08aa43248d8398c)
(cherry picked from commit 57bdc91de5543efd70af1d594d23de9848261a0e)
(cherry picked from commit c6ec6517b4c7f27ffbd181e61868aa012f180ba4)
(cherry picked from commit 2f03fc1b29c51ec2a46704957d1a89ebe58e4369)
(cherry picked from commit aaecd109664928eded31738acae2b9fa1202a06a)
(cherry picked from commit 7025ff7b0683ed0cbea4fb6b4ec639c15f9567d5)
(cherry picked from commit 15e6485ae1068e9261781e35ac4823eada1941e5)
(cherry picked from commit 5f7b23c65988c21a4487499416ed3221ee853bf9)
(cherry picked from commit a74d7ce6c5)
(cherry picked from commit f552e2fc56b5ea223d051d4102a5526389e88b5e)
(cherry picked from commit 9fa4a90baacd59ff0b16b35c30887a9bd1191ec4)
(cherry picked from commit a8be33da605a3e4dc18f01068e9767e45f1be339)
(cherry picked from commit 8b75a801a38a908f9398eb2c93bc90425591d99e)
(cherry picked from commit a69850b7aa76936f490a8d960767226d10629d0f)
(cherry picked from commit 94880f07e1aa2d8a082864448eace8693a57880d)
(cherry picked from commit 1d7ce2a39c841e77492ef08c0e86c3544ecca88d)
(cherry picked from commit 3af8757583ab9bf8fc5eb05ed1e6901a34682512)
(cherry picked from commit 0b818152096d58e402e3c8bbc2415695af0cdfb9)
(cherry picked from commit 7180a3222a00c33ec7dc9bcaa7e5a1e3ae924735)
(cherry picked from commit 069044f4a6dd7457662e5ca296c714275d16becf)
(cherry picked from commit dd1438f38995e397399d8957fca696f82b73707e)
(cherry picked from commit 4ae6b5d64d59902ce003199b5d652d6c4799ffcf)
(cherry picked from commit c572666d78d0f2cf2a33d5a50ea363430ac9a0a9)
(cherry picked from commit de0a4480bc3e40d61aa5342d93a2ef510abe3826)
(cherry picked from commit 60272ef4f73608407ffedd27892376b6c36a7069)
(cherry picked from commit a68fdde4becef625ce4e654ba6ea8ebee55251e5)
(cherry picked from commit ff448cf3a698dd45c713b734fce3e374c99940bf)
(cherry picked from commit 208498caaab339d4543abfb2b77de3c5793a73c8)
(cherry picked from commit 1abd3efad842f276f1c768ab40213200b62a0d3b)
(cherry picked from commit 818c7f4ca9)
(cherry picked from commit 1e552bbe44f79d9a2f5567a0180505b9a41d39d3)
(cherry picked from commit 88dbd6da1d7b4a7faf9dda2d25d68f3202c250a5)
(cherry picked from commit b1c86fb7775dd34511f4d5069db1bd3e5ef68e7a)
(cherry picked from commit 4741ea07033ca458c30b8ba9fe4b00a2e79f971a)
(cherry picked from commit 3c48dc408a7097abe309ca8c443efefca6085a6c)
(cherry picked from commit a5e5dd6567c27655cf7576109f48906e47ecf62f)
Backport #25701 by @CaiCandong
we refactored `userIDFromToken` for the token parsing part into a new
function `parseToken`. `parseToken` returns the string `token` from
request, and a boolean `ok` representing whether the token exists or
not. So we can distinguish between token non-existence and token
inconsistency in the `verfity` function, thus solving the problem of no
proper error message when the token is inconsistent.
close#24439
related #22119
Co-authored-by: caicandong <50507092+CaiCandong@users.noreply.github.com>
Co-authored-by: Jason Song <i@wolfogre.com>
Backport #25707 by @KN4CK3R
Fixes (?) #25538
Fixes https://codeberg.org/forgejo/forgejo/issues/972
Regression #23879#23879 introduced a change which prevents read access to packages if a
user is not a member of an organization.
That PR also contained a change which disallows package access if the
team unit is configured with "no access" for packages. I don't think
this change makes sense (at the moment). It may be relevant for private
orgs. But for public or limited orgs that's useless because an
unauthorized user would have more access rights than the team member.
This PR restores the old behaviour "If a user has read access for an
owner, they can read packages".
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Backport #22759 by @KN4CK3R
related #16865
This PR adds an accessibility check before mounting container blobs.
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: silverwind <me@silverwind.io>
Backport #25330
# The problem
There were many "path tricks":
* By default, Gitea uses its program directory as its work path
* Gitea tries to use the "work path" to guess its "custom path" and
"custom conf (app.ini)"
* Users might want to use other directories as work path
* The non-default work path should be passed to Gitea by GITEA_WORK_DIR
or "--work-path"
* But some Gitea processes are started without these values
* The "serv" process started by OpenSSH server
* The CLI sub-commands started by site admin
* The paths are guessed by SetCustomPathAndConf again and again
* The default values of "work path / custom path / custom conf" can be
changed when compiling
# The solution
* Use `InitWorkPathAndCommonConfig` to handle these path tricks, and use
test code to cover its behaviors.
* When Gitea's web server runs, write the WORK_PATH to "app.ini", this
value must be the most correct one, because if this value is not right,
users would find that the web UI doesn't work and then they should be
able to fix it.
* Then all other sub-commands can use the WORK_PATH in app.ini to
initialize their paths.
* By the way, when Gitea starts for git protocol, it shouldn't output
any log, otherwise the git protocol gets broken and client blocks
forever.
The "work path" priority is: WORK_PATH in app.ini > cmd arg --work-path
> env var GITEA_WORK_DIR > builtin default
The "app.ini" searching order is: cmd arg --config > cmd arg "work path
/ custom path" > env var "work path / custom path" > builtin default
## ⚠️ BREAKING
If your instance's "work path / custom path / custom conf" doesn't meet
the requirements (eg: work path must be absolute), Gitea will report a
fatal error and exit. You need to set these values according to the
error log.
Backport #25278 by @Zettat123
Fix#21072
Username Attribute is not a required item when creating an
authentication source. If Username Attribute is empty, the username
value of LDAP user cannot be read, so all users from LDAP will be marked
as inactive by mistake when synchronizing external users.
This PR improves the sync logic, if username is empty, the email address
will be used to find user.
Co-authored-by: Zettat123 <zettat123@gmail.com>
Backport #23911 by @lunny
Follow up #22405Fix#20703
This PR rewrites storage configuration read sequences with some breaks
and tests. It becomes more strict than before and also fixed some
inherit problems.
- Move storage's MinioConfig struct into setting, so after the
configuration loading, the values will be stored into the struct but not
still on some section.
- All storages configurations should be stored on one section,
configuration items cannot be overrided by multiple sections. The
prioioty of configuration is `[attachment]` > `[storage.attachments]` |
`[storage.customized]` > `[storage]` > `default`
- For extra override configuration items, currently are `SERVE_DIRECT`,
`MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another
section. The prioioty of the override configuration is `[attachment]` >
`[storage.attachments]` > `default`.
- Add more tests for storages configurations.
- Update the storage documentations.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
## Changes
- Adds the following high level access scopes, each with `read` and
`write` levels:
- `activitypub`
- `admin` (hidden if user is not a site admin)
- `misc`
- `notification`
- `organization`
- `package`
- `issue`
- `repository`
- `user`
- Adds new middleware function `tokenRequiresScopes()` in addition to
`reqToken()`
- `tokenRequiresScopes()` is used for each high-level api section
- _if_ a scoped token is present, checks that the required scope is
included based on the section and HTTP method
- `reqToken()` is used for individual routes
- checks that required authentication is present (but does not check
scope levels as this will already have been handled by
`tokenRequiresScopes()`
- Adds migration to convert old scoped access tokens to the new set of
scopes
- Updates the user interface for scope selection
### User interface example
<img width="903" alt="Screen Shot 2023-05-31 at 1 56 55 PM"
src="https://github.com/go-gitea/gitea/assets/23248839/654766ec-2143-4f59-9037-3b51600e32f3">
<img width="917" alt="Screen Shot 2023-05-31 at 1 56 43 PM"
src="https://github.com/go-gitea/gitea/assets/23248839/1ad64081-012c-4a73-b393-66b30352654c">
## tokenRequiresScopes Design Decision
- `tokenRequiresScopes()` was added to more reliably cover api routes.
For an incoming request, this function uses the given scope category
(say `AccessTokenScopeCategoryOrganization`) and the HTTP method (say
`DELETE`) and verifies that any scoped tokens in use include
`delete:organization`.
- `reqToken()` is used to enforce auth for individual routes that
require it. If a scoped token is not present for a request,
`tokenRequiresScopes()` will not return an error
## TODO
- [x] Alphabetize scope categories
- [x] Change 'public repos only' to a radio button (private vs public).
Also expand this to organizations
- [X] Disable token creation if no scopes selected. Alternatively, show
warning
- [x] `reqToken()` is missing from many `POST/DELETE` routes in the api.
`tokenRequiresScopes()` only checks that a given token has the correct
scope, `reqToken()` must be used to check that a token (or some other
auth) is present.
- _This should be addressed in this PR_
- [x] The migration should be reviewed very carefully in order to
minimize access changes to existing user tokens.
- _This should be addressed in this PR_
- [x] Link to api to swagger documentation, clarify what
read/write/delete levels correspond to
- [x] Review cases where more than one scope is needed as this directly
deviates from the api definition.
- _This should be addressed in this PR_
- For example:
```go
m.Group("/users/{username}/orgs", func() {
m.Get("", reqToken(), org.ListUserOrgs)
m.Get("/{org}/permissions", reqToken(), org.GetUserOrgsPermissions)
}, tokenRequiresScopes(auth_model.AccessTokenScopeCategoryUser,
auth_model.AccessTokenScopeCategoryOrganization),
context_service.UserAssignmentAPI())
```
## Future improvements
- [ ] Add required scopes to swagger documentation
- [ ] Redesign `reqToken()` to be opt-out rather than opt-in
- [ ] Subdivide scopes like `repository`
- [ ] Once a token is created, if it has no scopes, we should display
text instead of an empty bullet point
- [ ] If the 'public repos only' option is selected, should read
categories be selected by default
Closes#24501Closes#24799
Co-authored-by: Jonathan Tran <jon@allspice.io>
Co-authored-by: Kyle D <kdumontnu@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
The PKCE flow according to [RFC
7636](https://datatracker.ietf.org/doc/html/rfc7636) allows for secure
authorization without the requirement to provide a client secret for the
OAuth app.
It is implemented in Gitea since #5378 (v1.8.0), however without being
able to omit client secret.
Since #21316 Gitea supports setting client type at OAuth app
registration.
As public clients are already forced to use PKCE since #21316, in this
PR the client secret check is being skipped if a public client is
detected. As Gitea seems to implement PKCE authorization correctly
according to the spec, this would allow for PKCE flow without providing
a client secret.
Also add some docs for it, please check language as I'm not a native
English speaker.
Closes#17107Closes#25047
The admin config page has been broken for many many times, a little
refactoring would make this page panic.
So, add a test for it, and add another test to cover the 500 error page.
Co-authored-by: Giteabot <teabot@gitea.io>
This PR creates an API endpoint for creating/updating/deleting multiple
files in one API call similar to the solution provided by
[GitLab](https://docs.gitlab.com/ee/api/commits.html#create-a-commit-with-multiple-files-and-actions).
To archive this, the CreateOrUpdateRepoFile and DeleteRepoFIle functions
in files service are unified into one function supporting multiple files
and actions.
Resolves#14619
This adds the ability to pin important Issues and Pull Requests. You can
also move pinned Issues around to change their Position. Resolves#2175.
## Screenshots
The Design was mostly copied from the Projects Board.
## Implementation
This uses a new `pin_order` Column in the `issue` table. If the value is
set to 0, the Issue is not pinned. If it's set to a bigger value, the
value is the Position. 1 means it's the first pinned Issue, 2 means it's
the second one etc. This is dived into Issues and Pull requests for each
Repo.
## TODO
- [x] You can currently pin as many Issues as you want. Maybe we should
add a Limit, which is configurable. GitHub uses 3, but I prefer 6, as
this is better for bigger Projects, but I'm open for suggestions.
- [x] Pin and Unpin events need to be added to the Issue history.
- [x] Tests
- [x] Migration
**The feature itself is currently fully working, so tester who may find
weird edge cases are very welcome!**
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
## ⚠️ Breaking
The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.
Although many legacy options still work, it's encouraged to upgrade to
the new options.
The SMTP logger is deleted because SMTP is not suitable to collect logs.
If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.
## Description
Close#12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)
Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)
There is a new document (with examples): `logging-config.en-us.md`
This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.
## The old problems
The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.
## The new design
See `logger.go` for documents.
## Screenshot
<details>
</details>
## TODO
* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
This PR is a refactor at the beginning. And now it did 4 things.
- [x] Move renaming organizaiton and user logics into services layer and
merged as one function
- [x] Support rename a user capitalization only. For example, rename the
user from `Lunny` to `lunny`. We just need to change one table `user`
and others should not be touched.
- [x] Before this PR, some renaming were missed like `agit`
- [x] Fix bug the API reutrned from `http.StatusNoContent` to `http.StatusOK`
fix#12192 Support SSH for go get
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Giteabot <teabot@gitea.io>
Co-authored-by: mfk <mfk@hengwei.com.cn>
Co-authored-by: silverwind <me@silverwind.io>
Fixes#24145
To solve the bug, I added a "computed" `TargetBehind` field to the
`Release` model, which indicates the target branch of a release.
This is particularly useful if the target branch was deleted in the
meantime (or is empty).
I also did a micro-optimization in `calReleaseNumCommitsBehind`. Instead
of checking that a branch exists and then call `GetBranchCommit`, I
immediately call `GetBranchCommit` and handle the `git.ErrNotExist`
error.
This optimization is covered by the added unit test.
The `GetAllCommits` endpoint can be pretty slow, especially in repos
with a lot of commits. The issue is that it spends a lot of time
calculating information that may not be useful/needed by the user.
The `stat` param was previously added in #21337 to address this, by
allowing the user to disable the calculating stats for each commit. But
this has two issues:
1. The name `stat` is rather misleading, because disabling `stat`
disables the Stat **and** Files. This should be separated out into two
different params, because getting a list of affected files is much less
expensive than calculating the stats
2. There's still other costly information provided that the user may not
need, such as `Verification`
This PR, adds two parameters to the endpoint, `files` and `verification`
to allow the user to explicitly disable this information when listing
commits. The default behavior is true.
This fixes up https://github.com/go-gitea/gitea/pull/10446; in that, the
*expected* timezone was changed to the local timezone, but the computed
timezone was left in UTC.
The result was this failure, when run on a non-UTC system:
```
Diff:
--- Expected
+++ Actual
@@ -5,3 +5,3 @@
commitMsg: (string) (len=12) "init project",
- commitTime: (string) (len=29) "Wed, 14 Jun 2017 09:54:21 EDT"
+ commitTime: (string) (len=29) "Wed, 14 Jun 2017 13:54:21 UTC"
},
@@ -11,3 +11,3 @@
commitMsg: (string) (len=12) "init project",
- commitTime: (string) (len=29) "Wed, 14 Jun 2017 09:54:21 EDT"
+ commitTime: (string) (len=29) "Wed, 14 Jun 2017 13:54:21 UTC"
}
Test: TestViewRepo2
```
I assume this was probably missed since the CI servers all run in UTC?
The Format() string "Mon, 02 Jan 2006 15:04:05 UTC" was incorrect: 'UTC'
isn't recognized as a variable placeholder, but was just being copied
verbatim. It should use 'MST' in order to command Format() to output the
attached timezone, which is what `time.RFC1123` has.
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
Due to #24409 , we can now specify '--not' when getting all commits from
a repo to exclude commits from a different branch.
When I wrote that PR, I forgot to also update the code that counts the
number of commits in the repo. So now, if the --not option is used, it
may return too many commits, which can indicate that another page of
data is available when it is not.
This PR passes --not to the commands that count the number of commits in
a repo
This gives more "freshness" to the explore page. So it's not just the
same X users on the explore page by default, now it matches the same
sort as the repos on the explore page.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
