Commit graph

4052 commits

Author SHA1 Message Date
wxiaoguang
df5cf5ddbd
Avoid polluting config file when "save" (#25395)
That's a longstanding INI package problem: the "MustXxx" calls change
the option values, and the following "Save" will save a lot of garbage
options into the user's config file.

Ideally we should refactor the INI package to a clear solution, but it's
a huge work.

A clear workaround is what this PR does: when "Save", load a clear INI
instance and save it.

Partially fix #25377, the "install" page needs more fine tunes.
2023-06-21 10:31:40 +08:00
Kyle D
8220e50b56
Substitute variables in path names of template repos too (#25294)
### Summary

Extend the template variable substitution to replace file paths. This
can be helpful for setting up log files & directories that should match
the repository name.

### PR Changes

 - Move files matching glob pattern when setting up repos from template
- For security, added ~escaping~ sanitization for cross-platform support
and to prevent directory traversal (thanks @silverwind for the
reference)
 - Added unit testing for escaping function 
- Fixed the integration tests for repo template generation by passing
the repo_template_id
- Updated the integration testfiles to add some variable substitution &
assert the outputs

I had to fix the existing repo template integration test and extend it
to add a check for variable substitutions.

Example:

![image](https://github.com/go-gitea/gitea/assets/12700993/621feb09-0ef3-460e-afa8-da74cd84fa4e)
2023-06-20 21:14:47 +00:00
wxiaoguang
a1c5057fe8
Batch delete issue and improve tippy opts (#25253)
1. Add "batch delete" button for selected issues, close #22273
2. Address the review in
https://github.com/go-gitea/gitea/pull/25219#discussion_r1229266083
2023-06-19 15:46:50 +08:00
wxiaoguang
a5d0246fff
Avoid polluting the config (#25345)
Caught by #25330

Co-authored-by: Giteabot <teabot@gitea.io>
2023-06-18 16:10:44 +00:00
wxiaoguang
4e2f1ee58d
Refactor web package and context package (#25298)
1. The "web" package shouldn't depends on "modules/context" package,
instead, let each "web context" register themselves to the "web"
package.
2. The old Init/Free doesn't make sense, so simplify it
* The ctx in "Init(ctx)" is never used, and shouldn't be used that way
* The "Free" is never called and shouldn't be called because the SSPI
instance is shared

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-06-18 09:59:09 +02:00
wxiaoguang
b71cb7acdc
Use fetch to send requests to create issues/comments (#25258)
Follow #23290

Network error won't make content lost. And this is a much better
approach than "loading-button".

The UI is not perfect and there are still some TODOs, they can be done
in following PRs, not a must in this PR's scope.

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/c94ba958-aa46-4747-8ddf-6584deeed25c)

</details>
2023-06-16 06:32:43 +00:00
Lunny Xiao
cc3910d8c3
Fix index generation parallelly failure (#25235)
Fix #22109

---------

Co-authored-by: silverwind <me@silverwind.io>
2023-06-15 02:14:43 +02:00
HesterG
a43ea22479
Change form actions to fetch for submit review box (#25219)
Co-author: @wxiaoguang 

Close #25096 

The way to fix it in this PR is to change form submit to fetch using
formData, and add flags to avoid post repeatedly.
Should be able to apply to more forms that have the same issue after
this PR.

In the demo below, 'approve' is clicked several times, and then
'comment' is clicked several time after 'request changes' clicked.

After:


https://github.com/go-gitea/gitea/assets/17645053/beabeb1d-fe66-4b76-b048-4f022b4e83a0


Update: screenshots from /devtest

>
![image](https://user-images.githubusercontent.com/2114189/245680011-ee4231e0-a53d-4c2a-a9c2-71ccd98005cc.png)
> 
>
![image](https://user-images.githubusercontent.com/2114189/245680057-9215d348-63d8-406d-8828-17e171163aaa.png)
> 
>
![image](https://user-images.githubusercontent.com/2114189/245680148-89d7b3d1-d7b6-442f-b69e-eadaee112482.png)

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-06-14 16:01:37 +08:00
Lunny Xiao
d6dd6d641b
Fix all possible setting error related storages and added some tests (#23911)
Follow up #22405

Fix #20703 

This PR rewrites storage configuration read sequences with some breaks
and tests. It becomes more strict than before and also fixed some
inherit problems.

- Move storage's MinioConfig struct into setting, so after the
configuration loading, the values will be stored into the struct but not
still on some section.
- All storages configurations should be stored on one section,
configuration items cannot be overrided by multiple sections. The
prioioty of configuration is `[attachment]` > `[storage.attachments]` |
`[storage.customized]` > `[storage]` > `default`
- For extra override configuration items, currently are `SERVE_DIRECT`,
`MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another
section. The prioioty of the override configuration is `[attachment]` >
`[storage.attachments]` > `default`.
- Add more tests for storages configurations.
- Update the storage documentations.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-06-14 11:42:38 +08:00
silverwind
a51b115b0a
Use inline SVG for built-in OAuth providers (#25171)
The plan is that all built-in auth providers use inline SVG for more
flexibility in styling and to get the GitHub icon to follow
`currentcolor`. This only removes the `public/img/auth` directory and
adds the missing svgs to our svg build.

It should map the built-in providers to these SVGs and render them. If
the user has set a Icon URL, it should render that as an `img` tag
instead.

```
gitea-azure-ad
gitea-bitbucket
gitea-discord
gitea-dropbox
gitea-facebook
gitea-gitea
gitea-gitlab
gitea-google
gitea-mastodon
gitea-microsoftonline
gitea-nextcloud
gitea-twitter
gitea-yandex
octicon-mark-github
```

GitHub logo is now white again on dark theme:

<img width="431" alt="Screenshot 2023-06-12 at 21 45 34"
src="https://github.com/go-gitea/gitea/assets/115237/27a43504-d60a-4132-a502-336b25883e4d">

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-06-13 10:51:02 +00:00
yp05327
22a39bb961
Fix profile render when the README.md size is larger than 1024 bytes (#25131)
Fixes https://github.com/go-gitea/gitea/issues/25094

`GetBlobContent` will only get the first 1024 bytes, if the README.md
size is larger than 1024 bytes,
We can not render the rest of them.
After this fix, we should provide the limited size to read when call
`GetBlobContent`.

After:

![image](https://github.com/go-gitea/gitea/assets/18380374/22a42936-4cf8-40b4-a5c7-e384082beb0d)

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-06-13 09:02:25 +00:00
Jonathan Tran
f62cd2f473
Fix task list checkbox toggle to work with YAML front matter (#25184)
Fixes #25160.

`data-source-position` of checkboxes in a task list was incorrect
whenever there was YAML front matter. This would result in issue content
or PR descriptions getting corrupted with random `x` or space characters
when a user checked or unchecked a task.
2023-06-13 14:44:47 +08:00
Lunny Xiao
419804fd4d
Fix compatible for webhook ref type (#25195)
Fix #25185 
Caused by #24634
2023-06-13 06:05:28 +00:00
Lauris BH
9aaaf980f0
Fix open redirect check for more cases (#25143)
If redirect_to parameter has set value starting with `\\example.com`
redirect will be created with header `Location: /\\example.com` that
will redirect to example.com domain.
2023-06-08 14:08:14 +00:00
Denys Konovalov
eac1bddb8d
fix swagger documentation for multiple files API endpoint (#25110)
Fixes some issues with the swagger documentation for the new multiple
files API endpoint (#24887) which were overlooked when submitting the
original PR:

1. add some missing parameter descriptions
2. set correct `required` option for required parameters
3. change endpoint description to match it full functionality (every
kind of file modification is supported, not just creating and updating)
2023-06-07 23:49:58 +08:00
Wim
62ac3251fa
Remove stars when repo goes private (#19904)
Fixes #18600
2023-06-05 13:25:43 +00:00
Lunny Xiao
3d1fda737b
Use git command instead of the ini package to remove the origin remote (#25066) 2023-06-05 18:05:31 +08:00
Denys Konovalov
ca35dec18b
Add ability to set multiple redirect URIs in OAuth application UI (#25072)
OAuth applications can already have multiple redirect URIs if
created/edited over API.

This change allows for setting multiple redirect URIs through the UI as
a comma-separated list (e. g.
`https://example.org/redirect,https://redirect.example.org`)

<details>
<summary>Screenshots</summary>

![Bildschirmfoto vom 2023-06-04
17-14-40](https://github.com/go-gitea/gitea/assets/47871822/2206dc32-e7e4-4953-9ecb-e098890b3f54)
![Bildschirmfoto vom 2023-06-04
17-14-50](https://github.com/go-gitea/gitea/assets/47871822/cd97c73c-9310-44ee-a83a-b927a1ef94da)

</details>

Closes #25068
2023-06-05 15:00:12 +08:00
Jack Hay
18de83b2a3
Redesign Scoped Access Tokens (#24767)
## Changes
- Adds the following high level access scopes, each with `read` and
`write` levels:
    - `activitypub`
    - `admin` (hidden if user is not a site admin)
    - `misc`
    - `notification`
    - `organization`
    - `package`
    - `issue`
    - `repository`
    - `user`
- Adds new middleware function `tokenRequiresScopes()` in addition to
`reqToken()`
  -  `tokenRequiresScopes()` is used for each high-level api section
- _if_ a scoped token is present, checks that the required scope is
included based on the section and HTTP method
  - `reqToken()` is used for individual routes
- checks that required authentication is present (but does not check
scope levels as this will already have been handled by
`tokenRequiresScopes()`
- Adds migration to convert old scoped access tokens to the new set of
scopes
- Updates the user interface for scope selection

### User interface example
<img width="903" alt="Screen Shot 2023-05-31 at 1 56 55 PM"
src="https://github.com/go-gitea/gitea/assets/23248839/654766ec-2143-4f59-9037-3b51600e32f3">
<img width="917" alt="Screen Shot 2023-05-31 at 1 56 43 PM"
src="https://github.com/go-gitea/gitea/assets/23248839/1ad64081-012c-4a73-b393-66b30352654c">

## tokenRequiresScopes  Design Decision
- `tokenRequiresScopes()` was added to more reliably cover api routes.
For an incoming request, this function uses the given scope category
(say `AccessTokenScopeCategoryOrganization`) and the HTTP method (say
`DELETE`) and verifies that any scoped tokens in use include
`delete:organization`.
- `reqToken()` is used to enforce auth for individual routes that
require it. If a scoped token is not present for a request,
`tokenRequiresScopes()` will not return an error

## TODO
- [x] Alphabetize scope categories
- [x] Change 'public repos only' to a radio button (private vs public).
Also expand this to organizations
- [X] Disable token creation if no scopes selected. Alternatively, show
warning
- [x] `reqToken()` is missing from many `POST/DELETE` routes in the api.
`tokenRequiresScopes()` only checks that a given token has the correct
scope, `reqToken()` must be used to check that a token (or some other
auth) is present.
   -  _This should be addressed in this PR_
- [x] The migration should be reviewed very carefully in order to
minimize access changes to existing user tokens.
   - _This should be addressed in this PR_
- [x] Link to api to swagger documentation, clarify what
read/write/delete levels correspond to
- [x] Review cases where more than one scope is needed as this directly
deviates from the api definition.
   - _This should be addressed in this PR_
   - For example: 
   ```go
	m.Group("/users/{username}/orgs", func() {
		m.Get("", reqToken(), org.ListUserOrgs)
		m.Get("/{org}/permissions", reqToken(), org.GetUserOrgsPermissions)
}, tokenRequiresScopes(auth_model.AccessTokenScopeCategoryUser,
auth_model.AccessTokenScopeCategoryOrganization),
context_service.UserAssignmentAPI())
   ```

## Future improvements
- [ ] Add required scopes to swagger documentation
- [ ] Redesign `reqToken()` to be opt-out rather than opt-in
- [ ] Subdivide scopes like `repository`
- [ ] Once a token is created, if it has no scopes, we should display
text instead of an empty bullet point
- [ ] If the 'public repos only' option is selected, should read
categories be selected by default

Closes #24501
Closes #24799

Co-authored-by: Jonathan Tran <jon@allspice.io>
Co-authored-by: Kyle D <kdumontnu@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
2023-06-04 20:57:16 +02:00
wxiaoguang
520eb57d76
Use a separate admin page to show global stats, remove actions stat (#25062)
Before, Gitea shows the database table stats on the `admin dashboard`
page.

It has some problems:

* `count(*)` is quite heavy. If tables have many records, this blocks
loading the admin page blocks for a long time
* Some users had even reported issues that they can't visit their admin
page because this page causes blocking or `50x error (reverse proxy
timeout)`
* The `actions` stat is not useful. The table is simply too large. Does
it really matter if it contains 1,000,000 rows or 9,999,999 rows?
* The translation `admin.dashboard.statistic_info` is difficult to
maintain.

So, this PR uses a separate page to show the stats and removes the
`actions` stat.


![image](https://github.com/go-gitea/gitea/assets/2114189/babf7c61-b93b-4a62-bfaa-22983636427e)

## ⚠️ BREAKING

The `actions` Prometheus metrics collector has been removed for the
reasons mentioned beforehand.
Please do not rely on its output anymore.
2023-06-03 22:03:41 +08:00
wxiaoguang
de4a21fcb4
Refactor INI package (first step) (#25024)
The INI package has many bugs and quirks, and in fact it is
unmaintained.

This PR is the first step for the INI package refactoring: 

* Use Gitea's "config_provider" to provide INI access
* Deprecate the INI package by golangci.yml rule
2023-06-02 17:27:30 +08:00
谈笑风生间
a5acec329e
Rename NotifyPullReviewRequest to NotifyPullRequestReviewRequest (#24988)
after #24481

To keep consistent, maybe pull_request_review_request is better.

## what my pull request does

Rename NotifyPullReviewRequest to NotifyPullRequestReviewRequest.

<img width="1087" alt="image"
src="https://github.com/go-gitea/gitea/assets/75628309/2cf91f02-74f2-42fe-a998-0a039b821d8d">
2023-05-31 15:54:46 +08:00
silverwind
50bd7d0b24
Remove the service worker (#25010)
It's been disabled by default since 1.17
(https://github.com/go-gitea/gitea/pull/18914), and it never really
delivered any benefit except being another cache layer that has its own
unsolved invalidation issues. HTTP cache works, we don't need two cache
layers at the browser for assets.

## ⚠️ BREAKING

You can remove the config `[ui].USE_SERVICE_WORKER` from your `app.ini`
now.
2023-05-31 02:07:04 +00:00
Lunny Xiao
faae819f5d
revert the removed method to fix tmpl break on graph page (#25005)
Fix #24996 
Caused by #24634
2023-05-30 21:36:58 +08:00
wxiaoguang
ca5f302876
Fix admin config page error, use tests to cover the admin config and 500 error page (#24965)
The admin config page has been broken for many many times, a little
refactoring would make this page panic.

So, add a test for it, and add another test to cover the 500 error page.

Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-29 15:00:21 +00:00
Denys Konovalov
275d4b7e3f
API endpoint for changing/creating/deleting multiple files (#24887)
This PR creates an API endpoint for creating/updating/deleting multiple
files in one API call similar to the solution provided by
[GitLab](https://docs.gitlab.com/ee/api/commits.html#create-a-commit-with-multiple-files-and-actions).

To archive this, the CreateOrUpdateRepoFile and DeleteRepoFIle functions
in files service are unified into one function supporting multiple files
and actions.

Resolves #14619
2023-05-29 17:41:35 +08:00
wxiaoguang
84c8ab9fd1
Help to recover from corrupted levelqueue (#24912)
gitea.com experienced the corrupted LevelQueue bug again.

I think the problem is clear now: if the keys in LevelDB went
out-of-sync, the LevelQueue itself doesn't have the ability to recover,
eg:

* LevelQueue.Len() reports 100
* LevelQueue.LPop() reports ErrNotFound = errors.New("no key found")

So it needs to dive into the LevelDB to remove all keys to recover the
corrupted LevelQueue.

More comments are in TestCorruptedLevelQueue.
2023-05-29 10:52:32 +08:00
silverwind
c7612d178c
Remove meta tags theme-color and default-theme (#24960)
As discussed in
https://github.com/go-gitea/gitea/pull/24953#issuecomment-1565630156.

## ⚠️ BREAKING ⚠️

1. The `ui.THEME_COLOR_META_TAG` setting has been removed. If you still
need to set the `theme-color` meta tag, add it via
`$GITEA_CUSTOM/templates/custom/header.tmpl` instead.

2. The non-standard `default-theme` meta-tag added in
https://github.com/go-gitea/gitea/pull/13809 has been removed. Third
party code that needs to obtain the currently loaded theme should use
the `theme-<name>` class on the `<html>` node instead, which reflect the
currently active theme.
2023-05-28 22:33:17 +00:00
wxiaoguang
2f149c5c9d
Use [git.config] for reflog cleaning up (#24958)
Follow
https://github.com/go-gitea/gitea/pull/24860#discussion_r1200589651

Use `[git.config]` for reflog cleaning up, the new options are more
flexible.

*
https://git-scm.com/docs/git-config#Documentation/git-config.txt-corelogAllRefUpdates
*
https://git-scm.com/docs/git-config#Documentation/git-config.txt-gcreflogExpire

## ⚠️ BREAKING

The section `[git.reflog]` is now obsolete and its keys have been moved
to the following replacements:
- `[git.reflog].ENABLED` → `[git.config].core.logAllRefUpdates`
- `[git.reflog].EXPIRATION` → `[git.config].gc.reflogExpire`
2023-05-28 01:07:14 +00:00
wxiaoguang
0d54395fb5
Improve logger Pause handling (#24946)
The old EventWriter's Run does: 

```go
for {
    handlePause()
    select {
    case event <- Queue:
         write the log event ...
    }
}
```

So, if an event writer is started before the logger is paused, there is
a chance that the logger isn't paused for the first message.

The new logic is:

```go
for {
    select {
    case event <- Queue:
         handlePause()
         write the log event ...
    }
}
```

Then the event writer can be correctly paused
2023-05-27 22:35:44 +02:00
wxiaoguang
7314726bab
Do not output "Trace" level logs from process manager by default (#24952)
The old process manager's `Trace` function by default calls `log.Printf`
to output "trace" level logs. That's not ideal because by default the
trace level logs should not be outputted. In history it didn't cause
problems because there was no other call to the process manager before
the logger system's initialization.

But if there is any package using the process manager before the "Trace"
function gets assigned to the logger system's trace function, the
process manager will outputs unexpected verbose messages, this behavior
is not expected in most cases.

Now, the logger system also uses process manager to manage its goroutine
contexts, so it's the time to fix the old "trace" behavior: by default,
do not output the trace level messages. Fix #24951
2023-05-27 10:55:24 +00:00
wxiaoguang
18f26cfbf7
Improve queue and logger context (#24924)
Before there was a "graceful function": RunWithShutdownFns, it's mainly
for some modules which doesn't support context.

The old queue system doesn't work well with context, so the old queues
need it.

After the queue refactoring, the new queue works with context well, so,
use Golang context as much as possible, the `RunWithShutdownFns` could
be removed (replaced by RunWithCancel for context cancel mechanism), the
related code could be simplified.

This PR also fixes some legacy queue-init problems, eg:

* typo : archiver: "unable to create codes indexer queue" => "unable to
create repo-archive queue"
* no nil check for failed queues, which causes unfriendly panic

After this PR, many goroutines could have better display name:

![image](https://github.com/go-gitea/gitea/assets/2114189/701b2a9b-8065-4137-aeaa-0bda2b34604a)

![image](https://github.com/go-gitea/gitea/assets/2114189/f1d5f50f-0534-40f0-b0be-f2c9daa5fe92)
2023-05-26 07:31:55 +00:00
Lunny Xiao
f9cfd6ce5b
Use the type RefName for all the needed places and fix pull mirror sync bugs (#24634)
This PR replaces all string refName as a type `git.RefName` to make the
code more maintainable.

Fix #15367
Replaces #23070 
It also fixed a bug that tags are not sync because `git remote --prune
origin` will not remove local tags if remote removed.

We in fact should use `git fetch --prune --tags origin` but not `git
remote update origin` to do the sync.

Some answer from ChatGPT as ref.

> If the git fetch --prune --tags command is not working as expected,
there could be a few reasons why. Here are a few things to check:
> 
>Make sure that you have the latest version of Git installed on your
system. You can check the version by running git --version in your
terminal. If you have an outdated version, try updating Git and see if
that resolves the issue.
> 
>Check that your Git repository is properly configured to track the
remote repository's tags. You can check this by running git config
--get-all remote.origin.fetch and verifying that it includes
+refs/tags/*:refs/tags/*. If it does not, you can add it by running git
config --add remote.origin.fetch "+refs/tags/*:refs/tags/*".
> 
>Verify that the tags you are trying to prune actually exist on the
remote repository. You can do this by running git ls-remote --tags
origin to list all the tags on the remote repository.
> 
>Check if any local tags have been created that match the names of tags
on the remote repository. If so, these local tags may be preventing the
git fetch --prune --tags command from working properly. You can delete
local tags using the git tag -d command.

---------

Co-authored-by: delvh <dev.lh@web.de>
2023-05-26 01:04:48 +00:00
JakobDev
aaa1094663
Add the ability to pin Issues (#24406)
This adds the ability to pin important Issues and Pull Requests. You can
also move pinned Issues around to change their Position. Resolves #2175.

## Screenshots

![grafik](https://user-images.githubusercontent.com/15185051/235123207-0aa39869-bb48-45c3-abe2-ba1e836046ec.png)

![grafik](https://user-images.githubusercontent.com/15185051/235123297-152a16ea-a857-451d-9a42-61f2cd54dd75.png)

![grafik](https://user-images.githubusercontent.com/15185051/235640782-cbfe25ec-6254-479a-a3de-133e585d7a2d.png)

The Design was mostly copied from the Projects Board.

## Implementation
This uses a new `pin_order` Column in the `issue` table. If the value is
set to 0, the Issue is not pinned. If it's set to a bigger value, the
value is the Position. 1 means it's the first pinned Issue, 2 means it's
the second one etc. This is dived into Issues and Pull requests for each
Repo.

## TODO
- [x] You can currently pin as many Issues as you want. Maybe we should
add a Limit, which is configurable. GitHub uses 3, but I prefer 6, as
this is better for bigger Projects, but I'm open for suggestions.
- [x] Pin and Unpin events need to be added to the Issue history.
- [x] Tests
- [x] Migration

**The feature itself is currently fully working, so tester who may find
weird edge cases are very welcome!**

---------

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-25 15:17:19 +02:00
techknowlogick
033d92997f
Allow skipping forks and mirrors from being indexed (#23187)
This PR adds two new options to disable repo/code search indexing of
both forks and mirrors.

Related: #22842
2023-05-25 16:13:47 +08:00
wxiaoguang
5f39285d6d
Improve RunMode / dev mode (#24886)
1. non-dev mode is treated as prod mode, to protect users from
accidentally running in dev mode if there is a typo in this value.
2. in dev mode, do not need to really exit if there are template errors,
because the template errors could be fixed by developer soon and the
templates get reloaded, help:
* https://github.com/go-gitea/gitea/issues/24845#issuecomment-1557615382
3. Fine tune the mail template loading message.
2023-05-25 03:47:30 +00:00
谈笑风生间
309354c70e
New webhook trigger for receiving Pull Request review requests (#24481)
close https://github.com/go-gitea/gitea/issues/16321

Provided a webhook trigger for requesting someone to review the Pull
Request.

Some modifications have been made to the returned `PullRequestPayload`
based on the GitHub webhook settings, including:
- add a description of the current reviewer object as
`RequestedReviewer` .
- setting the action to either **review_requested** or
**review_request_removed** based on the operation.
- adding the `RequestedReviewers` field to the issues_model.PullRequest.
This field will be loaded into the PullRequest through
`LoadRequestedReviewers()` when `ToAPIPullRequest` is called.

After the Pull Request is merged, I will supplement the relevant
documentation.
2023-05-24 22:06:27 -04:00
wxiaoguang
395bb33e4c
Merge different languages for language stats (#24900)
Fix #24896

If users set different languages by `linguist-language`, the `stats` map
could be: `java: 100, Java: 200`.

Language stats are stored as case-insensitive in database and there is a
unique key.

So, the different language names should be merged to one unique name:
`Java: 300`
2023-05-24 19:37:36 +00:00
wxiaoguang
c21605951b
Make environment-to-ini support loading key value from file (#24832)
Replace #19857

Close #19856
Close #10311
Close #10123

Major changes:

1. Move a lot of code from `environment-to-ini.go` to `config_env.go` to
make them testable.
2. Add `__FILE` support
3. Update documents
4. Add tests
2023-05-24 11:37:22 +08:00
wxiaoguang
8080ace6fc
Support changing git config through app.ini, use diff.algorithm=histogram by default (#24860)
Close #13454 , Close #23255, Close #14697 (and maybe more related
issues)

Many users have the requirement to customize the git config. This PR
introduces an easy way: put the options in Gitea's app.ini
`[git.config]`, then the config options will be applied to git config.

And it can support more flexible default config values, eg: now
`diff.algorithm=histogram` by default. According to:
https://stackoverflow.com/a/32367597/4754037 , `histogram diff` is
efficient and doesn't like to cause server-side problems.

---------

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-23 16:30:19 +00:00
wxiaoguang
910bf31546
Fix flakey test in logger test (#24883)
Fix #24882

The goroutines are all asynchronized. So it needs a little "sleep" to
make sure the writer's goroutine has been paused before sending messages
to it.

Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-23 16:01:20 +00:00
wxiaoguang
16a766cba1
Do not call nil handler for a dummy queue (#24880)
A dummy queue doesn't really have a handler (see line 211), so the
`safeHandler` can safely drop all items
2023-05-23 13:40:55 +02:00
JakobDev
25dc1556cd
Add API for Label templates (#24602)
This adds API that allows getting the Label templates of the Gitea
Instance
2023-05-23 18:10:23 +08:00
wxiaoguang
abcf5a7b5e
Fix install page context, make the install page tests really test (#24858)
Fix #24856

Rename "context.contextKey" to "context.WebContextKey", this context is
for web context only. But the Context itself is not renamed, otherwise
it would cause a lot of changes (if we really want to rename it, there
could be a separate PR).

The old test code doesn't really test, the "install page" gets broken
not only one time, so use new test code to make sure the "install page"
could work.
2023-05-23 09:29:15 +08:00
Brecht Van Lommel
3588edbb08
Add gitea manager reload-templates command (#24843)
This can be useful to update custom templates in production mode, when
they are updated frequently and a full Gitea restart each time is
disruptive.
2023-05-22 23:51:40 +08:00
KN4CK3R
cdb088cec2
Add CRAN package registry (#22331)
This PR adds a [CRAN](https://cran.r-project.org/) package registry.

![grafik](https://user-images.githubusercontent.com/1666336/210450039-d6fa6f77-20cd-4741-89a8-1624def267f7.png)
2023-05-22 10:57:49 +08:00
wxiaoguang
ec2a01d1e2
Fix regression: access log template, gitea manager cli command (#24838)
Close #24836

![image](https://github.com/go-gitea/gitea/assets/2114189/95b025d2-f25f-4246-a08a-fe44ecb787a9)

![image](https://github.com/go-gitea/gitea/assets/2114189/c3afe1fa-2a23-420d-a016-3b67dcd04cd5)
2023-05-22 09:38:38 +08:00
wxiaoguang
2cb66fff60
Support wildcard in email domain allow/block list (#24831)
Replace #20257 (which is stale and incomplete)

Close #20255

Major changes:

* Deprecate the "WHITELIST", use "ALLOWLIST"
* Add wildcard support for EMAIL_DOMAIN_ALLOWLIST/EMAIL_DOMAIN_BLOCKLIST
* Update example config file and document
* Improve tests
2023-05-22 00:05:44 +00:00
wxiaoguang
4647660776
Rewrite logger system (#24726)
## ⚠️ Breaking

The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.

Although many legacy options still work, it's encouraged to upgrade to
the new options.

The SMTP logger is deleted because SMTP is not suitable to collect logs.

If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.

## Description

Close #12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)

Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)

There is a new document (with examples): `logging-config.en-us.md`

This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.

## The old problems

The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.


## The new design

See `logger.go` for documents.


## Screenshot

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff)


![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9)


![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee)

</details>

## TODO

* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)

---------

Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-21 22:35:11 +00:00
Brecht Van Lommel
268d121f4b
Fix video width overflow in markdown, and other changes to match img (#24834)
This change makes the CSS for `<video>` in markup match that of `<img>`,
and also allows additional attributes to be used. This way the width,
padding, alignment should work equally well for both.
2023-05-21 21:19:37 +00:00
wxiaoguang
6b33152b7d
Decouple the different contexts from each other (#24786)
Replace #16455

Close #21803

Mixing different Gitea contexts together causes some problems:

1. Unable to respond proper content when error occurs, eg: Web should
respond HTML while API should respond JSON
2. Unclear dependency, eg: it's unclear when Context is used in
APIContext, which fields should be initialized, which methods are
necessary.


To make things clear, this PR introduces a Base context, it only
provides basic Req/Resp/Data features.

This PR mainly moves code. There are still many legacy problems and
TODOs in code, leave unrelated changes to future PRs.
2023-05-21 09:50:53 +08:00
silverwind
32d9c47ec7
Add RTL rendering support to Markdown (#24816)
Support RTL content in Markdown:


![image](https://github.com/go-gitea/gitea/assets/115237/dedb1b0c-2f05-40dc-931a-0d9dc81f7c97)

Example document:
https://try.gitea.io/silverwind/symlink-test/src/branch/master/bidi-text.md
Same on GitHub:
https://github.com/silverwind/symlink-test/blob/master/bidi-text.md

`dir=auto` enables a browser heuristic that sets the text direction
automatically. It is the only way to get automatic text direction.

Ref: https://codeberg.org/Codeberg/Community/issues/1021

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-05-20 23:02:52 +02:00
Yarden Shoham
f5ce2ed292
Allow all URL schemes in Markdown links by default (#24805)
- Closes #21146
- Closes #16721

## ⚠️ BREAKING ⚠️
This changes the default behavior to now create links for any URL scheme
when the user uses the markdown form for links (`[label](URL)`), this
doesn't affect the rendering of inline links. To opt-out set the
`markdown.CUSTOM_URL_SCHEMES` setting to a list of allowed schemes, all
other schemes (except `http` and `https`) won't be allowed.

# Before

![image](https://github.com/go-gitea/gitea/assets/20454870/35fa18ce-7dda-4995-b5b3-3f360f38296d)

# After

![image](https://github.com/go-gitea/gitea/assets/20454870/0922216b-0b35-4b77-9919-21a5c21dd5d0)

---------

Signed-off-by: Yarden Shoham <git@yardenshoham.com>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-19 17:17:07 +02:00
Lunny Xiao
38cf43d060
Some refactors for issues stats (#24793)
This PR

- [x] Move some functions from `issues.go` to `issue_stats.go` and
`issue_label.go`
- [x] Remove duplicated issue options `UserIssueStatsOption` to keep
only one `IssuesOptions`
2023-05-19 22:17:48 +08:00
FuXiaoHei
c757765a9e
Implement actions artifacts (#22738)
Implement action artifacts server api.

This change is used for supporting
https://github.com/actions/upload-artifact and
https://github.com/actions/download-artifact in gitea actions. It can
run sample workflow from doc
https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts.
The api design is inspired by
https://github.com/nektos/act/blob/master/pkg/artifacts/server.go and
includes some changes from gitea internal structs and methods.

Actions artifacts contains two parts:

- Gitea server api and storage (this pr implement basic design without
some complex cases supports)
- Runner communicate with gitea server api (in comming)

Old pr https://github.com/go-gitea/gitea/pull/22345 is outdated after
actions merged. I create new pr from main branch.


![897f7694-3e0f-4f7c-bb4b-9936624ead45](https://user-images.githubusercontent.com/2142787/219382371-eb3cf810-e4e0-456b-a8ff-aecc2b1a1032.jpeg)

Add artifacts list in actions workflow page.
2023-05-19 21:37:57 +08:00
Jason Song
7985cde84d
Fix Actions being enabled accidentally (#24802)
Regression of #24536. If the user doesn't explicitly disable Actions, it
will be enabled.

1. Gitea will call `loadRepositoryFrom` before `loadActionsFrom`.

25d4f95df2/modules/setting/setting.go (L234-L237)
2. In `loadRepositoryFrom`,
`rootCfg.Section("actions").Key("ENABLED").MustBool(true)` will set
`actions.ENABLED` with `true`.

25d4f95df2/modules/setting/repository.go (L313-L315)
3. In `loadActionsFrom`, `rootCfg.Section("actions")` will get a section
with Actions enabled.

25d4f95df2/modules/setting/actions.go (L23-L26)

Although the cause of the problem was using `true` by copy-paste
mistake, it also surprised me that
**`rootCfg.Section("actions").Key("ENABLED").MustBool(true)` doesn't
only read, but also write.**
2023-05-19 19:35:12 +08:00
silverwind
655c890cbd
Fix TestMinioStorageIterator skip message (#24765)
Followup to https://github.com/go-gitea/gitea/pull/24762, fix this
message.
2023-05-17 16:22:17 +00:00
Lunny Xiao
6c8235dc98
Skip TestMinioStorageIterator on CI (#24762)
Fix https://github.com/go-gitea/gitea/pull/24691#issuecomment-1550987681

---------

Co-authored-by: silverwind <me@silverwind.io>
2023-05-17 10:00:58 +00:00
wxiaoguang
584c0789fa
Make mailer SMTP check have timed context (#24751)
Make mailer SMTP check have timed context

Otherwise Gitea may block for long time if the DNS request blocks.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-16 22:55:51 +02:00
zeripath
7565e5c3de
Implement systemd-notify protocol (#21151)
This PR adds support for the systemd notify protocol. Several status
messagess are provided. We should likely add a common notify/status
message for graceful.

Replaces #21140

Signed-off-by: Andrew Thornton <art27@cantab.net>

---------

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: ltdk <usr@ltdk.xyz>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-15 22:20:30 +00:00
wxiaoguang
99283415bc
Refactor Pull Mirror and fix out-of-sync bugs (#24732)
The "mirror" table and "repository" table might be out-of-sync in some
cases.

It means that "IsMirror=true" but "Mirror=nil"

This PR removes unnecessary "Mirror" field, rename "Mirror" to
"PullMirror" and fix nil panic bug.


Screenshot of changed templates:


![image](https://github.com/go-gitea/gitea/assets/2114189/c0f2bdfc-5911-43ea-b989-b19619de4235)



![image](https://github.com/go-gitea/gitea/assets/2114189/1078b41a-484f-4c06-8c2f-edb9e658275d)
2023-05-15 19:02:10 +00:00
KN4CK3R
5968c63a11
Add Go package registry (#24687)
Fixes #7608

This PR adds a Go package registry usable with the Go proxy protocol.

![grafik](https://github.com/go-gitea/gitea/assets/1666336/328feb5c-3df2-4f9d-8eae-fe3126d14c37)
2023-05-14 23:38:40 +08:00
FuXiaoHei
61ad4c607b
fix minio storage iterator path (#24691)
minio storage iterator shows different behavior with local fs iterator.

in local fs storage:

``` go
s.IterateObjects("prefix", func(path,obj)
     println(path) // show "prefix/xxx.file"
})
```

in minio storage:

```go
s.IterateObjects("prefix", func(path,obj)
     println(path) // show "xxx.file"
})
```

I think local fs is correct, minio use wrong `basePath` to trim storage
path prefix.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-13 22:33:25 +00:00
Yarden Shoham
4810fe55e3
Add status indicator on main home screen for each repo (#24638)
It will show the calculated commit status state of the latest commit on
the default branch for each repository in the dashboard repo list

- Closes #15620

# Before

![image](https://github.com/go-gitea/gitea/assets/20454870/aa1326c7-43c0-458a-a798-3102c766bcf9)

# After

![image](https://github.com/go-gitea/gitea/assets/20454870/8658cc03-2224-442a-b1c8-bf64126e4575)

---------

Signed-off-by: Yarden Shoham <git@yardenshoham.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-13 21:59:01 +00:00
wxiaoguang
82224c54e0
Improve avatar uploading / resizing / compressing, remove Fomantic card module (#24653)
Fixes: #8972
Fixes: #24263

And I think it also (partially) fix #24263 (no need to convert) ,
because users could upload any supported image format if it isn't larger
than AVATAR_MAX_ORIGIN_SIZE


The main idea: 

* if the uploaded file size is not larger than AVATAR_MAX_ORIGIN_SIZE,
use the origin
* if the resized size is larger than the origin, use the origin

Screenshots:

JPG:

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/70e98bb0-ecb9-4c4e-a89f-4a37d4e37f8e)

</details>

APNG:

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/9055135b-5e2d-4152-bd72-596fcb7c6671)


![image](https://github.com/go-gitea/gitea/assets/2114189/50364caf-f7f6-4241-a289-e485fe4cd582)

</details>

WebP (animated)

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/f642eb85-498a-49a5-86bf-0a7b04089ae0)

</details>

The only exception: if a WebP image is larger than MaxOriginSize and it
is animated, then current `webp` package can't decode it, so only in
this case it isn't supported. IMO no need to support such case: why a
user would upload a 1MB animated webp as avatar? crazy .....

---------

Co-authored-by: silverwind <me@silverwind.io>
2023-05-13 20:59:11 +02:00
wxiaoguang
a94a8d0ab1
Use standard HTTP library to serve files (#24693)
`http.ServeFile/ServeContent` handles `If-xxx`, `Content-Length`,
`Range` and `Etag` correctly

After this PR, storage files (eg: avatar) could be responded with
correct Content-Length.
2023-05-13 16:04:57 +02:00
KN4CK3R
9173e079ae
Add Alpine package registry (#23714)
This PR adds an Alpine package registry. You can follow [this
tutorial](https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package)
to build a *.apk package for testing.

This functionality is similar to the Debian registry (#22854) and
therefore shares some methods. I marked this PR as blocked because it
should be merged after #22854.


![grafik](https://user-images.githubusercontent.com/1666336/227779595-b76163aa-eea1-4a79-9583-775c24ad74e8.png)

---------

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-12 17:27:50 +00:00
wxiaoguang
bfa0fc2c98
Make testlogger output "TestLogger" prefix (#24683)
Make testlogger output "TestLogger" prefix instead of "Unknown Test"
when there is no running test case.
2023-05-12 14:20:29 +00:00
rune
4b80813341
Support SSH for go get (#24664)
fix #12192 Support SSH for go get

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Giteabot <teabot@gitea.io>
Co-authored-by: mfk <mfk@hengwei.com.cn>
Co-authored-by: silverwind <me@silverwind.io>
2023-05-12 09:44:37 +00:00
wxiaoguang
f6e029e6c7
Make repo migration cancelable and fix various bugs (#24605)
Replace #12917

Close #24601
Close #12845


![image](https://github.com/go-gitea/gitea/assets/2114189/39378118-064d-40fb-8396-4579ebf33917)


![image](https://github.com/go-gitea/gitea/assets/2114189/faf37418-191c-46a6-90a8-353141e00e2d)


![image](https://github.com/go-gitea/gitea/assets/2114189/fdc8ee4d-125f-4737-9990-89bcdf9eb388)


![image](https://github.com/go-gitea/gitea/assets/2114189/9a3bd2c2-fe20-4011-81f0-990ed869d139)

---------

Co-authored-by: Yarden Shoham <git@yardenshoham.com>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-11 08:25:46 +00:00
wxiaoguang
58dfaf3a75
Improve queue & process & stacktrace (#24636)
Although some features are mixed together in this PR, this PR is not
that large, and these features are all related.

Actually there are more than 70 lines are for a toy "test queue", so
this PR is quite simple.

Major features:

1. Allow site admin to clear a queue (remove all items in a queue)
* Because there is no transaction, the "unique queue" could be corrupted
in rare cases, that's unfixable.
* eg: the item is in the "set" but not in the "list", so the item would
never be able to be pushed into the queue.
* Now site admin could simply clear the queue, then everything becomes
correct, the lost items could be re-pushed into queue by future
operations.
3. Split the "admin/monitor" to separate pages
4. Allow to download diagnosis report
* In history, there were many users reporting that Gitea queue gets
stuck, or Gitea's CPU is 100%
    * With diagnosis report, maintainers could know what happens clearly

The diagnosis report sample:
[gitea-diagnosis-20230510-192913.zip](https://github.com/go-gitea/gitea/files/11441346/gitea-diagnosis-20230510-192913.zip)
, use "go tool pprof profile.dat" to view the report.


Screenshots:


![image](https://github.com/go-gitea/gitea/assets/2114189/320659b4-2eda-4def-8dc0-5ea08d578063)


![image](https://github.com/go-gitea/gitea/assets/2114189/c5c46fae-9dc0-44ca-8cd3-57beedc5035e)


![image](https://github.com/go-gitea/gitea/assets/2114189/6168a811-42a1-4e64-a263-0617a6c8c4fe)

---------

Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-11 07:45:47 +00:00
Kyle D
a54c8b4ce9
Kd/fix redis unit test (#24650)
Replaces #24641

Currently, unit tests fail when run locally (unless users have minio
instance running). This PR only requires redis unit tests if in CI.

- Only run redis unit tests when `CI` env variable is set
- Add minio as a service in unit tests actions
2023-05-11 10:09:37 +08:00
wxiaoguang
54f399c4df
Increase default LFS auth timeout from 20m to 24h (#24628)
According to the discussion with DanielGibson, the default "20m" seems
too short. It would make LFS fail if the file is large / network is
slow.

I think relaxing this timeout doesn't have side affect. So change the
default value to 24h, IMO that should be long enough.

## ⚠️ BREAKING

If admins want the previous timeout, they should set the setting
`[server].LFS_HTTP_AUTH_EXPIRY`.
2023-05-10 22:23:47 +08:00
Hester Gong
ea7954f069
Modify luminance calculation and extract related functions into single files (#24586)
Close #24508

Main changes:
As discussed in the issue

1. Change luminance calculation function to use [Relative
Luminance](https://www.w3.org/WAI/GL/wiki/Relative_luminance)
2. Move the luminance related functions into color.go/color.js
3. Add tests for both the files (Not sure if test cases are too many
now)

Before (tests included by `UseLightTextOnBackground` are labels started
with `##`):
https://try.gitea.io/HesterG/testrepo/labels

After:
<img width="1307" alt="Screen Shot 2023-05-08 at 13 37 55"
src="https://user-images.githubusercontent.com/17645053/236742562-fdfc3a4d-2fab-466b-9613-96f2bf96b4bc.png">
<img width="1289" alt="Screen Shot 2023-05-08 at 13 38 06"
src="https://user-images.githubusercontent.com/17645053/236742570-022db68e-cec0-43bb-888a-fc54f5332cc3.png">
<img width="1299" alt="Screen Shot 2023-05-08 at 13 38 20"
src="https://user-images.githubusercontent.com/17645053/236742572-9af1de45-fb7f-460b-828d-ba25fae20f51.png">

---------

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-10 11:19:03 +00:00
Lunny Xiao
c355728a6f
Remove unnecessary code (#24610)
As title, remove unnecessary code.
2023-05-10 04:57:06 +00:00
来自村里的小螃蟹
cd9a13ebb4
Create a branch directly from commit on the create branch API (#22956)
#### Added
- API: Create a branch directly from commit on the create branch API
- Added `old_ref_name` parameter to allow creating a new branch from a
specific commit, tag, or branch.
- Deprecated `old_branch_name` parameter in favor of the new
`old_ref_name` parameter.

---------

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-05-09 18:22:32 +08:00
wxiaoguang
023a048f52
Make repository response support HTTP range request (#24592)
Replace #20480
Replace #18448

Close #16414
2023-05-09 15:34:36 +08:00
wxiaoguang
def4956122
Improve Gitea's web context, decouple "issue template" code into service package (#24590)
1. Remove unused fields/methods in web context.
2. Make callers call target function directly instead of the light
wrapper like "IsUserRepoReaderSpecific"
3. The "issue template" code shouldn't be put in the "modules/context"
package, so move them to the service package.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-09 01:30:14 +02:00
wxiaoguang
6f9c278559
Rewrite queue (#24505)
# ⚠️ Breaking

Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).

If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.

Example:

```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```

Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.

# The problem

The old queue package has some legacy problems:

* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.

It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.

# The new queue package

It keeps using old config and concept as much as possible.

* It only contains two major kinds of concepts:
    * The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.

There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.

Almost ready for review.

TODO:

* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)

## Code coverage:

![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2023-05-08 19:49:59 +08:00
wxiaoguang
cb700aedd1
Split "modules/context.go" to separate files (#24569)
The "modules/context.go" is too large to maintain.

This PR splits it to separate files, eg: context_request.go,
context_response.go, context_serve.go

This PR will help:

1. The future refactoring for Gitea's web context (eg: simplify the context)
2. Introduce proper "range request" support
3. Introduce context function

This PR only moves code, doesn't change any logic.
2023-05-08 17:36:54 +08:00
Matthew Walowski
ff5629268c
Pass 'not' to commit count (#24473)
Due to #24409 , we can now specify '--not' when getting all commits from
a repo to exclude commits from a different branch.

When I wrote that PR, I forgot to also update the code that counts the
number of commits in the repo. So now, if the --not option is used, it
may return too many commits, which can indicate that another page of
data is available when it is not.

This PR passes --not to the commands that count the number of commits in
a repo
2023-05-08 07:10:53 +00:00
wxiaoguang
0bb52883eb
Improve decryption failure message (#24573)
Help some users like #16832 #1851

There are many users reporting similar problem: if the SECRET_KEY
mismatches, some operations (like 2FA login) only reports unclear 500
error and unclear "base64 decode error" log (some maintainers ever spent
a lot of time on debugging such problem)

The SECRET_KEY was not well-designed and it is also a kind of technical
debt. Since it couldn't be fixed easily, it's good to add clearer error
messages, then at least users could know what the real problem is.

---------

Co-authored-by: delvh <dev.lh@web.de>
2023-05-07 19:29:43 +08:00
wxiaoguang
56ae853ca0
Simplify template helper functions (#24570)
To avoid bloating the template helper functions, some functions could be
provided by type methods.

And the new code `data-line-type="{{.GetHTMLDiffLineType}}"` reads
better than `data-line-type="{{DiffLineTypeToStr .GetType}}"`


After the fix, screenshots (the same as before):

<details>


![image](https://user-images.githubusercontent.com/2114189/236657918-20ce01e0-1192-443e-aeb4-6b3fe1aa2102.png)


![image](https://user-images.githubusercontent.com/2114189/236657950-ee19727f-a1fc-4133-afc7-e5d1a8c1783f.png)

</details>
2023-05-07 09:49:46 +00:00
techknowlogick
4daf40505a
Sort users and orgs on explore by recency by default (#24279)
This gives more "freshness" to the explore page. So it's not just the
same X users on the explore page by default, now it matches the same
sort as the repos on the explore page.

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-05-06 22:04:55 +08:00
KN4CK3R
05209f0d1d
Add RPM registry (#23380)
Fixes #20751

This PR adds a RPM package registry. You can follow [this
tutorial](https://opensource.com/article/18/9/how-build-rpm-packages) to
build a *.rpm package for testing.

This functionality is similar to the Debian registry (#22854) and
therefore shares some methods. I marked this PR as blocked because it
should be merged after #22854.


![grafik](https://user-images.githubusercontent.com/1666336/223806549-d8784fd9-9d79-46a2-9ae2-f038594f636a.png)
2023-05-05 20:33:37 +00:00
Jason Song
ea1afb945d
Replace placeholders in licenses (#24354)
Replace #22117. Implement it in a more maintainable way.

Some licenses have placeholders e.g. the BSD licenses start with this
line:
```
Copyright (c) <year> <owner>. 
```
This PR replaces the placeholders with the correct value when initialize
a new repo.

### FAQ

- Why not use a regex?
It will be a pretty complicated regex which could be hard to maintain.

- There're still missing placeholders.
There are over 500 licenses, it's impossible for anyone to inspect all
of them alone. Please help to add them if you find any, and it is also
OK to leave them for the future.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-05 13:46:17 +00:00
sillyguodong
a866cb0cb9
Fix the permission of team's Actions unit issue (#24536)
close #24449

The unit of `Actions` should be contorlled not only by
`repository.DISABLED_REPO_UNITS` but also by `actions.ENABLED`
in the `app.ini`.
Previously, the permission of the team's `Actions` unit was not
controlled by `actions.Enabled`. So, even if the user sets
`actions.Enabled` to false, he can still select the permission of the
`Actions` unit for the team.

This PR makes the permissions of the team's `Actions` unit also
controlled by `actions.Enabled`. Just append`TypeActions` into
`DisabledRepoUnits` slice when initializing if `actions.Enabled` is
false.

### Changes:

If `Actions` is set disbaled in `app.ini`, like below:
```yaml
[actions]
ENABLED = false
```

1. If user try to create/edit a team, will prompt user that `Actions` is disabled.

![image](https://user-images.githubusercontent.com/33891828/236370415-961082b2-82d2-4d9e-8025-83872ad08cbb.png)

2. `actions` is not displayed in the sidebar on the team details page

![image](https://user-images.githubusercontent.com/33891828/236371817-f39f9bc9-5926-4b88-b5e6-d93617fcfb07.png)
2023-05-05 20:02:30 +08:00
wxiaoguang
5d77691d42
Improve template system and panic recovery (#24461)
Partially for #24457

Major changes:

1. The old `signedUserNameStringPointerKey` is quite hacky, use
`ctx.Data[SignedUser]` instead
2. Move duplicate code from `Contexter` to `CommonTemplateContextData`
3. Remove incorrect copying&pasting code `ctx.Data["Err_Password"] =
true` in API handlers
4. Use one unique `RenderPanicErrorPage` for panic error page rendering
5. Move `stripSlashesMiddleware` to be the first middleware
6. Install global panic recovery handler, it works for both `install`
and `web`
7. Make `500.tmpl` only depend minimal template functions/variables,
avoid triggering new panics

Screenshot:

<details>

![image](https://user-images.githubusercontent.com/2114189/235444895-cecbabb8-e7dc-4360-a31c-b982d11946a7.png)

</details>
2023-05-04 14:36:34 +08:00
oliverpool
75ea0d5dba
Faster git.GetDivergingCommits (#24482)
Using `git rev-list --left-right` is almost 2x faster than calling `git
rev-list` twice.

Co-authored-by: silverwind <me@silverwind.io>
2023-05-04 05:08:41 +00:00
Lunny Xiao
377a0a20f0
Merge setting.InitXXX into one function with options (#24389)
This PR will merge 3 Init functions on setting packages as 1 and
introduce an options struct.
2023-05-04 11:55:35 +08:00
zeripath
ad8631c069
Fix intermittent CI failure in EmptyQueue (#23753)
The ordering of the final token causing a close of the queue in this
test may be out of sync due to concurrency. Instead just use ensure that
the queue is closed when everything expected is done.

Fixes: https://github.com/go-gitea/gitea/issues/23608
Fixes: https://github.com/go-gitea/gitea/issues/23977

---------

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-05-03 21:37:30 -04:00
yp05327
dbb3736785
Fix incorrect webhook time and use relative-time to display it (#24477)
Fixes #24414
After click replay this webhook, it will display `now`

![image](https://user-images.githubusercontent.com/18380374/235559399-05a23927-13f5-442d-8f10-2c7cd24022a0.png)

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-03 19:53:43 -04:00
KN4CK3R
bf999e4069
Add Debian package registry (#24426)
Co-authored-by: @awkwardbunny

This PR adds a Debian package registry.
You can follow [this
tutorial](https://www.baeldung.com/linux/create-debian-package) to build
a *.deb package for testing.
Source packages are not supported at the moment and I did not find
documentation of the architecture "all" and how these packages should be
treated.


![grafik](https://user-images.githubusercontent.com/1666336/218126879-eb80a866-775c-4c8e-8529-5797203a64e6.png)

Part of #20751.

Revised copy of #22854.

---------

Co-authored-by: Brian Hong <brian@hongs.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-02 12:31:35 -04:00
silverwind
d6f8238492
Replace N/A with - everywhere (#24474)
Followup to https://github.com/go-gitea/gitea/pull/24427.

Reasoning is that `N/A` is specific to english while `-` is
language-neutral and does not need translation.

Before:
<img width="891" alt="Screenshot 2023-05-01 at 20 58 20"
src="https://user-images.githubusercontent.com/115237/235511592-8a36d0f2-34ff-4dbe-b642-67c0ade644fe.png">

After:
<img width="901" alt="Screenshot 2023-05-01 at 20 59 59"
src="https://user-images.githubusercontent.com/115237/235511594-d49f6d09-92e8-4e99-be7b-2a37f5d24129.png">
2023-05-02 05:54:29 -04:00
wxiaoguang
2a56666fd2
Fix incorrect CurrentUser check for docker rootless (#24441)
The IsRunUserMatchCurrentUser logic is fragile, the "SSH" config is not
ready when it executes.
2023-04-30 14:14:57 -04:00
wxiaoguang
e3750370df
Use globally shared HTMLRender (#24436)
The old `HTMLRender` is not ideal.

1. It shouldn't be initialized multiple times, it consumes a lot of
memory and is slow.
2. It shouldn't depend on short-lived requests, the `WatchLocalChanges`
needs a long-running context.
3. It doesn't make sense to use FuncsMap slice.


HTMLRender was designed to only work for GItea's specialized 400+
templates, so it's good to make it a global shared instance.
2023-04-30 08:22:23 -04:00
Yarden Shoham
cc84c58aff
Remove unused setting time.FORMAT (#24430)
It's loaded and then never used.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-04-29 22:51:43 +02:00
Matthew Walowski
f766b00293
Add ability to specify '--not' from GetAllCommits (#24409)
For my specific use case, I'd like to get all commits that are on one
branch but NOT on the other branch.

For instance, I'd like to get all the commits on `Branch1` that are not
also on `master` (I.e. all commits that were made after `Branch1` was
created).

This PR adds a `not` query param that gets passed down to the `git log`
command to allow the user to exclude items from `GetAllCommits`.

See [git
documentation](https://git-scm.com/docs/git-log#Documentation/git-log.txt---not)

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-04-29 08:34:14 -04:00
wxiaoguang
241b74f6c5
Improve template helper (#24417)
It seems that we really need the "context function" soon. So we should
clean up the helper functions first.

Major changes:

* Improve StringUtils and add JsonUtils
* Remove one-time-use helper functions like CompareLink
* Move other code (no change) to util_avatar/util_render/util_misc (no
need to propose changes for them)

I have tested the changed templates:


![image](https://user-images.githubusercontent.com/2114189/235283862-608dbf6b-2da3-4d06-8157-b523ca93edb4.png)


![image](https://user-images.githubusercontent.com/2114189/235283888-1dfc0471-e622-4d64-9d76-7859819580d3.png)


![image](https://user-images.githubusercontent.com/2114189/235283903-d559f14d-4abb-4a50-915f-2b9cbc381a7a.png)


![image](https://user-images.githubusercontent.com/2114189/235283955-b7b5adea-aca3-4758-b38a-3aae3f7c6048.png)

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-04-29 08:02:29 -04:00
silverwind
fc62992518
Skip known flaky queue tests on CI environment (#24419)
Random CI failures are annoying. It's better to just skip the affected
tests so maintainers can use their valuable time for more productive
topics.

Related: https://github.com/go-gitea/gitea/issues/23608
Related: https://github.com/go-gitea/gitea/issues/23977
Related: https://github.com/go-gitea/gitea/issues/18703
2023-04-29 11:53:11 +02:00