Commit graph

476 commits

Author SHA1 Message Date
Jason Song
a627b885c7
Support compression for Actions logs (#31761)
Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015

The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.

That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.

This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.

And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.

`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.

<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">

(cherry picked from commit 33cc5837a655ad544b936d4d040ca36d74092588)

Conflicts:
	assets/go-licenses.json
	go.mod
	go.sum
  resolved with make tidy
2024-08-13 06:51:49 +02:00
Gusted
a21128a734
[CHORE] Drop go-git support
See https://codeberg.org/forgejo/discussions/issues/164 for the
rationale and discussion of this change.

Everything related to the `go-git` dependency is dropped (Only a single
instance is left in a test file to test for an XSS, it requires crafting
an commit that Git itself refuses to craft). `_gogit` files have
been removed entirely, `go:build: !gogit` is removed, `XXX_nogogit.go` files
either have been renamed or had their code being merged into the
`XXX.go` file.
2024-08-12 19:11:09 +02:00
Renovate Bot
99d78fb9e7 Update x/tools to v0.24.0 2024-08-09 10:25:53 +00:00
Renovate Bot
64e56f0d0d Update module golang.org/x/crypto to v0.26.0 2024-08-08 00:03:21 +00:00
TheFox0x7
2e2a044493
Revert "Open telemetry integration (#3972)"
This reverts commit c738542201.
2024-08-07 11:22:43 +02:00
Renovate Bot
4b8726e599 Update module github.com/go-logr/logr to v1.4.2 2024-08-06 00:16:56 +00:00
Renovate Bot
eab599de41 Update module github.com/google/go-github/v57 to v63 2024-08-05 13:21:39 +00:00
Renovate Bot
8e3b33dd53
Update module golang.org/x/oauth2 to v0.22.0 2024-08-05 09:01:05 +02:00
Earl Warren
98457eb67d Merge pull request 'Update module golang.org/x/sys to v0.23.0 (forgejo)' (#4817) from renovate/forgejo-golang.org-x-sys-0.x into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4817
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
2024-08-05 06:59:14 +00:00
TheFox0x7
c738542201 Open telemetry integration (#3972)
This PR adds opentelemetry and chi wrapper to have basic instrumentation

<!--start release-notes-assistant-->

## Draft release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Features
  - [PR](https://codeberg.org/forgejo/forgejo/pulls/3972): <!--number 3972 --><!--line 0 --><!--description YWRkIHN1cHBvcnQgZm9yIGJhc2ljIHJlcXVlc3QgdHJhY2luZyB3aXRoIG9wZW50ZWxlbWV0cnk=-->add support for basic request tracing with opentelemetry<!--description-->
<!--end release-notes-assistant-->

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3972
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: TheFox0x7 <thefox0x7@gmail.com>
Co-committed-by: TheFox0x7 <thefox0x7@gmail.com>
2024-08-05 06:04:39 +00:00
Renovate Bot
2c95baffeb Update module golang.org/x/sys to v0.23.0 2024-08-05 02:04:33 +00:00
Renovate Bot
d0684334b3 Update module github.com/meilisearch/meilisearch-go to v0.27.2 2024-08-04 00:03:09 +00:00
Earl Warren
94f3589623
chore(ci): do not hardcode go version, use go.mod instead 2024-08-03 11:53:55 +02:00
Exploding Dragon
471265c4e0 Add signature support for the RPM module (#4780)
This pull request comes from https://github.com/go-gitea/gitea/pull/27069.

If the rpm package does not contain a matching gpg signature, the installation will fail. See ([gitea/gitea#27031](https://github.com/go-gitea/gitea/issues/27031)) , now auto-signing all new rpm uploads.

This option is turned off by default for compatibility.

<!--start release-notes-assistant-->

## Draft release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Features
  - [PR](https://codeberg.org/forgejo/forgejo/pulls/4780): <!--number 4780 --><!--line 0 --><!--description QWRkIHNpZ25hdHVyZSBzdXBwb3J0IGZvciB0aGUgUlBNIG1vZHVsZQ==-->Add signature support for the RPM module<!--description-->
<!--end release-notes-assistant-->

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4780
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Exploding Dragon <explodingfkl@gmail.com>
Co-committed-by: Exploding Dragon <explodingfkl@gmail.com>
2024-08-02 05:56:57 +00:00
Renovate Bot
ccdd5d375b Update module github.com/meilisearch/meilisearch-go to v0.27.1 2024-07-31 00:02:33 +00:00
Renovate Bot
ac61d697b6 Update module github.com/blevesearch/bleve/v2 to v2.4.2 2024-07-27 00:02:33 +00:00
Renovate Bot
1c63c47f5f Update module xorm.io/xorm to v1.3.9 2024-07-21 16:03:40 +00:00
Earl Warren
125e4832e0 Merge pull request 'Update module github.com/go-testfixtures/testfixtures/v3 to v3.12.0 (forgejo)' (#4485) from renovate/forgejo-github.com-go-testfixtures-testfixtures-v3-3.x into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4485
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
2024-07-17 14:07:10 +00:00
Earl Warren
c560595e1f
Revert "Update module github.com/redis/go-redis/v9 to v9.5.4"
This reverts commit dd6413d350.
2024-07-17 06:36:42 +02:00
Renovate Bot
6a3415abbb Update module github.com/minio/minio-go/v7 to v7.0.74 2024-07-16 02:05:23 +00:00
Renovate Bot
09c2ab9be8
Update module github.com/go-testfixtures/testfixtures/v3 to v3.12.0 2024-07-15 21:39:38 +02:00
Earl Warren
56ee58c239 Merge pull request '[CHORE] Use github.com/ProtonMail/go-crypto' (#4506) from gusted/proton-openpgp into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4506
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
2024-07-15 16:49:05 +00:00
Gusted
45341ee9ce
[CHORE] Use github.com/ProtonMail/go-crypto
- We were previously using `github.com/keybase/go-crypto`, because the
package for openpgp by Go itself is deprecated and no longer
maintained. This library provided a maintained version of the openpgp
package. However, it hasn't seen any activity for the last five years,
 and I would therefore consider this also unmaintained.
- This patch switches the package to `github.com/ProtonMail/go-crypto`
which provides a maintained version of the openpgp package and was
already being used in the tests.
- Adds unit tests, I've carefully checked the callstacks to ensure the
OpenPGP-related code was covered under either a unit test or integration
tests to avoid regression, as this can easily turn into security
vulnerabilities if a regression happens here.
- Small behavior update, revocations are now checked correctly instead
of checking if they merely exist and the expiry time of a subkey is used
if one is provided (this is just cosmetic and doesn't impact security).
- One more dependency eliminated :D
2024-07-15 17:27:37 +02:00
Gusted
45401e044f
[CHORE] Update jsonschema library to v6
- Update the `github.com/santhosh-tekuri/jsonschema` library from v5 to
v6.
- Update the migration loader function to a type, which is now required
in V6.
- `github.com/santhosh-tekuri/jsonschema/v6` was already used by gof3,
so removing the v5 library and using the v6 library reduces the binary
size of Forgejo.
  - Before: 95912040 bytes
  - After: 95706152 bytes
2024-07-15 17:20:50 +02:00
Renovate Bot
dd6413d350 Update module github.com/redis/go-redis/v9 to v9.5.4 2024-07-13 00:06:19 +00:00
Renovate Bot
6154befef6 Update module golang.org/x/tools to v0.23.0 2024-07-10 00:01:50 +00:00
Earl Warren
1efec2adab Merge pull request 'Update module github.com/buildkite/terminal-to-html/v3 to v3.14.0 (forgejo)' (#4429) from renovate/forgejo-github.com-buildkite-terminal-to-html-v3-3.x into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4429
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
2024-07-09 21:58:01 +00:00
Renovate Bot
a8f35f6edd Update module github.com/buildkite/terminal-to-html/v3 to v3.14.0 2024-07-09 00:03:10 +00:00
Renovate Bot
1630a94cfa Update module golang.org/x/net to v0.27.0 2024-07-08 00:05:03 +00:00
Renovate Bot
e4e2e91467 Update module golang.org/x/crypto to v0.25.0 2024-07-08 00:04:57 +00:00
Gusted
cf8f26d616
[CHORE] Remove github.com/yuin/goldmark-meta
- Remove a unused dependency. This dependency was added to handle YAML
'frontmatter' meta, parsing them and converting them to a table or
details in the resulting HTML. As can be read in the issue that reported
the behavior of YAML frontmatter being rendered literally,
https://github.com/go-gitea/gitea/issues/5377.
- It's an unused dependency as the codebase since then moved on to do this YAML
parsing and rendering on their own, this was implemented in
812cfd0ad9.
- Adds unit tests that was related to this functionality, to proof the
codebase already handles this and to prevent regressions.
2024-07-07 03:18:13 +02:00
Earl Warren
c5028d72a6 Merge pull request 'Update module github.com/microcosm-cc/bluemonday to v1.0.27' (#4333) from renovate/github.com-microcosm-cc-bluemonday-1.x into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4333
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
2024-07-05 05:25:08 +00:00
Earl Warren
860b33696d Merge pull request 'Update module github.com/buildkite/terminal-to-html/v3 to v3.13.0' (#4313) from renovate/github.com-buildkite-terminal-to-html-v3-3.x into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4313
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
2024-07-05 05:04:36 +00:00
Renovate Bot
a8df27e5a1
Update module github.com/microcosm-cc/bluemonday to v1.0.27 2024-07-05 06:36:35 +02:00
Renovate Bot
1c49047390 Update module google.golang.org/grpc to v1.65.0 2024-07-04 22:19:07 +00:00
Gusted
3eb178db49
[CHORE] Update terminal-to-html dependency
- Update the `github.com/buildkite/terminal-to-html/v3` dependency from
version v3.10.1 to v3.13.0.
- Version v3.12.0 introduced an incompatible change, the return type of
`AsHTML` changed from `[]byte` to `string`. That same version also
introduced streaming mode
https://github.com/buildkite/terminal-to-html/pull/126, which allows us
to avoid reading the whole input into memory.
- Closes #4313
2024-07-04 23:41:17 +02:00
Renovate Bot
319c4efbe7 Update module github.com/blevesearch/bleve/v2 to v2.4.1 2024-07-04 00:02:31 +00:00
Renovate Bot
4101260d6e Update module github.com/minio/minio-go/v7 to v7.0.73 2024-07-03 18:07:19 +00:00
Renovate Bot
fa1a853db0 Update dependency go to v1.22.5 2024-07-03 08:23:00 +00:00
Earl Warren
a3a8b0e7d1 Merge pull request 'Update module github.com/yuin/goldmark to v1.7.4' (#4240) from renovate/github.com-yuin-goldmark-1.x into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4240
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
2024-06-26 07:51:29 +00:00
Renovate Bot
d9b83719d6 Update module github.com/yuin/goldmark to v1.7.4 2024-06-26 06:19:55 +00:00
Renovate Bot
83d7be3447 Update module golang.org/x/image to v0.18.0 2024-06-26 06:19:47 +00:00
Earl Warren
91f16dfcb7
fix(security): GO-2024-2947
Vulnerability #1: GO-2024-2947
    Leak of sensitive information to log files in
    github.com/hashicorp/go-retryablehttp
  More info: https://pkg.go.dev/vuln/GO-2024-2947
  Module: github.com/hashicorp/go-retryablehttp
    Found in: github.com/hashicorp/go-retryablehttp@v0.7.5
    Fixed in: github.com/hashicorp/go-retryablehttp@v0.7.7
    Example traces found:
      #1: services/migrations/gitlab.go:500:74: migrations.GitlabDownloader.GetComments calls gitlab.DiscussionsService.ListMergeRequestDiscussions, which eventually calls retryablehttp.Client.Do
2024-06-26 07:35:19 +02:00
Renovate Bot
e91961224d
Update module github.com/yuin/goldmark to v1.7.3 2024-06-24 08:03:18 +02:00
Renovate Bot
11e847ac74 Update module github.com/go-chi/chi/v5 to v5.0.14 2024-06-23 00:03:30 +00:00
Renovate Bot
c07cc28d88 Update module code.forgejo.org/f3/gof3/v3 to v3.4.0 (#4196)
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| code.forgejo.org/f3/gof3/v3 | require | minor | `v3.3.1` -> `v3.4.0` |

---

### Configuration

📅 **Schedule**: Branch creation - "before 4am" (UTC), Automerge - "before 4am" (UTC).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

â™» **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MDkuMSIsInVwZGF0ZWRJblZlciI6IjM3LjQwOS4xIiwidGFyZ2V0QnJhbmNoIjoiZm9yZ2VqbyIsImxhYmVscyI6WyJkZXBlbmRlbmN5LXVwZ3JhZGUiXX0=-->

Co-authored-by: Twenty Panda <twenty-panda@posteo.com>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4196
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Renovate Bot <forgejo-renovate-action@forgejo.org>
Co-committed-by: Renovate Bot <forgejo-renovate-action@forgejo.org>
2024-06-21 13:51:59 +00:00
Renovate Bot
07e26508b0 Update module github.com/go-chi/chi/v5 to v5.0.13 2024-06-19 00:02:33 +00:00
Renovate Bot
a815eb552a
Update module github.com/gorilla/feeds to v1.2.0
Refs: https://github.com/go-gitea/gitea/pull/31400
2024-06-18 09:06:46 +02:00
Renovate Bot
aa9c164940 Update module github.com/yuin/goldmark to v1.7.2 2024-06-15 00:02:33 +00:00
Earl Warren
e99d3f7055
feat(F3): CLI: f3 mirror to convert to/from Forgejo
feat(F3): driver stub

feat(F3): util.Logger

feat(F3): driver compliance tests

feat(F3): driver/users implementation

feat(F3): driver/user implementation

feat(F3): driver/{projects,project} implementation

feat(F3): driver/{labels,label} implementation

feat(F3): driver/{milestones,milestone} implementation

feat(F3): driver/{repositories,repository} implementation

feat(F3): driver/{organizations,organization} implementation

feat(F3): driver/{releases,release} implementation

feat(F3): driver/{issues,issue} implementation

feat(F3): driver/{comments,comment} implementation

feat(F3): driver/{assets,asset} implementation

feat(F3): driver/{pullrequests,pullrequest} implementation

feat(F3): driver/{reviews,review} implementation

feat(F3): driver/{topics,topic} implementation

feat(F3): driver/{reactions,reaction} implementation

feat(F3): driver/{reviewComments,reviewComment} implementation

feat(F3): CLI: f3 mirror

chore(F3): move to code.forgejo.org

feat(f3): upgrade to gof3 3.1.0

repositories in pull requests are represented with a reference instead
of an owner/project pair of names
2024-06-14 12:52:12 +02:00