Commit graph

76 commits

Author SHA1 Message Date
Earl Warren
5442b0a6b1 Merge pull request '[v9.0/forgejo] feat: add architecture-specific removal support for arch package' (#5407) from bp-v9.0/forgejo-89742c4 into v9.0/forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/5407
2024-09-27 12:09:01 +00:00
Exploding Dragon
658ed564cb feat: add architecture-specific removal support for arch package (#5351)
- [x] add architecture-specific removal support
- [x] Fix upload competition
- [x] Fix not checking input when downloading

docs: https://codeberg.org/forgejo/docs/pulls/874

### Release notes

- [ ] I do not want this change to show in the release notes.

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/5351
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Exploding Dragon <explodingfkl@gmail.com>
Co-committed-by: Exploding Dragon <explodingfkl@gmail.com>
(cherry picked from commit 89742c4913)
2024-09-27 08:29:09 +00:00
KN4CK3R
232179aa3d Do not escape relative path in RPM primary index (#32038)
Fixes #32021

Do not escape the relative path.

(cherry picked from commit f528df944bb9436afcb9272add2ee0cccefbdb55)
(cherry picked from commit 0cafec4c7a)
2024-09-27 08:13:29 +00:00
Gusted
5a871f6095
[SEC] Ensure propagation of API scopes for Conan and Container authentication
- The Conan and Container packages use a different type of
authentication. It first authenticates via the regular way (api tokens
or user:password, handled via `auth.Basic`) and then generates a JWT
token that is used by the package software (such as Docker) to do the
action they wanted to do. This JWT token didn't properly propagate the
API scopes that the token was generated for, and thus could lead to a
'scope escalation' within the Conan and Container packages, read
access to write access.
- Store the API scope in the JWT token, so it can be propagated on
subsequent calls that uses that JWT token.
- Integration test added.
- Resolves #5128
2024-08-28 10:33:32 +02:00
Earl Warren
f19f31ac73
cron task to cleanup dangling container images with version sha256:*
Fixes: https://codeberg.org/forgejo/forgejo/issues/4378
2024-08-22 09:10:15 +02:00
Exploding Dragon
87d50eca87 feat: support grouping by any path for arch package (#4903)
Previous arch package grouping was not well-suited for complex or multi-architecture environments. It now supports the following content:

- Support grouping by any path.
- New support for packages in `xz` format.
- Fix clean up rules

<!--start release-notes-assistant-->

## Draft release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Features
  - [PR](https://codeberg.org/forgejo/forgejo/pulls/4903): <!--number 4903 --><!--line 0 --><!--description c3VwcG9ydCBncm91cGluZyBieSBhbnkgcGF0aCBmb3IgYXJjaCBwYWNrYWdl-->support grouping by any path for arch package<!--description-->
<!--end release-notes-assistant-->

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4903
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Exploding Dragon <explodingfkl@gmail.com>
Co-committed-by: Exploding Dragon <explodingfkl@gmail.com>
2024-08-11 10:35:11 +00:00
Exploding Dragon
f17194ca91 Arch packages implementation (#4785)
This PR is from https://github.com/go-gitea/gitea/pull/31037

This PR was originally created by @d1nch8g , and the original source code comes from https://ion.lc/core/gitea.

This PR adds a package registry for [Arch Linux](https://archlinux.org/) packages with support for package files, [signatures](https://wiki.archlinux.org/title/Pacman/Package_signing), and automatic [pacman-database](https://archlinux.org/pacman/repo-add.8.html) management.

Features:

1. Push any ` tar.zst ` package and Gitea sign it.
2. Delete endpoint for specific package version and all related files
3. Supports trust levels with `SigLevel = Required`.
4. Package UI with instructions to connect to the new pacman database and visualised package metadata

![](/attachments/810ca6df-bd20-44c2-bdf7-95e94886d750)

You can follow [this tutorial](https://wiki.archlinux.org/title/Creating_packages) to build a *.pkg.tar.zst package for testing

docs pr: https://codeberg.org/forgejo/docs/pulls/791

Co-authored-by: d1nch8g@ion.lc
Co-authored-by: @KN4CK3R
Co-authored-by: @mahlzahn
Co-authored-by: @silverwind
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4785
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Exploding Dragon <explodingfkl@gmail.com>
Co-committed-by: Exploding Dragon <explodingfkl@gmail.com>
2024-08-04 06:16:29 +00:00
Exploding Dragon
471265c4e0 Add signature support for the RPM module (#4780)
This pull request comes from https://github.com/go-gitea/gitea/pull/27069.

If the rpm package does not contain a matching gpg signature, the installation will fail. See ([gitea/gitea#27031](https://github.com/go-gitea/gitea/issues/27031)) , now auto-signing all new rpm uploads.

This option is turned off by default for compatibility.

<!--start release-notes-assistant-->

## Draft release notes
<!--URL:https://codeberg.org/forgejo/forgejo-->
- Features
  - [PR](https://codeberg.org/forgejo/forgejo/pulls/4780): <!--number 4780 --><!--line 0 --><!--description QWRkIHNpZ25hdHVyZSBzdXBwb3J0IGZvciB0aGUgUlBNIG1vZHVsZQ==-->Add signature support for the RPM module<!--description-->
<!--end release-notes-assistant-->

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4780
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Exploding Dragon <explodingfkl@gmail.com>
Co-committed-by: Exploding Dragon <explodingfkl@gmail.com>
2024-08-02 05:56:57 +00:00
Earl Warren
ff6aceaeac
feat: optimize the FindUnreferencedPackages package query
Replace a double select with a simple select.

The complication originates from the initial implementation which
deleted packages instead of selecting them. It was justified to
workaround a problem in MySQL. But it is just a waste of resources
when collecting a list of IDs.
2024-07-27 15:57:24 +02:00
Alexandre Oliveira
ef815db8f2 Remove hardcoded filenames for better readability 2024-07-17 23:20:48 +02:00
Gusted
45341ee9ce
[CHORE] Use github.com/ProtonMail/go-crypto
- We were previously using `github.com/keybase/go-crypto`, because the
package for openpgp by Go itself is deprecated and no longer
maintained. This library provided a maintained version of the openpgp
package. However, it hasn't seen any activity for the last five years,
 and I would therefore consider this also unmaintained.
- This patch switches the package to `github.com/ProtonMail/go-crypto`
which provides a maintained version of the openpgp package and was
already being used in the tests.
- Adds unit tests, I've carefully checked the callstacks to ensure the
OpenPGP-related code was covered under either a unit test or integration
tests to avoid regression, as this can easily turn into security
vulnerabilities if a regression happens here.
- Small behavior update, revocations are now checked correctly instead
of checking if they merely exist and the expiry time of a subkey is used
if one is provided (this is just cosmetic and doesn't impact security).
- One more dependency eliminated :D
2024-07-15 17:27:37 +02:00
oliverpool
dda88488ca packages: skip error logging on duplicate 2024-03-24 07:12:31 +01:00
Gusted
461f925554
[BUG] Reflect Cargo index state in settings
- Currently in the Cargo section of the packages setting menu two
buttons are always shown, "Initalize index" and "Rebuild index", however
only of these should be shown depending on the state of the index, if
there's no index the "Initalize index" button should be shown and if
there's an index the "Rebuild index" button should be shown. This patch
does exactly that.
- Resolves #2628
2024-03-20 09:17:49 +01:00
6543
e2371743d5
remove util.OptionalBool and related functions (#29513)
and migrate affected code

_last refactoring bits to replace **util.OptionalBool** with
**optional.Option[bool]**_

(cherry picked from commit a3f05d0d98408bb47333b19f505b21afcefa9e7c)

Conflicts:
	services/repository/branch.go
	trivial context conflict
2024-03-06 12:10:46 +08:00
wxiaoguang
62d3e5255f
Port "Use general token signing secret"
Port of https://github.com/go-gitea/gitea/pull/29205

Use a clearly defined "signing secret" for token signing.

(cherry picked from commit 8be198cdef0a486f417663b1fd6878458d7e5d92)
2024-02-19 20:49:37 +01:00
Robin Schoonover
c0ced0f0cd
Fix debian InRelease Acquire-By-Hash newline (#29204)
There is a missing newline when generating the debian apt repo InRelease
file, which results in output like:

```
[...]
Date: Wed, 14 Feb 2024 05:03:01 UTC
Acquire-By-Hash: yesMD5Sum:
 51a518dbddcd569ac3e0cebf330c800a 3018 main-dev/binary-amd64/Packages
[...]
```

It appears this would probably result in apt ignoring the
Acquire-By-Hash setting and not using the by-hash functionality,
although I'm not sure how to confirm it.

(cherry picked from commit 33400a02d4eb35a0656fd6d20fc56801de09b959)
2024-02-17 23:24:31 +01:00
Sergey Bugaev
2da233ad8b
Propagate install_if and provider_priority to APKINDEX (#28899)
Resolves https://github.com/go-gitea/gitea/issues/28704

Example of an entry in the generated `APKINDEX` file:

```
C:Q1xCO3H9LTTEbhKt9G1alSC87I56c=
P:hello
V:2.12-r1
A:x86_64
T:The GNU Hello program produces a familiar, friendly greeting
U:https://www.gnu.org/software/hello/
L:GPL-3.0-or-later
S:15403
I:36864
o:hello
m:
t:1705934118
D:so:libc.musl-x86_64.so.1
p:cmd:hello=2.12-r1
i:foobar=1.0 !baz
k:42
```

the `i:` and `k:` entries are new.

---------

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2024-02-05 05:52:56 +00:00
KN4CK3R
461d8b53c2
Fix some RPM registry flaws (#28782)
Related #26984
(https://github.com/go-gitea/gitea/pull/26984#issuecomment-1889588912)

Fix admin cleanup message.
Fix models `Get` not respecting default values.
Rebuild RPM repository files after cleanup.
Do not add RPM group to package version name.
Force stable sorting of Alpine/Debian/RPM repository data.
Fix missing deferred `Close`.
Add tests for multiple RPM groups.
Removed non-cached `ReplaceAllStringRegex`.

If there are multiple groups available, it's stated in the package
installation screen:

![grafik](https://github.com/go-gitea/gitea/assets/1666336/8f132760-882c-4ab8-9678-77e47dfc4415)
2024-01-19 11:37:10 +00:00
Mihir Joshi
b8270240bf
Fix reverting a merge commit failing (#28794)
Fixes #22236

---
Error occurring currently while trying to revert commit using read-tree
-m approach:
> 2022/12/26 16:04:43 ...rvices/pull/patch.go:240:AttemptThreeWayMerge()
[E] [63a9c61a] Unable to run read-tree -m! Error: exit status 128 -
fatal: this operation must be run in a work tree
> 	 - fatal: this operation must be run in a work tree

We need to clone a non-bare repository for `git read-tree -m` to work.

bb371aee6e
adds support to create a non-bare cloned temporary upload repository.

After cloning a non-bare temporary upload repository, we [set default
index](https://github.com/go-gitea/gitea/blob/main/services/repository/files/cherry_pick.go#L37)
(`git read-tree HEAD`).
This operation ends up resetting the git index file (see investigation
details below), due to which, we need to call `git update-index
--refresh` afterward.


Here's the diff of the index file before and after we execute
SetDefaultIndex: https://www.diffchecker.com/hyOP3eJy/

Notice the **ctime**, **mtime** are set to 0 after SetDefaultIndex.

You can reproduce the same behavior using these steps:
```bash
$ git clone https://try.gitea.io/me-heer/test.git -s -b main
$ cd test
$ git read-tree HEAD
$ git read-tree -m 1f085d7ed8 1f085d7ed8 9933caed00
error: Entry '1' not uptodate. Cannot merge.
```

After which, we can fix like this:
```
$ git update-index --refresh
$ git read-tree -m 1f085d7ed8 1f085d7ed8 9933caed00
```
2024-01-16 15:06:51 +00:00
Exploding Dragon
ba4d0b8ffb
Support for grouping RPMs using paths (#26984)
The current rpm repository places all packages in the same repository,
and different systems (el7,f34) may hit packages that do not belong to
this distribution ( #25304 ) , which now supports grouping of rpm.

![图片](https://github.com/go-gitea/gitea/assets/33776693/d1e1d99f-7799-4b2b-a19b-cb2a5c692914)

Fixes #25304 .
Fixes #27056 .

Refactor: [#25866](https://github.com/go-gitea/gitea/pull/25866)
2024-01-12 03:16:05 +00:00
Nanguan Lin
b6e0957b9c
Fix alpine package files are not rebuilt (#28638)
I noticed the `BuildAllRepositoryFiles` function under the Alpine folder
is unused and I thought it was a bug.
But I'm not sure about this. Was it on purpose?
2023-12-31 12:08:46 +08:00
Lunny Xiao
408a484224
Adjust object format interface (#28469)
- Remove `ObjectFormatID`
- Remove function `ObjectFormatFromID`.
- Use `Sha1ObjectFormat` directly but not a pointer because it's an
empty struct.
- Store `ObjectFormatName` in `repository` struct
2023-12-17 11:56:08 +00:00
Adam Majer
cbf923e87b
Abstract hash function usage (#28138)
Refactor Hash interfaces and centralize hash function. This will allow
easier introduction of different hash function later on.

This forms the "no-op" part of the SHA256 enablement patch.
2023-12-13 21:02:00 +00:00
KN4CK3R
ff5106d700
Fix possible nil pointer access (#28428)
There could be a nil pointer exception if the file is not found because
that specific error is suppressed but not handled.
2023-12-12 13:51:33 +00:00
KN4CK3R
dfa77ac020
Fix RPM/Debian signature key creation (#28352)
Fixes #28324

The name parameter can't contain some characters
(https://github.com/keybase/go-crypto/blob/master/openpgp/keys.go#L680)
but is optional. Therefore just use an empty string.
2023-12-05 00:48:01 +00:00
Lunny Xiao
f65977df3a
Revert "packages: Calculate package size quota using package creator ID instead of owner ID (#28007)" (#28049)
This reverts commit #28007 60522fc96f.
2023-11-14 16:03:56 +01:00
Danila Fominykh
60522fc96f
packages: Calculate package size quota using package creator ID instead of owner ID (#28007)
Changed behavior to calculate package quota limit using package `creator
ID` instead of `owner ID`.

Currently, users are allowed to create an unlimited number of
organizations, each of which has its own package limit quota, resulting
in the ability for users to have unlimited package space in different
organization scopes. This fix will calculate package quota based on
`package version creator ID` instead of `package version owner ID`
(which might be organization), so that users are not allowed to take
more space than configured package settings.

Also, there is a side case in which users can publish packages to a
specific package version, initially published by different user, taking
that user package size quota. Version in fix should be better because
the total amount of space is limited to the quota for users sharing the
same organization scope.
2023-11-14 00:38:50 +01:00
KN4CK3R
b36e2ca419
List all Debian package versions in Packages (#27786)
Closes #27783

This PR lists all and not only the latest package versions in the
`Packages` index.
2023-10-29 14:14:47 +00:00
KN4CK3R
ab3f6c1bac
Close all hashed buffers (#27787)
Add missing `.Close()` calls. The current code does not delete the
temporary files if the data grows over 32mb.
2023-10-25 21:24:24 +02:00
merlleu
796ff26e0e
Do not force creation of _cargo-index repo on publish (#27266)
Hello there,
Cargo Index over HTTP is now prefered over git for package updates: we
should not force users who do not need the GIT repo to have the repo
created/updated on each publish (it can still be created in the packages
settings).

The current behavior when publishing is to check if the repo exist and
create it on the fly if not, then update it's content.
Cargo HTTP Index does not rely on the repo itself so this will be
useless for everyone not using the git protocol for cargo registry.

This PR only disable the creation on the fly of the repo when publishing
a crate.

This is linked to #26844 (error 500 when trying to publish a crate if
user is missing write access to the repo) because it's now optional.

---------

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-10-24 03:26:38 +00:00
JakobDev
7047df36d4
Another round of db.DefaultContext refactor (#27103)
Part of #27065

---------

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-09-25 13:17:37 +00:00
JakobDev
c548dde205
More refactoring of db.DefaultContext (#27083)
Next step of #27065
2023-09-15 06:13:19 +00:00
Lunny Xiao
4f32abaf94
move repository deletion to service layer (#26948)
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-09-08 04:51:15 +00:00
Lunny Xiao
b9df9fa2e2
Move createrepository from module to service layer (#26927)
Repository creation depends on many models, so moving it to service
layer is better.
2023-09-06 12:08:51 +00:00
Lunny Xiao
540bf9fa6d
Move notification interface to services layer (#26915)
Extract from #22266
2023-09-05 18:37:47 +00:00
Chongyi Zheng
ac2f8c9ac6
Reduce some allocations in type conversion (#26772) 2023-08-29 00:43:16 +08:00
merlleu
a587d25261
Add auth-required to config.json for Cargo http registry (#26729)
Cargo registry-auth feature requires config.json to have a property
auth-required set to true in order to send token to all registry
requests.
This is ok for git index because you can manually edit the config.json
file to add the auth-required, but when using sparse
(setting index url to
"sparse+https://git.example.com/api/packages/{owner}/cargo/"), the
config.json is dynamically rendered, and does not reflect changes to the
config.json file in the repo.

I see two approaches:
- Serve the real config.json file when fetching the config.json on the
cargo service.
- Automatically detect if the registry requires authorization. (This is
what I implemented in this PR).

What the PR does:
- When a cargo index repository is created, on the config.json, set
auth-required to wether or not the repository is private.
- When the cargo/config.json endpoint is called, set auth-required to
wether or not the request was authorized using an API token.
2023-08-28 07:05:39 +00:00
KN4CK3R
0c6ae61229
Allow package cleanup from admin page (#25307)
Until now expired package data gets deleted daily by a cronjob. The
admin page shows the size of all packages and the size of unreferenced
data. The users (#25035, #20631) expect the deletion of this data if
they run the cronjob from the admin page but the job only deletes data
older than 24h.

This PR adds a new button which deletes all expired data.

![grafik](https://github.com/go-gitea/gitea/assets/1666336/b3e35d73-9496-4fa7-a20c-e5d30b1f6850)

---------

Co-authored-by: silverwind <me@silverwind.io>
2023-08-08 00:46:10 +00:00
KN4CK3R
2d3924d0e7
Prevent newline errors with Debian packages (#26332)
Fixes #26313
2023-08-05 10:59:52 +02:00
Peter Verraedt
48cd82b759
Fix version in rpm repodata/primary.xml.gz (#26009)
The version listed in rpm repodata should only contain the rpm version
(1.0.0) and not the combination of version and release (1.0.0-2). We
correct this behaviour in primary.xml.gz, filelists.xml.gz and
others.xml.gz.

Signed-off-by: Peter Verraedt <peter@verraedt.be>
2023-07-21 19:41:38 +00:00
harryzcy
0f9f6567bb
Bump github.com/golang-jwt/jwt to v5 (#25975)
Bumping `github.com/golang-jwt/jwt` from v4 to v5.

`github.com/golang-jwt/jwt` v5 is bringing some breaking changes:

- standard `Valid()` method on claims is removed. It's replaced by
`ClaimsValidator` interface implementing `Validator()` method instead,
which is called after standard validation. Gitea doesn't seem to be
using this logic.
- `jwt.Token` has a field `Valid`, so it's checked in `ParseToken`
function in `services/auth/source/oauth2/token.go`

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-07-19 09:57:10 +00:00
silverwind
88f835192d
Replace interface{} with any (#25686)
Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`.

Basically the same [as golang did](2580d0e08d).
2023-07-04 18:36:08 +00:00
KN4CK3R
c890454769
Add direct serving of package content (#25543)
Fixes #24723

Direct serving of content aka HTTP redirect is not mentioned in any of
the package registry specs but lots of official registries do that so it
should be supported by the usual clients.
2023-07-03 15:33:28 +02:00
KN4CK3R
cdb088cec2
Add CRAN package registry (#22331)
This PR adds a [CRAN](https://cran.r-project.org/) package registry.

![grafik](https://user-images.githubusercontent.com/1666336/210450039-d6fa6f77-20cd-4741-89a8-1624def267f7.png)
2023-05-22 10:57:49 +08:00
KN4CK3R
5968c63a11
Add Go package registry (#24687)
Fixes #7608

This PR adds a Go package registry usable with the Go proxy protocol.

![grafik](https://github.com/go-gitea/gitea/assets/1666336/328feb5c-3df2-4f9d-8eae-fe3126d14c37)
2023-05-14 23:38:40 +08:00
KN4CK3R
9173e079ae
Add Alpine package registry (#23714)
This PR adds an Alpine package registry. You can follow [this
tutorial](https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package)
to build a *.apk package for testing.

This functionality is similar to the Debian registry (#22854) and
therefore shares some methods. I marked this PR as blocked because it
should be merged after #22854.


![grafik](https://user-images.githubusercontent.com/1666336/227779595-b76163aa-eea1-4a79-9583-775c24ad74e8.png)

---------

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-12 17:27:50 +00:00
KN4CK3R
05209f0d1d
Add RPM registry (#23380)
Fixes #20751

This PR adds a RPM package registry. You can follow [this
tutorial](https://opensource.com/article/18/9/how-build-rpm-packages) to
build a *.rpm package for testing.

This functionality is similar to the Debian registry (#22854) and
therefore shares some methods. I marked this PR as blocked because it
should be merged after #22854.


![grafik](https://user-images.githubusercontent.com/1666336/223806549-d8784fd9-9d79-46a2-9ae2-f038594f636a.png)
2023-05-05 20:33:37 +00:00
KN4CK3R
723598b803
Implement Cargo HTTP index (#24452)
This implements the HTTP index
[RFC](https://rust-lang.github.io/rfcs/2789-sparse-index.html) for Cargo
registries.

Currently this is a preview feature and you need to use the nightly of
`cargo`:

`cargo +nightly -Z sparse-registry update`

See https://github.com/rust-lang/cargo/issues/9069 for more information.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-03 16:58:43 -04:00
KN4CK3R
bf999e4069
Add Debian package registry (#24426)
Co-authored-by: @awkwardbunny

This PR adds a Debian package registry.
You can follow [this
tutorial](https://www.baeldung.com/linux/create-debian-package) to build
a *.deb package for testing.
Source packages are not supported at the moment and I did not find
documentation of the architecture "all" and how these packages should be
treated.


![grafik](https://user-images.githubusercontent.com/1666336/218126879-eb80a866-775c-4c8e-8529-5797203a64e6.png)

Part of #20751.

Revised copy of #22854.

---------

Co-authored-by: Brian Hong <brian@hongs.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-02 12:31:35 -04:00
Yarden Shoham
c0ddec8a2a
Revert "Add Debian package registry" (#24412)
Reverts go-gitea/gitea#22854
2023-04-28 18:06:41 -04:00