- Databases are one of the most important parts of Forgejo, every
interaction with Forgejo uses the database in one way or another.
Therefore, it is important to maintain the database and recognize when
Forgejo is not doing well with the database. Forgejo already has the
option to log *every* SQL query along with its execution time, but
monitoring becomes impractical for larger instances and takes up
unnecessary storage in the logs.
- Add a QoL enhancement that allows instance administrators to specify a
threshold value beyond which query execution time is logged as a warning
in the xorm logger. The default value is a conservative five seconds to
avoid this becoming a source of spam in the logs.
- The use case for this patch is that with an instance the size of Codeberg, monitoring SQL logs is not very fruitful and most of them are uninteresting. Recently, in the context of persistent deadlock issues (https://codeberg.org/forgejo/forgejo/issues/220), I have noticed that certain queries hold locks on tables like comment and issue for several seconds. This patch helps to identify which queries these are and when they happen.
- Added unit test.
(cherry picked from commit 24bbe7886fb4cb9a38c8dab8c44f4c9cbfa25481)
(cherry picked from commit 6e29145b3c1455498531593d38e6a914941a12cb)
(cherry picked from commit 63731e30712872bd2395eb3cf36d9996e5793645)
(cherry picked from commit 3ce1a097369c132654de70df707b867e47bd1c40)
(cherry picked from commit a64426907de788cc0937a7a2b16af4d2f26f7fe6)
(cherry picked from commit 4b1921569156445c58d9889602733da5934c7b95)
(cherry picked from commit e6356744359fa947c049827d60c2ea0e277e03dc)
(cherry picked from commit 9cf501f1af4cd870221cef6af489618785b71186)
(cherry picked from commit 0d6b934eba1c0e9b27b364791113aae816b6b366)
(cherry picked from commit 4b6c2738795002887844a106f2fed2ef1673eed1)
(cherry picked from commit b50517139cc62f214c1629ef2fd9bcaa37b46202)
(cherry picked from commit 6546dd1fc946e620a02b6d1afed7d5ac50655fa8)
(cherry picked from commit 3eda6890e6b840237f675a2873c102a6fc86b8f1)
[GITEA] Add slow SQL query warning (squash) document the setting
(cherry picked from commit ce38599c5141c7fc6bc054819f5ff1c1b45bda1f)
(cherry picked from commit 794aa67c68c8e24ac7301eb7ef767c6e2499a78d)
(cherry picked from commit 8227673deb1b93015f56e446b27c52a0013eba29)
(cherry picked from commit 8854d1d4dda72304f43ea8aa61a941701a5deede)
(cherry picked from commit 9121a0e21f7c5de89ba2d8afa9054c5fbf210fb1)
(cherry picked from commit 41bae2e425)
Replaces `Gitea` with `Forgejo` in the default config settings for new installs.
This will not affect existing installs.
Co-authored-by: Caesar Schinas <caesar@caesarschinas.com>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/140
Co-authored-by: Caesar Schinas <caesar@noreply.codeberg.org>
Co-committed-by: Caesar Schinas <caesar@noreply.codeberg.org>
(cherry picked from commit ca1319aa16128516e50dabdc8e8cadc23eb71d2f)
(cherry picked from commit 52a4d238a0b5bbea28b28e55e35f28c51ecbb2de)
(cherry picked from commit f63536538cfe4b010ebb5a6323f4b5e5b6ec7232)
Conflicts:
web_src/js/features/install.js
(cherry picked from commit 861cc434e129f3fbd932ee36067c560e754dab9a)
(cherry picked from commit 0e6ea60c802d6cfd95dea4aad5df96bb6e4bc7a5)
(cherry picked from commit 0cbc0ec15d9d952b0ecfb68a36bf58cbae0c43dd)
(cherry picked from commit 3cc19b0ae214ae163f23efa52ab0aa53eb7c935b)
(cherry picked from commit 50fcb885fe2f87a27e6ff778cdd0c7cd97bbe9e9)
(cherry picked from commit f6039d4df481fc58b8db3e863158665d24cac847)
(cherry picked from commit 5ae5c6ba2deefe829f768900f5e1bbcbe9389e15)
(cherry picked from commit f0b565e0ed7fe52f0264e92c49736b487b9bff8a)
(cherry picked from commit adbd4d2015e8e2c4789849c84cfa4032acd40b7f)
(cherry picked from commit d26c540ffdbfb7ec83412635576ab39653d1b531)
(cherry picked from commit 6df6781b42ea126d029ba9e85485dbcb9bf6601d)
(cherry picked from commit b6fb56e1c407195bedfe09f91ecb6537024d5189)
(cherry picked from commit bb4f98a0ca7515aa6c44529df0573195f779a643)
(cherry picked from commit 6779229f2732f6791aba3bd1ba51a338a88ca1f3)
(cherry picked from commit c216c85aee552aa15e9d6316002ee9e170de64d7)
(cherry picked from commit dff780bced1dc78ddd7fa4952766969ee528c538)
(cherry picked from commit 4e036aa3b6d1bc5f2041c3b30ec289cb082fe824)
(cherry picked from commit 8b3bc3e8a64edaba64cb759ea31248eea7307937)
(cherry picked from commit 1e4d8523321efaf6c5febdd77ba5150effe5c1ac)
(cherry picked from commit 07a15d18447bb03bb04001f1f65305670d1467ef)
(cherry picked from commit fb44b3e10d685dd180f37678a3e2a64ee641f2b2)
(cherry picked from commit b212d833190ce59230b3fc288c1aac5106cf33a0)
(cherry picked from commit 5754971be5c37e97f3165878af3760117e40af01)
(cherry picked from commit 0c43b4e82cab028fae1b709c549251d63a3ffc04)
Conflicts:
routers/install/install.go
https://codeberg.org/forgejo/forgejo/pulls/1351
(cherry picked from commit 2e22a7208a0f149afc3794425c7b5b2b0181a939)
(cherry picked from commit 676b0a8a48f99d751b677b9eca5b57238b5003a5)
(cherry picked from commit bc4a8bf9bcb75e92b7802c477b1bb14a047344a6)
(cherry picked from commit 5e09a4e174cc6d62b08fa718d0309275effadbd8)
(cherry picked from commit 712c52a32a3b6ce9aafaefaf9e63729522d1e0c5)
(cherry picked from commit ba3d93cc4a5dca5d0daef647fab557e0136925e2)
(cherry picked from commit a5a0396abc512834ca7b0b247277e4d753a87659)
(cherry picked from commit c1f389f0cb237dcce05db4ade342d48e260ae144)
(cherry picked from commit 87d4d2da9faf9cacdd6f0441914c806efe4d1e36)
(cherry picked from commit 89d7559054c8d99d1eb4bb151f3514cbb865e82f)
(cherry picked from commit 7698cc0c8553d0135d41344ece12f18ad8fc15aa)
(cherry picked from commit 69e24c60e1)
Backport #27723 by @mpldr
This patchset changes the connection string builder to use net.URL and
the host/port parser to use the stdlib function for splitting host from
port. It also adds a footnote about a potentially required portnumber
for postgres UNIX sockets.
Fixes: #24552
Co-authored-by: Moritz Poldrack <33086936+mpldr@users.noreply.github.com>
In modern days, there is no reason to make users set "charset" anymore.
Close#25378
## ⚠️ BREAKING
The key `[database].CHARSET` was removed completely as every newer
(>10years) MySQL database supports `utf8mb4` already.
There is a (deliberately) undocumented new fallback option if anyone
still needs to use it, but we don't recommend using it as it simply
causes problems.
## ⚠️ Breaking
The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.
Although many legacy options still work, it's encouraged to upgrade to
the new options.
The SMTP logger is deleted because SMTP is not suitable to collect logs.
If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.
## Description
Close#12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)
Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)
There is a new document (with examples): `logging-config.en-us.md`
This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.
## The old problems
The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.
## The new design
See `logger.go` for documents.
## Screenshot
<details>
</details>
## TODO
* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
Creating pid file should not belong to setting package and only web
command needs that. So this PR moves pidfile creation from setting
package to web command package to keep setting package more readable.
I marked this as `break` because the PIDFile path moved. For those who
have used the pid build argument, it has to be changed.
---------
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Replace #23350.
Refactor `setting.Database.UseMySQL` to
`setting.Database.Type.IsMySQL()`.
To avoid mismatching between `Type` and `UseXXX`.
This refactor can fix the bug mentioned in #23350, so it should be
backported.
Some bugs caused by less unit tests in fundamental packages. This PR
refactor `setting` package so that create a unit test will be easier
than before.
- All `LoadFromXXX` files has been splited as two functions, one is
`InitProviderFromXXX` and `LoadCommonSettings`. The first functions will
only include the code to create or new a ini file. The second function
will load common settings.
- It also renames all functions in setting from `newXXXService` to
`loadXXXSetting` or `loadXXXFrom` to make the function name less
confusing.
- Move `XORMLog` to `SQLLog` because it's a better name for that.
Maybe we should finally move these `loadXXXSetting` into the `XXXInit`
function? Any idea?
---------
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
Gitea will migrate the database model version automatically, but it
should be able to be disabled and keep Gitea shutdown if the version is
not matched.
Change all license headers to comply with REUSE specification.
Fix#16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Use default values when provided values are empty
- When provided values are empty like `:3000` would imply that host is
empty, use the default value.
- Resolves#20316
* Update database.go
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Refactor install page (db type)
* set correct default DB HOST for different DB TYPE
* remove legacy TiDB from documents
* unify the usage of DB TYPE, in code we only use "mysql". "MySQL" is only shown to users for friendly name.
* Gitea can use TiDB via MySQL protocol
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
One perennial issue is users running the install page,
changing the database dialect and then suffering with issues
This PR simply resets all of the database.Use flags on
initDBConfig. This should prevent this issue from occuring.
Fix#13788Fix#5480
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Avatar support minio
* Support repo avatar minio storage
* Add missing migration
* Fix bug
* Fix test
* Add test for minio store type on avatars and repo avatars; Add documents
* Fix bug
* Fix bug
* Add back missed avatar link method
* refactor codes
* Simplify the codes
* Code improvements
* Fix lint
* Fix test mysql
* Fix test mysql
* Fix test mysql
* Fix settings
* Fix test
* fix test
* Fix bug
MySQL in its infinite wisdom determines that UTF8 does not
mean UTF8. Our install scripts know about this and will set
CHARSET to utf8mb4 if we users choose this but... users who
do not explicitly set this variable will default to utf8mb3
without knowing it.
This PR changes the unset CHARSET value to utf8mb4 if users
choose to use mysql.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add support for database schema
* Require setting search_path for the db user
* Add schema setting to admin/config.tmpl
* Use a schema different from default for psql tests
* Update postgres scripts to use custom schema
* Update to xorm/core 0.7.3 and xorm/xorm c37aff9b3a
* Fix migration test
Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Expose db.SetMaxOpenConns and allow other dbs to set their connection params
* Add note about port exhaustion
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
