This is a long-needed test that tests the whole pipeline from Nginx, Relay, to Kafka, and Snuba. The final missing piece is testing the Symbolicator integration.
This PR is also a follow up to #576 as it didn't solve the Relay issues fully (the earlier fix was a coincidence or is not as reliable as it seemed).
Fixes#486 (finally?).
This patch adds `INTERNAL_IPS` definition to `sentry.conf.py` by sniffing the network from eth0 and relies on this for trusted Relays instead of the ALLOWLISTED PKs. This removes the necessity of syncing Relay PKs to `sentry.conf.py`.
This PR needs getsentry/sentry#19798 to work.
I would like to be able to customize the configuration for my Sentry 10 symbolicator instance, which this change allows me to easily do.
See related: https://github.com/getsentry/symbolicator/issues/245
Co-authored-by: Burak Yigit Kaya <ben@byk.im>
Hi,
I've been through quite a few different ways of implementing this fix and settled on creating a variable to store the output of checking whether the zookeeper copy target folder exists and copying the snapshot file based on the copy target folder existing. I've ran quite a few manual tests for each option as well. Currently the PR sits on Option 3 from the below options.
**Option 1**
Judging from the [Jira issue](https://issues.apache.org/jira/browse/ZOOKEEPER-3056), it seems like the work around for zookeeper upgrades could be omitted entirely since the issue relates to upgrades from v3.4.10 to v3.5.4. I've tested removing the zookeeper workaround entirely and that install ran smoothly on a clean install of Sentry (no existing data) as well as an install of Sentry that currently has very minimal amount of log entries (roughly 100 log entries). Could we possibly remove the workaround entirely?
**Option 2**
The second option was to simply add a check to the currently [existing line](https://github.com/getsentry/onpremise/blob/master/install.sh#L178) of whether the copy target folder exists and perform the snapshot file copy only if the copy target folder exists. This is the least amount of code and possibly the simpler fix while also setting the `ZOOKEEPER_SNAPSHOT_TRUST_EMPTY` env var to `true`, however, some unnecessary calculations will be done to determine the `ZOOKEEPER_LOG_FILE_COUNT` and `ZOOKEEPER_SNAPSHOT_FILE_COUNT`.
**Option 3**
I've created a variable to store whether the copy target folder exists and proceed with the zookeeper upgrade workaround only if the copy target folder exists. This means that if the copy target folder does not exist, the env var `ZOOKEEPER_SNAPSHOT_TRUST_EMPTY` will not be set.
Fixes#547.
Co-authored-by: chamirb <chamirb@globalkinetic.com>
Tested this in a Ubuntu VM. The output of `date` is not too pretty but at least localized (so D/M vs M/D is not confusing)
## What is the TSDB migration?
We're effectively deprecating all TSDB backends but `sentry.tsdb.redissnuba.RedisSnubaTSDB`. We cannot reasonably support any other backend due to the fact that we would have to reimplement each of the backends in Relay, which is written in a different language. Also, like with deprecating mysql support, we don't really have the capacity to support things we do not use ourselves.
## Migration
`install.sh` should rewrite your configuration automatically and define a cutover date such that no data is lost. Before the cutover date, data is written to two backends at once, Redis and Snuba, and read from one, Redis. After the cutover date, event-related metrics will be read from Snuba which matches what we have on sentry.io.
## Manual migration guide for TSDB
In case `install.sh` is unable to migrate your files you will be given basic instructions on the console that essentially tell you to completely delete all TSDB config and paste the new, standard one. If for some reason you cannot say goodbye to your existing TSDB config, please create a new issue in this repo and cc @untitaker on it.
This PR disables checking for the SSE4.2 flag, which may not be present if the server is running in a VM even if SSE4.2 is in fact available to use (ClickHouse/ClickHouse#20). The KVM check was obtained from a [comment](https://github.com/ClickHouse/ClickHouse/issues/20#issuecomment-226849297) in that issue.
If SSE4.2 is not actually available in the VM, then the installation script may fail in an unpredictable way. Perhaps we can add a keyword argument to the script to let the user decide if they want to skip the check.
This patch does two things:
1. Separate creating of Relay credentials from syncing them to Sentry config
2. Has a more flexible Relay credentials check and sync: look for the quoted public key in the config file, if it is there, assume this must be correctly set as it is very unlikely to have that random key in a different context with quotes around. The second one is to allow having other whitelisted relay keys by using an append method when adding the new key.
Update exit -1 to exit 1 in line with best practices
Fixes premature exit of the script when checking for sse42 support. The
script would exit before displaying the reason for failure to the user.
With getsentry/snuba#709 merged, we no longer need to randomly try forever creating Kafka topics, which were a prerequisite for creating Clickhouse tables. This patch also removes the preceding (and obsolete) `docker-compose up` statement which was used to "speed up" the `snuba bootstrap` command in the hopes to fix it.
Since we are [moving the eventstream into a proper db migration](https://github.com/getsentry/sentry/pull/16226), we need Snuba and its friends ready at the time of `sentry upgrade` command. This patch does exactly that.
Co-authored-by: Lyn Nagara <lyn.nagara@gmail.com>
This change makes the install script create a new install log file, `sentry_install_log-<ISO_TIMESTAMP>.txt`, for each run and records all the output there for future reference.
This adds a fully automated `install.sh` that does the following:
- Checks minimum Docker and `docker-compose` versions
- Checks minimum RAM available to Docker containers
- Removes potential user errors in install steps
