samizdat issueshttps://0xacab.org/rysiek/samizdat/-/issues2020-10-05T17:55:30Zhttps://0xacab.org/rysiek/samizdat/-/issues/69Plugin and ServiceWorker config in a separate config file2020-10-05T17:55:30ZMichał "rysiek" WoźniakPlugin and ServiceWorker config in a separate config fileSince the ServiceWorker can import other files (which it does for plugins), it can also import a configuration file that sets everything up, making deployment way easier (no need to edit code).Since the ServiceWorker can import other files (which it does for plugins), it can also import a configuration file that sets everything up, making deployment way easier (no need to edit code).BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/40Stop relying on pubkey in gun+ipfs plugin2020-10-05T17:53:41ZMichał "rysiek" WoźniakStop relying on pubkey in gun+ipfs pluginWe need a better strategy of getting the Gun user pubkey to use in the `gun+ipfs` plugin.We need a better strategy of getting the Gun user pubkey to use in the `gun+ipfs` plugin.BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/71Sanity in console output2020-10-05T02:24:59ZMichał "rysiek" WoźniakSanity in console outputSamizdat's console output has become verbose and hard to follow enough as to make debugging difficult. We can do better.
Ideas:
- debug levels configurable per-plugin via `SamizdatConfig` (now that we [have it](#69))
- using [`console...Samizdat's console output has become verbose and hard to follow enough as to make debugging difficult. We can do better.
Ideas:
- debug levels configurable per-plugin via `SamizdatConfig` (now that we [have it](#69))
- using [`console.debug`](https://developer.mozilla.org/en-US/docs/Web/API/Console/debug) instead of `console.log`
- using [console groups](https://developer.mozilla.org/en-US/docs/Web/API/console#Using_groups_in_the_console)BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/64Gateway IPNS plugin improvements2020-10-04T00:00:13ZMichał "rysiek" WoźniakGateway IPNS plugin improvementsThe plugin is currently very naively implemented. Needed improvements:
- [ ] verify the hash of downloaded data (relevant: [`ipfs-only-hash`](https://www.npmjs.com/package/ipfs-only-hash))
- [x] use 2-3 gateways at once for a single re...The plugin is currently very naively implemented. Needed improvements:
- [ ] verify the hash of downloaded data (relevant: [`ipfs-only-hash`](https://www.npmjs.com/package/ipfs-only-hash))
- [x] use 2-3 gateways at once for a single resource (gateways are slow, whichever resolves first wins!)
- ~~add a reasonable timeout so that another plugin can pick the ball if it gets dropped by `gateway-ipns`~~
*we're implementing global timeouts in the ServiceWorker instead (#65)*BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/70Configurable plugin set and strategy2020-09-30T02:29:56ZMichał "rysiek" WoźniakConfigurable plugin set and strategySince we already have a configuration system for the Service Worker (#69), we can implement a way to configure:
- [x] what plugins are loaded
- [x] what is the content retrieval strategy (that is, what order are the plugins being deplo...Since we already have a configuration system for the Service Worker (#69), we can implement a way to configure:
- [x] what plugins are loaded
- [x] what is the content retrieval strategy (that is, what order are the plugins being deployed to fetch the content)
- [x] simple order (expressed as an array)
- [x] support for more complex structures with steps involving "any of these plugins" approaches (expressed as an array inside the simple ordered array, perhaps?)BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/67Improve/standardize/clarify language around transports, plugins, etc.2020-09-21T09:22:24ZMichał "rysiek" WoźniakImprove/standardize/clarify language around transports, plugins, etc.From Doc Edward Morbius [over at the Fediverse](https://mastodon.social/web/statuses/104899337763392473):
>>>
So, suggested clarifications (I'm not sure which are accurate):
or any HTTP(S) site able to host the content
or one or more ...From Doc Edward Morbius [over at the Fediverse](https://mastodon.social/web/statuses/104899337763392473):
>>>
So, suggested clarifications (I'm not sure which are accurate):
or any HTTP(S) site able to host the content
or one or more HTTP(S) sites also hosting the content
or one or more previously established HTTP(S) sites also hosting the content
or one or more previously established HTTPS sites which will dynamically and automatically rehost the content
or one or more previously established sites which will dynamically and automatically rehost the content using
or any any of a mesh netwoork of sites configured to dynamically host the content
Questions being:
What site(s) can / will host content?
What protocol(s) are used? HTTP(S)? Any of a set of TCP data transports? UDP? TLS-capable only? Tor? IPFS is apparently available.
Are these preconfigured and prepopulated? Preconfigured and dynamically populated? Dynamically configured?
Is the user at all aware of alternate sourcing? Are there redirects, or is this all ServiceWorker majick? (I have NFC about Serviceworker stuff, even after scanning docs).
Existing language is ... opaque.
>>>BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/54Document possible privacy gains from using Samizdat2020-09-21T01:19:52ZMichał "rysiek" WoźniakDocument possible privacy gains from using SamizdatSamizdat can be used not just to circumvent Internet censorship, but also to provide some (limited but perhaps useful) protection to website visitors, if the ServiceWorker is configured such that it does *not* run the direct `fetch()` pl...Samizdat can be used not just to circumvent Internet censorship, but also to provide some (limited but perhaps useful) protection to website visitors, if the ServiceWorker is configured such that it does *not* run the direct `fetch()` plugin at all, relying instead on cache (if present) and on other plugins.
This would mean that after the visitor visits the Samizdat-enabled site, all requests avoid sing `fetch()` and thus (depending on plugins used) potentially completely obscure the fact that the person is trying to visit the blocked site.Michał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/17More advanced handling of blocked content, timeouts, and user impatience2020-09-19T20:22:26ZMichał "rysiek" WoźniakMore advanced handling of blocked content, timeouts, and user impatienceI the long term we need a more advanced way of handling blocked content and long delays when fetching content from decentralized sources.
Ideas include:
- get content from cache immediately (if available), display a message to the user...I the long term we need a more advanced way of handling blocked content and long delays when fetching content from decentralized sources.
Ideas include:
- get content from cache immediately (if available), display a message to the user allowing them reloading from a decentralized source
- set a timeout to a reasonable delay and if it hits display minimal content informing the user they need to wait a bit more, and proceeding to fetch content from decentralised sources
This is definitely for much later, not in the PoC phase.BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/65Reasonable timeout for `plugin.fetch()` in the ServiceWorker2020-09-19T19:05:34ZMichał "rysiek" WoźniakReasonable timeout for `plugin.fetch()` in the ServiceWorkerCurrently the ServiceWorker just runs any plugin (including the initial HTTPS `fetch()`) without any explicit timeouts. This is sub-optimal, for example when:
- the site is censored by `DROP`ing the packets on some firewall (instead of ...Currently the ServiceWorker just runs any plugin (including the initial HTTPS `fetch()`) without any explicit timeouts. This is sub-optimal, for example when:
- the site is censored by `DROP`ing the packets on some firewall (instead of rejecting them);
- the site is experiencing technical issues and effectively takes forever to load;
- any of the plugins (looking at you, `IPFS`) takes inordinate amount of time to return either the content, or an error.
We need to be able to wait for some time, but then just go with another plugin, before the browser times out on us.BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/62Gun 0.2020.520 not usable2020-09-15T01:55:08ZMichał "rysiek" WoźniakGun 0.2020.520 not usableJob [#164219](https://0xacab.org/rysiek/samizdat/-/jobs/164219) failed for 859229f8f196d789d7328630c14ee9f02472f9f3: this might be related to the recent Gun upgrade, or general Gun unreliability (ref. #2).Job [#164219](https://0xacab.org/rysiek/samizdat/-/jobs/164219) failed for 859229f8f196d789d7328630c14ee9f02472f9f3: this might be related to the recent Gun upgrade, or general Gun unreliability (ref. #2).https://0xacab.org/rysiek/samizdat/-/issues/61Update dependencies2020-09-15T01:25:02ZMichał "rysiek" WoźniakUpdate dependenciesWe're using [an older version of Gun (`0.2020.116`)](https://0xacab.org/rysiek/samizdat/-/blob/688887a0a2c547093341cabdfcf8d24c67234d2c/samizdat-cli/package.json#L13) than [the latest available (`0.2020.520`)](https://www.npmjs.com/packa...We're using [an older version of Gun (`0.2020.116`)](https://0xacab.org/rysiek/samizdat/-/blob/688887a0a2c547093341cabdfcf8d24c67234d2c/samizdat-cli/package.json#L13) than [the latest available (`0.2020.520`)](https://www.npmjs.com/package/gun). Same is probably also true for [SEA, IPFS, and WebRTC](https://0xacab.org/rysiek/samizdat/-/tree/master/lib).
Time for some upgrades.
- ~~Gun~~
- ~~SEA~~
- [x] `go-ipfs` `0.4.22` -> `0.6.0`
- [x] [`ipfs.js`](https://js.ipfs.io/) `0.40.0` -> `0.50.2`
- [x] [`is-ipfs`](https://www.npmjs.com/package/is-ipfs) `0.6.3` -> `2.0.0`
- ~~WebRTC~~BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/58Fix HTTP 504 errors in the pipeline2020-09-13T15:57:38ZMichał "rysiek" WoźniakFix HTTP 504 errors in the pipelineWe tend to [get some `HTTP 504` error codes](https://0xacab.org/rysiek/samizdat/-/jobs/163615#L211) in the pipeline. This is *probably* because we're hitting one particular `IPFS` gateway *a lot* with each pipeline run.
One solution wou...We tend to [get some `HTTP 504` error codes](https://0xacab.org/rysiek/samizdat/-/jobs/163615#L211) in the pipeline. This is *probably* because we're hitting one particular `IPFS` gateway *a lot* with each pipeline run.
One solution would be to use a [few different gateways](https://ipfs.github.io/public-gateway-checker/) and spread the load. This would also allow us to hit another gateway in case we experience a problem from one of them, and thus checking if it's a gateway problem, or problem with the actual content.BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/60Improve npm use in CI/CD pipelines2020-09-10T00:57:14ZMichał "rysiek" WoźniakImprove npm use in CI/CD pipelinesWe should be:
- [x] using [`npm ci`](https://docs.npmjs.com/cli/ci.html) instead of `npm install`
- [x] [enable caching](https://docs.gitlab.com/ee/ci/caching/#caching-nodejs-dependencies) to speed up CI/CD pipelinesWe should be:
- [x] using [`npm ci`](https://docs.npmjs.com/cli/ci.html) instead of `npm install`
- [x] [enable caching](https://docs.gitlab.com/ee/ci/caching/#caching-nodejs-dependencies) to speed up CI/CD pipelinesMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/59CLI: --peer option for setting the peer(s) explicitly2020-09-08T23:46:11ZMichał "rysiek" WoźniakCLI: --peer option for setting the peer(s) explicitly`samizdat-cli` needs an `--peer` option for setting the Gun peer(s) explicitly. This is necessary for [local testing]( #56 ), for example.`samizdat-cli` needs an `--peer` option for setting the Gun peer(s) explicitly. This is necessary for [local testing]( #56 ), for example.BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/34Gun user management in samizdat-cli2020-09-08T18:29:58ZMichał "rysiek" WoźniakGun user management in samizdat-cliWe need decent Gun user management in `samizdat-cli`. Specifically:
1. [x] user registration
1. [x] checking if a user exists
1. [x] getting a user's pubkey
1. [x] user deletion
The last two *could* be combined into a single function, ...We need decent Gun user management in `samizdat-cli`. Specifically:
1. [x] user registration
1. [x] checking if a user exists
1. [x] getting a user's pubkey
1. [x] user deletion
The last two *could* be combined into a single function, returning the pubkey if the user exists, and exiting with an error otherwise.BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/47Job Failed #153491: Gun is causing problems again2020-07-12T01:02:15ZMichał "rysiek" WoźniakJob Failed #153491: Gun is causing problems againJob [#153491](https://0xacab.org/rysiek/samizdat/-/jobs/153491) failed for 589ada5dea21088dd65b745ff04f436c2b175346: Gun seems to be causing problems.Job [#153491](https://0xacab.org/rysiek/samizdat/-/jobs/153491) failed for 589ada5dea21088dd65b745ff04f436c2b175346: Gun seems to be causing problems.Michał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/44Display the Mastodon #Samizdat hashtag on samizdat.is2020-02-10T21:33:39ZMichał "rysiek" WoźniakDisplay the Mastodon #Samizdat hashtag on samizdat.isBecause why not.Because why not.Michał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/43Make Samizdat messages disappear automagically2020-02-02T01:11:38ZMichał "rysiek" WoźniakMake Samizdat messages disappear automagicallyCurrently Samizdat messages do not disappear automagically. That's annoying. A sane (5s?) timeout would be in order.Currently Samizdat messages do not disappear automagically. That's annoying. A sane (5s?) timeout would be in order.BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/42Inform the user when background content fetching finishes (but no retrieved c...2020-02-02T01:11:38ZMichał "rysiek" WoźniakInform the user when background content fetching finishes (but no retrieved content is new)Currently Samizdat displays two messages to the user:
- "`Some content seems blocked or unavailable. Attempting to retrieve it via Samizdat`"
when a regular HTTPS `fetch()` fails and plugins get deployed in the background
- "`New...Currently Samizdat displays two messages to the user:
- "`Some content seems blocked or unavailable. Attempting to retrieve it via Samizdat`"
when a regular HTTPS `fetch()` fails and plugins get deployed in the background
- "`Newer version of this page is available; please reload to see it.`"
when all background plugins finish and there is content retrieved that is deemed "fresher" than what was found in cache
For completeness, we need a message displayed when background fetching is done but content is not deemed fresher.BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/33Stop relying on OCCRP's test Gun instance2020-01-31T19:43:51ZMichał "rysiek" WoźniakStop relying on OCCRP's test Gun instanceWe're relying heavily on OCCRP's test Gun instance (`http://cdn.test.occrp.org:8080/gun`); it's all over the code. We should *probably* spin up our own.We're relying heavily on OCCRP's test Gun instance (`http://cdn.test.occrp.org:8080/gun`); it's all over the code. We should *probably* spin up our own.BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniak