samizdat issueshttps://0xacab.org/rysiek/samizdat/-/issues2019-12-10T12:47:49Zhttps://0xacab.org/rysiek/samizdat/-/issues/28Consider using Snowflake as a transport plugin?2019-12-10T12:47:49ZMichał "rysiek" WoźniakConsider using Snowflake as a transport plugin?Might be relevant: https://trac.torproject.org/projects/tor/wiki/doc/SnowflakeMight be relevant: https://trac.torproject.org/projects/tor/wiki/doc/Snowflakehttps://0xacab.org/rysiek/samizdat/-/issues/78BitFrost + Samizdat?2021-08-24T20:45:30ZMichał "rysiek" WoźniakBitFrost + Samizdat?This seems very on-topic: https://www.qurium.org/bifrost/
Sounds like something that could be the backing-store for Samizdat. Or, put differently, sounds like something that could benefit from Samizdat automagically fetching data from B...This seems very on-topic: https://www.qurium.org/bifrost/
Sounds like something that could be the backing-store for Samizdat. Or, put differently, sounds like something that could benefit from Samizdat automagically fetching data from BotFrost if the origin site is blocked.Michał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/77Investigate JMAP as a potential transport plugin2021-04-06T17:40:09ZMichał "rysiek" WoźniakInvestigate JMAP as a potential transport plugin[JMAP](https://jmap.io/) is a thing, works in the browser, and could allow Samizdat to use JMAP-enabled mail hosts as transports. Question is, does anyone actually use it?..[JMAP](https://jmap.io/) is a thing, works in the browser, and could allow Samizdat to use JMAP-enabled mail hosts as transports. Question is, does anyone actually use it?..Michał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/76Other work in the area2020-12-29T22:58:24ZMichał "rysiek" WoźniakOther work in the areaThis issue will gather links to similar projects or related developments.This issue will gather links to similar projects or related developments.https://0xacab.org/rysiek/samizdat/-/issues/75Potential plugin: cjdns2020-10-19T23:35:24ZMichał "rysiek" WoźniakPotential plugin: cjdnsPotentially, [`cjdns`](https://github.com/cjdelisle/cjdns) could be an interesting transport plugin tech. Pros include public-key routing and an active large network of nodes ([Hyperboria](https://hyperboria.net/)).Potentially, [`cjdns`](https://github.com/cjdelisle/cjdns) could be an interesting transport plugin tech. Pros include public-key routing and an active large network of nodes ([Hyperboria](https://hyperboria.net/)).https://0xacab.org/rysiek/samizdat/-/issues/74Censorship-circumvention for the config2020-10-05T17:53:41ZMichał "rysiek" WoźniakCensorship-circumvention for the configCurrently, all config options are kept in `config.js` and imported directly in `service-worker.js`, which means it's considered a part of the Service Worker by the browser and only allowed to update via direct HTTPS `fetch()`.
We need t...Currently, all config options are kept in `config.js` and imported directly in `service-worker.js`, which means it's considered a part of the Service Worker by the browser and only allowed to update via direct HTTPS `fetch()`.
We need to be able to fetch updated config settings (by fetching `config.js` or via some other means) and update the Service Worker configuration using it. This probably requires going through the cache, and storing *specific config options* there.
Not all config options can be meaningfully changed this way, however. For example, the set of JS files (and thus, plugins) that is loaded by the Service Worker cannot change without a direct HTTPS `fetch()` (which is a limitation imposed by the API specs); if implemented right, their order could be different, however. This could create a situation where a new config requires loading of files that have not been loaded and breaks Samizdat for someone for whom the original site is blocked.
tl;dr this is trickyMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/73Make it possible to explicitly configure each plugin when loading2020-09-29T22:37:11ZMichał "rysiek" WoźniakMake it possible to explicitly configure each plugin when loadingCurrently a plugin file gets loaded and a plugin immediately instantiated, with a hard-coded `SamizdatConfig` path (based on plugin name) used for config. This needs to change.
This is relevant for #72Currently a plugin file gets loaded and a plugin immediately instantiated, with a hard-coded `SamizdatConfig` path (based on plugin name) used for config. This needs to change.
This is relevant for #72Michał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/72Allow loading the same plugin multiple times with different config2020-10-04T00:48:07ZMichał "rysiek" WoźniakAllow loading the same plugin multiple times with different configWe need a way to load the same plugin multiple times with different configs. For instance, we might want to try using a few different IPNS identities, or different HTTPS gateways, etc.
This is also relevant for #70.We need a way to load the same plugin multiple times with different configs. For instance, we might want to try using a few different IPNS identities, or different HTTPS gateways, etc.
This is also relevant for #70.Michał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/71Sanity in console output2020-10-05T02:24:59ZMichał "rysiek" WoźniakSanity in console outputSamizdat's console output has become verbose and hard to follow enough as to make debugging difficult. We can do better.
Ideas:
- debug levels configurable per-plugin via `SamizdatConfig` (now that we [have it](#69))
- using [`console...Samizdat's console output has become verbose and hard to follow enough as to make debugging difficult. We can do better.
Ideas:
- debug levels configurable per-plugin via `SamizdatConfig` (now that we [have it](#69))
- using [`console.debug`](https://developer.mozilla.org/en-US/docs/Web/API/Console/debug) instead of `console.log`
- using [console groups](https://developer.mozilla.org/en-US/docs/Web/API/console#Using_groups_in_the_console)BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/70Configurable plugin set and strategy2020-09-30T02:29:56ZMichał "rysiek" WoźniakConfigurable plugin set and strategySince we already have a configuration system for the Service Worker (#69), we can implement a way to configure:
- [x] what plugins are loaded
- [x] what is the content retrieval strategy (that is, what order are the plugins being deplo...Since we already have a configuration system for the Service Worker (#69), we can implement a way to configure:
- [x] what plugins are loaded
- [x] what is the content retrieval strategy (that is, what order are the plugins being deployed to fetch the content)
- [x] simple order (expressed as an array)
- [x] support for more complex structures with steps involving "any of these plugins" approaches (expressed as an array inside the simple ordered array, perhaps?)BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/69Plugin and ServiceWorker config in a separate config file2020-10-05T17:55:30ZMichał "rysiek" WoźniakPlugin and ServiceWorker config in a separate config fileSince the ServiceWorker can import other files (which it does for plugins), it can also import a configuration file that sets everything up, making deployment way easier (no need to edit code).Since the ServiceWorker can import other files (which it does for plugins), it can also import a configuration file that sets everything up, making deployment way easier (no need to edit code).BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/68Split the CI/CD pipeline into separate stages/jobs2020-09-21T23:57:57ZMichał "rysiek" WoźniakSplit the CI/CD pipeline into separate stages/jobsCurrently we have the whole of deployment and publishing as a single job. That's not helping as far as fixing issues is concerned (debugging a problem somewhere down the line requires a full run).
Separating it into jobs also means thin...Currently we have the whole of deployment and publishing as a single job. That's not helping as far as fixing issues is concerned (debugging a problem somewhere down the line requires a full run).
Separating it into jobs also means things could happen simultaneously (for instance, Gun and IPNS updates).BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/67Improve/standardize/clarify language around transports, plugins, etc.2020-09-21T09:22:24ZMichał "rysiek" WoźniakImprove/standardize/clarify language around transports, plugins, etc.From Doc Edward Morbius [over at the Fediverse](https://mastodon.social/web/statuses/104899337763392473):
>>>
So, suggested clarifications (I'm not sure which are accurate):
or any HTTP(S) site able to host the content
or one or more ...From Doc Edward Morbius [over at the Fediverse](https://mastodon.social/web/statuses/104899337763392473):
>>>
So, suggested clarifications (I'm not sure which are accurate):
or any HTTP(S) site able to host the content
or one or more HTTP(S) sites also hosting the content
or one or more previously established HTTP(S) sites also hosting the content
or one or more previously established HTTPS sites which will dynamically and automatically rehost the content
or one or more previously established sites which will dynamically and automatically rehost the content using
or any any of a mesh netwoork of sites configured to dynamically host the content
Questions being:
What site(s) can / will host content?
What protocol(s) are used? HTTP(S)? Any of a set of TCP data transports? UDP? TLS-capable only? Tor? IPFS is apparently available.
Are these preconfigured and prepopulated? Preconfigured and dynamically populated? Dynamically configured?
Is the user at all aware of alternate sourcing? Are there redirects, or is this all ServiceWorker majick? (I have NFC about Serviceworker stuff, even after scanning docs).
Existing language is ... opaque.
>>>BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/66Consider implementing means of detecting content deployed by an adversary2020-09-23T02:04:06ZMichał "rysiek" WoźniakConsider implementing means of detecting content deployed by an adversaryCurrently Samizdat is unable to notice if, for whatever reason, content returned by any transport plugin (`fetch`, or any other that does not inherently provide end-to-end verification of content) is has been maliciously modified.
**Sc...Currently Samizdat is unable to notice if, for whatever reason, content returned by any transport plugin (`fetch`, or any other that does not inherently provide end-to-end verification of content) is has been maliciously modified.
**Scenario 1:**
1. Website is deployed with Samizdat in the current default configuration (`fetch`->`cache`->`gun+ipfs`).
1. An adversary takes over the original domain, and deploys a new SSL certificate
1. The adversary then deploys their own versions of some content (`index.html`, for example)
1. When a user visits the site, `fetch` succeeds and the adversary-controlled `index.html` is displayed; alternative transports are not ever used for that file.
**Scenario 2:**
1. Website is deployed with Samizdat configured to pull content from Google Drive as an alternative endpoint, in case original website is unavailable.
1. An adversary gains access to the Google Drive folder by whatever means and modifies content.
1. Then, the adversary blocks the original domain.
1. Content modified by the adversary is now served to users.
This *could* be mitigated to some extent with some for of content signing, at least for HTML/CSS/JS, but at a cost of added complexity. Perhaps it could be implemented as an optional plugin, which would wrap any other plugin, and verify the content signature against a known public key of some sort. If the signature does not match (or is absent), throw an error.
The signature could be added as a comment in the last line of text-based files, for example. Headers won't work, since in case of most plugins there is no way to control the headers.https://0xacab.org/rysiek/samizdat/-/issues/65Reasonable timeout for `plugin.fetch()` in the ServiceWorker2020-09-19T19:05:34ZMichał "rysiek" WoźniakReasonable timeout for `plugin.fetch()` in the ServiceWorkerCurrently the ServiceWorker just runs any plugin (including the initial HTTPS `fetch()`) without any explicit timeouts. This is sub-optimal, for example when:
- the site is censored by `DROP`ing the packets on some firewall (instead of ...Currently the ServiceWorker just runs any plugin (including the initial HTTPS `fetch()`) without any explicit timeouts. This is sub-optimal, for example when:
- the site is censored by `DROP`ing the packets on some firewall (instead of rejecting them);
- the site is experiencing technical issues and effectively takes forever to load;
- any of the plugins (looking at you, `IPFS`) takes inordinate amount of time to return either the content, or an error.
We need to be able to wait for some time, but then just go with another plugin, before the browser times out on us.BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/64Gateway IPNS plugin improvements2020-10-04T00:00:13ZMichał "rysiek" WoźniakGateway IPNS plugin improvementsThe plugin is currently very naively implemented. Needed improvements:
- [ ] verify the hash of downloaded data (relevant: [`ipfs-only-hash`](https://www.npmjs.com/package/ipfs-only-hash))
- [x] use 2-3 gateways at once for a single re...The plugin is currently very naively implemented. Needed improvements:
- [ ] verify the hash of downloaded data (relevant: [`ipfs-only-hash`](https://www.npmjs.com/package/ipfs-only-hash))
- [x] use 2-3 gateways at once for a single resource (gateways are slow, whichever resolves first wins!)
- ~~add a reasonable timeout so that another plugin can pick the ball if it gets dropped by `gateway-ipns`~~
*we're implementing global timeouts in the ServiceWorker instead (#65)*BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/63Pure Gun transport plugin?2020-09-16T23:18:08ZMichał "rysiek" WoźniakPure Gun transport plugin?Seems like it should be possible to have a pure Gun transport plugin:
>>>
Michał Woźniak
@rysiekpl_gitlab
Sep 15 22:43
>> @rysiekpl_gitlab I've synced even video files as big as 30MB, and part of the goal of the new version is to see ho...Seems like it should be possible to have a pure Gun transport plugin:
>>>
Michał Woźniak
@rysiekpl_gitlab
Sep 15 22:43
>> @rysiekpl_gitlab I've synced even video files as big as 30MB, and part of the goal of the new version is to see how far I can push this limit ( @QVDev at one point was near live-streaming video across 4G) if you wanna help with the Infinity Scroll test... once that is passing I want to apply it to video.
>
> Right. When you say "synced", how exactly? I assume not using put(), right? Is there some API I am missing?
gunchatbridge
@gunchatbridge
22:06
> [D] marknadal: <@!105969993933422592>
> @rysiekpl_gitlab yes using .put( 😛 not saying it was the best idea but works to some extent and I'm trying to make it better over time. https://github.com/amark/gun/blob/master/examples/basic/upload.html
>>>
Not sure how useful that would be. Gun still does not have a public node network, and is still rather... unstable.https://0xacab.org/rysiek/samizdat/-/issues/62Gun 0.2020.520 not usable2020-09-15T01:55:08ZMichał "rysiek" WoźniakGun 0.2020.520 not usableJob [#164219](https://0xacab.org/rysiek/samizdat/-/jobs/164219) failed for 859229f8f196d789d7328630c14ee9f02472f9f3: this might be related to the recent Gun upgrade, or general Gun unreliability (ref. #2).Job [#164219](https://0xacab.org/rysiek/samizdat/-/jobs/164219) failed for 859229f8f196d789d7328630c14ee9f02472f9f3: this might be related to the recent Gun upgrade, or general Gun unreliability (ref. #2).https://0xacab.org/rysiek/samizdat/-/issues/61Update dependencies2020-09-15T01:25:02ZMichał "rysiek" WoźniakUpdate dependenciesWe're using [an older version of Gun (`0.2020.116`)](https://0xacab.org/rysiek/samizdat/-/blob/688887a0a2c547093341cabdfcf8d24c67234d2c/samizdat-cli/package.json#L13) than [the latest available (`0.2020.520`)](https://www.npmjs.com/packa...We're using [an older version of Gun (`0.2020.116`)](https://0xacab.org/rysiek/samizdat/-/blob/688887a0a2c547093341cabdfcf8d24c67234d2c/samizdat-cli/package.json#L13) than [the latest available (`0.2020.520`)](https://www.npmjs.com/package/gun). Same is probably also true for [SEA, IPFS, and WebRTC](https://0xacab.org/rysiek/samizdat/-/tree/master/lib).
Time for some upgrades.
- ~~Gun~~
- ~~SEA~~
- [x] `go-ipfs` `0.4.22` -> `0.6.0`
- [x] [`ipfs.js`](https://js.ipfs.io/) `0.40.0` -> `0.50.2`
- [x] [`is-ipfs`](https://www.npmjs.com/package/is-ipfs) `0.6.3` -> `2.0.0`
- ~~WebRTC~~BetaMichał "rysiek" WoźniakMichał "rysiek" Woźniakhttps://0xacab.org/rysiek/samizdat/-/issues/60Improve npm use in CI/CD pipelines2020-09-10T00:57:14ZMichał "rysiek" WoźniakImprove npm use in CI/CD pipelinesWe should be:
- [x] using [`npm ci`](https://docs.npmjs.com/cli/ci.html) instead of `npm install`
- [x] [enable caching](https://docs.gitlab.com/ee/ci/caching/#caching-nodejs-dependencies) to speed up CI/CD pipelinesWe should be:
- [x] using [`npm ci`](https://docs.npmjs.com/cli/ci.html) instead of `npm install`
- [x] [enable caching](https://docs.gitlab.com/ee/ci/caching/#caching-nodejs-dependencies) to speed up CI/CD pipelinesMichał "rysiek" WoźniakMichał "rysiek" Woźniak