home / github

Menu
  • Search all tables
  • GraphQL API

issues

Table actions
  • GraphQL API for issues

61 rows where comments = 6, repo = 107914493 and state = "closed" sorted by updated_at descending

✖
✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: user, milestone, author_association, created_at (date), updated_at (date), closed_at (date)

type 2

  • issue 51
  • pull 10

state 1

  • closed · 61 ✖

repo 1

  • datasette · 61 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1907655261 I_kwDOBm6k_c5xtIJd 2193 "Test DATASETTE_LOAD_PLUGINS" test shows errors but did not fail the CI run simonw 9599 closed 0     6 2023-09-21T19:49:34Z 2023-09-21T21:56:43Z 2023-09-21T21:56:43Z OWNER  

That passed on 3.8 but should have failed: https://github.com/simonw/datasette/actions/runs/6266341481/job/17017099801 - the "Test DATASETTE_LOAD_PLUGINS" test shows errors but did not fail the CI run.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/2057#issuecomment-1730201226

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2193/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1886791100 I_kwDOBm6k_c5wdiW8 2180 Plugin hook: `actors_from_ids()` simonw 9599 closed 0     6 2023-09-08T01:16:41Z 2023-09-10T17:44:14Z 2023-09-08T04:28:03Z OWNER  

In building Datasette Cloud we realized that a bunch of the features we are building need a way of resolving an actor ID to the actual actor, in order to display something more interesting than just an integer ID.

Social plugins in particular need this - comments by X, CSV uploaded by X, that kind of thing.

I think the solution is a new plugin hook: actors_from_ids(datasette, ids) which can return a list of actor dictionaries.

The default implementation can return [{"id": "..."}] for the IDs passed to it.

Pluggy has a firstresult=True option which is relevant here, since this is the first plugin hook we will have implemented where only one plugin should provide an answer.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2180/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1874327336 PR_kwDOBm6k_c5ZLMSe 2165 DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins simonw 9599 closed 0     6 2023-08-30T20:33:30Z 2023-08-30T22:12:25Z 2023-08-30T22:12:25Z OWNER simonw/datasette/pulls/2165
  • 2164

TODO:

  • [x] Automated tests
  • [ ] Documentation
  • [x] Make sure DATASETTE_LOAD_PLUGINS='' works for loading zero plugins
datasette 107914493 pull    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2165/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
685806511 MDU6SXNzdWU2ODU4MDY1MTE= 950 Private/secret databases: database files that are only visible to plugins simonw 9599 closed 0     6 2020-08-25T20:46:17Z 2023-08-24T22:26:09Z 2023-08-24T22:26:08Z OWNER  

In thinking about the best way to implement https://github.com/simonw/datasette-auth-passwords/issues/6 (SQL-backed user accounts for datasette-auth-passwords) I realized that there are a few different use-cases where a plugin might want to store data that isn't visible to regular Datasette users:

  • Storing password hashes
  • Storing API tokens
  • Storing secrets that are used for data import integrations (secrets for talking to the Twitter API for example)

Idea: allow one or more private database files to be attached to Datasette, something like this:

datasette github.db linkedin.db -s secrets.db -m metadata.yml

The secrets.db file would not be visible using any of the Datasette's usual interface or API routes - but plugins would be able to run queries against it.

So datasette-auth-passwords might then be configured like this:

yaml plugins: datasette-auth-passwords: database: secrets sql: "select password_hash from passwords where username = :username" The plugin could even refuse to operate against a database that hadn't been loaded as a secret database.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/950/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1615862295 I_kwDOBm6k_c5gUBoX 2036 `publish cloudrun` reuses image tags, which can lead to very surprising deploy problems simonw 9599 closed 0     6 2023-03-08T20:11:44Z 2023-03-08T20:57:34Z 2023-03-08T20:57:34Z OWNER  

See this issue: - https://github.com/simonw/datasette.io/issues/141

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2036/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1495431932 I_kwDOBm6k_c5ZInr8 1951 `datasette.create_token(...)` method for creating signed API tokens simonw 9599 closed 0   Datasette 1.0a2 8711695 6 2022-12-14T01:25:34Z 2022-12-14T02:43:45Z 2022-12-14T02:42:05Z OWNER  

I need this for: - #1947

And I can refactor this to use it too: - #1855

By making this a documented internal API it can be used by other plugins too. It's also going to be really useful for writing tests.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1951/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1486011362 PR_kwDOBm6k_c5E3XqB 1940 register_permissions() plugin hook simonw 9599 closed 0   Datasette 1.0a2 8711695 6 2022-12-09T05:09:28Z 2022-12-13T02:05:55Z 2022-12-13T02:05:54Z OWNER simonw/datasette/pulls/1940

Refs #1939

From this comment: https://github.com/simonw/datasette/issues/1939#issuecomment-1343872168

  • [x] Unit test for the registration plugin hook itself
  • [x] Use them in check_permission_actions_are_documented test in conftest.py
  • [x] Add description field to Permissions (and update tests and docs)
  • [x] Documentation for datasette.permissions dictionary
  • [x] If no default= provided in call to permission_allowed() then use default from datasette.permissions list
  • [x] Remove default= from a bunch of places
  • [x] Throw an error if two permissions are registered with the same name or abbreviation (but other attributes differ)
  • [x] Update authentication and permissions documentation to explain that permissions are now registered and have a registered default

:books: Documentation preview :books:: https://datasette--1940.org.readthedocs.build/en/1940/

datasette 107914493 pull    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1940/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1468709531 I_kwDOBm6k_c5Xirqb 1915 Interactive demo of Datasette 1.0 write APIs simonw 9599 closed 0     6 2022-11-29T21:16:03Z 2022-11-30T04:05:46Z 2022-11-30T04:05:46Z OWNER  

I'm going to try to get this working on https://latest.datasette.io/ - it already has a way for people to sign in as root, but none of the databases there are writable.

So I'm going to build a plugin which adds a writable named in-memory database.

And some kind of mechanism for clearing out that database on a regular basis - maybe tables in that database get deleted automatically an hour after they are created?

(Would be neat to display their time-left-until-deleted too)

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1915/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1363440999 I_kwDOBm6k_c5RRHVn 1804 Ability to set a custom facet_size per table simonw 9599 closed 0     6 2022-09-06T15:11:40Z 2022-09-07T00:21:56Z 2022-09-06T18:06:53Z OWNER  

Suggestion from Discord: https://discord.com/channels/823971286308356157/823971286941302908/1016725586351247430

Is it possible to limit the facet size per database or even per table?

This is a really good idea, it could be done in metadata.yml.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1804/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1084193403 PR_kwDOBm6k_c4wDKmb 1574 introduce new option for datasette package to use a slim base image fs111 33631 closed 0     6 2021-12-19T21:18:19Z 2022-08-15T08:49:31Z 2022-08-15T08:49:31Z NONE simonw/datasette/pulls/1574

The official python images on docker hub come with a slim variant that is significantly smaller than the default. The diff does not change the default, but allows to switch to the slim variant with commandline switch (--slim-base-image)

Size comparison:

``` $ datasette package some.db -t fat --install "datasette-basemap datasette-cluster-map"

$ datasette package some.db -t slim --slim-base-image --install "datasette-basemap datasette-cluster-map"

$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE fat latest 807b393ace0d 9 seconds ago 978MB slim latest 31bc5e63505c 8 minutes ago 191MB

```

datasette 107914493 pull    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1574/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1223699280 I_kwDOBm6k_c5I8CtQ 1739 .db downloads should be served with an ETag simonw 9599 closed 0     6 2022-05-03T05:11:21Z 2022-05-04T18:21:18Z 2022-05-03T14:59:51Z OWNER  

I noticed that my Pyodide Datasette prototype is downloading the same database file every single time rather than browser caching it:

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1739/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1174423568 I_kwDOBm6k_c5GAEgQ 1670 Ship Datasette 0.61 simonw 9599 closed 0     6 2022-03-20T02:47:54Z 2022-03-23T18:32:32Z 2022-03-23T18:32:03Z OWNER  

Let the alpha bake for a while, since #1668 is a big last-minute change.

After shipping, release a new datasette-hashed-urls that depends on it, also this:

  • https://github.com/simonw/datasette-hashed-urls/issues/11
datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1670/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1126604194 I_kwDOBm6k_c5DJp2i 1632 datasette one.db one.db opens database twice, as one and one_2 simonw 9599 closed 0   Datasette 1.0 3268330 6 2022-02-07T23:14:47Z 2022-03-19T04:04:49Z 2022-02-07T23:50:01Z OWNER  

% mkdir /tmp/data % cp ~/Dropbox/Development/datasette/fixtures.db /tmp/data % datasette /tmp/data/*.db /tmp/data/created.db --create -p 8852 ... INFO: Uvicorn running on http://127.0.0.1:8852 (Press CTRL+C to quit) ^CINFO: Shutting down % datasette /tmp/data/*.db /tmp/data/created.db --create -p 8852 ... INFO: 127.0.0.1:49533 - "GET / HTTP/1.1" 200 OK The first time I ran Datasette I got two databases - fixtures and created

BUT... when I ran Datasette the second time it looked like this:

This is the same result you get if you run:

datasette /tmp/data/fixtures.db /tmp/data/created.db /tmp/data/created.db

This is caused by this Datasette issue: - https://github.com/simonw/datasette/issues/509

So... either I teach Datasette to de-duplicate multiple identical file paths passed to the command, or I can't use /data/*.db in the Dockerfile here and I need to go back to other solutions for the challenge described in this comment: https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1031971831

Originally posted by @simonw in https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1032029874

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1632/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1087913724 I_kwDOBm6k_c5A2D78 1577 Drop support for Python 3.6 simonw 9599 closed 0   Datasette 1.0 3268330 6 2021-12-23T18:17:03Z 2022-01-25T23:30:03Z 2022-01-20T04:31:41Z OWNER  

Original title: Decide when to drop support for Python 3.6

context_vars can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/

Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days?

Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1576#issuecomment-999878907

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1577/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1083669410 I_kwDOBm6k_c5Al3ui 1566 Release Datasette 0.60 simonw 9599 closed 0   Datasette 0.60 7571612 6 2021-12-17T22:58:12Z 2022-01-14T01:59:55Z 2022-01-14T01:59:55Z OWNER  

Using this as a tracking issue. I'm hoping to get the bulk of the JSON redesign work from the refactor in #1554 in for this release.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1566/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1076388044 I_kwDOBm6k_c5AKGDM 1547 Writable canned queries fail to load custom templates wragge 127565 closed 0   Datasette 0.60 7571612 6 2021-12-10T03:31:48Z 2022-01-13T22:27:59Z 2021-12-19T21:12:00Z CONTRIBUTOR  

I've created a canned query with "write": true set. I've also created a custom template for it, but the template doesn't seem to be found. If I look in the HTML I see (stock_exchange is the db name):

<!-- Templates considered: query-stock_exchange.html, *query.html -->

My non-writeable canned queries pick up custom templates as expected, and if I look at their HTML I see the canned query name added to the templates considered (the canned query here is date_search):

<!-- Templates considered: query-stock_exchange-date_search.html, query-stock_exchange.html, *query.html -->

So it seems like the writeable canned query is behaving differently for some reason. Is it an authentication thing? I'm using the built in --root authentication.

Thanks!

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1547/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
1082743068 PR_kwDOBm6k_c4v-izc 1559 filters_from_request plugin hook, now used in TableView simonw 9599 closed 0     6 2021-12-16T23:59:33Z 2021-12-17T23:09:41Z 2021-12-17T19:02:15Z OWNER simonw/datasette/pulls/1559

New plugin hook, refs #473

Used it to extract the logic from TableView that handles _search and _through and _where - refs #1518

datasette 107914493 pull    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1559/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
1052851176 I_kwDOBm6k_c4-wTvo 1507 ReadTheDocs build failed for 0.59.2 release simonw 9599 closed 0     6 2021-11-14T05:24:34Z 2021-11-14T05:41:55Z 2021-11-14T05:41:55Z OWNER  

I had to cancel the 0.59.2 release because ReadTheDocs was failing to build the documentation.

https://readthedocs.org/projects/datasette/builds/15268454/

``` /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m sphinx -T -b html -d _build/doctrees -D language=en . _build/html Running Sphinx v1.8.5 loading translations [en]... done making output directory... building [mo]: targets for 0 po files that are out of date building [html]: targets for 27 source files that are out of date updating environment: 27 added, 0 changed, 0 removed reading sources... [ 3%] authentication

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/cmd/build.py", line 304, in build_main app.build(args.force_all, filenames) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/application.py", line 341, in build self.builder.build_update() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/builders/init.py", line 347, in build_update len(to_build)) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/builders/init.py", line 360, in build updated_docnames = set(self.read()) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/builders/init.py", line 468, in read self._read_serial(docnames) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/builders/init.py", line 490, in _read_serial self.read_doc(docname) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/builders/init.py", line 534, in read_doc doctree = read_doc(self.app, self.env, self.env.doc2path(docname)) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/io.py", line 318, in read_doc pub.publish() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/docutils/core.py", line 219, in publish self.apply_transforms() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/docutils/core.py", line 200, in apply_transforms self.document.transformer.apply_transforms() File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/transforms/init.py", line 90, in apply_transforms Transformer.apply_transforms(self) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/docutils/transforms/init.py", line 171, in apply_transforms transform.apply(**kwargs) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/transforms/init.py", line 245, in apply apply_source_workaround(n) File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/util/nodes.py", line 94, in apply_source_workaround for classifier in reversed(node.parent.traverse(nodes.classifier)): TypeError: argument to reversed() must be a sequence

Exception occurred: File "/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/lib/python2.7/site-packages/sphinx/util/nodes.py", line 94, in apply_source_workaround for classifier in reversed(node.parent.traverse(nodes.classifier)): TypeError: argument to reversed() must be a sequence The full traceback has been saved in /tmp/sphinx-err-vkl0oE.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks! ```

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1507/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
974987856 MDU6SXNzdWU5NzQ5ODc4NTY= 1442 Mechanism to cause specific branches to deploy their own demos simonw 9599 closed 0     6 2021-08-19T19:41:39Z 2021-08-19T21:11:45Z 2021-08-19T21:09:40Z OWNER  

A useful capability would be if it was super-easy to say "any pushes to branch X should be deployed to latest-X.datasette.io".

I'd like to use this for the column query information work in #1434

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1442/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
940077168 MDU6SXNzdWU5NDAwNzcxNjg= 1389 "searchmode": "raw" in table metadata simonw 9599 closed 0     6 2021-07-08T17:32:10Z 2021-07-10T18:33:13Z 2021-07-10T18:33:13Z OWNER  

http://localhost:8001/index/summary?_search=language%3Aeng&_sort=title&_searchmode=raw

But I'm not able to manage it in the metadata file. Here is mine (note that the sort column is taken into account) Here it is:

``` { "databases": { "index": { "tables": { "summary": { "sort": "title", "searchmode": "raw" } } } } }

Originally posted by @Krazybug in https://github.com/simonw/datasette/issues/759#issuecomment-624860451

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1389/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
395236066 MDU6SXNzdWUzOTUyMzYwNjY= 393 CSV export in "Advanced export" pane doesn't respect query ltrgoddard 1727065 closed 0     6 2019-01-02T12:39:41Z 2021-06-17T18:14:24Z 2019-01-03T02:44:10Z NONE  

It looks like there's an inconsistency when exporting to CSV via the the web interface. Say I'm looking at songs released in 1989 in the classic-rock/classic-rock-song-list table from the Five Thirty Eight data. The JSON and CSV export links at the top of the page both give me filtered data using Release+Year__exact=1989 in the URL. In the Advanced export tab, though, the CSV option gives me the whole data set, while the JSON options preserve the query.

It may be that this is intended behaviour related to the streaming CSV stuff discussed here, but if that's the case then I think it should be a little clearer.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/393/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
724369025 MDExOlB1bGxSZXF1ZXN0NTA1NzY5NDYy 1031 Fallback to databases in inspect-data.json when no -i options are passed frankier 299380 closed 0     6 2020-10-19T07:51:06Z 2021-03-29T01:46:45Z 2021-03-29T00:23:41Z FIRST_TIME_CONTRIBUTOR simonw/datasette/pulls/1031

Currenlty Datasette.__init__ checks immutables against None to decide whether to fallback to inspect-data.json. This patch modifies the serve command to pass None when no -i options are passed so this fallback works correctly.

datasette 107914493 pull    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1031/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
826613352 MDExOlB1bGxSZXF1ZXN0NTg4NjAxNjI3 1254 Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions durkie 3200608 closed 0     6 2021-03-09T20:49:08Z 2021-03-10T18:27:45Z 2021-03-09T22:04:23Z NONE simonw/datasette/pulls/1254

This requires adding the RT Topology library (Spatialite changed to RT Topology from LWGEOM between 4.4 and 5.0), as well as upgrading the GEOS version (which is the reason for switching to python:3.7.10-slim-buster as the base image.)

autoconf and libtool are added to build RT Topology, and Spatialite is now built with --disable-minizip (minizip wasn't an option in 4.4 and I didn't want to add another dependency) and --disable-dependency-tracking which, according to Spatialite, "speeds up one-time builds"

datasette 107914493 pull    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1254/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
753767911 MDExOlB1bGxSZXF1ZXN0NTI5NzgzMjc1 1117 Support for generated columns simonw 9599 closed 0     6 2020-11-30T20:10:46Z 2020-11-30T22:23:19Z 2020-11-30T21:29:58Z OWNER simonw/datasette/pulls/1117

Refs #1116. My first attempt at this worked on my laptop but broke in CI, so I'm going to iterate on it in a pull request instead.

datasette 107914493 pull    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1117/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
714449879 MDU6SXNzdWU3MTQ0NDk4Nzk= 992 Change "--config foo:bar" to "--setting foo bar" simonw 9599 closed 0   Datasette 0.52 6055094 6 2020-10-05T01:27:45Z 2020-11-24T20:01:54Z 2020-11-24T20:01:54Z OWNER  

I designed the config format before I had a good feel for CLI design using Click. --config max_page_size 2000 is better than --config max_page_size:2000.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/992/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
733796942 MDU6SXNzdWU3MzM3OTY5NDI= 1075 PrefixedUrlString mechanism broke everything simonw 9599 closed 0   0.51 6026070 6 2020-10-31T19:58:05Z 2020-10-31T20:48:51Z 2020-10-31T20:48:51Z OWNER  

Added in 7a67bc7a569509d65b3a8661e0ad2c65f0b09166 refs #1026. Lots of tests are failing now.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1075/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
732905360 MDU6SXNzdWU3MzI5MDUzNjA= 1067 Table actions menu on view pages, not on query pages simonw 9599 closed 0   0.51 6026070 6 2020-10-30T05:56:39Z 2020-10-31T17:51:31Z 2020-10-31T17:40:14Z OWNER  

Follow-on from #1066.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1067/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
733303548 MDExOlB1bGxSZXF1ZXN0NTEzMTA2MDI2 1069 load_template() plugin hook simonw 9599 closed 0   0.51 6026070 6 2020-10-30T15:59:45Z 2020-10-30T17:47:20Z 2020-10-30T17:47:19Z OWNER simonw/datasette/pulls/1069

Refs #1042

datasette 107914493 pull    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1069/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
729096595 MDU6SXNzdWU3MjkwOTY1OTU= 1051 Better display of binary data on arbitrary query results page simonw 9599 closed 0     6 2020-10-25T19:38:06Z 2020-10-29T22:12:16Z 2020-10-29T22:01:39Z OWNER  

https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+101

Problem: if these were larger fields that HTML page could have multiple megabytes of Python binary string representations on it.

It should behave more like the regular table view does:

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1051/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
718723543 MDU6SXNzdWU3MTg3MjM1NDM= 1014 Add Link: pagination HTTP headers simonw 9599 closed 0   0.51 6026070 6 2020-10-10T23:42:40Z 2020-10-23T19:44:05Z 2020-10-11T00:18:51Z OWNER  

Spun off from #782. These can go on all of the JSON endpoints that support pagination.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1014/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
710506708 MDU6SXNzdWU3MTA1MDY3MDg= 978 Rendering glitch with column headings on mobile simonw 9599 closed 0   Datasette 0.50 5971510 6 2020-09-28T19:04:45Z 2020-10-08T23:54:40Z 2020-09-28T22:43:01Z OWNER  

https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_parse%28%2210+october+2020+3pm%22%29%2C%0D%0A++dateutil_easter%28%222020%22%29%2C%0D%0A++dateutil_parse_fuzzy%28%22This+is+due+10+september%22%29%2C%0D%0A++dateutil_parse%28%221%2F2%2F2020%22%29%2C%0D%0A++dateutil_parse%28%222020-03-04%22%29%2C%0D%0A++dateutil_parse_dayfirst%28%222020-03-04%22%29%2C%0D%0A++dateutil_easter%282020%29

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/978/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
314506446 MDU6SXNzdWUzMTQ1MDY0NDY= 214 Ability for plugins to define extra JavaScript and CSS simonw 9599 closed 0     6 2018-04-16T05:29:34Z 2020-09-30T20:36:11Z 2018-04-18T03:13:03Z OWNER  

This can hook in to the existing extra_css_urls and extra_js_urls mechanism:

https://github.com/simonw/datasette/blob/b2955d9065ea019500c7d072bcd9d49d1967f051/datasette/app.py#L304-L305

The plugins should be able to bundle their own assets though, so it will also have to integrate with the /static/ static mounts mechanism somehow:

https://github.com/simonw/datasette/blob/b2955d9065ea019500c7d072bcd9d49d1967f051/datasette/app.py#L1255-L1257

Refs #14

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/214/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
679779797 MDU6SXNzdWU2Nzk3Nzk3OTc= 939 extra_ plugin hooks should take the same arguments simonw 9599 closed 0     6 2020-08-16T16:04:54Z 2020-08-16T18:25:05Z 2020-08-16T16:50:29Z OWNER  
  • [x] extra_css_urls(template, database, table, datasette)
  • [x] extra_js_urls(template, database, table, datasette)
  • [x] extra_body_script(template, database, table, view_name, datasette)
  • [x] extra_template_vars(template, database, table, view_name, request, datasette)

Originally posted by @simonw in https://github.com/simonw/datasette/issues/938#issuecomment-674544691

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/939/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
677326155 MDU6SXNzdWU2NzczMjYxNTU= 930 Datasette sdist is missing templates (hence broken when installing from Homebrew) simonw 9599 closed 0     6 2020-08-12T02:20:16Z 2020-08-12T03:30:59Z 2020-08-12T03:30:59Z OWNER  

Pretty nasty bug this: I'm getting 500 errors for all pages that try to render a template after installing the newly released Datasette 0.47 - both from pip install and via Homebrew.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/930/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
644309017 MDU6SXNzdWU2NDQzMDkwMTc= 864 datasette.add_message() doesn't work inside plugins simonw 9599 closed 0   Datasette 0.45 5533512 6 2020-06-24T04:30:06Z 2020-06-29T00:51:01Z 2020-06-29T00:51:01Z OWNER  

Similar problem to #863 - calling datasette.add_message() in a view registered using the register_routes() plugin hook doesn't work, because the code that writes accumulated messages to the ds_messages signed cookie lives in the BaseView class here:

https://github.com/simonw/datasette/blob/28bb1c51897f3956861755e345e18b8e0b1423ac/datasette/views/base.py#L94-L97

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/864/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
637342551 MDU6SXNzdWU2MzczNDI1NTE= 834 startup() plugin hook simonw 9599 closed 0   Datasette 0.45 5533512 6 2020-06-11T21:48:14Z 2020-06-28T19:38:50Z 2020-06-13T17:56:12Z OWNER  

It might be useful to have an startup hook which gets passed the datasette object as soon as Datasette has finished initializing.

My initial use-case for this is configuration verification - checking that the "plugins" configuration block for this plugin contains valid details.

I imagine there are plenty of other potential uses for this as well.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/834/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
529429214 MDU6SXNzdWU1Mjk0MjkyMTQ= 642 Provide a cookiecutter template for creating new plugins simonw 9599 closed 0   Datasette 1.0 3268330 6 2019-11-27T15:46:36Z 2020-06-20T03:20:33Z 2020-06-20T03:20:25Z OWNER  

See this conversation: https://twitter.com/psychemedia/status/1199707352540368896

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/642/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
638241779 MDU6SXNzdWU2MzgyNDE3Nzk= 846 "Too many open files" error running tests simonw 9599 closed 0     6 2020-06-13T22:11:40Z 2020-06-14T00:26:31Z 2020-06-14T00:26:31Z OWNER  

I got this on my laptop: ```pytest ... /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.7/site-packages/jinja2/loaders.py:171: in get_source f = open_if_exists(filename)


filename = '/Users/simon/Dropbox/Development/datasette/datasette/templates/400.html', mode = 'rb'

def open_if_exists(filename, mode='rb'):
    """Returns a file descriptor for the filename if that file exists,
    otherwise `None`.
    """
    try:
      return open(filename, mode)

E OSError: [Errno 24] Too many open files: '/Users/simon/Dropbox/Development/datasette/datasette/templates/400.html'

/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.7/site-packages/jinja2/utils.py:154: OSError ``` Based on the conversation in https://github.com/pytest-dev/pytest/issues/2970 I'm worried that my tests are opening too many files without closing them.

In particular... I call sqlite3.connect(filepath) a LOT - and I don't ever call conn.close() on those opened connections:

https://github.com/simonw/datasette/blob/cf7a2bdb404734910ec07abc7571351a2d934828/datasette/database.py#L58-L60

Could this be resulting in my tests eventually opening too many unclosed file handles? How could I confirm this?

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/846/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
631932926 MDU6SXNzdWU2MzE5MzI5MjY= 801 allow_by_query setting for configuring permissions with a SQL statement simonw 9599 closed 0   Datasette 1.0 3268330 6 2020-06-05T20:30:19Z 2020-06-11T18:58:56Z 2020-06-11T18:58:49Z OWNER  

Idea: an "allow_sql" key with a SQL query that gets passed the actor JSON as :actor and can extract the relevant keys from it and return 1 or 0.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/698#issuecomment-639787304

See also #800

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/801/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
636426530 MDU6SXNzdWU2MzY0MjY1MzA= 829 Ability to set ds_actor cookie such that it expires simonw 9599 closed 0   Datasette 0.44 5512395 6 2020-06-10T17:31:40Z 2020-06-10T19:41:35Z 2020-06-10T19:40:05Z OWNER  

I need this for datasette-auth-github: https://github.com/simonw/datasette-auth-github/issues/62#issuecomment-642152076

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/829/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
632673972 MDU6SXNzdWU2MzI2NzM5NzI= 804 python tests/fixtures.py command has a bug simonw 9599 closed 0   Datasette 0.44 5512395 6 2020-06-06T19:17:36Z 2020-06-09T20:01:30Z 2020-06-09T19:58:34Z OWNER  

This command is meant to write out fixtures.db, metadata.json and a plugins directory: $ python tests/fixtures.py /tmp/fixtures.db /tmp/metadata.json /tmp/plugins/ Test tables written to /tmp/fixtures.db - metadata written to /tmp/metadata.json Traceback (most recent call last): File "tests/fixtures.py", line 833, in <module> ("my_plugin.py", PLUGIN1), NameError: name 'PLUGIN1' is not defined

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/804/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
635147716 MDU6SXNzdWU2MzUxNDc3MTY= 825 Way to enable a default=False permission for anonymous users simonw 9599 closed 0   Datasette 0.44 5512395 6 2020-06-09T06:26:27Z 2020-06-09T17:19:19Z 2020-06-09T17:01:10Z OWNER  

I'd like plugins to be able to ship with a default that says "anonymous users cannot do this", but allow site administrators to over-ride that such that anonymous users can use the feature after all.

This is tricky because right now the anonymous user doesn't have an actor dictionary at all, so there's no key to match to an allow block.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/825/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
634139848 MDU6SXNzdWU2MzQxMzk4NDg= 813 Mechanism for specifying allow_sql permission in metadata.json simonw 9599 closed 0   Datasette 0.44 5512395 6 2020-06-08T04:57:19Z 2020-06-09T00:09:57Z 2020-06-09T00:07:19Z OWNER  

Split from #811. It would be useful if finely-grained permissions configured in metadata.json could be used to specify if a user is allowed to execute arbitrary SQL queries.

We have a permission check call for this already: https://github.com/simonw/datasette/blob/9397d718345c4b35d2a5c55bfcbd1468876b5ab9/datasette/views/database.py#L159

But there's currently no way to implement this check without writing a plugin.

I think a "allow_sql": {...} block at the database level in metadata.json (sibling to the current "allow" block for that database implemented in #811) would be a good option for this.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/813/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
585633142 MDU6SXNzdWU1ODU2MzMxNDI= 706 Documentation for the "request" object simonw 9599 closed 0   Datasette 1.0 3268330 6 2020-03-22T02:55:50Z 2020-05-30T13:20:00Z 2020-05-27T22:31:22Z OWNER  

Since that object is passed to the extra_template_vars hooks AND the classes registered by register_facet_classes it should be part of the documented interface on https://datasette.readthedocs.io/en/stable/internals.html

I could also start passing it to the register_output_renderer callback.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/706/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
573583971 MDU6SXNzdWU1NzM1ODM5NzE= 689 "Templates considered" comment broken in >=0.35 chrishas35 35075 closed 0     6 2020-03-01T17:31:21Z 2020-04-05T19:39:44Z 2020-04-05T19:39:44Z NONE  

Noticed that the "Templates Considered" comment is missing in 0.37. Believe I traced it back to #664 as you can see it in https://v0-34.datasette.io/ but not https://v0-35.datasette.io/. Looking at the template context debug between the two you can see what is missing from 0.35 vs. 0.34:

```diff < "datasette_version": "0.34", < "app_css_hash": "ffa51a", < "select_templates": [ < "*index.html" < ], < "zip": "<class 'zip'>", < "body_scripts": [], < "extra_css_urls": "<generator object BaseView._asset_urls at 0x7f6529ac05f0>", < "extra_js_urls": "<generator object BaseView._asset_urls at 0x7f6529ac0660>", < "format_bytes": "<function format_bytes at 0x7f652a1588b0>", < "database_url": "<bound method BaseView.database_url of \<datasette.views.index.IndexView object at 0x7f6529b03e50>>", < "database_color": "<bound method BaseView.database_color of \<datasette.views.index.IndexView object at 0x7f6529b03e50>>"


"datasette_version": "0.35",
"database_url": "<bound method BaseView.database_url of <datasette.views.index.IndexView object at 0x7f6140dacd90>>",
"database_color": "<bound method BaseView.database_color of <datasette.views.index.IndexView object at 0x7f6140dacd90>>"

```

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/689/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
592829135 MDU6SXNzdWU1OTI4MjkxMzU= 713 Support YAML in metadata - metadata.yaml simonw 9599 closed 0     6 2020-04-02T18:10:05Z 2020-04-02T19:36:17Z 2020-04-02T19:30:55Z OWNER  

I was originally going to do this with a plugin - see #357 - but the more I work with metadata.json the more I want it to just accept YAML as an optional alternative to JSON.

The best example why is still this one: https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/master/russian-ads-metadata.yaml

YAML is just SO much better than JSON for multi-line strings - in particular HTML and SQL, both of which are common in metadata.json files.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/713/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
513008936 MDU6SXNzdWU1MTMwMDg5MzY= 608 Improve UI of "datasette publish cloudrun" to reduce chances of accidentally over-writing a service simonw 9599 closed 0     6 2019-10-27T19:21:28Z 2019-11-08T02:51:36Z 2019-11-08T02:48:46Z OWNER  

The concept of a "service" in Cloud Run is crucial: if you deploy to the same service, you will over-write what you deployed there last!

As such, I'd like to make service a required positional argument for publish cloudrun:

datasette publish cloudrun my-service one.db two.db three.db
datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/608/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
512996469 MDU6SXNzdWU1MTI5OTY0Njk= 607 Ways to improve fuzzy search speed on larger data sets? zeluspudding 8431341 closed 0     6 2019-10-27T17:31:37Z 2019-11-07T03:38:10Z 2019-11-07T03:38:10Z NONE  

I have an sqlite table with 16 million rows in it. Having read @simonw article "Fast Autocomplete Search for Your Website" I was curious to try datasette to see what kind of query performance I could get out of it. In truth I don't need to do full text search since all I would like to do is give my users a way to search for the names of investors such as "Warren Buffet", or "Tim Cook" (who's names are in a single column).

On the first search, Datasette takes over 20 seconds to return all records associated with elon musk:

If I rerun the same search, it then takes almost 9 seconds:

That's far to slow to implement an autocomplete feature. I could reduce the latency by making a special table of only unique investor names, thereby reducing the search space to less than a million rows (then I'd need to implement a way to add only new investor names to the table as I received new data.. about 4,000 rows a day). If I did that, I'm still concerned the new table wouldn't be lean enough to lookup investor names quickly. Plus, even if I can implement the autocomplete feature, I would still finally have to lookup records for that investors which would take between 8 - 20 seconds.

Are there any tricks for speeding this up?

Here's my hardware:

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/607/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
459621683 MDU6SXNzdWU0NTk2MjE2ODM= 521 Easier way of creating custom row templates simonw 9599 closed 0     6 2019-06-23T21:49:27Z 2019-07-03T03:23:56Z 2019-07-03T03:23:56Z OWNER  

I was messing around with a custom _rows_and_columns.html template and ended up with this: ```html {% for row in display_rows %}


{% for cell in row %} {% if cell.column == "First_Name" %}

{{ cell.value }} {% elif cell.column == "Last_Name" %} {{ cell.value }}

{% elif cell.column == "Short_Description" %}

{{ cell.column }}: {{ cell.value }}

{% else %} {{ cell.column }}: {{ cell.value }}    {% endif %} {% endfor %}

{% endfor %} This is nasty. I'd like to be able to do something like this instead: {% for row in display_rows %}

{{ row["First_Name"] }} {{ row["Last_Name"] }}

... ```

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/521/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
349827640 MDU6SXNzdWUzNDk4Mjc2NDA= 359 Faceted browse against a JSON list of tags simonw 9599 closed 0     6 2018-08-12T17:01:14Z 2019-05-29T21:39:12Z 2019-05-03T00:21:44Z OWNER  

If a table has a ["foo", "bar", "baz"] JSON column allow that to be faceted against.

  • [x] Support ?column__arraycontains=x filter queries
  • [x] Support ?_facet_array=column faceting
datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/359/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
322787470 MDU6SXNzdWUzMjI3ODc0NzA= 259 inspect() should detect many-to-many relationships simonw 9599 closed 0     6 2018-05-14T12:03:58Z 2019-05-23T03:55:37Z 2019-05-23T03:55:37Z OWNER  

Relates to #255 - in particular supporting facets across M2M relationships.

It should be possible for .inspect() to notice when a table has two foreign keys to two different tables, and assume that this means there is a M2M relationship between those tables.

When rendering a table with a m2m relationship we could display the first X associated records as a comma separated list of hyperlinks in a new column on the table view, with a column name derived from the table on the other side.

Since SQLite doesn't have RANK or an equivalent of https://www.xaprb.com/blog/2006/12/02/how-to-number-rows-in-mysql/ this would be implemented as N+1 queries (one query per cell that we want to display an m2m summary). This should be OK in SQLite: https://sqlite.org/np1queryprob.html

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/259/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
310533258 MDU6SXNzdWUzMTA1MzMyNTg= 191 Figure out how to bundle a more up-to-date SQLite simonw 9599 closed 0     6 2018-04-02T16:33:25Z 2018-07-10T17:46:13Z 2018-07-10T17:46:13Z OWNER  

The version of SQLite that ships with Python 3 is a bit limited - it doesn't support row values for example https://www.sqlite.org/rowvalue.html

Figure out how to bundle a more recent SQLite engine with datasette. We need to figure out two cases:

  • Bundling a recent version in a Dockerfile build. I expect this to be quite easy.
  • Making a more recent version available to people hacking around in Mac OS X. I have no idea how to start on this.

I want it working on Mac OS X too because I don't want to force Docker as a dependency for anyone who just want to hack around with Datasette a little and run the test suite.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/191/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
333086005 MDU6SXNzdWUzMzMwODYwMDU= 313 Deploy demo of Datasette on every commit that passes tests simonw 9599 closed 0     6 2018-06-17T19:19:12Z 2018-06-17T21:52:58Z 2018-06-17T21:52:58Z OWNER  

We can use Travis CI and Zeit Now to ensure there is always a live demo of current master. We can ship archived demos for releases as well.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/313/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
314455877 MDExOlB1bGxSZXF1ZXN0MTgxNzIzMzAz 209 Don't duplicate simple primary keys in the link column russss 45057 closed 0     6 2018-04-15T21:56:15Z 2018-04-18T08:40:37Z 2018-04-18T01:13:04Z CONTRIBUTOR simonw/datasette/pulls/209

When there's a simple (single-column) primary key, it looks weird to duplicate it in the link column.

This change removes the second PK column and treats the link column as if it were the PK column from a header/sorting perspective.

This might make it a bit more difficult to tell what the link for the row is, I'm not sure yet. I feel like the alternative is to change the link column to just have the text "view" or something, instead of repeating the PK. (I doubt it makes much more sense with compound PKs.)

Bonus change in this PR: fix urlencoding of links in the displayed HTML.

Before:

After:

datasette 107914493 pull    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/209/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
314471743 MDU6SXNzdWUzMTQ0NzE3NDM= 211 Load plugins from a `--plugins-dir=plugins/` directory simonw 9599 closed 0     6 2018-04-16T01:17:43Z 2018-04-16T05:22:02Z 2018-04-16T05:22:02Z OWNER  

In #14 and 33c7c53ff87c2 I've added working support for setuptools entry_points plugins. These can be installed from PyPI using pip install ....

I imagine some projects will benefit from being able to add plugins without first publishing them to PyPI. Datasette already supports loading custom templates like so:

datasette serve --template-dir=mytemplates/ mydb.db

I propose an additional option, --plugins-dir= which specifies a directory full of blah.py files which will be loaded into Datasette when the application server starts.

datasette serve --plugins-dir=myplugins/ mydb.db

This will also need to be supported by datasette publish as those Python files should be copied up as part of the deployment.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/211/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
280315352 MDU6SXNzdWUyODAzMTUzNTI= 167 Nasty bug: last column not being correctly displayed simonw 9599 closed 0   Custom templates edition 2949431 6 2017-12-07T23:23:46Z 2017-12-10T01:00:21Z 2017-12-10T01:00:20Z OWNER  

e.g. https://datasette-bwnojrhmmg.now.sh/dk3-bde9a9a/dk?source__contains=http

The JSON output shows that the column is there, but is being displayed incorrectly:

https://datasette-bwnojrhmmg.now.sh/dk3-bde9a9a/dk.jsono?source__contains=http

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/167/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
274284246 MDExOlB1bGxSZXF1ZXN0MTUyODcwMDMw 104 [WIP] Add publish to heroku support jacobian 21148 closed 0     6 2017-11-15T19:56:22Z 2017-11-21T20:55:05Z 2017-11-21T20:55:05Z CONTRIBUTOR simonw/datasette/pulls/104

Refs #90

datasette 107914493 pull    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/104/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
0  
273678673 MDU6SXNzdWUyNzM2Nzg2NzM= 85 Detect foreign keys and use them to link HTML pages together simonw 9599 closed 0   Foreign key edition 2919870 6 2017-11-14T06:12:05Z 2017-11-19T06:08:19Z 2017-11-19T06:08:19Z OWNER  

https://stackoverflow.com/a/44430157/6083 documents the PRAGMA needed to extract foreign key references for a table.

At a minimum we can link column values known to be foreign keys to the corresponding row page. We could try to summarize the linked row in some way too - somehow extracting a sensible link title, maybe based on additional configuration in the metadata.json file.

Still todo:

  • [x] Fix it to csvs-to-sqlite refactoring command correctly creates primary key on generated tables
  • [x] Ship new csvs-to-sqlite with refactoring command
  • [x] Refactor column logic to be more predictable in our templates (the rowid special case)
  • [x] Mechanism by which table metadata can specify the "label" column for a table
  • [x] Automatically set the label column as the first column that isn't a primary key (falling back on primary key)
  • [x] Code which runs a "select id, label from table where id in (...)" query as part of the tableview and populates a lookup dictionary
  • [x] Modify templates to use values from that lookup dictionary
datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/85/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
267726219 MDU6SXNzdWUyNjc3MjYyMTk= 16 Default HTML/CSS needs to look reasonable and be responsive simonw 9599 closed 0   Ship first public release 2857392 6 2017-10-23T16:05:22Z 2017-11-11T20:19:07Z 2017-11-11T20:19:07Z OWNER  

Version one should have the following characteristics:

  • Looks OK
  • Works great on mobile
  • Loads extremely fast
  • No JavaScript! At least not in v1.
datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/16/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
267788884 MDU6SXNzdWUyNjc3ODg4ODQ= 23 Support Django-style filters in querystring arguments simonw 9599 closed 0   Ship first public release 2857392 6 2017-10-23T19:29:42Z 2017-10-25T04:23:03Z 2017-10-25T04:23:02Z OWNER  

e.g

/database/table?name__contains=Simon&age__gte=4

Same format as Django: double underscore as the split.

If you need to match against a column that happens to contain a double underscore in its official name, do this:

/database/table?weird__column__exact=Simon

__exact is the default operation if none is supplied.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/23/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed
267513424 MDU6SXNzdWUyNjc1MTM0MjQ= 1 Addressable pages for every row in a table simonw 9599 closed 0   Ship first public release 2857392 6 2017-10-23T00:44:16Z 2017-10-24T14:11:04Z 2017-10-24T14:11:03Z OWNER  
/database-name-7sha256/table-name/compound-pk
/database-name-7sha256/table-name/compound-pk.json

Tricky part will be figuring out what the private key is - especially since it could be a compound primary key and it might involve different data types.

datasette 107914493 issue    
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);
Powered by Datasette · Queries took 460.041ms · About: github-to-sqlite
  • Sort ascending
  • Sort descending
  • Facet by this
  • Hide this column
  • Show all columns
  • Show not-blank rows