issues

Table actions

372 rows where comments = 0 and user = 9599 sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: milestone, author_association, draft, state_reason, created_at (date), updated_at (date), closed_at (date)

type 2

state 2

id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1978022687 I_kwDOBm6k_c515jsf 2204 request.post_body() can only be called once simonw 9599 open 0     0 2023-11-05T23:22:03Z 2023-11-05T23:23:23Z   OWNER  

This code here:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/utils/asgi.py#L127-L135

It consumes the messages, which means if you try to call it a second time you won't be able to get at the body.

This is efficient - we don't end up with a request object property with potentially megabytes of content that we never look at again - but it's inconvenient for cases like middleware or functions where we don't know if the body has been consumed yet or not.

Potential solution: set request._body the first time it is called, and return that on subsequent calls.

Potential optimization: only do this for bodies that are shorter than a certain threshold - maybe 1MB - and raise an exception if you attempt to call post_body() multiple times against one of those larger bodies.

I'm a bit nervous about that option though, since it could result in errors that don't show up in testing but do show up in production.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2204/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1977155641 I_kwDOCGYnMM512QA5 601 Move plugin directory into documentation simonw 9599 open 0     0 2023-11-04T04:07:52Z 2023-11-04T04:07:52Z   OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/601/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1895266807 I_kwDOBm6k_c5w93n3 2184 Design decision - should configuration be exposed at /-/config ? simonw 9599 open 0     0 2023-09-13T21:07:08Z 2023-09-13T21:07:38Z   OWNER  

This made me think. That {"$env": "ENV_VAR"} hack was introduced back here:

The problem it was solving was that metadata was visible to everyone with access to the instance at /-/metadata but plugins clearly needed a way to set secret settings.

Now that this stuff is moving to config, we have some decisions to make:

  1. Add /-/config to let people see the configuration of their instance, and keep the $env trick for secret settings.
  2. Say all configuration aside from metadata is secret and make $env optional or ditch it entirely.
  3. Allow plugins to announce which of their configuration options are secret so we can automatically redact them from /-/config

I've found /-/metadata extraordinarily useful as a user of Datasette - it really helps me understand exactly what's going on if I run into any problems with a plugin, if I can quickly check what the settings look like.

So I'm leaning towards option 1 or 3.

Originally posted by @simonw in https://github.com/simonw/datasette/pull/2183#discussion_r1325076924

Also refs: - #2093

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2184/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1888477283 I_kwDOC8SPRc5wj-Bj 38 Run `rebuild_fts` after building the index simonw 9599 open 0     0 2023-09-08T23:17:45Z 2023-09-08T23:17:45Z   MEMBER  

In: - https://github.com/simonw/datasette.io/issues/152#issuecomment-1712323347

This turned out to be the fix:

bash dogsheep-beta index dogsheep-index.db templates/dogsheep-beta.yml sqlite-utils rebuild-fts dogsheep-index.db

 dogsheep-beta 197431109 issue     
{
    "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/38/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1886649402 I_kwDOBm6k_c5wc_w6 2179 Flaky test: test_hidden_sqlite_stat1_table simonw 9599 closed 0     0 2023-09-07T22:48:43Z 2023-09-07T22:51:19Z 2023-09-07T22:51:19Z OWNER  

This test here: https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/tests/test_api.py#L1011-L1020

It failed for me like this:

E AssertionError: assert [('normal', False), ('sqlite_stat1', True), ('sqlite_stat4', True)] in ([('normal', False), ('sqlite_stat1', True)],)

Looks like some builds of SQLite include a sqlite_stat4 table.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2179/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1884333600 PR_kwDOBm6k_c5Zszqk 2175 Test against Python 3.12 preview simonw 9599 closed 0     0 2023-09-06T16:09:05Z 2023-09-06T16:16:28Z 2023-09-06T16:16:27Z OWNER simonw/datasette/pulls/2175

https://dev.to/hugovk/help-test-python-312-beta-1508/

:books: Documentation preview :books:: https://datasette--2175.org.readthedocs.build/en/2175/

 datasette 107914493 pull     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2175/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1876407598 I_kwDOBm6k_c5v17Uu 2169 execute-sql on a database should imply view-database/view-permission simonw 9599 closed 0     0 2023-08-31T22:45:56Z 2023-08-31T22:46:28Z 2023-08-31T22:46:28Z OWNER  

I noticed that a token with execute-sql permission alone did not work, because it was not allowed to view the instance of the database.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2169/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1875739055 I_kwDOBm6k_c5vzYGv 2167 Document return type of await ds.permission_allowed() simonw 9599 open 0     0 2023-08-31T15:14:23Z 2023-08-31T15:14:23Z   OWNER  

The return type isn't documented here: https://github.com/simonw/datasette/blob/4c3ef033110407f3b3dbce501659d523724985e0/docs/internals.rst#L327-L350

On inspecting the code I'm not 100% sure if it's possible for this. method to return None, or if it can only return True or False. Need to confirm that.

https://github.com/simonw/datasette/blob/4c3ef033110407f3b3dbce501659d523724985e0/datasette/app.py#L822C15-L853

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2167/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
594237015 MDU6SXNzdWU1OTQyMzcwMTU= 718 Plugin idea: datasette-redirects simonw 9599 open 0     0 2020-04-05T03:41:38Z 2023-08-30T22:17:31Z   OWNER  

I just had to write a one-off custom plugin to redirect niche-musems.com to www.niche-museums.com (https://github.com/simonw/museums/issues/21) - it would be great if this kind of thing could be handled by a configurable plugin.

https://github.com/simonw/museums/blob/6b1faf00c463b2228860d4d62d104b11935e01b1/plugins/redirect_www.py

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/718/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   reopened
1795051447 I_kwDOBm6k_c5q_k-3 2097 Drop Python 3.7 simonw 9599 closed 0     0 2023-07-08T18:39:44Z 2023-08-23T18:18:00Z 2023-08-23T18:18:00Z OWNER  

I'm going to drop Python 3.7.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1153#issuecomment-1627455892

It's not supported any more: https://devguide.python.org/versions/

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2097/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1822940964 I_kwDOBm6k_c5sp98k 2115 Ensure all tests pass against new query view JSON simonw 9599 closed 0   Datasette 1.0a3 9700784 0 2023-07-26T18:25:20Z 2023-08-08T02:01:39Z 2023-08-08T02:01:38Z OWNER  

  • 2109

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2115/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1823428714 I_kwDOBm6k_c5sr1Bq 2120 Add __all__ to datasette/__init__.py simonw 9599 open 0     0 2023-07-27T01:07:10Z 2023-07-27T01:07:10Z   OWNER  

Currently looks like this: https://github.com/simonw/datasette/blob/08181823990a71ffa5a1b57b37259198eaa43e06/datasette/init.py#L1-L6

Adding __all__ = ["Permission", "Forbidden"...] would let me get rid of those # noqa comments.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2120/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1786243905 I_kwDOCGYnMM5qd-tB 564 Document that running `db.transform()` tidies up the schema indentation simonw 9599 closed 0     0 2023-07-03T13:59:28Z 2023-07-22T22:15:34Z 2023-07-22T22:15:34Z OWNER  

... and it turns out running .transform() with no arguments still fixes the format of the schema!

```pycon

db["log"].add_column("foo", str)

<Table log (id, name2, age, weight, foo)> >>> db["log"].add_column("bar", str) <Table log (id, name2, age, weight, foo, bar)> >>> db["log"].add_column("baz", str) <Table log (id, name2, age, weight, foo, bar, baz)> >>> print(db["log"].schema) CREATE TABLE "log" ( [id] INTEGER PRIMARY KEY, [name2] TEXT, [age] INTEGER, [weight] FLOAT , [foo] TEXT, [bar] TEXT, [baz] TEXT) >>> db["log"].transform() <Table log (id, name2, age, weight, foo, bar, baz)> >>> print(db["log"].schema) CREATE TABLE "log" ( [id] INTEGER PRIMARY KEY, [name2] TEXT, [age] INTEGER, [weight] FLOAT, [foo] TEXT, [bar] TEXT, [baz] TEXT ) ``` _Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1618347727_
 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/564/reactions",
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 1
}
   completed
1816857105 I_kwDOCGYnMM5sSwoR 570 `sqlite-utils install -e` option simonw 9599 closed 0     0 2023-07-22T18:32:23Z 2023-07-22T18:55:59Z 2023-07-22T18:32:56Z OWNER  

As seen in LLM.

Needed while working on: - #567

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/570/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1808116827 I_kwDOBm6k_c5rxaxb 2103 data attribute on Datasette tables exposing the primary key of the row simonw 9599 open 0     0 2023-07-17T16:18:25Z 2023-07-17T16:18:25Z   OWNER  

Maybe put it on the <tr> but probably better to go on the td.type-pk.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2103/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1773450152 I_kwDOCGYnMM5ptLOo 559 sqlean support simonw 9599 closed 0     0 2023-06-25T19:27:26Z 2023-06-25T23:25:53Z 2023-06-25T23:25:53Z OWNER  

If sqlean is available, use that.

Refs: - https://github.com/nalgeon/sqlean.py/issues/1#issuecomment-1605707788

This will provide a good workaround for: - #235

sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/559/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1764792125 I_kwDOBm6k_c5pMJc9 2086 Show information on startup in directory configuration mode simonw 9599 open 0     0 2023-06-20T07:13:33Z 2023-06-20T07:13:33Z   OWNER  

https://discord.com/channels/823971286308356157/823971286941302908/1120516587036889098

One thing that would be helpful would be message at launch indicating a metadata.json is getting picked up. I'm using directory mode and was editing the wrong file for awhile before I realize nothing I was doing was having any effect.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2086/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1701018909 I_kwDOCGYnMM5lY30d 543 Tests broken on Windows due to new convert() lambda names simonw 9599 closed 0     0 2023-05-08T22:11:29Z 2023-05-08T22:19:04Z 2023-05-08T22:19:04Z OWNER  

https://github.com/simonw/sqlite-utils/actions/runs/4920084038/jobs/8788501314 python sql = 'update [example] set [dt] = lambda_-9223371942137158589([dt]);' From: - #526

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/543/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1665510265 I_kwDOBm6k_c5jRat5 2060 Clean up a bunch of warnings from ruff simonw 9599 open 0     0 2023-04-13T01:23:02Z 2023-04-13T01:23:02Z   OWNER  

See: - #2056

ruff spots a bunch of warnings about things like unused variables - would be good to clean up as many of these as possible.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2060/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1661617056 I_kwDODD6af85jCkOg 15 ambiguous column name: createdAt - on checkin_details view simonw 9599 closed 0     0 2023-04-11T01:07:47Z 2023-04-11T03:16:37Z 2023-04-11T03:16:37Z MEMBER  

It looks like Swarm changed their schema and now both venues and checkins have createdAt fields.

Which breaks this view: https://github.com/dogsheep/swarm-to-sqlite/blob/719b6e96a016d0ca8b316d3bed9c2a7a0cb499ee/swarm_to_sqlite/utils.py#L171-L188

 swarm-to-sqlite 205429375 issue     
{
    "url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/15/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1649793525 I_kwDOBm6k_c5iVdn1 2051 `?_extra=row_urls` for table pages simonw 9599 open 0     0 2023-03-31T17:58:36Z 2023-03-31T17:58:36Z   OWNER  

Provides URLs to the JSON version of those rows. Maybe it persists the ?_shape= option too? Not sure about that.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2051/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1641013220 I_kwDOBm6k_c5hz9_k 2045 First column on a view page has no facet option in cog menu simonw 9599 open 0   Datasette 1.0 3268330 0 2023-03-26T18:02:47Z 2023-03-26T18:02:48Z   OWNER  

e.g. first column on this page - cog menu has no option to facet.

https://datasette.io/content/tools

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2045/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1636616315 I_kwDOBm6k_c5hjMh7 2042 Gather feedback on new ?_extra= design simonw 9599 open 0     0 2023-03-22T23:07:43Z 2023-03-22T23:08:19Z   OWNER  

Now that I've landed: - #1999

See also: - #262

I want to get some feedback from people on the design of the new ?_extra= feature, before freezing it into Datasette 1.0.

The big change is that the default JSON representation is now MUCH slimmer - it only gives you keys for "next" and "rows", where rows is a list of JSON objects (not a list of arrays as was previously the default) - for example https://latest.datasette.io/fixtures/sortable.json

If you want extra stuff you can ask for it with the new ?_extra= parameter - e.g. https://latest.datasette.io/fixtures/sortable.json?_extra=columns&_extra=suggested_facets

You can use ?_extra=extras to see a list of available extras: https://latest.datasette.io/fixtures/sortable.json?_extra=extras

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2042/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1617962395 I_kwDOJHON9s5gcCWb 10 Include schema in README simonw 9599 closed 0     0 2023-03-09T20:38:59Z 2023-03-09T20:48:18Z 2023-03-09T20:48:18Z MEMBER  

As seen in other tools like https://github.com/simonw/git-history

 apple-notes-to-sqlite 611552758 issue     
{
    "url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/10/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1617938730 I_kwDOJHON9s5gb8kq 9 Default to just storing plaintext, store HTML if `--html` is passed simonw 9599 open 0     0 2023-03-09T20:19:06Z 2023-03-09T20:19:06Z   MEMBER  

The full body version of the notes can get HUGE, due to embedded images. It turns out for my own purposes I'm usually happy with just the plaintext version.

I'm tempted to say you don't get HTML unless you pass a --html option.

 apple-notes-to-sqlite 611552758 issue     
{
    "url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/9/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1616440856 I_kwDOJHON9s5gWO4Y 5 Configure full text search simonw 9599 open 0     0 2023-03-09T05:20:46Z 2023-03-09T05:20:46Z   MEMBER  

FTS would be useful.

Maybe even extract the plain text from the notes to make that index easier to create, rather than creating it against the HTML. Can use the plaintext property for that.

 apple-notes-to-sqlite 611552758 issue     
{
    "url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/5/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1616422013 I_kwDOJHON9s5gWKR9 3 `apple-notes-to-sqlite --dump` option simonw 9599 closed 0     0 2023-03-09T05:05:49Z 2023-03-09T05:06:14Z 2023-03-09T05:06:14Z MEMBER  

Option that doesn't write to the database at all, it just outputs all the notes to stdout as newline-delimited JSON.

 apple-notes-to-sqlite 611552758 issue     
{
    "url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/3/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1564774831 I_kwDOBm6k_c5dRJGv 2012 Missing space in database summary simonw 9599 open 0     0 2023-01-31T18:01:13Z 2023-01-31T18:01:13Z   OWNER  

Spotted this on an instance index page:

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2012/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1554032168 I_kwDOBm6k_c5coKYo 2002 Document how actors are displayed simonw 9599 open 0     0 2023-01-24T00:08:49Z 2023-01-24T00:08:49Z   OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/2002/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1516815571 I_kwDOBm6k_c5aaMTT 1975 _col=id can cause id column to export twice in CSV export simonw 9599 open 0     0 2023-01-03T00:25:15Z 2023-01-03T00:25:21Z   OWNER  

https://datasette.simonwillison.net/simonwillisonblog/blog_entry.csv?_col=id&_col=title&_col=body&_labels=on&_size=1

csv id,id,title,body 1,1,WaSP Phase II,"<p>The <a href=""http://www.webstandards.org/"">Web Standards</a> project has launched Phase II.</p>" That should not have two id columns.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1975/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1515186569 I_kwDOBm6k_c5aT-mJ 1972 Fix Sphinx warning about extlink extension simonw 9599 closed 0     0 2022-12-31T19:12:04Z 2022-12-31T19:13:26Z 2022-12-31T19:13:26Z OWNER  

[sphinx-autobuild] > sphinx-build -b html /Users/simon/Dropbox/Development/datasette/docs /Users/simon/Dropbox/Development/datasette/docs/_build Running Sphinx v5.3.0 loading pickled environment... done WARNING: extlinks: Sphinx-6.0 will require a caption string to contain exactly one '%s' and all other '%' need to be escaped as '%%'.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1971#issuecomment-1368266904

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1972/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1499387139 PR_kwDOBm6k_c5FmNXs 1961 Use click.echo() instead of print() for --root option simonw 9599 closed 0     0 2022-12-16T00:54:56Z 2022-12-16T00:55:19Z 2022-12-16T00:55:18Z OWNER simonw/datasette/pulls/1961

This ensures the URL is output correctly when running under Docker.

Closes #1958

:books: Documentation preview :books:: https://datasette--1961.org.readthedocs.build/en/1961/

 datasette 107914493 pull     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1961/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1468495358 I_kwDOBm6k_c5Xh3X- 1910 Check incoming column types on various write APIs simonw 9599 open 0   Datasette 1.0a-next 8755003 0 2022-11-29T18:09:10Z 2022-12-13T05:29:09Z   OWNER  

I do think this needs type checking - I just tried and you really can send a string to an integer column and have it work, which feels bad.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1863#issuecomment-1331089156

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1910/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1447465004 I_kwDOBm6k_c5WRpAs 1889 Ability to create new tokens via the API simonw 9599 open 0   Datasette 1.0a-next 8755003 0 2022-11-14T06:21:36Z 2022-12-13T05:29:08Z   OWNER  

Refs: - #1850

Initially I decided that the API shouldn't be able to create new tokens at all - I don't like the idea of an API token holder creating themselves additional tokens.

Then I realized that two of the API features are specifically more useful if you can generate fresh tokens via the API:

  • Tokes that expire after a time limit are MUCH more useful if they can be automatically generated
  • Likewise, tokens that are restricted to a subset of permissions (see #1855) make more sense to be generated like this, especially in conjunction with expiry times
 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1889/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1216436131 I_kwDOBm6k_c5IgVej 1721 Implement plugin hooks: `register_table_extras`, `register_row_extras`, `register_query_extras` simonw 9599 open 0   Datasette 1.0a-next 8755003 0 2022-04-26T20:21:49Z 2022-12-13T05:29:07Z   OWNER  

Designed in: - #1720

Part of: - #262 - #1709

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1721/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1493306655 I_kwDOBm6k_c5ZAg0f 1945 `view-instance` should not be checked for /-/actor.json simonw 9599 closed 0   Datasette 1.0a2 8711695 0 2022-12-13T04:01:46Z 2022-12-13T04:11:56Z 2022-12-13T04:11:56Z OWNER  

Spotted this while testing:

  • 1855

    export TOKEN=$(datasette create-token root --secret s -a foo) curl -H "Authorization: Bearer $TOKEN" http://localhost:8002/-/actor.json Returned a Forbidden error (and not in JSON either).
 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1945/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1487764628 I_kwDOCGYnMM5YrXyU 518 flake8 ValueError: Error code '#' supplied to 'extend-ignore' option... simonw 9599 closed 0     0 2022-12-10T01:30:24Z 2022-12-10T01:36:46Z 2022-12-10T01:36:46Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/518/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1479920517 I_kwDOBm6k_c5YNcuF 1934 Return number of ignored/replaced items from /-/insert simonw 9599 open 0   Datasette 1.0 3268330 0 2022-12-06T19:01:58Z 2022-12-06T19:02:03Z   OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1934/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1456013930 I_kwDOBm6k_c5WyQJq 1906 Extract publish Heroku support to a plugin simonw 9599 open 0   Datasette 1.0 3268330 0 2022-11-19T00:02:51Z 2022-11-19T00:03:10Z   OWNER  

This is a strong argument for extracting the Heroku support out to a plugin - it would allow this to be fixed with a plugin release without needing to push a full release of Datasette itself.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1905#issuecomment-1320678715

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1906/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1455932972 I_kwDOBm6k_c5Wx8Ys 1904 Datasette Lite tests failing due to httpx upgrade simonw 9599 closed 0   Datasette 1.0a0 8658075 0 2022-11-18T22:49:31Z 2022-11-18T22:57:48Z 2022-11-18T22:52:22Z OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1904/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1454532488 I_kwDOBm6k_c5WsmeI 1902 Document {% block crumbs %} for plugin authors simonw 9599 open 0   Datasette 1.0 3268330 0 2022-11-18T06:16:30Z 2022-11-18T06:16:39Z   OWNER  

For datasette-copyable I want to show breadcrumbs that take database/instance permissions into account, so I'm removing {% block nav %} entirely and replacing it with this:

html+jinja {% block crumbs %} {{ crumbs.nav(request=request, database=database, table=table) }} {% endblock %}

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1901#issuecomment-1319588163

I should document this.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1902/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1424980545 I_kwDOBm6k_c5U73pB 1861 request.headers.get("Content-Type") fails simonw 9599 open 0     0 2022-10-27T03:39:12Z 2022-10-27T03:39:12Z   OWNER  

Turns out this is case-sensitive, needs to be:

request.headers.get("content-type") != "application/json"

That's not great usability. It should be case insensitive.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1861/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1406860394 I_kwDOBm6k_c5T2vxq 1841 Drop format_bytes for Jinja filesizeformat filter simonw 9599 open 0     0 2022-10-12T22:06:34Z 2022-10-12T22:06:34Z   OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1841/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1399933513 I_kwDOBm6k_c5TcUpJ 1833 Ability to submit long queries by POST simonw 9599 open 0     0 2022-10-06T16:03:26Z 2022-10-06T16:18:00Z   OWNER  

Datasette doesn't limit URL lengths but some common web proxies do - the one in front of Google Cloud Run for example limits to 8KB total for incoming request headers: https://cloud.google.com/load-balancing/docs/quotas#https-lb-header-limits

This means longer SQL queries can break!

Need an optional mechanism for submitting queries by POST instead.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1833/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1396977994 I_kwDOBm6k_c5TRDFK 1830 Add documentation for writing tests with signed actor cookies simonw 9599 open 0     0 2022-10-04T23:51:26Z 2022-10-04T23:51:26Z   OWNER  

I use this pattirn in a lot of plugin tests, e.g. https://github.com/simonw/datasette-edit-templates/blob/087f6a6cabc20020f2b0524f11aa3a7836320848/tests/test_edit_templates.py#L55-L58 python actor = ds.sign({"a": {"id": "root"}}, "actor") response1 = await ds.client.get( "/-/edit-templates/_footer.html", cookies={"ds_actor": actor} ) I should add this to the documentation on this page: https://docs.datasette.io/en/latest/testing_plugins.html

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1830/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1378636455 I_kwDOBm6k_c5SLFKn 1815 `datasette publish provider .` to publish whole directory, similar to configuration directory mode simonw 9599 open 0     0 2022-09-19T23:28:59Z 2022-09-19T23:29:11Z   OWNER  

I haven't done this with any of my other datasette publish tools, but I do think it's a good idea. Being able to publish the entire directory - with templates and plugins and metadata - does seem very useful to me.

Originally posted by @simonw in https://github.com/simonw/datasette-publish-fly/issues/23#issuecomment-1251673489

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1815/reactions",
    "total_count": 2,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1359604075 I_kwDOCGYnMM5RCelr 481 Idea: `sqlite-utils create-table tablename --sql "select ..."` simonw 9599 open 0     0 2022-09-02T01:41:24Z 2022-09-02T01:42:08Z   OWNER  

Could offer syntactic sugar for:

sql create table foo as select * from bar

sqlite-utils create-table data.db foo --sql "select * from bar" https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/481/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1353088849 I_kwDOBm6k_c5Qpn9R 1795 Consider automatically cleaning up curly quotes in searches simonw 9599 open 0     0 2022-08-27T16:35:25Z 2022-08-27T16:35:25Z   OWNER  

If your phone helpfully adds curly quotes for you then phrase searches against FTS won't work: “Rebecca Sugar”

In regular (not ?_searchmode=raw search mode Datasette could clean these up for you to help avoid that mistake.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1795/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1345561209 I_kwDOBm6k_c5QM6J5 1790 A better HTML title for canned query pages simonw 9599 open 0     0 2022-08-21T18:27:46Z 2022-08-21T18:27:46Z   OWNER  

https://scotrail.datasette.io/scotrail/assemble_sentence?terms=This+train+is+formed+of%2Cbomb+which

Current title is:

scotrail: with phrases as ( select key, value from json_each('["' || replace(:terms, ',', '","') || '"]')),matches as (select phrases.key, phrases.value, ( select File from announcements where announcements.Transcription like '%' || trim(phrases.value) || '%' order by length(announcements.Transcription) limit 1 ) as Filefrom phrases),results as ( select key, announcements.Transcription, announcements.mp3 from announcements join matches on announcements.File = matches.File order by key)select 'Combined sentence:' as mp3, group_concat(Transcription, ' ') as Transcription, -1 as keyfrom results unionselect mp3, Transcription, keyfrom resultsorder by key

I think a better title would be:

scotrail: assemble_sentence, terms = This train is formed of,bomb which

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1790/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1343732788 I_kwDOBm6k_c5QF7w0 1788 Make it more obvious that Datasette publish can publish multiple databases simonw 9599 closed 0     0 2022-08-18T22:57:51Z 2022-08-18T23:06:16Z 2022-08-18T23:06:16Z OWNER  

Feedback initially for datasette-publish-fly but it applies to the others too.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1788/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1342374388 PR_kwDOCGYnMM49Wv9T 466 Use Read the Docs action v1 (#463) simonw 9599 closed 0     0 2022-08-17T23:11:50Z 2022-08-17T23:11:54Z 2022-08-17T23:11:54Z OWNER simonw/sqlite-utils/pulls/466

Read the Docs repository was renamed from readthedocs/readthedocs-preview to readthedocs/actions/. Now, the preview action is under readthedocs/actions/preview and is tagged as v1

 sqlite-utils 140912432 pull     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/466/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1340900019 I_kwDOBm6k_c5P7IKz 1785 Can't use cog menu to facet by first column in a view simonw 9599 open 0     0 2022-08-16T21:27:23Z 2022-08-16T21:27:23Z   OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1785/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1266329095 I_kwDOBm6k_c5LeqYH 1756 Mechanism for creating databases in WAL mode simonw 9599 open 0     0 2022-06-09T15:39:28Z 2022-06-09T15:39:28Z   OWNER  

The --create option currently creates databases if they are missing, but does not enable WAL mode for them.

It turns out WAL mode is useful for databases that are accepting writes!

I think a --create-wal option that both creates them AND sets WAL mode on any that are created would be a good idea.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1756/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1251739062 I_kwDOBm6k_c5KnAW2 1752 Research if I can drop Janus simonw 9599 open 0     0 2022-05-28T22:46:52Z 2022-05-28T22:46:52Z   OWNER  

It seems to me Janus dependency is not necessary, async with app.database_write_mutex(): out = await app.transaction(func) may be enough.

Comment here: https://lobste.rs/s/fki4tj/architecture_notes_datasette#c_a2ihon

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1752/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1243715381 I_kwDOCGYnMM5KIZc1 436 Add "copy to clipboard" button to code examples in documentation simonw 9599 closed 0     0 2022-05-20T21:53:23Z 2022-05-20T21:57:53Z 2022-05-20T21:57:53Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/436/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1237871948 I_kwDOBm6k_c5JyG1M 1743 `datasette.utils.to_css_class()` should be a documented internal simonw 9599 open 0     0 2022-05-16T23:57:26Z 2022-05-16T23:57:26Z   OWNER  

Because I'm using it in this plugin: - https://github.com/simonw/datasette-upload-dbs/issues/1

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1743/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1223312279 PR_kwDOBm6k_c43MIU0 1736 Clean up compatibility with Pyodide simonw 9599 closed 0     0 2022-05-02T20:14:38Z 2022-05-02T20:15:28Z 2022-05-02T20:15:27Z OWNER simonw/datasette/pulls/1736

Closes #1735, closes #1733

 datasette 107914493 pull     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1736/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1219398983 I_kwDOBm6k_c5Iro1H 1730 SQL tracing should much more closely track the SQL query execution simonw 9599 open 0     0 2022-04-28T22:41:04Z 2022-04-28T22:41:10Z   OWNER  

In #1727 I realized that the SQL tracing was measuring a whole bunch of stuff outside of the SQL query itself.

I started experimenting with this fix for that but it didn't work - I got back an empty JSON array of traces for some reason:

```diff diff --git a/datasette/database.py b/datasette/database.py index ba594a8..d7f9172 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -7,7 +7,7 @@ import sys import threading import uuid

-from .tracer import trace +from .tracer import trace, trace_child_tasks from .utils import ( detect_fts, detect_primary_keys, @@ -207,30 +207,31 @@ class Database: time_limit_ms = custom_time_limit

         with sqlite_timelimit(conn, time_limit_ms):
  • try:
  • cursor = conn.cursor()
  • cursor.execute(sql, params if params is not None else {})
  • max_returned_rows = self.ds.max_returned_rows
  • if max_returned_rows == page_size:
  • max_returned_rows += 1
  • if max_returned_rows and truncate:
  • rows = cursor.fetchmany(max_returned_rows + 1)
  • truncated = len(rows) > max_returned_rows
  • rows = rows[:max_returned_rows]
  • else:
  • rows = cursor.fetchall()
  • truncated = False
  • except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:
  • if e.args == ("interrupted",):
  • raise QueryInterrupted(e, sql, params)
  • if log_sql_errors:
  • sys.stderr.write(
  • "ERROR: conn={}, sql = {}, params = {}: {}\n".format(
  • conn, repr(sql), params, e
  • with trace("sql", database=self.name, sql=sql.strip(), params=params):
  • try:
  • cursor = conn.cursor()
  • cursor.execute(sql, params if params is not None else {})
  • max_returned_rows = self.ds.max_returned_rows
  • if max_returned_rows == page_size:
  • max_returned_rows += 1
  • if max_returned_rows and truncate:
  • rows = cursor.fetchmany(max_returned_rows + 1)
  • truncated = len(rows) > max_returned_rows
  • rows = rows[:max_returned_rows]
  • else:
  • rows = cursor.fetchall()
  • truncated = False
  • except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:
  • if e.args == ("interrupted",):
  • raise QueryInterrupted(e, sql, params)
  • if log_sql_errors:
  • sys.stderr.write(
  • "ERROR: conn={}, sql = {}, params = {}: {}\n".format(
  • conn, repr(sql), params, e
  • ) )
  • )
  • sys.stderr.flush()
  • raise
  • sys.stderr.flush()

  • raise

         if truncate:
         return Results(rows, truncated, cursor.description)

    @@ -238,9 +239,8 @@ class Database: else: return Results(rows, False, cursor.description)

  • with trace("sql", database=self.name, sql=sql.strip(), params=params):

  • results = await self.execute_fn(sql_operation_in_thread)
  • return results
  • with trace_child_tasks():

  • return await self.execute_fn(sql_operation_in_thread)

    @property def size(self): ```

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1727#issuecomment-1111602802

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1730/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1217014076 I_kwDOBm6k_c5Iiik8 1726 Security page in the documentation simonw 9599 open 0     0 2022-04-27T08:43:30Z 2022-04-27T08:43:30Z   OWNER  

A page talking about how to run Datasette securely, and security concerns to take into account.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1726/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1216622905 I_kwDOBm6k_c5IhDE5 1725 Performance question - what is happening in this gap? simonw 9599 open 0     0 2022-04-27T00:21:11Z 2022-04-27T00:21:11Z   OWNER  

Trace from https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_trace=1&_facet=committer

What's going on in that gap? Can I improve the tracing output to show some non-SQL queries to figure that out?

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1725/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1216479167 I_kwDOBm6k_c5Igf-_ 1722 `db.primary_keys()` and `db.table_columns()` don't show up in traces simonw 9599 open 0     0 2022-04-26T21:08:36Z 2022-04-26T21:08:36Z   OWNER  

Noticed this while working on: - #1715

This code here isn't showing up in traces: https://github.com/simonw/datasette/blob/579f59dcec43a91dd7d404e00b87a00afd8515f2/datasette/views/table.py#L218-L220

Because those functions don't use the regular trace-instrumented db.execute() code path - they work directly against a connection instead: https://github.com/simonw/datasette/blob/579f59dcec43a91dd7d404e00b87a00afd8515f2/datasette/utils/init.py#L610-L626

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1722/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1197925865 I_kwDOBm6k_c5HZuXp 1704 File PRs against incompatible plugins pinning to datasette<1.0 simonw 9599 open 0   Datasette 1.0 3268330 0 2022-04-08T23:15:30Z 2022-04-08T23:15:30Z   OWNER  

As part of the preparation for the 1.0 release, test all existing known plugins against the alpha.

For any that break, submit a PR suggesting they pin to a version <1.0 - and include a link to the documentation on how to upgrade the plugin for 1.0.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1704/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1196327155 I_kwDOBm6k_c5HToDz 1702 Be more consistent with column quoting simonw 9599 open 0     0 2022-04-07T16:59:20Z 2022-04-07T16:59:20Z   OWNER  

This tutorial made me notice that Datasette is pretty inconsistent with how column quoting works: https://datasette.io/tutorials/learn-sql

It has examples of each of "table_name" and [table_name] and table_name, and it uses single quoted values too.

Datasette should generate SQL as consistently as possible to support learners.

That tutorial should also provide a tiny bit of extra information about what's going on here.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1702/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1194790504 I_kwDOBm6k_c5HNw5o 1701 Use + for spaces instead of ~20 simonw 9599 closed 0   Datasette 1.0 3268330 0 2022-04-06T15:40:48Z 2022-04-06T15:55:10Z 2022-04-06T15:55:05Z OWNER  

Tilde encoding introduced in #1657 means that database files with spaces in the name - e.g. the Apple Mail Envelope Index database - end up with URLs like this:

http://127.0.0.1:8001/Envelope~20Index

I think this would be prettier:

http://127.0.0.1:9933/Envelope+Index
datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1701/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1182143895 I_kwDOBm6k_c5GdhWX 1691 Bug in pytest-httpx example simonw 9599 closed 0     0 2022-03-26T22:45:30Z 2022-03-26T22:46:09Z 2022-03-26T22:46:09Z OWNER  

https://docs.datasette.io/en/0.61.1/testing_plugins.html#testing-outbound-http-calls-with-pytest-httpx says:

python async def test_outbound_http_call(httpx_mock): httpx_mock.add_response( url='https://www.example.com/', data='Hello world', ) That's wrong - data= should be text=.

https://github.com/Colin-b/pytest_httpx/blob/v0.20.0/README.md#reply-with-custom-body

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1691/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1181364043 I_kwDOBm6k_c5Gai9L 1687 Make show_json.html or a similar mechanism stable for plugins simonw 9599 open 0     0 2022-03-25T23:42:45Z 2022-03-25T23:42:45Z   OWNER  

I used show_json.html in the new datasette-packages plugin, which means it will break if that template changes: - https://github.com/simonw/datasette-packages/issues/3

It would be useful if it (or something like it) was documented and stable for plugins to use.

Also relevant: - #878

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1687/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1179928510 I_kwDOBm6k_c5GVEe- 1683 allow_facet: False should be respected by column cog menu simonw 9599 closed 0     0 2022-03-24T19:05:06Z 2022-03-24T19:16:36Z 2022-03-24T19:16:36Z OWNER  

The column cog menu currently shows "Facet by this" even if faceting is disabled for the Datasette instance.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1683/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
953218043 MDU6SXNzdWU5NTMyMTgwNDM= 1403 Labels explaining what hidden tables are for simonw 9599 open 0     0 2021-07-26T19:29:22Z 2022-03-21T22:20:37Z   OWNER  

A reasonable question: "What are those hidden tables for?"

This could be answered by adding a small piece of explanatory text to each table - based on if it's related to FTS or to SpatiaLite or configured to be hidden for some other reason.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1403/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1175894898 I_kwDOBm6k_c5GFrty 1680 Consider simplifying permissions for 1.0 simonw 9599 open 0   Datasette 1.0 3268330 0 2022-03-21T20:17:29Z 2022-03-21T20:17:29Z   OWNER  

Permission checks right now can express one of three opinions:

  • False means "so not grant this permisson"
  • True means "grant this permission"
  • None means "I have no opinion"

But... there's also a concept of a "default" for a given permission check, which might be False or True.

I worry this is too complicated. Could this be simplified before 1.0? In particular the default concept.

See also: - #1676

datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1680/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1122413719 I_kwDOBm6k_c5C5qyX 1621 Test against Python 3.11 dev version simonw 9599 closed 0   Datasette 1.0 3268330 0 2022-02-02T21:38:57Z 2022-03-19T04:04:49Z 2022-02-02T21:58:54Z OWNER  

To avoid another surprise like we got with 3.10: https://simonwillison.net/2021/Oct/9/finding-and-reporting-a-bug/

From a quick GitHub code search it looks like 3.11-dev should work: https://cs.github.com/urllib3/urllib3/blob/7bec77e81aa0a194c98381053225813f5347c9d2/.github/workflows/ci.yml#L60

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1621/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1148638868 I_kwDOBm6k_c5EdtaU 1639 Make datasette-redirect-forbidden unneccessary simonw 9599 open 0     0 2022-02-23T22:18:46Z 2022-02-23T22:18:46Z   OWNER  

I wrote datasette-redirect-forbidden today because I needed 403 errors to redirect to /-/login and it was the quickest way to solve that problem.

This should be a feature of Datasette core.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1639/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1142107925 I_kwDOBm6k_c5EEy8V 1638 `filters_from_request` plugin hook docs should mention that returning an async function is allowed simonw 9599 open 0     0 2022-02-18T00:08:26Z 2022-02-18T00:08:26Z   OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1638/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1125576543 I_kwDOBm6k_c5DFu9f 1630 Review datasette.utils and decide which functions should be documented for 1.0 simonw 9599 open 0   Datasette 1.0 3268330 0 2022-02-07T06:39:52Z 2022-02-07T06:39:52Z   OWNER  

Follows: - #1176

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1630/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1123851690 I_kwDOCGYnMM5C_J2q 396 mypy failure, sqlite_utils/utils.py:56 simonw 9599 closed 0     0 2022-02-04T06:08:09Z 2022-02-04T06:10:33Z 2022-02-04T06:10:33Z OWNER  

https://github.com/simonw/sqlite-utils/runs/5062725880?check_suite_focus=true

sqlite_utils/utils.py:56: error: Incompatible return value type (got "None", expected "str")

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/396/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1122557010 I_kwDOBm6k_c5C6NxS 1627 Get the tests passing against Windows simonw 9599 open 0     0 2022-02-03T01:23:06Z 2022-02-03T01:23:32Z   OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1627/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1122450452 I_kwDOBm6k_c5C5zwU 1625 Try running tests against macOS and Windows in addition to Ubuntu simonw 9599 open 0     0 2022-02-02T22:25:57Z 2022-02-02T22:25:57Z   OWNER  

I already do this for sqlite-utils: https://github.com/simonw/sqlite-utils/blob/3.22.1/.github/workflows/test.yml

Related: - #1617 - #1545

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1625/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1099897648 I_kwDOCGYnMM5Bjxsw 384 Add examples to every `--help` simonw 9599 closed 0     0 2022-01-12T05:31:25Z 2022-01-26T03:15:02Z 2022-01-26T03:15:02Z OWNER  

Everything on https://sqlite-utils.datasette.io/en/stable/cli-reference.html would benefit from an example.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/384/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1107557831 I_kwDOCGYnMM5CA_3H 386 Better "contributing" documentation simonw 9599 closed 0     0 2022-01-19T02:11:48Z 2022-01-19T02:15:21Z 2022-01-19T02:15:21Z OWNER  

This page jumps straight into running the tests: https://sqlite-utils.datasette.io/en/latest/contributing.html

It should add a little more about expected collaboration styles - opening an issue before filing a pull request - and probably link to https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/386/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1102966378 I_kwDOBm6k_c5Bve5q 1599 Add architecture documentation simonw 9599 open 0     0 2022-01-14T04:55:38Z 2022-01-14T04:56:03Z   OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1599/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1102568047 I_kwDOBm6k_c5Bt9pv 1596 Documentation page warning of changes coming in 1.0 simonw 9599 open 0     0 2022-01-13T23:26:04Z 2022-01-13T23:26:04Z   OWNER  

I should start this relatively soon.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1596/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1082746149 I_kwDOBm6k_c5AiWUl 1560 Table page title has "where where" in it simonw 9599 closed 0   Datasette 0.60 7571612 0 2021-12-17T00:05:48Z 2022-01-13T22:28:35Z 2022-01-13T22:20:15Z OWNER  

Just noticed this while working on #1518.

% curl -s 'https://latest.datasette.io/fixtures/facetable?_sort=pk&on_earth__exact=1' | grep -C 1 '<title>' <head> <title>fixtures: facetable: 14 rows where where on_earth = 1 sorted by pk</title>

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1560/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1083927147 I_kwDOBm6k_c5Am2pr 1571 Track number of executions for execute_write_many() in traces simonw 9599 closed 0   Datasette 0.60 7571612 0 2021-12-18T19:16:17Z 2022-01-13T22:27:49Z 2021-12-19T20:30:40Z OWNER  

Spotted while working on #1555

There's no indication there of how many times execute_write_many() executed the SQL.

Solving this is a tiny bit tricky because params_seq is an iterator that we don't want to exhaust before passing it to conn.executemany() - so we need to instead wrap it in something that counts how many times it was called.

But then we need a way to attach that to the trace here: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/database.py#L115-L122

So probably need to redesign the trace() decorator to allow extra pairs to be attached to it within the with statement.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1571/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1084007781 I_kwDOBm6k_c5AnKVl 1572 "Query took" should be "Queries took" simonw 9599 closed 0   Datasette 0.60 7571612 0 2021-12-19T04:03:00Z 2022-01-13T22:27:43Z 2021-12-19T04:03:24Z OWNER  

This is misleading, since usually there have been more than one query executed:

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1572/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1083718998 I_kwDOBm6k_c5AmD1W 1567 Remove undocumented sqlite_functions mechanism simonw 9599 closed 0   Datasette 0.60 7571612 0 2021-12-18T01:51:10Z 2022-01-13T22:27:04Z 2021-12-18T01:54:46Z OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1567/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1098544628 I_kwDOCGYnMM5BenX0 379 CLI options for running ANALYZE simonw 9599 closed 0   3.21 7558727 0 2022-01-11T01:09:16Z 2022-01-11T01:38:01Z 2022-01-11T01:36:48Z OWNER  

The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865

  • [x] sqlite-utils analyze command
  • [x] sqlite-utils create-index --analyze option (see #365)
  • [x] sqlite-utils insert --analyze option
  • [x] sqlite-utils upsert --analyze option

In #378 I also added .delete_where(..., analyze=True) but there isn't currently a sqlite-utils delete-where CLI command - deletions via CLI are expected to be handled using SQL queries.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/379/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1098309897 I_kwDOCGYnMM5BduEJ 378 analyze=True parameter for some methods simonw 9599 closed 0   3.21 7558727 0 2022-01-10T19:54:52Z 2022-01-11T01:08:11Z 2022-01-11T01:08:09Z OWNER  

This would cause ANALYZE to be run against the relevant table at the end of executing the method.

Having browsed the API reference I think the methods that would benefit from an analyze=True parameter are:

  • [x] table.create_index
  • [x] table.insert_all
  • [x] table.upsert_all
  • [x] table.delete_where

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/378/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1097436959 I_kwDOCGYnMM5BaY8f 376 `--nl` mode should ignore blank lines simonw 9599 closed 0   3.21 7558727 0 2022-01-10T04:10:54Z 2022-01-10T19:27:41Z 2022-01-10T04:12:46Z OWNER  

Spotted this while manually testing #364 - there's no reason --nl should crash if you feed it an empty line in between JSON objects.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/376/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1094974713 I_kwDOCGYnMM5BQ_z5 362 upsert --detect-types is broken simonw 9599 closed 0     0 2022-01-06T05:12:10Z 2022-01-06T06:54:45Z 2022-01-06T06:28:34Z OWNER  

Noticed this thanks to syntax highlighting in VS Code showing an unused variable - need to fix it and add a test.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/362/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1090798237 I_kwDOCGYnMM5BBEKd 359 Use RETURNING if available to populate last_pk simonw 9599 open 0     0 2021-12-29T23:43:23Z 2021-12-29T23:43:23Z   OWNER  

Inspired by this: https://news.ycombinator.com/item?id=29729283

Because SQLite is effectively serializing all the writes for us, we have zero locking in our code. We used to have to lock when inserting new items (to get the LastInsertRowId), but the newer version of SQLite supports the RETURNING keyword, so we don't even have to lock on inserts now.

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/359/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
741231849 MDU6SXNzdWU3NDEyMzE4NDk= 1087 Idea: ?_extra=urls for getting back URLs to useful things simonw 9599 open 0     0 2020-11-12T02:55:41Z 2021-12-15T18:06:16Z   OWNER  

Working on https://github.com/simonw/datasette-search-all/issues/10 made me realize that sometimes it can be difficult to calculate the URL for a database, table or row within Datasette.

It would be useful to have an optional extra JSON extension (using ?_extra= from #262) that can help with this.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1087/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1067775061 I_kwDOBm6k_c4_pPRV 1539 Research PRAGMA query_only simonw 9599 open 0     0 2021-11-30T23:30:24Z 2021-11-30T23:30:24Z   OWNER  

https://www.sqlite.org/pragma.html#pragma_query_only

The query_only pragma prevents data changes on database files when enabled. When this pragma is enabled, any attempt to CREATE, DELETE, DROP, INSERT, or UPDATE will result in an SQLITE_READONLY error. However, the database is not truly read-only. You can still run a checkpoint or a COMMIT and the return value of the sqlite3_db_readonly() routine is not affected.

Would it be worth adding this as an extra protection against accidental writes to a DB file over a read-only connection?

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1539/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1059209412 I_kwDOBm6k_c4_IkDE 1523 Come up with a more elegant solution for base_url than ds.urls.path() simonw 9599 open 0     0 2021-11-20T19:05:22Z 2021-11-20T19:05:22Z   OWNER  

While fixing #1519 I added a lot of ugly code that looks like this: https://github.com/simonw/datasette/blob/08947fa76433d18988aa1ee1d929bd8320c75fe2/datasette/facets.py#L228-L230

See these two commits in particular: fe687fd0207c4c56c4778d3e92e3505fc4b18172 and 08947fa76433d18988aa1ee1d929bd8320c75fe2

It would be great to come up with a less verbose and error-prone way of handling this problem.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1523/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1056746091 I_kwDOBm6k_c4-_Kpr 1515 Handle foreign keys that point to a non-existent table simonw 9599 open 0     0 2021-11-17T23:40:13Z 2021-11-18T01:31:56Z   OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1515/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1053136495 I_kwDOCGYnMM4-xZZv 341 `hash_id: Optional[Any]` should be `hash_id: Optional[str]` simonw 9599 closed 0     0 2021-11-15T02:12:39Z 2021-11-15T02:19:31Z 2021-11-15T02:19:31Z OWNER   sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/341/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1053087862 I_kwDOCGYnMM4-xNh2 338 dict, list, tuple should all map to TEXT simonw 9599 closed 0     0 2021-11-15T00:28:01Z 2021-11-15T00:36:03Z 2021-11-15T00:36:03Z OWNER  

This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947

So the COLUMN_TYPE_MAPPING should include those too - right now it looks like this: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L165-L188

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/322#issuecomment-968401459

 sqlite-utils 140912432 issue     
{
    "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/338/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1051277222 I_kwDOBm6k_c4-qTem 1504 Link to ?_size=max at bottom of table page simonw 9599 open 0     0 2021-11-11T19:06:33Z 2021-11-11T19:06:33Z   OWNER  

This can have text such as "Show 1,000 rows per page", based on the max size limit setting. Would make it easier for people to see more data at once without having to know how to hack the URL, similar to the ... for facet sizes I added in #1337.

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1504/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1026664511 PR_kwDOBm6k_c4tNtoe 1490 Upgrade to httpx 0.20 simonw 9599 closed 0     0 2021-10-14T17:51:05Z 2021-10-14T18:03:45Z 2021-10-14T18:03:44Z OWNER simonw/datasette/pulls/1490

Refs #1488

datasette 107914493 pull     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1490/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
1021849766 I_kwDOBm6k_c486DCm 1483 Running a search on page 2 of results should not preserve ?_next= simonw 9599 closed 0     0 2021-10-10T01:18:12Z 2021-10-13T21:08:10Z 2021-10-13T21:08:10Z OWNER   datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1483/reactions",
    "total_count": 2,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
1006016302 I_kwDOBm6k_c479pcu 1477 Consider adding request to the documented default template context simonw 9599 open 0     0 2021-09-24T02:34:09Z 2021-09-24T02:34:09Z   OWNER  

I made a plugin for this today but I think perhaps it should be a default thing instead: https://datasette.io/plugins/datasette-template-request

 datasette 107914493 issue     
{
    "url": "https://api.github.com/repos/simonw/datasette/issues/1477/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
975158266 MDU6SXNzdWU5NzUxNTgyNjY= 19 table activity_summary has no column named appleMoveTime simonw 9599 closed 0     0 2021-08-20T00:46:44Z 2021-08-20T00:54:34Z 2021-08-20T00:54:34Z MEMBER  

Got this error today against a fresh export:

table activity_summary has no column named appleMoveTime
healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/19/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);