> .
>
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325373747,Build Dockerfile with recent Sqlite + Spatialite,
https://github.com/simonw/datasette/pull/279#issuecomment-391077700,https://api.github.com/repos/simonw/datasette/issues/279,391077700,MDEyOklzc3VlQ29tbWVudDM5MTA3NzcwMA==,198537,rgieseke,2018-05-22T17:38:17Z,2018-05-22T17:38:17Z,CONTRIBUTOR,"Alright, that should work now -- let me know if you would prefer any different behaviour.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325352370,Add version number support with Versioneer,
https://github.com/simonw/datasette/pull/279#issuecomment-391073267,https://api.github.com/repos/simonw/datasette/issues/279,391073267,MDEyOklzc3VlQ29tbWVudDM5MTA3MzI2Nw==,198537,rgieseke,2018-05-22T17:24:16Z,2018-05-22T17:24:16Z,CONTRIBUTOR,"Sorry, just realised you rely on `version` being a module ...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325352370,Add version number support with Versioneer,
https://github.com/simonw/datasette/pull/279#issuecomment-391073009,https://api.github.com/repos/simonw/datasette/issues/279,391073009,MDEyOklzc3VlQ29tbWVudDM5MTA3MzAwOQ==,198537,rgieseke,2018-05-22T17:23:26Z,2018-05-22T17:23:26Z,CONTRIBUTOR,"> I think I prefer the aesthetics of just ""0.22"" for the version string if it's a tagged release with no additional changes - does that work?
Yes! That's the default versioneer behaviour.
> I'd like to continue to provide a tuple that can be imported from the version.py module as well, as seen here:
Should work now, it can be a two (for a tagged version), three or four items tuple.
```
In [2]: datasette.__version__
Out[2]: '0.12+292.ga70c2a8.dirty'
In [3]: datasette.__version_info__
Out[3]: ('0', '12+292', 'ga70c2a8', 'dirty')
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325352370,Add version number support with Versioneer,
https://github.com/simonw/datasette/pull/280#issuecomment-391059008,https://api.github.com/repos/simonw/datasette/issues/280,391059008,MDEyOklzc3VlQ29tbWVudDM5MTA1OTAwOA==,565628,r4vi,2018-05-22T16:40:27Z,2018-05-22T16:40:27Z,CONTRIBUTOR,"```python
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.23.1'
>>>
```
running the above in the container seems to show 3.23.1 too so maybe we don't need pysqlite3 at all?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325373747,Build Dockerfile with recent Sqlite + Spatialite,
https://github.com/simonw/datasette/issues/276#issuecomment-391050113,https://api.github.com/repos/simonw/datasette/issues/276,391050113,MDEyOklzc3VlQ29tbWVudDM5MTA1MDExMw==,45057,russss,2018-05-22T16:13:00Z,2018-05-22T16:13:00Z,CONTRIBUTOR,"Yup, I'll have a think about it. My current thoughts are for spatialite we'll need to hook into the following places:
* Inspection, so we can detect which columns are geometry columns. (We also currently ignore spatialite tables during inspection, it may be worth moving that to the plugin as well.)
* After data load, so we can convert WKB into the correct intermediate format for display. The alternative here is to alter the select SQL itself and get spatialite to do this conversion, but that strikes me as a bit more complex and possibly not as useful.
* HTML rendering.
* Querying?
The rendering and querying hooks could also potentially be used to move the units support into a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,Handle spatialite geometry columns better,
https://github.com/simonw/datasette/issues/276#issuecomment-390795067,https://api.github.com/repos/simonw/datasette/issues/276,390795067,MDEyOklzc3VlQ29tbWVudDM5MDc5NTA2Nw==,45057,russss,2018-05-21T21:55:57Z,2018-05-21T21:55:57Z,CONTRIBUTOR,"Well, we do have the capability to detect spatialite so my intention certainly wasn't to require it.
I can see the advantage of having it as a plugin but it does touch a number of points in the code. I think I'm going to attack this by refactoring the necessary bits and seeing where that leads (which was my plan anyway).
I think my main concern is - if I add certain plugin hooks for this, is anything else ever going to use them? I'm not sure I have an answer to that question yet, either way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,Handle spatialite geometry columns better,
https://github.com/simonw/datasette/issues/273#issuecomment-390250253,https://api.github.com/repos/simonw/datasette/issues/273,390250253,MDEyOklzc3VlQ29tbWVudDM5MDI1MDI1Mw==,198537,rgieseke,2018-05-18T15:49:52Z,2018-05-18T15:49:52Z,CONTRIBUTOR,"Shouldn't [versioneer](https://github.com/warner/python-versioneer) do that?
E.g. 0.21+2.g1076c97
You'd need to install via `pip install git+https://github.com/simow/datasette.git` though, this does a temp git clone.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324451322,Figure out a way to have /-/version return current git commit hash,
https://github.com/simonw/datasette/pull/209#issuecomment-381905593,https://api.github.com/repos/simonw/datasette/issues/209,381905593,MDEyOklzc3VlQ29tbWVudDM4MTkwNTU5Mw==,45057,russss,2018-04-17T08:50:28Z,2018-04-17T08:50:28Z,CONTRIBUTOR,"I've added another commit which puts classes a class on each `` by default with its column name, and I've also made the PK column bold.
Unfortunately the tests are still failing on 3.6, which is weird. I can't reproduce locally...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314455877, Don't duplicate simple primary keys in the link column,
https://github.com/simonw/datasette/pull/209#issuecomment-381738137,https://api.github.com/repos/simonw/datasette/issues/209,381738137,MDEyOklzc3VlQ29tbWVudDM4MTczODEzNw==,45057,russss,2018-04-16T20:27:43Z,2018-04-16T20:27:43Z,CONTRIBUTOR,"Tests now fixed, honest. The failing test on Travis looks like an intermittent sqlite failure which should resolve itself on a retry...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314455877, Don't duplicate simple primary keys in the link column,
https://github.com/simonw/datasette/issues/203#issuecomment-381763651,https://api.github.com/repos/simonw/datasette/issues/203,381763651,MDEyOklzc3VlQ29tbWVudDM4MTc2MzY1MQ==,45057,russss,2018-04-16T21:59:17Z,2018-04-16T21:59:17Z,CONTRIBUTOR,"Ah, I had no idea you could bind python functions into sqlite!
I think the primary purpose of this issue has been served now - I'm going to close this and create a new issue for the only bit of this that hasn't been touched yet, which is (optionally) exposing units in the JSON API.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",313837303,Support for units,
https://github.com/simonw/datasette/pull/209#issuecomment-381441392,https://api.github.com/repos/simonw/datasette/issues/209,381441392,MDEyOklzc3VlQ29tbWVudDM4MTQ0MTM5Mg==,45057,russss,2018-04-15T21:59:15Z,2018-04-15T21:59:15Z,CONTRIBUTOR,"I suspected this would cause some test failures, but I'll wait for opinions before attempting to fix them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314455877, Don't duplicate simple primary keys in the link column,
https://github.com/simonw/datasette/pull/205#issuecomment-381332222,https://api.github.com/repos/simonw/datasette/issues/205,381332222,MDEyOklzc3VlQ29tbWVudDM4MTMzMjIyMg==,45057,russss,2018-04-14T14:16:35Z,2018-04-14T14:16:35Z,CONTRIBUTOR,I've added some tests and that docs link.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",314319372,Support filtering with units and more,
https://github.com/simonw/datasette/issues/203#issuecomment-381315675,https://api.github.com/repos/simonw/datasette/issues/203,381315675,MDEyOklzc3VlQ29tbWVudDM4MTMxNTY3NQ==,45057,russss,2018-04-14T09:14:45Z,2018-04-14T09:27:30Z,CONTRIBUTOR,"> I'd like to figure out a sensible opt-in way to expose this in the JSON output as well. Maybe with a &_units=true parameter?
From a machine-readable perspective I'm not sure why it would be useful to decorate the values with units. Edit: Should have had some coffee first. It's clearly useful for stuff like map rendering!
I agree that the unit metadata should definitely be exposed in the JSON.
> In #204 you said ""I'd like to add support for using units when querying but this is PR is pretty usable as-is."" - I'm fascinated to hear more about how this could work.
I'm thinking about a couple of approaches here. I think the simplest one is: if the column has a unit attached, optionally accept units in query fields:
```python
column_units = ureg(""Hz"") # Create a unit object for the column's unit
query_variable = ureg(""4 GHz"") # Supplied query variable
# Now we can convert the query units into column units before querying
supplied_value.to(column_units).magnitude
> 4000000000.0
# If the user doesn't supply units, pint just returns the plain
# number and we can query as usual assuming it's the base unit
query_variable = ureg(""50"")
query_variable
> 50
isinstance(query_variable, numbers.Number)
> True
```
This also lets us do some nice unit conversion on querying:
```python
column_units = ureg(""m"")
query_variable = ureg(""50 ft"")
supplied_value.to(column_units)
>
```
The alternative would be to provide a dropdown of units next to the query field (so a ""Hz"" field would give you ""kHz"", ""MHz"", ""GHz""). Although this would be clearer to the user, it isn't so easy - we'd need to know more about the context of the field to give you sensible SI prefixes (I'm not so interested in nanoHertz, for example).
You also lose the bonus of being able to convert - although pint will happily show you all the compatible units, it again suffers from a lack of context:
```python
ureg(""m"").compatible_units()
> frozenset({,
,
,
,
,
,
,
,
,
,
,
})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",313837303,Support for units,
https://github.com/simonw/datasette/issues/125#issuecomment-381361734,https://api.github.com/repos/simonw/datasette/issues/125,381361734,MDEyOklzc3VlQ29tbWVudDM4MTM2MTczNA==,45057,russss,2018-04-14T21:26:30Z,2018-04-14T21:26:30Z,CONTRIBUTOR,"FWIW I am now doing this on my WTR app (instead of silently limiting maps to 1000).
[Telefonica](https://wtr-api.herokuapp.com/wtr-663ea99/licensee/18325) now has about 4000 markers and good old [BT](https://wtr-api.herokuapp.com/wtr-663ea99/licensee/8412) has 22,000 or so.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275135393,Plot rows on a map with Leaflet and Leaflet.markercluster,
https://github.com/simonw/datasette/pull/202#issuecomment-381237440,https://api.github.com/repos/simonw/datasette/issues/202,381237440,MDEyOklzc3VlQ29tbWVudDM4MTIzNzQ0MA==,45057,russss,2018-04-13T19:22:53Z,2018-04-13T19:22:53Z,CONTRIBUTOR,I spotted you'd mentioned that in #184 but only after I'd written the patch!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",313785206,Raise 404 on nonexistent table URLs,
https://github.com/simonw/datasette/issues/203#issuecomment-380966565,https://api.github.com/repos/simonw/datasette/issues/203,380966565,MDEyOklzc3VlQ29tbWVudDM4MDk2NjU2NQ==,45057,russss,2018-04-12T22:43:08Z,2018-04-12T22:43:08Z,CONTRIBUTOR,"Looks like [pint](https://pint.readthedocs.io/en/latest/tutorial.html) is pretty good at this.
```python
In [1]: import pint
In [2]: ureg = pint.UnitRegistry()
In [3]: q = 3e6 * ureg('Hz')
In [4]: '{:~P}'.format(q.to_compact())
Out[4]: '3.0 MHz'
In [5]: q = 0.3 * ureg('m')
In [5]: '{:~P}'.format(q.to_compact())
Out[5]: '300.0 mm'
In [6]: q = 5 * ureg('')
In [7]: '{:~P}'.format(q.to_compact())
Out[7]: '5'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",313837303,Support for units,
https://github.com/simonw/datasette/pull/200#issuecomment-380608372,https://api.github.com/repos/simonw/datasette/issues/200,380608372,MDEyOklzc3VlQ29tbWVudDM4MDYwODM3Mg==,45057,russss,2018-04-11T21:55:46Z,2018-04-11T21:55:46Z,CONTRIBUTOR,"> I think the most reliable way to detect spatialite is to run `SELECT AddGeometryColumn(1, 2, 3, 4, 5);` against a `:memory:` database and see if it throws an exception
Or just see if there's a `geometry_columns` table? I think that's quite unlikely to be added by accident (and it's an OGC standard). It also tells you if Spatialite is installed in the database rather than just loaded.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",313494458,Hide Spatialite system tables,
https://github.com/simonw/datasette/issues/179#issuecomment-360535979,https://api.github.com/repos/simonw/datasette/issues/179,360535979,MDEyOklzc3VlQ29tbWVudDM2MDUzNTk3OQ==,82988,psychemedia,2018-01-25T17:18:24Z,2018-01-25T17:18:24Z,CONTRIBUTOR,"To summarise that thread:
- expose full `metadata.json` object to the index page template, eg to allow tables to be referred to by name;
- ability to import multiple `metadata.json` files, eg to allow metadata files created for a specific SQLite db to be reused in a datasette referring to several database files;
It could also be useful to allow users to import a python file containing custom functions that can that be loaded into scope and made available to custom templates.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",288438570,More metadata options for template authors ,
https://github.com/simonw/datasette/issues/14#issuecomment-346244871,https://api.github.com/repos/simonw/datasette/issues/14,346244871,MDEyOklzc3VlQ29tbWVudDM0NjI0NDg3MQ==,21148,jacobian,2017-11-22T05:06:30Z,2017-11-22T05:06:30Z,CONTRIBUTOR,"I'd also suggest taking a look at [stevedore](https://docs.openstack.org/stevedore/latest/), which has a ton of tools for doing plugin stuff. I've had good luck with it in the past.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267707940,Datasette Plugins,
https://github.com/simonw/datasette/pull/104#issuecomment-346124764,https://api.github.com/repos/simonw/datasette/issues/104,346124764,MDEyOklzc3VlQ29tbWVudDM0NjEyNDc2NA==,21148,jacobian,2017-11-21T18:52:14Z,2017-11-21T18:52:14Z,CONTRIBUTOR,"OK, now this should work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274284246,[WIP] Add publish to heroku support,
https://github.com/simonw/datasette/pull/104#issuecomment-346124073,https://api.github.com/repos/simonw/datasette/issues/104,346124073,MDEyOklzc3VlQ29tbWVudDM0NjEyNDA3Mw==,21148,jacobian,2017-11-21T18:49:55Z,2017-11-21T18:49:55Z,CONTRIBUTOR,"Actually hang on, don't merge - there are some bugs that #141 masked when I tested this out elsewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274284246,[WIP] Add publish to heroku support,
https://github.com/simonw/datasette/pull/104#issuecomment-346116745,https://api.github.com/repos/simonw/datasette/issues/104,346116745,MDEyOklzc3VlQ29tbWVudDM0NjExNjc0NQ==,21148,jacobian,2017-11-21T18:23:25Z,2017-11-21T18:23:25Z,CONTRIBUTOR,"@simonw ready for a review and merge if you want.
There's still some nasty duplicated code in cli.py and utils.py, which is just going to get worse if/when we start adding any other deploy targets (and I want to do one for cloud.gov, at least). I think there's an opportunity for some refactoring here. I'm happy to do that now as part of this PR, or if you merge this first I'll do it in a different one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274284246,[WIP] Add publish to heroku support,
https://github.com/simonw/datasette/issues/27#issuecomment-345652450,https://api.github.com/repos/simonw/datasette/issues/27,345652450,MDEyOklzc3VlQ29tbWVudDM0NTY1MjQ1MA==,198537,rgieseke,2017-11-20T10:19:39Z,2017-11-20T10:19:39Z,CONTRIBUTOR,"If Data Package metadata gets adopted (#105) the views spec work might also be worth a look:
http://frictionlessdata.io/specs/views/
http://datahub.io/docs/features/views
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267886330,Ability to plot a simple graph,
https://github.com/simonw/datasette/issues/105#issuecomment-345503897,https://api.github.com/repos/simonw/datasette/issues/105,345503897,MDEyOklzc3VlQ29tbWVudDM0NTUwMzg5Nw==,198537,rgieseke,2017-11-19T09:38:08Z,2017-11-19T09:38:08Z,CONTRIBUTOR,"Thanks, I wrote this very simple reader because the default approach as described on the Datahub pages seemed to complicated. I had metadata from the `datapackage.json` attached to the returned DataFrames but removed this due to some attribute handling change in the latest Pandas version.
This could also be useful for getting from Data Package to SQL db: https://github.com/frictionlessdata/tableschema-sql-py
I maintain a few climate science related dataset at https://github.com/openclimatedata/
The Data Retriever (mainly ecological data) by @ethanwhite et al. is also using the Data Package format for metadata and has some tooling for different dbs:
https://frictionlessdata.io/articles/the-data-retriever/
https://github.com/weecology/retriever
The Open Power System Data project also has a couple of datasets that show nicely how CSV is great for assembling and then already make SQLite files available. It's one of the first data sets I tried with Datasette, perfect for the use case of getting an API for putting power stations on a map ...
https://data.open-power-system-data.org/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274314940,Consider data-package as a format for metadata,
https://github.com/simonw/datasette/pull/104#issuecomment-345452669,https://api.github.com/repos/simonw/datasette/issues/104,345452669,MDEyOklzc3VlQ29tbWVudDM0NTQ1MjY2OQ==,21148,jacobian,2017-11-18T16:18:45Z,2017-11-18T16:18:45Z,CONTRIBUTOR,"I'd like to do a bit of cleanup, and some error checking in case heroku/heroku-builds isn't installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274284246,[WIP] Add publish to heroku support,
https://github.com/simonw/datasette/pull/107#issuecomment-345117690,https://api.github.com/repos/simonw/datasette/issues/107,345117690,MDEyOklzc3VlQ29tbWVudDM0NTExNzY5MA==,3433657,raynae,2017-11-17T01:29:41Z,2017-11-17T01:29:41Z,CONTRIBUTOR,"Thanks for bearing with me. I was getting a message about my branch diverging when I tried to push after rebasing, so I merged master into isnull, seems like that did the trick. Let me know if I should make any corrections.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274343647,add support for ?field__isnull=1,
https://github.com/simonw/datasette/issues/46#issuecomment-345002908,https://api.github.com/repos/simonw/datasette/issues/46,345002908,MDEyOklzc3VlQ29tbWVudDM0NTAwMjkwOA==,54999,ingenieroariel,2017-11-16T17:47:49Z,2017-11-16T17:47:49Z,CONTRIBUTOR,I'll try to find alternatives to the Dockerfile option - I also think we should not use that old one without sources or license.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",271301468,Dockerfile should build more recent SQLite with FTS5 and spatialite support,
https://github.com/simonw/datasette/pull/107#issuecomment-344811268,https://api.github.com/repos/simonw/datasette/issues/107,344811268,MDEyOklzc3VlQ29tbWVudDM0NDgxMTI2OA==,3433657,raynae,2017-11-16T04:17:45Z,2017-11-16T04:17:45Z,CONTRIBUTOR,"Thanks for the guidance. I added a unit test and made a slight change to utils.py.
I didn't realize this, but evidently string.format only complains if you supply less arguments than there are format placeholders, so the original commit worked, but was adding a superfluous named param.
I added a conditional that prevents the named param from being created and ensures the correct number of args are passed to sting.format. It has the side effect of hiding the SQL query in /templates/table.html when there are no other where clauses--not sure if that's the desired outcome here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274343647,add support for ?field__isnull=1,
https://github.com/simonw/datasette/issues/46#issuecomment-344810525,https://api.github.com/repos/simonw/datasette/issues/46,344810525,MDEyOklzc3VlQ29tbWVudDM0NDgxMDUyNQ==,54999,ingenieroariel,2017-11-16T04:11:25Z,2017-11-16T04:11:25Z,CONTRIBUTOR,"@simonw On the spatialite support, here is some info to make it work and a screenshot:
![]()
I used the following Dockerfile:
```
FROM prolocutor/python3-sqlite-ext:3.5.1-spatialite as build
RUN mkdir /code
ADD . /code/
RUN pip install /code/
EXPOSE 8001
CMD [""datasette"", ""serve"", ""/code/ne.sqlite"", ""--host"", ""0.0.0.0""]
```
and added this to `prepare_connection`:
```
conn.enable_load_extension(True)
conn.execute(""SELECT load_extension('/usr/local/lib/mod_spatialite.so')"")
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",271301468,Dockerfile should build more recent SQLite with FTS5 and spatialite support,
https://github.com/simonw/datasette/pull/104#issuecomment-344710204,https://api.github.com/repos/simonw/datasette/issues/104,344710204,MDEyOklzc3VlQ29tbWVudDM0NDcxMDIwNA==,21148,jacobian,2017-11-15T19:57:50Z,2017-11-15T19:57:50Z,CONTRIBUTOR,"A first basic stab at making this work, just to prove the approach. Right now this requires [a Heroku CLI plugin](https://github.com/heroku/heroku-builds), which seems pretty unreasonable. I think this can be replaced with direct API calls, which could clean up a lot of things. But I wanted to prove it worked first, and it does.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274284246,[WIP] Add publish to heroku support,
https://github.com/simonw/datasette/issues/88#issuecomment-344430689,https://api.github.com/repos/simonw/datasette/issues/88,344430689,MDEyOklzc3VlQ29tbWVudDM0NDQzMDY4OQ==,15543,tomdyson,2017-11-14T23:08:22Z,2017-11-14T23:08:22Z,CONTRIBUTOR,"> I'm getting an internal server error on http://run.plnkr.co/preview/cj9zlf1qc0003414y90ajkwpk/ at the moment
Sorry about that - here's a working version on Netlify:
https://nhs-england-map.netlify.com","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273775212,Add NHS England Hospitals example to wiki,
https://github.com/simonw/datasette/pull/81#issuecomment-344125441,https://api.github.com/repos/simonw/datasette/issues/81,344125441,MDEyOklzc3VlQ29tbWVudDM0NDEyNTQ0MQ==,50527,jefftriplett,2017-11-14T02:24:54Z,2017-11-14T02:24:54Z,CONTRIBUTOR,"Oops, if I jumped the gun. I saw the project in my github activity feed and saw some low hanging fruit :) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273595473,:fire: Removes DS_Store,
https://github.com/simonw/datasette/issues/57#issuecomment-344151223,https://api.github.com/repos/simonw/datasette/issues/57,344151223,MDEyOklzc3VlQ29tbWVudDM0NDE1MTIyMw==,247192,macropin,2017-11-14T05:32:28Z,2017-11-14T05:33:03Z,CONTRIBUTOR,"The pattern is called ""multi-stage builds"". And the result is a svelte 226MB image (201MB for 3.6-slim) vs 700MB+ for the full image. It's possible to get it even smaller, but that takes a lot more work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273127694,Ship a Docker image of the whole thing,
https://github.com/simonw/datasette/issues/57#issuecomment-344147583,https://api.github.com/repos/simonw/datasette/issues/57,344147583,MDEyOklzc3VlQ29tbWVudDM0NDE0NzU4Mw==,247192,macropin,2017-11-14T05:03:47Z,2017-11-14T05:03:47Z,CONTRIBUTOR,"Let me know if you'd like a PR. The image is usable as
`docker run --rm -t -i -p 9000:8001 -v $(pwd)/db:/db datasette datasette serve /db/chinook.db`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273127694,Ship a Docker image of the whole thing,
https://github.com/simonw/datasette/issues/57#issuecomment-344145265,https://api.github.com/repos/simonw/datasette/issues/57,344145265,MDEyOklzc3VlQ29tbWVudDM0NDE0NTI2NQ==,247192,macropin,2017-11-14T04:45:38Z,2017-11-14T04:45:38Z,CONTRIBUTOR,"I'm happy to contribute this. Just let me know if you want a Dockerfile for development or production purposes, or both.
If it's prod then we can just pip install the source from pypi, otherwise for dev we'll need a `requirements.txt` to speed up rebuilds.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273127694,Ship a Docker image of the whole thing,
https://github.com/dogsheep/twitter-to-sqlite/issues/58#issuecomment-910121331,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/58,910121331,IC_kwDODEm0Qs42P1lz,42904,rubenv,2021-09-01T09:49:33Z,2021-09-01T09:49:33Z,CONTRIBUTOR,"Found the cause, it's the other commands. PR #59 submitted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",984939366,"Error: Use either --since or --since_id, not both - still broken",
https://github.com/dogsheep/dogsheep-photos/issues/3#issuecomment-934372104,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3,934372104,IC_kwDOD079W843sWMI,41546558,RhetTbull,2021-10-05T12:38:24Z,2021-10-05T12:38:24Z,CONTRIBUTOR,"As dogsheep-photos already uses [osxphotos](https://github.com/RhetTbull/osxphotos) to load photos you can access the EXIF data via osxphotos. Apple Photos imports a small subset of EXIF data at the time the photo is imported and osxphotos provides this via the [exif_info](https://github.com/RhetTbull/osxphotos#exifinfo) property. If you want the full EXIF data, osxphotos also provides a wrapper around [exiftool](https://github.com/RhetTbull/osxphotos#exiftool).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533481,"Import EXIF data into SQLite - lens used, ISO, aperture etc",
https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843975536,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843975536,IC_kwDOCGYnMM5t6NVw,16437338,tkhattra,2023-12-07T01:17:05Z,2023-12-07T01:17:05Z,CONTRIBUTOR,Apologies - I pushed a fix that addresses the mypy failures.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,Add more STRICT table support,
https://github.com/simonw/sqlite-utils/issues/344#issuecomment-1815825863,https://api.github.com/repos/simonw/sqlite-utils/issues/344,1815825863,IC_kwDOCGYnMM5sO03H,16437338,tkhattra,2023-11-17T06:44:49Z,2023-11-17T06:44:49Z,CONTRIBUTOR,"hello Simon,
I've added more STRICT table support per https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776 in changeset https://github.com/simonw/sqlite-utils/commit/e4b9b582cdb4e48430865f8739f341bc8017c1e4.
It also fixes table.transform() to preserve STRICT mode.
Please pull if you deem appropriate. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables,
https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1747231893,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1747231893,IC_kwDOCGYnMM5oJKSV,62745,spookylukey,2023-10-04T16:15:09Z,2023-10-04T16:28:21Z,CONTRIBUTOR,"I confirm the bug, as above, and that @jonafato 's patch fixes it for me. However, it's not the right fix. The problem is that ProgressBar is being used in the wrong way. This also results in two lines being printed instead of one, like this:
```
[#######-----------------------------] 20%
[####################################] 100%%
```
The bug is reproducible for me in any terminal, including Gnome Terminal and Guake, and VSCode. With VSCode I can use this launch.json to reproduce it:
```json
{
""version"": ""0.2.0"",
""configurations"": [
{
""name"": ""Python: Module"",
""type"": ""python"",
""request"": ""launch"",
""module"": ""sqlite_utils"",
""justMyCode"": false,
""args"": [""insert"", ""test.db"", ""test"", ""--csv"", ""tests/sniff/example1.csv""]
}
]
}
```
[edit - deleted my analysis of why the current code is wrong, which was confused and confusing]","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1668113177,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1668113177,IC_kwDOCGYnMM5jbWMZ,25778,eyeseast,2023-08-07T15:41:49Z,2023-08-07T15:41:49Z,CONTRIBUTOR,I wonder if this should be two hooks: input and output. The current `--csv` (and `--tsv`) options apply to both. Haven't looked at how it's implemented. Or maybe it's one hook that returns a format for reading and for writing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648339661,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648339661,IC_kwDOCGYnMM5iP6rN,25778,eyeseast,2023-07-24T17:44:30Z,2023-07-24T17:44:30Z,CONTRIBUTOR,"> A related feature would be support for plugins to add new ways of ingesting data - currently sqlite-utils insert works against JSON, newline-JSON, CSV and TSV.
This is my goal, to have one plugin that handles input and output symmetrically. I'd like to be able to do something like this:
```sh
sqlite-utils insert data.db table file.geojson --format geojson
# ... explore and manipulate in Datasette
sqlite-utils query data.db ... --format geojson > output.geojson
```
This would work especially well with [datasette-query-files](https://github.com/eyeseast/datasette-query-files), since I already have the queries I need saved in standalone SQL files.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646656283,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646656283,IC_kwDOCGYnMM5iJfsb,25778,eyeseast,2023-07-22T19:32:24Z,2023-07-22T19:32:24Z,CONTRIBUTOR,Cool. I might try to add a geojson plugin that handles both input and output. That would help me out a lot. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638910473,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1638910473,IC_kwDOCGYnMM5hr8oJ,15178711,asg017,2023-07-17T21:27:41Z,2023-07-17T21:27:41Z,CONTRIBUTOR,"Another use-case: I want to make a `sqlite-utils` plugin that'll help me insert data into Datasette.
```bash
sqlite-utils insert-datasette \
--token $DATASETTE_API_KEY \
https://latest.datasette.io/fixtures/my-table \
'select ...'
```
This could also be a datasette plugin (ex `datasette upload-data ...`, but you can also think of `sqlite-utils` plugins that upload to S3, a postgres table, other DBMS's, etc.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1642808866,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1642808866,IC_kwDOCGYnMM5h60Yi,25778,eyeseast,2023-07-19T21:54:27Z,2023-07-19T21:54:27Z,CONTRIBUTOR,Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1590531892,https://api.github.com/repos/simonw/sqlite-utils/issues/557,1590531892,IC_kwDOCGYnMM5ezZc0,7908073,chapmanjacobd,2023-06-14T06:09:21Z,2023-06-14T06:09:21Z,CONTRIBUTOR,"I put together a [simple script](https://github.com/chapmanjacobd/library/blob/42129c5ebe15f9d74653c0f5ca4ed0c991d383e0/xklb/scripts/dedupe_db.py) to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help
```
CREATE TABLE my_table (
id INTEGER PRIMARY KEY,
column1 TEXT,
column2 TEXT,
column3 TEXT
);
INSERT INTO my_table (column1, column2, column3)
VALUES
('Value 1', 'Duplicate 1', 'Duplicate A'),
('Value 2', 'Duplicate 2', 'Duplicate B'),
('Value 3', 'Duplicate 2', 'Duplicate C'),
('Value 4', 'Duplicate 3', 'Duplicate D'),
('Value 5', 'Duplicate 3', 'Duplicate E'),
('Value 6', 'Duplicate 3', 'Duplicate F');
```
```
library dedupe-db test.db my_table --bk column2
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740150327,Aliased ROWID option for tables created from alter=True commands,
https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1577355134,https://api.github.com/repos/simonw/sqlite-utils/issues/557,1577355134,IC_kwDOCGYnMM5eBId-,7908073,chapmanjacobd,2023-06-05T19:26:26Z,2023-06-05T19:26:26Z,CONTRIBUTOR,"this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use `upsert_all`, `insert_all`, etc without having to write DDL to create tables. The meat of the issue is that SQLITE doesn't make rowid stable between vacuums so it is not possible to take shortcuts","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740150327,Aliased ROWID option for tables created from alter=True commands,
https://github.com/simonw/sqlite-utils/issues/529#issuecomment-1592110694,https://api.github.com/repos/simonw/sqlite-utils/issues/529,1592110694,IC_kwDOCGYnMM5e5a5m,7908073,chapmanjacobd,2023-06-14T23:11:47Z,2023-06-14T23:12:12Z,CONTRIBUTOR,"sorry i was wrong. `sqlite-utils --raw-lines` works correctly
```
sqlite-utils --raw-lines :memory: ""SELECT * FROM (VALUES ('test'), ('line2'))"" | cat -A
test$
line2$
sqlite-utils --csv --no-headers :memory: ""SELECT * FROM (VALUES ('test'), ('line2'))"" | cat -A
test$
line2$
```
I think this was fixed somewhat recently","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1581090327,Microsoft line endings,
https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592052320,https://api.github.com/repos/simonw/sqlite-utils/issues/535,1592052320,IC_kwDOCGYnMM5e5Mpg,7908073,chapmanjacobd,2023-06-14T22:05:28Z,2023-06-14T22:05:28Z,CONTRIBUTOR,piping to `jq` is good enough usually,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1655860104,rows: --transpose or psql extended view-like functionality,
https://github.com/simonw/sqlite-utils/issues/555#issuecomment-1592047502,https://api.github.com/repos/simonw/sqlite-utils/issues/555,1592047502,IC_kwDOCGYnMM5e5LeO,7908073,chapmanjacobd,2023-06-14T22:00:10Z,2023-06-14T22:01:57Z,CONTRIBUTOR,"You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python.
That might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the `NOT IN` style of query...
You may also try chunking like this:
```py
def chunks(lst, n) -> Generator:
for i in range(0, len(lst), n):
yield lst[i : i + n]
SQLITE_PARAM_LIMIT = 32765
data = []
chunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT)
for ids in chunked:
data.expand(
list(
db.query(
f""""""SELECT * from videos
WHERE id in (""""""
+ "","".join([""?""] * len(ids))
+ "")"",
(*ids,),
)
)
)
```
but that actually won't work with your `NOT IN` requirements. You need to query the full resultset to check any row.
Since you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1733198948,Filter table by a large bunch of ids,
https://github.com/simonw/sqlite-utils/issues/556#issuecomment-1575310378,https://api.github.com/repos/simonw/sqlite-utils/issues/556,1575310378,IC_kwDOCGYnMM5d5VQq,601708,mcint,2023-06-04T01:21:15Z,2023-06-04T01:21:15Z,CONTRIBUTOR,"I've resolved my use, with the line-buffered output and while read loop for line buffered input, but I leave this here so the incremental saving or line-buffered use-case can be explicitly handled or rejected (or deferred).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740026046,Support storing incrementally piped values,
https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1540900733,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1540900733,IC_kwDOCGYnMM5b2Ed9,167893,mcarpenter,2023-05-09T21:15:05Z,2023-05-09T21:15:05Z,CONTRIBUTOR,"Sorry, I completely missed your first comment whilst on Easter break.
This looks like a good practical compromise before v4. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values,
https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1501017004,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1501017004,IC_kwDOCGYnMM5Zd7Os,25778,eyeseast,2023-04-09T01:49:43Z,2023-04-09T01:49:43Z,CONTRIBUTOR,I'm going to close this in favor of #536. Will try a cleaner approach to custom paths once that one is merge.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon,
https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465315726,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1465315726,IC_kwDOCGYnMM5XVvGO,25778,eyeseast,2023-03-12T22:21:56Z,2023-03-12T22:21:56Z,CONTRIBUTOR,"Exactly, that's what I was running into. On my M2 MacBook, SpatiaLite ends up in what is -- for the moment -- a non-standard location, so even when I passed in the location with `--load-extension`, I still hit an error on `create-spatial-index`.
What I learned doing this originally is that SQLite needs to load the extension for each connection, even if all the SpatiaLite stuff is already in the database. So that's why `init_spatialite()` gets called again.
Here's the code where I hit the error: https://github.com/eyeseast/boston-parcels/blob/main/Makefile#L30 It works using this branch.
I'm not attached to this solution if you can think of something better. And I'm not sure, TBH, my test would actually catch what I'm after here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon,
https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1444474487,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1444474487,IC_kwDOCGYnMM5WGO53,167893,mcarpenter,2023-02-24T20:57:43Z,2023-02-24T22:22:18Z,CONTRIBUTOR,"I think I see what is happening here, although I haven't quite work out a fix yet. Usually:
* `click.progressbar.render_progress()` renders the cursor invisible on each invocation (update of the bar)
* When the progress bar goes out of scope, the `__exit()__` method is invoked, which calls `render_finish()` to make the cursor re-appear.
(See terminal escape sequences `BEFORE_BAR` and `AFTER_BAR` in click).
However the sqlite-utils `utils.file_progress` context manager wraps `click.progressbar` and yields an instance of a helper class:
``` python
@contextlib.contextmanager
def file_progress(file, silent=False, **kwargs):
...
with click.progressbar(length=file_length, **kwargs) as bar:
yield UpdateWrapper(file, bar.update)
```
The yielded `UpdateWrapper` goes out of scope quickly and `click.progressbar.__exit__()` is called. The cursor is made un-invisible. Hoewever `bar` is still live and so when the caller iterates on the yielded wrapper this invokes the bar's update method, calling `render_progress()`, each time printing the ""make cursor invisible"" escape code. The `progressbar.__exit__` function is not called again, so the cursor doesn't re-appear.
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor,
https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1435318713,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1435318713,IC_kwDOCGYnMM5VjTm5,167893,mcarpenter,2023-02-17T21:55:01Z,2023-02-17T21:55:01Z,CONTRIBUTOR,"Meanwhile, a cheap workaround is to invalidate the registered function cache:
``` python
table.convert(...)
db._registered_functions = set()
table.convert(...)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail,
https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1421571810,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1421571810,IC_kwDOCGYnMM5Uu3bi,167893,mcarpenter,2023-02-07T22:43:09Z,2023-02-07T22:43:09Z,CONTRIBUTOR,"Hey, isn't this essentially the same issue as #448 ?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419357290,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1419357290,IC_kwDOCGYnMM5Umaxq,25778,eyeseast,2023-02-06T16:21:44Z,2023-02-06T16:21:44Z,CONTRIBUTOR,SQLite doesn't have a native `DATETIME` type. It stores dates internally as strings and then has [functions](https://www.sqlite.org/lang_datefunc.html) to work with date-like strings. Yes it's weird.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1423387341,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1423387341,IC_kwDOCGYnMM5U1yrN,167893,mcarpenter,2023-02-08T23:48:52Z,2023-02-09T00:17:30Z,CONTRIBUTOR,PR below,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail,
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-1404070841,https://api.github.com/repos/simonw/sqlite-utils/issues/203,1404070841,IC_kwDOCGYnMM5TsGu5,536941,fgregg,2023-01-25T18:47:18Z,2023-01-25T18:47:18Z,CONTRIBUTOR,i'll adopt this PR to make the changes @simonw suggested https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys,
https://github.com/simonw/sqlite-utils/issues/523#issuecomment-1407264466,https://api.github.com/repos/simonw/sqlite-utils/issues/523,1407264466,IC_kwDOCGYnMM5T4SbS,536941,fgregg,2023-01-28T02:41:14Z,2023-01-28T02:41:14Z,CONTRIBUTOR,"I also often then run another little script to cast all empty strings to null, but i save that for another issue if this gets accepted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560651350,Feature request: trim all leading and trailing white space for all columns for all tables in a database,
https://github.com/simonw/sqlite-utils/issues/510#issuecomment-1318777114,https://api.github.com/repos/simonw/sqlite-utils/issues/510,1318777114,IC_kwDOCGYnMM5OmvEa,7908073,chapmanjacobd,2022-11-17T15:09:47Z,2022-11-17T15:09:47Z,CONTRIBUTOR,"why close? is the only problem that the _config table that incorrectly says 4 for fts5? if so, that's still something that should be fixed","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434911255,Cannot enable FTS5 despite it being available,
https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304320521,https://api.github.com/repos/simonw/sqlite-utils/issues/511,1304320521,IC_kwDOCGYnMM5NvloJ,7908073,chapmanjacobd,2022-11-04T22:54:09Z,2022-11-04T22:59:54Z,CONTRIBUTOR,I ran `PRAGMA integrity_check` and it returned `ok`. but then I tried restoring from a backup and I didn't get this `IntegrityError: constraint failed` error. So I think it was just something wrong with my database. If it happens again I will first try to reindex and see if that fixes the issue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1436539554,"[insert_all, upsert_all] IntegrityError: constraint failed",
https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304078945,https://api.github.com/repos/simonw/sqlite-utils/issues/511,1304078945,IC_kwDOCGYnMM5Nuqph,7908073,chapmanjacobd,2022-11-04T19:38:36Z,2022-11-04T20:13:17Z,CONTRIBUTOR,"Even more bizarre, the source db only has one record and the target table has no conflicting record:
```
875 0.3s lb:/ (main|✚2) [0|0]🌺 sqlite-utils tube_71.db 'select * from media where path = ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz""' | jq
[
{
""size"": null,
""time_created"": null,
""play_count"": 1,
""language"": null,
""view_count"": null,
""width"": null,
""height"": null,
""fps"": null,
""average_rating"": null,
""live_status"": null,
""age_limit"": null,
""uploader"": null,
""time_played"": 0,
""path"": ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"",
""id"": ""088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz/074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv"",
""ie_key"": ""ArchiveOrg"",
""playlist_path"": ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"",
""duration"": 1424.05,
""tags"": null,
""title"": ""074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv""
}
]
875 0.3s lb:/ (main|✚2) [0|0]🥧 sqlite-utils video.db 'select * from media where path = ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz""' | jq
[]
```
I've been able to use this code successfully several times before so not sure what's causing the issue.
I guess the way that I'm handling multiple databases is an issue, though it hasn't ever inserted into the source db, not sure what's different. The only reasonable explanation is that it is trying to insert into the source db from the source db for some reason? Or maybe sqlite3 is checking the source db for primary key violation because the table name is the same","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1436539554,"[insert_all, upsert_all] IntegrityError: constraint failed",
https://github.com/simonw/sqlite-utils/issues/50#issuecomment-1303660293,https://api.github.com/repos/simonw/sqlite-utils/issues/50,1303660293,IC_kwDOCGYnMM5NtEcF,7908073,chapmanjacobd,2022-11-04T14:38:36Z,2022-11-04T14:38:36Z,CONTRIBUTOR,where did you see the limit as 999? I believe the limit has been 32766 for quite some time. If you could detect which one this could speed up batch insert of some types of data significantly,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473083260,"""Too many SQL variables"" on large inserts",
https://github.com/simonw/sqlite-utils/pull/508#issuecomment-1297788531,https://api.github.com/repos/simonw/sqlite-utils/issues/508,1297788531,IC_kwDOCGYnMM5NWq5z,7908073,chapmanjacobd,2022-10-31T22:54:33Z,2022-11-17T15:11:16Z,CONTRIBUTOR,"Maybe this is actually a problem in the python sqlite bindings. Given [SQLITE's stance on this](https://www.sqlite.org/invalidutf.html) they should probably use `encode('utf-8', 'surrogatepass')`. As far as I understand the error here won't actually be resolved by this PR as-is. We would need to modify the data with `surrogateescape`... :/ or modify the sqlite3 module to use `surrogatepass`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430563092,Allow surrogates in parameters,
https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1297703307,https://api.github.com/repos/simonw/sqlite-utils/issues/448,1297703307,IC_kwDOCGYnMM5NWWGL,167893,mcarpenter,2022-10-31T21:23:51Z,2022-10-31T21:27:32Z,CONTRIBUTOR,"The Windows aspect is a red herring: OP's sample above produces the same error on Linux. (Though I don't know what's going on with the CI).
The same error can also be obtained by passing an `io` from a file opened in non-binary mode (`'r'` as opposed to `'rb'`) to `rows_from_file()`. This is how I got here.
The fix for my case is easy: open the file in mode `'rb'`. The analagous fix for OP's problem also works: use `BytesIO` in place of `StringIO`.
Minimal test case (derived from [utils.py](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/utils.py#L304)):
``` python
import io
from typing import cast
#fp = io.StringIO(""id,name\n1,Cleo"") # error
fp = io.BytesIO(bytes(""id,name\n1,Cleo"", encoding='utf-8')) # okay
reader = io.BufferedReader(cast(io.RawIOBase, fp))
reader.peek(1) # exception thrown here
```
I see the signature of `rows_from_file()` correctly has `fp: BinaryIO` but I guess you'd need either a runtime type check for that (not all `io`s have `mode()`), or to catch the `AttributeError` on `peek()` to produce a better error for users. Neither option is ideal.
Some thoughts on testing binary-ness of `io`s in this SO question: https://stackoverflow.com/questions/44584829/how-to-determine-if-file-is-opened-in-binary-or-text-mode","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279144769,Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto',
https://github.com/simonw/sqlite-utils/issues/507#issuecomment-1297859539,https://api.github.com/repos/simonw/sqlite-utils/issues/507,1297859539,IC_kwDOCGYnMM5NW8PT,7908073,chapmanjacobd,2022-11-01T00:40:16Z,2022-11-01T00:40:16Z,CONTRIBUTOR,"Ideally people could fix their data if they run into this issue.
If you are using filenames try [convmv](https://linux.die.net/man/1/convmv)
```
convmv --preserve-mtimes -f utf8 -t utf8 --notest -i -r .
```
maybe this script will also help:
```py
import argparse, shutil
from pathlib import Path
import ftfy
from xklb import utils
from xklb.utils import log
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument(""paths"", nargs='*')
parser.add_argument(""--verbose"", ""-v"", action=""count"", default=0)
args = parser.parse_args()
log.info(utils.dict_filter_bool(args.__dict__))
return args
def rename_invalid_paths() -> None:
args = parse_args()
for path in args.paths:
log.info(path)
for p in sorted([str(p) for p in Path(path).rglob(""*"")], key=len):
fixed = ftfy.fix_text(p, uncurl_quotes=False).replace(""\r\n"", ""\n"").replace(""\r"", ""\n"").replace(""\n"", """")
if p != fixed:
try:
shutil.move(p, fixed)
except FileNotFoundError:
log.warning(""FileNotFound. %s"", p)
else:
log.info(fixed)
if __name__ == ""__main__"":
rename_invalid_paths()
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430325103,conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character,
https://github.com/simonw/sqlite-utils/pull/499#issuecomment-1292401308,https://api.github.com/repos/simonw/sqlite-utils/issues/499,1292401308,IC_kwDOCGYnMM5NCHqc,7908073,chapmanjacobd,2022-10-26T17:54:26Z,2022-10-26T17:54:51Z,CONTRIBUTOR,"The problem with how it is currently is that the transformed fts table _will_ return incorrect results (unless the table was only 1 row or something), even if create_triggers was enabled previously. Maybe the simplest solution is to disable fts on a transformed table rather than try to recreate it? Thoughts?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1405196044,feat: recreate fts triggers after table transform,
https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1264223554,https://api.github.com/repos/simonw/sqlite-utils/issues/409,1264223554,IC_kwDOCGYnMM5LWoVC,7908073,chapmanjacobd,2022-10-01T03:42:50Z,2022-10-01T03:42:50Z,CONTRIBUTOR,oh weird. it inserts into db2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149661489,`with db:` for transactions,
https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1264223363,https://api.github.com/repos/simonw/sqlite-utils/issues/409,1264223363,IC_kwDOCGYnMM5LWoSD,7908073,chapmanjacobd,2022-10-01T03:41:45Z,2022-10-01T03:41:45Z,CONTRIBUTOR,"```
pytest xklb/check.py --pdb
xklb/check.py:11: in test_transaction
assert list(db2[""t""].rows) == []
E AssertionError: assert [{'foo': 1}] == []
E + where [{'foo': 1}] = list()
E + where = .rows
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/xk/github/xk/lb/xklb/check.py(11)test_transaction()
9 with db1.conn:
10 db1[""t""].insert({""foo"": 1})
---> 11 assert list(db2[""t""].rows) == []
12 assert list(db2[""t""].rows) == [{""foo"": 1}]
```
It fails because it is already inserted.
btw if you put these two lines in you pyproject.toml you can get `ipdb` in pytest
```
[tool.pytest.ini_options]
addopts = ""--pdbcls=IPython.terminal.debugger:TerminalPdb --ignore=tests/data --capture=tee-sys --log-cli-level=ERROR""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149661489,`with db:` for transactions,
https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1264219650,https://api.github.com/repos/simonw/sqlite-utils/issues/493,1264219650,IC_kwDOCGYnMM5LWnYC,7908073,chapmanjacobd,2022-10-01T03:22:50Z,2022-10-01T03:23:58Z,CONTRIBUTOR,"this is likely what you are looking for: https://stackoverflow.com/a/51076749/697964
but yeah I would say just disable smart quotes","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386562662,Tiny typographical error in install/uninstall docs,
https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1264218914,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1264218914,IC_kwDOCGYnMM5LWnMi,7908073,chapmanjacobd,2022-10-01T03:18:36Z,2023-06-14T22:14:24Z,CONTRIBUTOR,"> some good concrete use-cases in mind
I actually found myself wanting something like this the past couple days. The use-case was databases with slightly different schema but same table names.
here is a full script:
```
import argparse
from pathlib import Path
from sqlite_utils import Database
def connect(args, conn=None, **kwargs) -> Database:
db = Database(conn or args.database, **kwargs)
with db.conn:
db.conn.execute(""PRAGMA main.cache_size = 8000"")
return db
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument(""database"")
parser.add_argument(""dbs_folder"")
parser.add_argument(""--db"", ""-db"", help=argparse.SUPPRESS)
parser.add_argument(""--verbose"", ""-v"", action=""count"", default=0)
args = parser.parse_args()
if args.db:
args.database = args.db
Path(args.database).touch()
args.db = connect(args)
return args
def merge_db(args, source_db):
source_db = str(Path(source_db).resolve())
s_db = connect(argparse.Namespace(database=source_db, verbose = args.verbose))
for table in s_db.table_names():
data = s_db[table].rows
args.db[table].insert_all(data, alter=True, replace=True)
args.db.conn.commit()
def merge_directory():
args = parse_args()
source_dbs = list(Path(args.dbs_folder).glob('*.db'))
for s_db in source_dbs:
merge_db(args, s_db)
if __name__ == '__main__':
merge_directory()
```
edit: I've made some improvements to this and put it on PyPI:
```
$ pip install xklb
$ lb merge-db -h
usage: library merge-dbs DEST_DB SOURCE_DB ... [--only-target-columns] [--only-new-rows] [--upsert] [--pk PK ...] [--table TABLE ...]
Merge-DBs will insert new rows from source dbs to target db, table by table. If primary key(s) are provided,
and there is an existing row with the same PK, the default action is to delete the existing row and insert the new row
replacing all existing fields.
Upsert mode will update matching PK rows such that if a source row has a NULL field and
the destination row has a value then the value will be preserved instead of changed to the source row's NULL value.
Ignore mode (--only-new-rows) will insert only rows which don't already exist in the destination db
Test first by using temp databases as the destination db.
Try out different modes / flags until you are satisfied with the behavior of the program
library merge-dbs --pk path (mktemp --suffix .db) tv.db movies.db
Merge database data and tables
library merge-dbs --upsert --pk path video.db tv.db movies.db
library merge-dbs --only-target-columns --only-new-rows --table media,playlists --pk path audio-fts.db audio.db
library merge-dbs --pk id --only-tables subreddits reddit/81_New_Music.db audio.db
library merge-dbs --only-new-rows --pk subreddit,path --only-tables reddit_posts reddit/81_New_Music.db audio.db -v
positional arguments:
database
source_dbs
```
Also if you want to dedupe a table based on a ""business key"" which isn't explicitly your primary key(s) you can run this:
```
$ lb dedupe-db -h
usage: library dedupe-dbs DATABASE TABLE --bk BUSINESS_KEYS [--pk PRIMARY_KEYS] [--only-columns COLUMNS]
Dedupe your database (not to be confused with the dedupe subcommand)
It should not need to be said but *backup* your database before trying this tool!
Dedupe-DB will help remove duplicate rows based on non-primary-key business keys
library dedupe-db ./video.db media --bk path
If --primary-keys is not provided table metadata primary keys will be used
If --only-columns is not provided all non-primary and non-business key columns will be upserted
positional arguments:
database
table
options:
-h, --help show this help message and exit
--skip-0
--only-columns ONLY_COLUMNS
Comma separated column names to upsert
--primary-keys PRIMARY_KEYS, --pk PRIMARY_KEYS
Comma separated primary keys
--business-keys BUSINESS_KEYS, --bk BUSINESS_KEYS
Comma separated business keys
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables,
https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258712931,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258712931,IC_kwDOCGYnMM5LBm9j,25778,eyeseast,2022-09-26T22:31:58Z,2022-09-26T22:31:58Z,CONTRIBUTOR,"Right. The backup command will copy tables completely, but in the case of conflicting table names, the destination gets overwritten silently. That might not be what you want here. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables,
https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258508215,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258508215,IC_kwDOCGYnMM5LA0-3,25778,eyeseast,2022-09-26T19:22:14Z,2022-09-26T19:22:14Z,CONTRIBUTOR,"This might be fairly straightforward using SQLite's backup utility: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables,
https://github.com/simonw/sqlite-utils/pull/498#issuecomment-1274153135,https://api.github.com/repos/simonw/sqlite-utils/issues/498,1274153135,IC_kwDOCGYnMM5L8giv,7908073,chapmanjacobd,2022-10-11T06:34:31Z,2022-10-11T06:34:31Z,CONTRIBUTOR,nevermind it was because I was running `db[table].transform`. The fts tables would still be there but the triggers would be dropped,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1404013495,fix: enable-fts permanently save triggers,
https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1252898131,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1252898131,IC_kwDOCGYnMM5KrbVT,7908073,chapmanjacobd,2022-09-20T20:51:21Z,2022-09-20T20:56:07Z,CONTRIBUTOR,"When I run `reset` it fixes my terminal. I suspect it is related to the progress bar
https://linux.die.net/man/1/reset
```
950 1s /m/d/03_Downloads 🐑 echo $TERM
xterm-kitty
▓░▒░ /m/d/03_Downloads 🌏 kitty -v
kitty 0.26.2 created by Kovid Goyal
$ sqlite-utils insert test.db facility facility-boundary-us-all.csv --csv
blah blah blah (no offense)
$
$ reset
$
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor,
https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1256858763,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1256858763,IC_kwDOCGYnMM5K6iSL,7908073,chapmanjacobd,2022-09-24T04:50:59Z,2022-09-24T04:52:08Z,CONTRIBUTOR,"Instead of outputting binary data to stdout the interface might be better like this
```
sqlite-utils merge animals.db cats.db dogs.db
```
similar to `zip`, `ogr2ogr`, etc
Actually I think this might already be possible within `ogr2ogr`. I don't believe spatial data is a requirement though it might add an `ogc_id` column or something
```
cp cats.db animals.db
ogr2ogr -append animals.db dogs.db
ogr2ogr -append animals.db another.db
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables,
https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232356302,https://api.github.com/repos/simonw/sqlite-utils/issues/480,1232356302,IC_kwDOCGYnMM5JdEPO,7908073,chapmanjacobd,2022-08-31T01:51:49Z,2022-08-31T01:51:49Z,CONTRIBUTOR,Thanks for pointing me to the right place,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1355433619,search_sql add include_rank option,
https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224382336,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224382336,IC_kwDOCGYnMM5I-peA,50527,jefftriplett,2022-08-23T17:16:13Z,2022-08-23T17:16:13Z,CONTRIBUTOR,"> Should passing `alter=True` also drop any columns that aren't included in the new table structure?
>
> It could even spot column types that aren't correct and fix those.
>
> Is that consistent with the expectations set by how `alter=True` works elsewhere?
I would lean towards not dropping them (or making a `drop=True` or `drop_columns=True`or `drop_missing_columns=True`) to work with existing tables easier.
I do like that sqlite-utils mostly just works with existing tables but it's also nice to add to existing fields in a few cases.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns,
https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1179579878,https://api.github.com/repos/simonw/sqlite-utils/issues/449,1179579878,IC_kwDOCGYnMM5GTvXm,1690072,davidleejy,2022-07-09T17:41:32Z,2022-07-09T17:41:50Z,CONTRIBUTOR,Learnt that the types in Sqlite-utils differ somewhat from those in Sqlite. I've changed my test to account for this difference and the test has passed successfully. I will submit a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279863844,Utilities for duplicating tables and creating a table with the results of a query,
https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1190277829,https://api.github.com/repos/simonw/sqlite-utils/issues/456,1190277829,IC_kwDOCGYnMM5G8jLF,536941,fgregg,2022-07-20T13:19:15Z,2022-07-20T13:19:15Z,CONTRIBUTOR,hadley wickham's melt and reshape could be good inspo: http://had.co.nz/reshape/introduction.pdf,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1310243385,feature request: pivot command,
https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1190272780,https://api.github.com/repos/simonw/sqlite-utils/issues/456,1190272780,IC_kwDOCGYnMM5G8h8M,536941,fgregg,2022-07-20T13:14:54Z,2022-07-20T13:14:54Z,CONTRIBUTOR,"for example, i have data on votes that look like this:
| ballot_id | option_id | choice |
|-|-|-|
| 1 | 1 | 0 |
| 1 | 2 | 1 |
| 1 | 3 | 0 |
| 1 | 4 | 1 |
| 2 | 1 | 1 |
| 2 | 2 | 0 |
| 2 | 3 | 1 |
| 2 | 4 | 0 |
and i want to reshape from this long form to this wide form:
| ballot_id | option_id_1 | option_id_2 | option_id_3 | option_id_ 4|
|-|-|-|-| -|
| 1 | 0 | 1 | 0 | 1 |
| 2 | 1 | 0 | 1| 0 |
i could do such a think like this.
```sql
select ballot_id,
sum(choice) filter (where option_id = 1) as option_id_1,
sum(choice) filter (where option_id = 2) as option_id_2,
sum(choice) filter (where option_id = 3) as option_id_3,
sum(choice) filter (where option_id = 4) as option_id_4
from vote
group by ballot_id
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1310243385,feature request: pivot command,
https://github.com/simonw/sqlite-utils/issues/423#issuecomment-1189010812,https://api.github.com/repos/simonw/sqlite-utils/issues/423,1189010812,IC_kwDOCGYnMM5G3t18,536941,fgregg,2022-07-19T12:47:39Z,2022-07-19T12:47:39Z,CONTRIBUTOR,just ran into this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1199158210,.extract() doesn't set foreign key when extracted columns contain NULL value,
https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1174027079,https://api.github.com/repos/simonw/sqlite-utils/issues/449,1174027079,IC_kwDOCGYnMM5F-jtH,1690072,davidleejy,2022-07-04T17:33:04Z,2022-07-04T17:48:43Z,CONTRIBUTOR,"I've written the code and test. Would you be able to advise how to compare table columns in a pytest function properly? Experiencing a challenge when comparing columns.
Test:
```python
def test_duplicate(fresh_db):
table = fresh_db.create_table(
""table1"",
{
""text_col"": str,
""float_col"": float,
""int_col"": int,
""bool_col"": bool,
""bytes_col"": bytes,
""datetime_col"": datetime.datetime,
},
)
dt = datetime.datetime.now()
b = bytes('hello world', 'utf-8')
data = {""text_col"": ""Cleo"",
""float_col"": 3.14,
""int_col"": -2,
""bool_col"": True,
""bytes_col"": b,
""datetime_col"": str(dt)}
table1 = fresh_db[""table1""]
row_id = table1.insert(data).last_rowid
table1.duplicate('table2')
table2 = fresh_db[""table2""]
assert data == table2.get(row_id)
assert table1.columns == table2.columns # FAILS HERE
```
Result:
Failure is due to column types being named differently -- e.g. 'FLOAT' vs 'REAL', 'INTEGER' vs 'INT'. How should I go about comparing columns while accounting for equivalent types?
Or did I miss out something in my duplication code correctly? Here's how I did it: in `db.py`, I've added the following code:
```python
class Table(Queryable):
[...]
def duplicate(
self,
name_new: str
) -> ""Table"":
""""""
Duplicate this table in this database.
:param name_new: Name of new table.
""""""
assert self.exists()
with self.db.conn:
sql = ""CREATE TABLE [{new_table}] AS SELECT * FROM [{table}];"".format(
new_table = name_new,
table = self.name,
)
self.db.execute(sql)
return self.db[name_new]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279863844,Utilities for duplicating tables and creating a table with the results of a query,
https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1077671779,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1077671779,IC_kwDOCGYnMM5AO_dj,25778,eyeseast,2022-03-24T14:11:33Z,2022-03-24T14:11:43Z,CONTRIBUTOR,"Coming back to this. I was about to add a utility function to [datasette-geojson]() to convert lat/lng columns to geometries. Thankfully I googled first. There's a SpatiaLite function for this: [MakePoint](https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html#p0).
```sql
select MakePoint(longitude, latitude) as geometry from places;
```
I'm not sure if that would work with `conversions`, since it needs two columns, but it's an option for tables that already have latitude, longitude columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code",
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-1067981656,https://api.github.com/repos/simonw/sqlite-utils/issues/131,1067981656,IC_kwDOCGYnMM4_qBtY,25778,eyeseast,2022-03-15T13:21:42Z,2022-03-15T13:21:42Z,CONTRIBUTOR,"Just ran into this issue last night. I have a big table that's _mostly_ numbers, but also a zip code column in a state where ZIP codes start with 0. Would be great to run something like this:
```sh
sqlite-utils insert data.db places file.csv --csv --detect-types --type zipcode text
```
Maybe I'll take a crack at this one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types,
https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065477258,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065477258,IC_kwDOCGYnMM4_geSK,25778,eyeseast,2022-03-11T20:14:59Z,2022-03-11T20:14:59Z,CONTRIBUTOR,"Good call on adding this to `create-table`, especially for stored columns. Having the stored/virtual split might make this tricky to implement, but I haven't gone any farther than thinking about what the CLI looks like. I'm going to try making the SQL side work first and figure that'll tell me more about what it needs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns,
https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059647114,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059647114,IC_kwDOCGYnMM4_KO6K,25778,eyeseast,2022-03-05T01:54:24Z,2022-03-05T01:54:24Z,CONTRIBUTOR,"I haven't tried this, but it looks like Pandas has a method for this: https://pandas.pydata.org/docs/reference/api/pandas.read_sql_query.html
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration,
https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1035057014,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1035057014,IC_kwDOCGYnMM49sbd2,25778,eyeseast,2022-02-10T15:30:28Z,2022-02-10T15:30:40Z,CONTRIBUTOR,"Yeah, the CLI experience is probably where any kind of multi-column, configured setup is going to fall apart. Sticking with GIS examples, one way I might think about this is using the [fiona CLI](https://fiona.readthedocs.io/en/latest/cli.html):
```sh
# assuming a database is already created and has SpatiaLite
fio cat boundary.shp | sqlite-utils insert boundaries --conversion geometry GeometryGeoJSON -
```
Anyway, very interested to see where you land here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism,
https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1033332570,https://api.github.com/repos/simonw/sqlite-utils/issues/403,1033332570,IC_kwDOCGYnMM49l2da,536941,fgregg,2022-02-09T04:22:43Z,2022-02-09T04:22:43Z,CONTRIBUTOR,dddoooope,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126692066,Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`,
https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032732242,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1032732242,IC_kwDOCGYnMM49jj5S,25778,eyeseast,2022-02-08T15:26:59Z,2022-02-08T15:26:59Z,CONTRIBUTOR,"What if you did something like this:
```python
class Conversion:
def __init__(self, *args, **kwargs):
""Put whatever settings you need here""
def python(self, row, column, value): # not sure on args here
""Python step to transform value""
return value
def sql(self, row, column, value):
""Return the actual sql that goes in the insert/update step, and maybe params""
# value is the return of self.python()
return value, []
```
This way, you're always passing an instance, which has methods that do the conversion. (Or you're passing a SQL string, as you would now.) The `__init__` could take column names, or SRID, or whatever other setup state you need per row, but the row is getting processed with the `python` and `sql` methods (or whatever you want to call them). This is pretty rough, so do what you will with names and args and such.
You'd then use it like this:
```python
# subclass might be unneeded here, if methods are present
class LngLatConversion(Conversion):
def __init__(self, x=""longitude"", y=""latitude""):
self.x = x
self.y = y
def python(self, row, column, value):
x = row[self.x]
y = row[self.y]
return x, y
def sql(self, row, column, value):
# value is now a tuple, returned above
s = ""GeomFromText(POINT(? ?))""
return s, value
table.insert_all(rows, conversions={""point"": LngLatConversion(""lng"", ""lat""))}
```
I haven't thought through all the implementation details here, and it'll probably break in ways I haven't foreseen, but wanted to get this idea out of my head. Hope it helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism,
https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032126353,https://api.github.com/repos/simonw/sqlite-utils/issues/403,1032126353,IC_kwDOCGYnMM49hP-R,536941,fgregg,2022-02-08T01:45:15Z,2022-02-08T01:45:31Z,CONTRIBUTOR,"you can hack something like this to achieve this result:
`sqlite-utils convert my_database my_table rowid ""{'id': value}"" --multi`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126692066,Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`,
https://github.com/simonw/sqlite-utils/issues/26#issuecomment-1032120014,https://api.github.com/repos/simonw/sqlite-utils/issues/26,1032120014,IC_kwDOCGYnMM49hObO,536941,fgregg,2022-02-08T01:32:34Z,2022-02-08T01:32:34Z,CONTRIBUTOR,"if you are curious about prior art, https://github.com/jsnell/json-to-multicsv is really good!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455486286,Mechanism for turning nested JSON into foreign keys / many-to-many,
https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031779460,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1031779460,IC_kwDOCGYnMM49f7SE,25778,eyeseast,2022-02-07T18:24:56Z,2022-02-07T18:24:56Z,CONTRIBUTOR,"I wonder if there's any overlap with the goals here and the `sqlite3` module's concept of adapters and converters: https://docs.python.org/3/library/sqlite3.html#sqlite-and-python-types
I'm not sure that's _exactly_ what we're talking about here, but it might be a parallel with some useful ideas to borrow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism,
https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031791783,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1031791783,IC_kwDOCGYnMM49f-Sn,25778,eyeseast,2022-02-07T18:37:40Z,2022-02-07T18:37:40Z,CONTRIBUTOR,"I've never used it either, but it's interesting, right? Feel like I should try it for something.
I'm trying to get my head around how this conversions feature might work, because I really like the idea of it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism,
https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030629879,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030629879,IC_kwDOCGYnMM49bin3,25778,eyeseast,2022-02-05T13:57:33Z,2022-02-05T19:49:38Z,CONTRIBUTOR,"I'm mostly using [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite) at the moment. Even with shapefiles, I'm usually converting to GeoJSON and projecting to EPSG:4326 (with [ogr2ogr](https://gdal.org/programs/ogr2ogr.html)) first.
I think an open question here is how much you want to leave to external libraries and how much you want here. My thinking has been that adding Spatialite helpers here would make external stuff easier, but it would be nice to have some standard way to insert geometries.
I'm in the middle of adding GeoJSON and Spatialite support to [geocode-sqlite](https://github.com/eyeseast/geocode-sqlite), and that will probably use WKT. Since that's all points, I think I can just make the string inline. But for polygons, I'd generally use Shapely, which probably isn't a dependency you want to add to sqlite-utils.
I've also been trying to get some of the approaches [here](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=Supporting+GeoJSON) to work, but haven't had any success so far.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI,
https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740826,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740826,IC_kwDOCGYnMM49b9ta,25778,eyeseast,2022-02-06T02:59:10Z,2022-02-06T02:59:10Z,CONTRIBUTOR,"All this said, I don't think it's unreasonable to point people to dedicated tools like `geojson-to-sqlite`. If I'm dealing with a bunch of GeoJSON or Shapefiles, I need to something to read those anyway (or I need to figure out virtual tables). But something like this might make it easier to build those libraries, or standardize the underlying parts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code",
https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740653,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740653,IC_kwDOCGYnMM49b9qt,25778,eyeseast,2022-02-06T02:57:17Z,2022-02-06T02:57:17Z,CONTRIBUTOR,"I like the idea of having stock conversions you could import. I'd actually move them to a dedicated module (call it `sqlite_utils.conversions` or something), because it's different from other utilities. Maybe they even take configuration, or they're composable.
```python
from sqlite_utils.conversions import LongitudeLatitude
db[""places""].insert(
{
""name"": ""London"",
""lng"": -0.118092,
""lat"": 51.509865,
},
conversions={""point"": LongitudeLatitude(""lng"", ""lat"")},
)
```
I would definitely use that for every CSV I get with lat/lng columns where I actually need GeoJSON.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code",
https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030741289,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030741289,IC_kwDOCGYnMM49b90p,25778,eyeseast,2022-02-06T03:03:43Z,2022-02-06T03:03:43Z,CONTRIBUTOR,"> I wonder if there are any interesting non-geospatial canned conversions that it would be worth including?
Off the top of my head:
- Un-nesting JSON objects into columns
- Splitting arrays
- Normalizing dates and times
- URL munging with `urlparse`
- Converting strings to numbers
Some of this is easy enough with SQL functions, some is easier in Python. Maybe that's where having pre-built classes gets really handy, because it saves you from thinking about which way it's implemented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code",
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1030002502,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1030002502,IC_kwDOCGYnMM49ZJdG,25778,eyeseast,2022-02-04T13:50:19Z,2022-02-04T13:50:19Z,CONTRIBUTOR,Awesome. Thanks for your help getting it in. Will now look at adding CLI versions of this. It's going to be super helpful on a bunch of my projects.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029370537,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029370537,IC_kwDOCGYnMM49WvKp,25778,eyeseast,2022-02-03T20:25:58Z,2022-02-03T20:25:58Z,CONTRIBUTOR,"OK, I moved all the GIS helpers into `db.py` as methods on `Database` and `Table`, and I put `find_spatialite` back in `utils.py`. I deleted `gis.py`, since there's nothing left it. Docs and tests are updated and passing.
I think this is better.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029338360,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029338360,IC_kwDOCGYnMM49WnT4,25778,eyeseast,2022-02-03T19:43:56Z,2022-02-03T19:43:56Z,CONTRIBUTOR,"Works for me. I was just looking at how the FTS extensions work and they're just methods, too. So this can be consistent with that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029326568,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029326568,IC_kwDOCGYnMM49Wkbo,25778,eyeseast,2022-02-03T19:28:26Z,2022-02-03T19:28:26Z,CONTRIBUTOR,"> `from sqlite_utils.utils import find_spatialite` is part of the documented API already:
>
> https://sqlite-utils.datasette.io/en/3.22.1/python-api.html#finding-spatialite
>
> To avoid needing to bump the major version number to 4 to indicate a backwards incompatible change, we should keep a `from .gis import find_spatialite` line at the top of `utils.py` such that any existing code with that documented import continues to work.
This is fixed now. I had to take out the type annotations for `Database` and `Table` to avoid a circular import, but that's fine and may be moot if these become class methods.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029317527,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029317527,IC_kwDOCGYnMM49WiOX,25778,eyeseast,2022-02-03T19:18:02Z,2022-02-03T19:18:02Z,CONTRIBUTOR,"Taking part of the conversation from #385 here.
> Would sqlite-utils add-geometry-column ... be a good CLI enhancement. for example?
Yes. And also `sqlite-utils create-spatial-index` would be great to have. My plan would be to add those once the Python API is settled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029306428,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029306428,IC_kwDOCGYnMM49Wfg8,25778,eyeseast,2022-02-03T19:03:43Z,2022-02-03T19:03:43Z,CONTRIBUTOR,"I thought about adding these as methods on `Database` and `Table`, and I'm back and forth on it for the same reasons you are. It's certainly cleaner, and it's clearer what you're operating on. I could go either way.
I do sort of like having all the Spatialite stuff in its own module, just because it's built around an extension you might not have or want, but I don't know if that's a good reason to have a different API.
You could have `init_spatialite` add methods to `Database` and `Table`, so they're only there if you have Spatialite set up. Is that too clever? It feels too clever.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029180984,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029180984,IC_kwDOCGYnMM49WA44,25778,eyeseast,2022-02-03T16:42:04Z,2022-02-03T16:42:04Z,CONTRIBUTOR,Fixed my spelling. That's a useful thing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029175907,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029175907,IC_kwDOCGYnMM49V_pj,25778,eyeseast,2022-02-03T16:36:54Z,2022-02-03T16:36:54Z,CONTRIBUTOR,"@simonw Not sure if you've seen this, but any chance you can run the tests?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods,
https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1038336591,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1038336591,IC_kwDOCGYnMM4948JP,25778,eyeseast,2022-02-13T18:48:21Z,2022-02-13T18:49:49Z,CONTRIBUTOR,"Been chipping away at this between other things and realized `sqlite-utils init-spatialite` is probably unnecessary. Any of the other commands requires running `db.init_spatialite` to have the extension functions available, and that will do everything `init-spatialite` would do.
I think it's probably worth keeping a SpatiaLite flag on `create-database` in case you wanted to create all the spatial metadata up front. Otherwise, it's going to get added the first time you run `add-geometry-column` or `create-spatial-index`, which is probably fine in most cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1013698557,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1013698557,IC_kwDOCGYnMM48a8_9,25778,eyeseast,2022-01-15T15:15:22Z,2022-01-15T15:15:22Z,CONTRIBUTOR,@simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012413729,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012413729,IC_kwDOCGYnMM48WDUh,25778,eyeseast,2022-01-13T18:50:00Z,2022-01-13T18:50:00Z,CONTRIBUTOR,"One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012253198,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012253198,IC_kwDOCGYnMM48VcIO,25778,eyeseast,2022-01-13T15:39:14Z,2022-01-13T15:39:14Z,CONTRIBUTOR,"Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (`gis.py` or something) just so it's easier to find later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012230212,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012230212,IC_kwDOCGYnMM48VWhE,25778,eyeseast,2022-01-13T15:15:13Z,2022-01-13T15:15:13Z,CONTRIBUTOR,"Some proposals I'd add to sqlite-utils:
Some version of this, from [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite/blob/main/geojson_to_sqlite/utils.py#L124-L130):
```python
def init_spatialite(db, lib):
db.conn.enable_load_extension(True)
db.conn.load_extension(lib)
# Initialize SpatiaLite if not yet initialized
if ""spatial_ref_sys"" in db.table_names():
return
db.conn.execute(""select InitSpatialMetadata(1)"")
```
Also a function for creating a spatial index:
```python
db.conn.execute(""select CreateSpatialIndex(?, ?)"", [table, ""geometry""])
```
I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like:
```sh
sqlite-utils spatial-index spatial.db table-name column-name
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012158895,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012158895,IC_kwDOCGYnMM48VFGv,25778,eyeseast,2022-01-13T13:55:59Z,2022-01-13T13:55:59Z,CONTRIBUTOR,"Came here to add this. I might pick it up.
Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009548580,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1009548580,IC_kwDOCGYnMM48LH0k,536941,fgregg,2022-01-11T02:43:34Z,2022-01-11T02:43:34Z,CONTRIBUTOR,thanks so much! always a pleasure to see how you work through these things,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008275546,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008275546,IC_kwDOCGYnMM48GRBa,536941,fgregg,2022-01-09T11:01:15Z,2022-01-09T13:37:51Z,CONTRIBUTOR,"i don’t want to be such a partisan for analyze, but the query planner deciding *not* to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice.
the original poster in that stack overflow doesn’t say there’s a performance regression ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008166084,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008166084,IC_kwDOCGYnMM48F2TE,536941,fgregg,2022-01-08T22:32:47Z,2022-01-08T22:32:47Z,CONTRIBUTOR,or using “ pragma optimize”,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008161965,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008161965,IC_kwDOCGYnMM48F1St,536941,fgregg,2022-01-08T22:02:56Z,2022-01-08T22:02:56Z,CONTRIBUTOR,"for options 2 and 3, i would worry about discoverablity.
in other db’s it is not necessary to explicitly call analyze for most indices. ie for postgres
> The system regularly collects statistics on all of a table's columns. Newly-created non-expression indexes can immediately use these statistics to determine an index's usefulness.
i suppose i would propose raising a warning if the stats table is created that explains what is going on and informs users about a —no-analyze argument.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164116,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008164116,IC_kwDOCGYnMM48F10U,536941,fgregg,2022-01-08T22:18:57Z,2022-01-08T22:18:57Z,CONTRIBUTOR,"the table with the query ran so bad was about 50k.
i think the scenario should not be worse than no stats.
i also did not know that sqlite was so different from postgres and needed an explicit analyze call.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164786,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008164786,IC_kwDOCGYnMM48F1-y,536941,fgregg,2022-01-08T22:24:19Z,2022-01-08T22:24:19Z,CONTRIBUTOR,the out-of-date scenario you describe could be addressed by automatically adding an analyze to the insert or convert commands if they implicate an index,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007636709,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007636709,IC_kwDOCGYnMM48D1Dl,536941,fgregg,2022-01-07T18:28:33Z,2022-01-07T18:29:43Z,CONTRIBUTOR,"i added an index to one table with sqlite-utils, and then a query that used to take about 1 second started taking hundreds of seconds.
running analyze got me back to sub second speed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991405755,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991405755,IC_kwDOCGYnMM47F6a7,536941,fgregg,2021-12-11T01:38:29Z,2021-12-11T01:38:29Z,CONTRIBUTOR,"wow! that's awesome! thanks so much, @simonw!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983155079,https://api.github.com/repos/simonw/sqlite-utils/issues/348,983155079,IC_kwDOCGYnMM46mcGH,25778,eyeseast,2021-12-01T00:28:40Z,2021-12-01T00:28:40Z,CONTRIBUTOR,"I'd use this. Right now, I tend to do `touch my.db` and then `enable-wal` or whatever else, but I'm never sure if that's a bad idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database,
https://github.com/simonw/sqlite-utils/issues/26#issuecomment-964205475,https://api.github.com/repos/simonw/sqlite-utils/issues/26,964205475,IC_kwDOCGYnMM45eJuj,536941,fgregg,2021-11-09T14:31:29Z,2021-11-09T14:31:29Z,CONTRIBUTOR,i was just reaching for a tool to do this this morning,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455486286,Mechanism for turning nested JSON into foreign keys / many-to-many,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-953911245,https://api.github.com/repos/simonw/sqlite-utils/issues/242,953911245,IC_kwDOCGYnMM4424fN,25778,eyeseast,2021-10-28T14:37:55Z,2021-10-28T14:37:55Z,CONTRIBUTOR,"I've been thinking about this a bit lately, doing a project that involves moving a lot of data in and out of SQLite files, datasette and GeoJSON. This has me leaning toward the idea that something like [`datasette query`](https://github.com/simonw/datasette/issues/1356) would be a better place to do async queries.
I know there's a lot of overlap in sqlite-utils and datasette, and maybe keeping sqlite-utils synchronous would let datasette be entirely async and give a cleaner separation of implementations.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/pull/326#issuecomment-916119657,https://api.github.com/repos/simonw/sqlite-utils/issues/326,916119657,IC_kwDOCGYnMM42muBp,191622,meatcar,2021-09-09T13:54:10Z,2021-09-09T13:54:10Z,CONTRIBUTOR,dupe of #293?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991237645,Test against 3.10-dev,
https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040998433,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040998433,IC_kwDOCGYnMM4-DGAh,25778,eyeseast,2022-02-16T01:29:39Z,2022-02-16T01:29:39Z,CONTRIBUTOR,Happy to do it and have it in the library. Going to use it a bunch. This whole SpatiaLite toolchain become a huge part of my work in the past year.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI,
https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040580250,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040580250,IC_kwDOCGYnMM4-Bf6a,25778,eyeseast,2022-02-15T17:40:00Z,2022-02-15T17:40:00Z,CONTRIBUTOR,@simonw I think this is ready for a look.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI,
https://github.com/simonw/datasette/issues/2213#issuecomment-1843072926,https://api.github.com/repos/simonw/datasette/issues/2213,1843072926,IC_kwDOBm6k_c5t2w-e,536941,fgregg,2023-12-06T15:05:44Z,2023-12-06T15:05:44Z,CONTRIBUTOR,"it probably does not make sense to gzip large sqlite database files on the fly. it can take many seconds to gzip a large file and you either have to have this big thing in memory, or write it to disk, which some deployment environments will not like.
i wonder if it would make sense to gzip the databases as part of the datasette publish process. it would be very cool to statically serve those as if they dynamically zipped (i.e. serve the filename example.db, not example.db.zip, and rely on the browser to expand).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2028698018,feature request: gzip compression of database downloads,
https://github.com/simonw/datasette/pull/2209#issuecomment-1812750369,https://api.github.com/repos/simonw/datasette/issues/2209,1812750369,IC_kwDOBm6k_c5sDGAh,198537,rgieseke,2023-11-15T15:29:37Z,2023-11-15T15:29:37Z,CONTRIBUTOR,"Looks like tests are passing now but there is an issue with yaml loading and/or cog.
https://github.com/simonw/datasette/actions/runs/6879299298/job/18710911166?pr=2209","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994861266,Fix query for suggested facets with column named value,
https://github.com/simonw/datasette/pull/2209#issuecomment-1812623778,https://api.github.com/repos/simonw/datasette/issues/2209,1812623778,IC_kwDOBm6k_c5sCnGi,198537,rgieseke,2023-11-15T14:22:42Z,2023-11-15T15:24:09Z,CONTRIBUTOR,"Whoops, looks like I forgot to check for other places where the 'facetable' table is used in the tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994861266,Fix query for suggested facets with column named value,
https://github.com/simonw/datasette/issues/2208#issuecomment-1812617851,https://api.github.com/repos/simonw/datasette/issues/2208,1812617851,IC_kwDOBm6k_c5sClp7,198537,rgieseke,2023-11-15T14:18:58Z,2023-11-15T14:18:58Z,CONTRIBUTOR,"Without aliases:
The proposed fix in #2209 also works when the 'value' column is actually facetable (just added another value in the 'value' column).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994857251,No suggested facets when a column named 'value' is included,
https://github.com/simonw/datasette/pull/2202#issuecomment-1801876943,https://api.github.com/repos/simonw/datasette/issues/2202,1801876943,IC_kwDOBm6k_c5rZnXP,49699333,dependabot[bot],2023-11-08T13:19:00Z,2023-11-08T13:19:00Z,CONTRIBUTOR,Superseded by #2206.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1959278971,Bump the python-packages group with 1 update,"{""id"": 29110, ""slug"": ""dependabot"", ""node_id"": ""MDM6QXBwMjkxMTA="", ""owner"": {""login"": ""github"", ""id"": 9919, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjk5MTk="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/9919?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/github"", ""html_url"": ""https://github.com/github"", ""followers_url"": ""https://api.github.com/users/github/followers"", ""following_url"": ""https://api.github.com/users/github/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/github/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/github/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/github/subscriptions"", ""organizations_url"": ""https://api.github.com/users/github/orgs"", ""repos_url"": ""https://api.github.com/users/github/repos"", ""events_url"": ""https://api.github.com/users/github/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/github/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Dependabot"", ""description"": """", ""external_url"": ""https://dependabot-api.githubapp.com"", ""html_url"": ""https://github.com/apps/dependabot"", ""created_at"": ""2019-04-16T22:34:25Z"", ""updated_at"": ""2023-10-12T13:35:09Z"", ""permissions"": {""checks"": ""write"", ""contents"": ""write"", ""issues"": ""write"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""read"", ""vulnerability_alerts"": ""read"", ""workflows"": ""write""}, ""events"": [""check_suite"", ""issues"", ""issue_comment"", ""label"", ""pull_request"", ""pull_request_review"", ""pull_request_review_comment"", ""repository""]}"
https://github.com/simonw/datasette/issues/1655#issuecomment-1767219901,https://api.github.com/repos/simonw/datasette/issues/1655,1767219901,IC_kwDOBm6k_c5pVaK9,536941,fgregg,2023-10-17T21:29:03Z,2023-10-17T21:29:03Z,CONTRIBUTOR,@yejiyang why don’t you move this discussion to my fork to spare simon’s notifications ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data,
https://github.com/simonw/datasette/issues/1655#issuecomment-1766994810,https://api.github.com/repos/simonw/datasette/issues/1655,1766994810,IC_kwDOBm6k_c5pUjN6,536941,fgregg,2023-10-17T19:01:59Z,2023-10-17T19:01:59Z,CONTRIBUTOR,"hi @yejiyang, have your tried using my fork of datasette: https://github.com/fgregg/datasette/tree/no_limit_csv_publish
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data,
https://github.com/simonw/datasette/pull/2200#issuecomment-1777228352,https://api.github.com/repos/simonw/datasette/issues/2200,1777228352,IC_kwDOBm6k_c5p7lpA,49699333,dependabot[bot],2023-10-24T13:40:25Z,2023-10-24T13:40:25Z,CONTRIBUTOR,Superseded by #2202.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1949756141,Bump the python-packages group with 1 update,
https://github.com/simonw/datasette/issues/2199#issuecomment-1760401731,https://api.github.com/repos/simonw/datasette/issues/2199,1760401731,IC_kwDOBm6k_c5o7ZlD,15178711,asg017,2023-10-12T21:41:42Z,2023-10-12T21:41:42Z,CONTRIBUTOR,"I dig it - I was thinking an Observable notebook where you paste your `metadata.json`/`metadata.yaml` and it would generate the new metadata + datasette.yaml files, but an extensible `datasette upgrade` plugin would be nice for future plugins.
One thing to think about: If someone has comments in their original `metadata.yaml`, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml,
https://github.com/simonw/datasette/pull/2155#issuecomment-1737363182,https://api.github.com/repos/simonw/datasette/issues/2155,1737363182,IC_kwDOBm6k_c5njg7u,418191,jaywgraves,2023-09-27T13:05:41Z,2023-09-27T13:05:41Z,CONTRIBUTOR,I'm hitting the #2123 issue and I just patched my local version with this and it seems to work fine.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865572575,Fix hupper.start_reloader entry point,
https://github.com/simonw/datasette/pull/2190#issuecomment-1729961503,https://api.github.com/repos/simonw/datasette/issues/2190,1729961503,IC_kwDOBm6k_c5nHR4f,15178711,asg017,2023-09-21T16:56:57Z,2023-09-21T16:56:57Z,CONTRIBUTOR,TODO: add similar checks for permissions/allow/canned queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,"Raise an exception if a ""plugins"" block exists in metadata.json",
https://github.com/simonw/datasette/issues/2188#issuecomment-1722848454,https://api.github.com/repos/simonw/datasette/issues/2188,1722848454,IC_kwDOBm6k_c5msJTG,15178711,asg017,2023-09-18T06:58:53Z,2023-09-18T06:58:53Z,CONTRIBUTOR,"Thinking about this more, here a list of things I imagine a ""compile-to-sql"" plugin would want to do:
1. Attach itself to the SQL code editor (switch from SQL -> PRQL/Logica, additional syntax highlighting)
2. Add ""Query using PRQL"" buttons in various parts of Datasette's UI, like `/dbname` page
3. Use `$LANGUAGE=` instead of `sql=` in the JSON API and the SQL results pages
4. Have their own dedicated code editor page
1) and 2) would be difficult to do with current plugin hooks, unless we add the concept of ""slots"" and get the JS plugin support in. 3) could maybe be done with the [`asgi_wrapper(datasette)`](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) hook? And 4) ca n be done easily with the `register_routes()` hooks.
So it really only sounds like extending the SQL editor will be the hard part. In #2094 I want to add JavaScript plugin hooks for extending the SQL editor, which may work here.
If I get the time/motivation, I might try out a `datasette-prql` extension, just because I like playing with it. It'd be really cool if I can get the `asgi_wrapper()` hook to work right there...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1900026059,"Plugin Hooks for ""compile to SQL"" languages",
https://github.com/simonw/datasette/issues/1191#issuecomment-1722845490,https://api.github.com/repos/simonw/datasette/issues/1191,1722845490,IC_kwDOBm6k_c5msIky,15178711,asg017,2023-09-18T06:55:52Z,2023-09-18T06:55:52Z,CONTRIBUTOR,"One note here: this feature could be called ""slots"", similar to [Layout Slots](https://vitepress.dev/guide/extending-default-theme#layout-slots) in Vitepress.
In Vitepress, you can add custom components/widget/gadgets into determined named ""slots"", like so:
```
doc-top
doc-bottom
doc-footer-before
doc-before
doc-after
...
```
Would be great to do in both Python and Javascript, with the upcoming JavaScript API #2052. In `datasette-write-ui`, all we do is add a few ""Insert row"" and ""edit this row"" buttons and that required completely capturing the `table.html` template, which isn't great for other plugins. But having ""slots"" like `table-footer-before` or `table-row-id` or something would be great to work with.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/pull/2182#issuecomment-1719451803,https://api.github.com/repos/simonw/datasette/issues/2182,1719451803,IC_kwDOBm6k_c5mfMCb,49699333,dependabot[bot],2023-09-14T13:27:26Z,2023-09-14T13:27:26Z,CONTRIBUTOR,"Looks like these dependencies are updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1890593563,Bump the python-packages group with 2 updates,
https://github.com/simonw/datasette/pull/2183#issuecomment-1716801971,https://api.github.com/repos/simonw/datasette/issues/2183,1716801971,IC_kwDOBm6k_c5mVFGz,15178711,asg017,2023-09-13T01:34:01Z,2023-09-13T01:34:01Z,CONTRIBUTOR,"@simonw docs are finished, this is ready for review!
One thing: I added ""Configuration"" as a top-level item in the documentation site, at the very bottom. Not sure if this is the best, maybe it can be named ""datasette.yaml Configuration"" or something similar?
Mostly because ""Configuration"" by itself can mean many things, but adding ""datasette.yaml"" would make it pretty clear it's about that specific file, and is easier to scan. I'd also be fine with using ""datasette.yaml"" instead of ""datasette.json"", since writing in YAML is much more forgiving (and advanced users will know JSON is also supported)
Also, maybe this is a chance to consolidate the docs a bit? I think ""Settings"", ""Configuration"", ""Metadata"", and ""Authentication and permissions"" should possibly be under the same section. Maybe even consolidate the different Plugin pages that exist?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891212159,`datasette.yaml` plugin support,
https://github.com/simonw/datasette/issues/2157#issuecomment-1700291967,https://api.github.com/repos/simonw/datasette/issues/2157,1700291967,IC_kwDOBm6k_c5lWGV_,15178711,asg017,2023-08-31T02:45:56Z,2023-08-31T02:45:56Z,CONTRIBUTOR,"@simonw what do you think about adding a `DATASETTE_INTERNAL_DB_PATH` env variable, where when defined, is the default location of the internal DB? This means when the `--internal` flag is NOT provided, Datasette would check to see if `DATASETTE_INTERNAL_DB_PATH` exists, and if so, uses that as the internal database (and would fallback to an ephemeral memory database)
My rationale: some plugins may require, or strongly encourage, a persistent internal database (`datasette-comments`, `datasette-bookmarks`, `datasette-link-shortener`, etc.). However, for users that have a global installation of Datasette (say from `brew install` or a global `pip install`), it would be annoying having to specify `--internal` every time. So instead, they can just add `export DATASETTE_INTERNAL_DB_PATH=""/path/to/internal.db""` to their bashrc/zshrc/whereever to not have to worry about `--internal`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865869205,"Proposal: Make the `_internal` database persistent, customizable, and hidden",
https://github.com/simonw/datasette/pull/2148#issuecomment-1696591957,https://api.github.com/repos/simonw/datasette/issues/2148,1696591957,IC_kwDOBm6k_c5lH_BV,49699333,dependabot[bot],2023-08-29T00:15:29Z,2023-08-29T00:15:29Z,CONTRIBUTOR,This pull request was built based on a group rule. Closing it will not ignore any of these versions in future pull requests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2152#issuecomment-1695736691,https://api.github.com/repos/simonw/datasette/issues/2152,1695736691,IC_kwDOBm6k_c5lEuNz,49699333,dependabot[bot],2023-08-28T13:49:35Z,2023-08-28T13:49:35Z,CONTRIBUTOR,Superseded by #2160.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865174661,Bump the python-packages group with 3 updates,
https://github.com/simonw/datasette/pull/2148#issuecomment-1689198413,https://api.github.com/repos/simonw/datasette/issues/2148,1689198413,IC_kwDOBm6k_c5krx9N,49699333,dependabot[bot],2023-08-23T02:57:55Z,2023-08-23T02:57:55Z,CONTRIBUTOR,"Looks like this PR has been edited by someone other than Dependabot. That means Dependabot can't rebase it - sorry!
If you're happy for Dependabot to recreate it from scratch, overwriting any edits, you can request `@dependabot recreate`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/issues/2093#issuecomment-1688532012,https://api.github.com/repos/simonw/datasette/issues/2093,1688532012,IC_kwDOBm6k_c5kpPQs,15178711,asg017,2023-08-22T16:21:40Z,2023-08-22T16:21:40Z,CONTRIBUTOR,"OK Here's the gameplan for this, which is closely tied to #2143 :
- We will add a new `datasette.json`/`datasette.yaml` configuration file to datasette, which combines settings/plugin config/permissions/canned queries into a new file format
- Metadata will NOT be a part of this file
- TOML support is not planned, but maybe we can create a separate issue for support TOML with JSON/YAML
- The `settings.json` file will be deprecated, and the `--config` arg will be brought back.
- Command line arguments can still be used to overwrite values (ex `--setting` will overwrite settings in `datasette.yaml`
The format of `datasette.json` will follow what Simon listed here: https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426
Here's the current implementation plan:
1. Add a new `--config` flag and port over `""settings""` into a new datasette.json config file, remove settings.json
2. Add top-level plugin config support to `datasette.json`
3. Figure out database/table structure of config `datasette.json`
4. Port over database/table level plugin config support `datasette.json`
5. Port over permissions/auth settings to `datasette.json`
6. Deprecate non-metadata values in `metadata.json`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/issues/2145#issuecomment-1686745094,https://api.github.com/repos/simonw/datasette/issues/2145,1686745094,IC_kwDOBm6k_c5kibAG,15178711,asg017,2023-08-21T17:30:01Z,2023-08-21T17:30:01Z,CONTRIBUTOR,"Another point: The new Datasette write API should refuse to insert a row with a NULL primary key. That will likely decrease the likelihood someone find themselves with NULLs in their primary keys, at least with Datasette users. Especially buggy code that uses the write API, like our `datasette-write-ui` bug that led to this issue.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/pull/2144#issuecomment-1686366557,https://api.github.com/repos/simonw/datasette/issues/2144,1686366557,IC_kwDOBm6k_c5kg-ld,49699333,dependabot[bot],2023-08-21T13:48:15Z,2023-08-21T13:48:15Z,CONTRIBUTOR,Superseded by #2148.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856760386,Bump the python-packages group with 3 updates,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684496274,https://api.github.com/repos/simonw/datasette/issues/2143,1684496274,IC_kwDOBm6k_c5kZ1-S,15178711,asg017,2023-08-18T22:30:45Z,2023-08-18T22:30:45Z,CONTRIBUTOR,"> That said, I do really like a bias towards settings that can be changed at runtime
Does this include things like `--settings` values or plugin config? I can totally see being able to update metadata without restarting, but not sure if that would work well with `--setting`, plugin config, or auth/permissions stuff.
Well it could work with `--setting` and auth/permissions, with a lot of core changes. But changing plugin config on the fly could be challenging, for plugin authors. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684205563,https://api.github.com/repos/simonw/datasette/issues/2143,1684205563,IC_kwDOBm6k_c5kYu_7,15178711,asg017,2023-08-18T17:12:54Z,2023-08-18T17:12:54Z,CONTRIBUTOR,"Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.
Though I imagine Python imports might make this complex to do, and json/yaml is already supported and pretty easy to write
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684202932,https://api.github.com/repos/simonw/datasette/issues/2143,1684202932,IC_kwDOBm6k_c5kYuW0,15178711,asg017,2023-08-18T17:10:21Z,2023-08-18T17:10:21Z,CONTRIBUTOR,"I agree with all your points!
I think the best solution would be having a `datasette.json` config file, where you ""configure"" your datasette instances, with settings, permissions/auth, plugin configuration, and table settings (sortable column, label columns, etc.). Which #2093 would do.
Then optionally, you have a `metadata.json`, or use `datasette_metadata`, or some other plugin to define metadata (ex the future [sqlite-docs](https://github.com/asg017/sqlite-docs) plugin).
Everything in `datasette.json` could also be overwritten by CLI flags, like `--setting key value`, `--plugin xxxx key value`.
We could even completely remove `settings.json` in favor or just `datasette.json`. Mostly because I think the less files the better, especially if they have generic names like `settings.json` or `config.json`. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/pull/2142#issuecomment-1683950031,https://api.github.com/repos/simonw/datasette/issues/2142,1683950031,IC_kwDOBm6k_c5kXwnP,49699333,dependabot[bot],2023-08-18T13:49:24Z,2023-08-18T13:49:24Z,CONTRIBUTOR,"Looks like these dependencies are updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1854970601,Bump the python-packages group with 2 updates,
https://github.com/simonw/datasette/pull/2141#issuecomment-1682256251,https://api.github.com/repos/simonw/datasette/issues/2141,1682256251,IC_kwDOBm6k_c5kRTF7,49699333,dependabot[bot],2023-08-17T13:07:43Z,2023-08-17T13:07:43Z,CONTRIBUTOR,"Looks like blacken-docs is updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1853289039,Bump the python-packages group with 1 update,
https://github.com/simonw/datasette/pull/2125#issuecomment-1668187546,https://api.github.com/repos/simonw/datasette/issues/2125,1668187546,IC_kwDOBm6k_c5jboWa,49699333,dependabot[bot],2023-08-07T16:20:26Z,2023-08-07T16:20:26Z,CONTRIBUTOR,"Looks like sphinx is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1833193570,Bump sphinx from 6.1.3 to 7.1.2,
https://github.com/simonw/datasette/pull/2121#issuecomment-1668186872,https://api.github.com/repos/simonw/datasette/issues/2121,1668186872,IC_kwDOBm6k_c5jboL4,49699333,dependabot[bot],2023-08-07T16:20:19Z,2023-08-07T16:20:19Z,CONTRIBUTOR,"Looks like furo is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1824399610,Bump furo from 2023.3.27 to 2023.7.26,
https://github.com/simonw/datasette/pull/2098#issuecomment-1668186815,https://api.github.com/repos/simonw/datasette/issues/2098,1668186815,IC_kwDOBm6k_c5jboK_,49699333,dependabot[bot],2023-08-07T16:20:18Z,2023-08-07T16:20:18Z,CONTRIBUTOR,"Looks like blacken-docs is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1796830110,Bump blacken-docs from 1.14.0 to 1.15.0,
https://github.com/simonw/datasette/pull/2124#issuecomment-1662215579,https://api.github.com/repos/simonw/datasette/issues/2124,1662215579,IC_kwDOBm6k_c5jE2Wb,49699333,dependabot[bot],2023-08-02T13:28:43Z,2023-08-02T13:28:43Z,CONTRIBUTOR,Superseded by #2125.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1826424151,Bump sphinx from 6.1.3 to 7.1.1,
https://github.com/simonw/datasette/pull/2107#issuecomment-1655678215,https://api.github.com/repos/simonw/datasette/issues/2107,1655678215,IC_kwDOBm6k_c5ir6UH,49699333,dependabot[bot],2023-07-28T13:23:16Z,2023-07-28T13:23:16Z,CONTRIBUTOR,Superseded by #2124.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1820346348,Bump sphinx from 6.1.3 to 7.1.0,
https://github.com/simonw/datasette/pull/2077#issuecomment-1653652665,https://api.github.com/repos/simonw/datasette/issues/2077,1653652665,IC_kwDOBm6k_c5ikLy5,49699333,dependabot[bot],2023-07-27T13:40:52Z,2023-07-27T13:40:52Z,CONTRIBUTOR,Superseded by #2121.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1719759468,Bump furo from 2023.3.27 to 2023.5.20,
https://github.com/simonw/datasette/pull/2075#issuecomment-1649849249,https://api.github.com/repos/simonw/datasette/issues/2075,1649849249,IC_kwDOBm6k_c5iVrOh,49699333,dependabot[bot],2023-07-25T13:28:35Z,2023-07-25T13:28:35Z,CONTRIBUTOR,Superseded by #2107.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1710164693,Bump sphinx from 6.1.3 to 7.0.1,
https://github.com/simonw/datasette/pull/2052#issuecomment-1630776144,https://api.github.com/repos/simonw/datasette/issues/2052,1630776144,IC_kwDOBm6k_c5hM6tQ,9020979,hydrosquall,2023-07-11T12:54:03Z,2023-07-11T12:54:03Z,CONTRIBUTOR,"Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the `table` and `database` views. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/issues/2104#issuecomment-1641082395,https://api.github.com/repos/simonw/datasette/issues/2104,1641082395,IC_kwDOBm6k_c5h0O4b,15178711,asg017,2023-07-18T22:41:37Z,2023-07-18T22:41:37Z,CONTRIBUTOR,"For filtering virtual table's ""shadow tables"" (ex the FTS5 _content and most the spatialite tables), you can use `pragma_table_list` (first appeared in SQLite 3.37 (2021-11-27), which has a `type` column that calls out `type=""shadow""` tables https://www.sqlite.org/pragma.html#pragma_table_list","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1808215339,Tables starting with an underscore should be treated as hidden,
https://github.com/simonw/datasette/issues/2087#issuecomment-1616853644,https://api.github.com/repos/simonw/datasette/issues/2087,1616853644,IC_kwDOBm6k_c5gXzqM,15178711,asg017,2023-07-02T22:00:48Z,2023-07-02T22:00:48Z,CONTRIBUTOR,"I just saw in the docs that Dasette auto-detects `settings.json`:
> settings.json - settings that would normally be passed using --setting - here they should be stored as a JSON object of key/value pairs
> [*Source*](https://docs.datasette.io/en/stable/settings.html#:~:text=settings.json%20%2D%20settings%20that%20would%20normally%20be%20passed%20using%20%2D%2Dsetting%20%2D%20here%20they%20should%20be%20stored%20as%20a%20JSON%20object%20of%20key/value%20pairs)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1765870617,`--settings settings.json` option,
https://github.com/simonw/datasette/issues/2093#issuecomment-1616286848,https://api.github.com/repos/simonw/datasette/issues/2093,1616286848,IC_kwDOBm6k_c5gVpSA,15178711,asg017,2023-07-02T02:17:46Z,2023-07-02T02:17:46Z,CONTRIBUTOR,"Storing metadata in the database won't be required. I imagine there'll be many different ways to store metadata, including any possible `datasette_metadata` or sqlite-docs, or the older metadata.json way.
The next question will be how precedence should work - i'd imagine metadata.json > plugins > datasette_metadata > sqlite-docs","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/pull/2052#issuecomment-1615997736,https://api.github.com/repos/simonw/datasette/issues/2052,1615997736,IC_kwDOBm6k_c5gUiso,9020979,hydrosquall,2023-07-01T16:55:24Z,2023-07-01T16:55:24Z,CONTRIBUTOR,"> Ok @hydrosquall a couple things before this PR should be good to go:
Thank you @asg017 ! I've pushed both suggested changes onto this branch.
> Not sure how difficult it'll be to inject it server-side
If we are OK with having a build system, it would free me up to do do many things! We could make datasette-manager.js a server-side rendered file as a ""template"" instead of having it as a static JS file, but I'm not sure it's worth the extra jump in complexity / loss of syntax highlighting in the JS file.
In the short-term, I could see an intermediary solution where a unit test in the preferred language was able to read both `version.py` and `datasette-manager.js`, and make sure that the strings versions are in sync. (This assumes that we want the manager and datasette's versions to be synced, and not decoupled). Since the version is not changing very often, a ""manual sync"" might be good enough.
> In terms of how to integrate this into Datasette, a few options I can see working:
This sounds good to me. I'm not sure how to add a settings flag, but will be interested to see the PR that adds support for it.
> I'm also curious to see how ""plugins for a plugin' would work
I'm comfortable to wait until we have a realistic usecase for this. In the short term, I think we could give plugins a way to grant access to a ""public API of other plugins"", and also ask to be notified when plugins with other names have loaded, but don't picture the datasette manager getting more involved than that.
> here's a list of Simon's Datasette plugins that use ""extra_js_urls()""
Neat, thanks for compiling this list! Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?
> I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.)
I look forward to trying this out 👍
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1616095810,https://api.github.com/repos/simonw/datasette/issues/2052,1616095810,IC_kwDOBm6k_c5gU6pC,15178711,asg017,2023-07-01T20:31:31Z,2023-07-01T20:31:31Z,CONTRIBUTOR,"> Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?
I just did a github search for `user:simonw ""def extra_js_urls(""` ! Though I'm sure other plugins made by people other than Simon also exist out there https://github.com/search?q=user%3Asimonw+%22def+extra_js_urls%28%22&type=code","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/issues/2093#issuecomment-1613896210,https://api.github.com/repos/simonw/datasette/issues/2093,1613896210,IC_kwDOBm6k_c5gMhoS,15178711,asg017,2023-06-29T22:53:33Z,2023-06-29T22:53:33Z,CONTRIBUTOR,"Maybe we can have a separate issue for revamping `metadata.json`? A `datasette_metadata` table or the `sqlite-docs` extension seem like two reasonable additions that we can work through. Storing metadata inside a SQLite database makes sense, but I don't think storing `datasette.*` style config (ex ports, settings, etc.) inside a SQLite DB makes sense, since it's very environment-dependent","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/issues/2093#issuecomment-1613895188,https://api.github.com/repos/simonw/datasette/issues/2093,1613895188,IC_kwDOBm6k_c5gMhYU,15178711,asg017,2023-06-29T22:51:53Z,2023-06-29T22:51:53Z,CONTRIBUTOR,"I agree with not liking `metadata.json` stuff in a `datasette.*` config file. Editing description of a table/column in a file like `datasette.*` seems odd to me.
Though since plugin configuration currently lives in `metadata.json`, I think it should be removed from there and placed in `datasette.*`, at least for top-level config like `datasette-auth-github`'s config. Keeping `metadata.json` strictly for documentation/licensing/column units makes sense to me, but anything plugin related should be in some config file, like `datasette.*`.
And ya, supporting both `datasette.*` and CLI flags makes a lot of sense to me. Any `--setting` flag should override anything in `datasette.*` for easier debugging, with possibly a warning message so people don't get confused. Same with `--port` and a port defined in `datasette.*`","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/pull/2052#issuecomment-1613778296,https://api.github.com/repos/simonw/datasette/issues/2052,1613778296,IC_kwDOBm6k_c5gME14,15178711,asg017,2023-06-29T20:36:09Z,2023-06-29T20:36:09Z,CONTRIBUTOR,"Ok @hydrosquall a couple things before this PR should be good to go:
- Can we move `datasette/static/table-example-plugins.js` into `demos/plugins/static`?
- For `datasetteManager.VERSION`, can we fill that in or just comment it out for now? Not sure how difficult it'll be to inject it server-side. I imagine we could also have a small build process with esbuild/rollup that just injects a version string into `manager.js` directly, so we don't have to worry about server-rendering (but that can be a future PR)
In terms of how to integrate this into Datasette, a few options I can see working:
- Push this as-is and figure it out before the next release
- Hide this feature behind a settings flag (`--setting unstable-js-plugins on`) and use that setting to hide/show `` in `base.html`
I'll let @simonw decide which one to work with. I kindof like the idea of having an ""unstable"" opt-in process to enable JS plugins, to give us time to try it out with a wide variety of plugins until we feel its ready.
I'm also curious to see how ""plugins for a plugin' would work, like #1542. For example, if the leaflet plugin showed default markers, but also included its own hook for other plugins to add more markers/styling. I'm imagine that the individual plugin would re-create their own plugin system compared to this, since handling ""plugins of plugins"" at the top with Datasette seems really convoluted.
Also for posterity, here's a list of Simon's Datasette plugins that use ""extra_js_urls()"", which probably means they can be ported/re-written to use this new plugin system:
- [`datasette-vega`](https://github.com/simonw/datasette-vega/blob/00de059ab1ef77394ba9f9547abfacf966c479c4/datasette_vega/__init__.py#L25)
- [`datasette-cluster-map`](https://github.com/simonw/datasette-cluster-map/blob/795d25ad9ff6cba0307191f44fecc8f8070bef5c/datasette_cluster_map/__init__.py#L14)
- [`datasette-leaflet-geojson`](https://github.com/simonw/datasette-leaflet-geojson/blob/64713aa497750400b9ac2c12e8bb6ffab8eb77f3/datasette_leaflet_geojson/__init__.py#L47)
- [`datasette-pretty-traces`](https://github.com/simonw/datasette-pretty-traces/blob/5219d65eca3d7d7a73bb9d3120df42fe046a1315/datasette_pretty_traces/__init__.py#L5)
- [`datasette-youtube-embed`](https://github.com/simonw/datasette-youtube-embed/blob/4b4a0d7e58ebe15f47e9baf68beb9908c1d899da/datasette_youtube_embed/__init__.py#L55)
- [`datasette-leaflet-freedraw`](https://github.com/simonw/datasette-leaflet-freedraw/blob/8f28c2c2080ec9d29f18386cc6a2573a1c8fbde7/datasette_leaflet_freedraw/__init__.py#L66)
- [`datasette-hovercards`](https://github.com/simonw/datasette-hovercards/blob/9439ba46b7140fb03223faff0d21aeba5615a287/datasette_hovercards/__init__.py#L5)
- [`datasette-mp3-audio`](https://github.com/simonw/datasette-mp3-audio/blob/4402168792f452a46ab7b488e40ec49cd4b12185/datasette_mp3_audio/__init__.py#L6)
- [`datasette-geojson-map`](https://github.com/simonw/datasette-geojson-map/blob/32af5f1fd1a07278bbf8071fbb20a61e0f613246/datasette_geojson_map/__init__.py#L30)","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1606352600,https://api.github.com/repos/simonw/datasette/issues/2052,1606352600,IC_kwDOBm6k_c5fvv7Y,15178711,asg017,2023-06-26T00:17:04Z,2023-06-26T00:17:04Z,CONTRIBUTOR,":wave: would love to see this get merged soon! I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.), and this would help out a bunch","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1585149909,https://api.github.com/repos/simonw/datasette/issues/2052,1585149909,IC_kwDOBm6k_c5ee3fV,9020979,hydrosquall,2023-06-09T21:35:00Z,2023-06-09T21:35:00Z,CONTRIBUTOR,"Thanks @cldellow for the thoughtful comments! These are all things that I'll keep in mind as we figure out how/if this API is actually used by plugin authors once it's actually out in the world.
> Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.
Ah, I understand now! Thanks for explaining. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1548617257,https://api.github.com/repos/simonw/datasette/issues/2052,1548617257,IC_kwDOBm6k_c5cTgYp,193185,cldellow,2023-05-15T21:32:20Z,2023-05-15T21:32:20Z,CONTRIBUTOR,"> Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays?
The latter - that you could return a promise of arrays, so it parallels the [""await me maybe"" pattern in Datasette](https://simonwillison.net/2020/Sep/2/await-me-maybe/), where you can return either a value, a callable or an awaitable.
> I have a hunch that what you're describing might be achievable without adding Promises to the API with something
Oops, I did a poor job explaining. Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.
This isn't that big of a deal - it'd be a nice ergonomic improvement, but nowhere near as a big of an improvement as having an officially sanctioned way to add stuff to the column menus in the first place.
This could also be layered on in a future commit without breaking v1 users, too, so it's not at all urgent.
> especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag
Ah, this is maybe the the key point. Since it's all hosted inside Datasette, Datasette can provide some arbitrary sugar to make it easier to work with.
My experience with async scripts in JS is that people sometimes don't understand the race conditions inherent to them. If they copy/paste from a tutorial, it does just work. But then they'll delete half the code, and by chance it still works on their machine/Datasette templates, and now someone's headed for an annoying debugging session -- maybe them, maybe someone else who tries to re-use their plugin.
Again, a fairly minor thing, though.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2068#issuecomment-1547911570,https://api.github.com/repos/simonw/datasette/issues/2068,1547911570,IC_kwDOBm6k_c5cQ0GS,49699333,dependabot[bot],2023-05-15T13:59:35Z,2023-05-15T13:59:35Z,CONTRIBUTOR,Superseded by #2075.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1690842199,Bump sphinx from 6.1.3 to 7.0.0,
https://github.com/simonw/datasette/pull/2052#issuecomment-1546362374,https://api.github.com/repos/simonw/datasette/issues/2052,1546362374,IC_kwDOBm6k_c5cK54G,9020979,hydrosquall,2023-05-12T22:09:03Z,2023-05-12T22:09:03Z,CONTRIBUTOR,"Hey @cldellow , thanks for the thoughtful feedback and describing the ""lazy facets"" feature!
It sounds like the [postTask](https://developer.mozilla.org/en-US/docs/Web/API/Scheduler/postTask) API might be relevant for the types of network request scheduling you have in mind.
Addressing your points inline below:
> It might also be nice if the plugins could return Promises.
Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like `makeColumnActions` or `makeAboveTablePanelConfigs` supported returning a promise of arrays instead only returning plain arrays?
I think what you're describing can be achievable, but I want to make sure I do so in a way that addresses your need / keeps the complexity of the plugin core system at a level this is approachable .
I have a hunch that what you're describing might be achievable without adding Promises to the API with something like
```
fetch('/api/with-custom-facets').then(myFacets => {
// reusing the go() idiom
go(manager, myFacets);
})
```
but I'd like to confirm if that's the case before investigating adding support.
> bulletproof plugin registration code that is robust against the order in which the script tags load
Yes, I think what you wrote looks right to me! While it looks a little bit verbose compared to the second example, I'm hoping we can mitigate the cost of that during this API incubation phase by making it an easy-to-copy paste code snippet.
I haven't heard of the GA queing pattern before, thanks for the example. I won't have time to implement of proof of concept in the next few weeks, but I took some time to think through the pros/cons to decide whether we may want to add this in a future release:
I can see that this approach brings advantages
- Plugin developers don't need to know the name of the datasette initialization event to start their plugin
- Pushing a function to an array probably is easier (definitely more concise) than adding a document event listener
- One less event listener sitting in memory
It also has some minor costs
- A malicious plugin could choose to (or accidentally) mess with the order of the queue if multiple scripts are lined up
- Some risk in encouraging people to mutate global state
- (not a cost, more a moot point): changing this API may not make a meaningful difference if we're discussing whether people enter 2 vs 5 lines of code, especially if those lines are encapsulated by a function we provide (maybe something that's available on the `window` provided by Datasette as an inline script tag).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1530822437,https://api.github.com/repos/simonw/datasette/issues/2052,1530822437,IC_kwDOBm6k_c5bPn8l,193185,cldellow,2023-05-02T03:35:30Z,2023-05-02T16:02:38Z,CONTRIBUTOR,"Also, just checking - is this how I'd write bulletproof plugin registration code that is robust against the order in which the script tags load (eg if both my code and the Datasette code are loaded via a `` tag)?
```js
if (window.__DATASETTE__)
go(window.__DATASETTE__);
else
document.addEventListener(""datasette_init"", (evt) => go(evt.detail));
function go(manager) {
manager.registerPlugin(...)
}
```
I don't know if it'd make sense, but you could also consider the asynchronous queuing pattern that Google Analytics uses (see [this Stack Overflow post](https://stackoverflow.com/questions/6963779/whats-the-name-of-google-analytics-async-design-pattern-and-where-is-it-used) for more details):
```js
__DATASETTE__ = __DATASETTE__ || [];
__DATASETTE__.push(go);
function go(manager) {
manager.registerPlugin(...);
}
```","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1530817667,https://api.github.com/repos/simonw/datasette/issues/2052,1530817667,IC_kwDOBm6k_c5bPmyD,193185,cldellow,2023-05-02T03:24:53Z,2023-05-02T03:24:53Z,CONTRIBUTOR,"Thanks for putting this together! I've been slammed with work/personal stuff so haven't been able to actually prototype anything with this. :(
tl;dr: I think this would be useful immediately as is. It might also be nice if the plugins could return `Promise`s.
The long version: I read the design notes and example plugin. I think I'd be able to use this in [datasette-ui-extras](https://github.com/cldellow/datasette-ui-extras) for my lazy-facets feature.
The lazy-facets feature tries to provide a snappier user experience. It does this by altering how suggested facets work.
First, at page render time:
(A) it lies to Datasette and claims that no columns support facets, this avoids the lengthy delays/timeouts that can happen if the dataset is large.
(B) there's a python plugin that implements the [extra_body_script](https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette) hook, to write out the list of column names for future use by JavaScript
Second, at page load time: there is some JavaScript that:
(C) makes AJAX requests to suggest facets for each column - it makes 1 request per column, using the data from (B)
(D) wires up the column menus to add Facet-by-this options for each facet
With the currently proposed plugin scheme, I think (D) could be moved into the plugin. I'd do the ajax requests, then register the plugin.
If the plugin scheme also supported promises, I think (B) and (C) could also be moved into the plugin.
Does that make sense? Sorry for the wall of text!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2064#issuecomment-1529737426,https://api.github.com/repos/simonw/datasette/issues/2064,1529737426,IC_kwDOBm6k_c5bLfDS,49699333,dependabot[bot],2023-05-01T13:58:50Z,2023-05-01T13:58:50Z,CONTRIBUTOR,Superseded by #2068.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1683229834,Bump sphinx from 6.1.3 to 6.2.1,
https://github.com/simonw/datasette/pull/2063#issuecomment-1521837780,https://api.github.com/repos/simonw/datasette/issues/2063,1521837780,IC_kwDOBm6k_c5atWbU,49699333,dependabot[bot],2023-04-25T13:57:52Z,2023-04-25T13:57:52Z,CONTRIBUTOR,Superseded by #2064.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1681339696,Bump sphinx from 6.1.3 to 6.2.0,
https://github.com/simonw/datasette/pull/2052#issuecomment-1515694393,https://api.github.com/repos/simonw/datasette/issues/2052,1515694393,IC_kwDOBm6k_c5aV6k5,9020979,hydrosquall,2023-04-20T04:25:55Z,2023-04-20T04:25:55Z,CONTRIBUTOR,"Thanks for the thoughtful review and generous examples @asg017 ! I'll make the changes you suggested soon. Bonus thoughts inlined below.
> comments
These were very much appreciated, it's important to a plugin system that details like this feel right! I'll address them in batch later in the week.
> I know TypeScript can be a little controversial
FWIW I am in favor of doing Typescript - I just wanted to keep the initial set of files in this PR as simple as possible to review. Really appreciate you scaffolding this initial set of types + I think it would be a welcome addition to maintain a set of types.d.ts files.
I'm entertaining the idea of writing the actual source code in Typescript as long as the compiled output is readable b/c it can be tricky to keep the types and plain JS files in sync. Curious if you have encountered projects that are good at preventing drift.
> Maybe they should have more ""action-y"" names
This is a great observation. I'm inclined towards something like `make*` or `build*` since to me `add*` make me think the thing the method is attached to is being mutated, but I agree that any of these may be clearer than the current `get*` setup. I'll go through and update these.
> Maybe we can make it easier to do pure-js datasette plugins?
I really like this idea! It'll be easier to get contributors if they don't have to touch the python side at _all_.
> And then do the PERMITTED_VIEWS filtering in JS rather than Python.
One cost of doing this is that pages that won't use the JS would still have to load the unused code (given that I'm not sending up anything complex like lazy loading). But hopefully the manager core size is close to negligible, and it won't be a big deal. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1510423215,https://api.github.com/repos/simonw/datasette/issues/2052,1510423215,IC_kwDOBm6k_c5aBzqv,9020979,hydrosquall,2023-04-16T16:12:59Z,2023-04-16T16:12:59Z,CONTRIBUTOR,"## Research notes
- I stuck to the ""minimal dependencies"" ethos of datasette (no React, Typescript, JS linting, etc).
- Main threads on JS plugin development
- Main: sketch of pluggy-inspired system: https://github.com/simonw/datasette/issues/983
- Main: provide locations in Datasette HTML that are designed for multiple plugins to safely cooperate with each other (starting with the panel, but eventually could extend to ""search boxes""): https://github.com/simonw/datasette/issues/1191
- Main: HTML hooks for JS plugin authors: https://github.com/simonw/datasette/issues/987
- Prior threads on JS plugins in Datasette for future design directions
- Idea: pass useful strings to JS plugins: https://github.com/simonw/datasette/issues/1565
- Idea: help with plugin dependency loading: https://github.com/simonw/datasette/issues/1542 . (IMO - the plugin providing the dependency can emit an event once it's done. Other plugins can listen for it, or ask the manager to inform them when the dependency is available).
- Idea: help plugins to manage state in shareable URLs (plugins shouldn't have to interact with the URL directly, should have some basic insulation from clobbering each others' keys): https://github.com/simonw/datasette/issues/1144
- Articles on plugins reviewed
- https://css-tricks.com/designing-a-javascript-plugin-system/
- Plugin/Extension systems reviewed (mostly JS).
- Yarn: https://yarnpkg.com/advanced/plugin-tutorial
- Tappable https://github.com/webpack/tapable (used by Auto, webpack)
- Pluggy: https://pluggy.readthedocs.io/en/stable/
- VSCode: https://code.visualstudio.com/api/get-started/your-first-extension
- Chrome: https://developer.chrome.com/docs/extensions/reference/
- Figma/Figjam Widget: https://www.figma.com/widget-docs/
- Datadog Apps: [Programming Model](https://github.com/DataDog/apps/blob/master/docs/en/programming-model.md)
- Storybook: https://storybook.js.org/docs/react/addons/addons-api","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1510423051,https://api.github.com/repos/simonw/datasette/issues/2052,1510423051,IC_kwDOBm6k_c5aBzoL,9020979,hydrosquall,2023-04-16T16:12:14Z,2023-04-20T05:14:39Z,CONTRIBUTOR,"# Javascript Plugin Docs (alpha)
## Motivation
The Datasette JS Plugin API allows developers to add interactive features to the UI, without having to modify the Python source code.
## Setup
No external/NPM dependencies are needed.
Plugin behavior is coordinated by the Datasette `manager`. Every page has 1 `manager`.
There are 2 ways to add your plugin to the `manager`.
1. Read `window.__DATASETTE__` if the manager was already loaded.
```js
const manager = window.__DATASETTE__;
```
2. Wait for the `datasette_init` event to fire if your code was loaded before the manager is ready.
```js
document.addEventListener(""datasette_init"", function (evt) {
const { detail: manager } = evt;
// register plugin here
});
```
3. Add plugin to the manager by calling `manager.registerPlugin` in a JS file. Each plugin will supply 1 or more hooks with
- unique name (`YOUR_PLUGIN_NAME`)
- a numeric version (starting at `0.1`),
- configuration value, the details vary by hook. (In this example, `getColumnActions` takes a function)
```js
manager.registerPlugin(""YOUR_PLUGIN_NAME"", {
version: 0.1,
makeColumnActions: (columnMeta) => {
return [
{
label: ""Copy name to clipboard"",
// evt = native click event
onClick: (evt) => copyToClipboard(columnMeta.column),
}
];
},
});
```
There are 2 plugin hooks available to `manager.registerPlugin`:
- `makeColumnActions` - Add items to the cog menu for headers on datasette table pages
- `makeAboveTablePanelConfigs` - Add items to ""tabbed"" panel above the `` on pages that use the Datasette table template.
While there are additional properties on the `manager`, but it's not advised to depend on them directly as the shape is subject to change.
4. To make your JS file available as a Datasette plugin from the Python side, you can add a python file resembling [this](https://github.com/simonw/datasette/pull/2052/files#diff-c5ecf3d22075a60d04a4e95da2e15c612cf1bc84e38d777b67ba60dbd156e293) to your plugins directory. Note that you could host your JS file anywhere, it doesn't have to be served from the Datasette statics folder.
I welcome ideas for more hooks, or feedback on the current design!
## Examples
See the [example plugins file](https://github.com/simonw/datasette/blob/2d92b9328022d86505261bcdac419b6ed9cb2236/datasette/static/table-example-plugins.js) for additional examples.
## Hooks API Guide
### `makeAboveTablePanelConfigs`
Provide a function with a list of panel objects. Each panel object should contain
1. A unique string `id`
2. A string `label` for the tab
3. A `render` function. The first argument is reference to an HTML [Element](https://developer.mozilla.org/en-US/docs/Web/API/Element).
Example:
```js
manager.registerPlugin(""panel-plugin-graphs"", {
version: 0.1,
makeAboveTablePanelConfigs: () => {
return [
{
id: 'first-panel',
label: ""My new panel"",
render: node => {
const description = document.createElement('p');
description.innerText = 'Hello world';
node.appendChild(description);
}
}
];
},
});
```
### `makeColumnActions`
Provide a function that returns a list of action objects. Each action object has
1. A string `label` for the menu dropdown label
2. An onClick `render` function.
Example:
```js
manager.registerPlugin(""column-name-plugin"", {
version: 0.1,
getColumnActions: (columnMeta) => {
// Info about selected column.
const { columnName, columnNotNull, columnType, isPk } = columnMeta;
return [
{
label: ""Copy name to clipboard"",
onClick: (evt) => copyToClipboard(column),
}
];
},
});
```
The getColumnActions callback has access to an object with metadata about the clicked column. These fields include:
- columnName: string (name of the column)
- columnNotNull: boolean
- columnType: sqlite datatype enum (text, number, etc)
- isPk: Whether this is the primary key: boolean
You can use this column metadata to customize the action config objects (for example, handling different summaries for text vs number columns).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1509461324,https://api.github.com/repos/simonw/datasette/issues/2052,1509461324,IC_kwDOBm6k_c5Z-I1M,9020979,hydrosquall,2023-04-15T01:57:06Z,2023-04-15T01:57:06Z,CONTRIBUTOR,Notes from 1:1 - it _is_ possible to pass in URL params into a ObservableHQ notebook: https://observablehq.com/@bherbertlc/pass-values-as-url-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2014#issuecomment-1487999503,https://api.github.com/repos/simonw/datasette/issues/2014,1487999503,IC_kwDOBm6k_c5YsRIP,49699333,dependabot[bot],2023-03-29T06:09:11Z,2023-03-29T06:09:11Z,CONTRIBUTOR,Superseded by #2047.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1566081801,Bump black from 22.12.0 to 23.1.0,
https://github.com/simonw/datasette/pull/2043#issuecomment-1486944644,https://api.github.com/repos/simonw/datasette/issues/2043,1486944644,IC_kwDOBm6k_c5YoPmE,49699333,dependabot[bot],2023-03-28T13:58:20Z,2023-03-28T13:58:20Z,CONTRIBUTOR,Superseded by #2046.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1639446870,Bump furo from 2022.12.7 to 2023.3.23,
https://github.com/simonw/datasette/issues/2033#issuecomment-1457172180,https://api.github.com/repos/simonw/datasette/issues/2033,1457172180,IC_kwDOBm6k_c5W2q7U,25778,eyeseast,2023-03-06T22:54:52Z,2023-03-06T22:54:52Z,CONTRIBUTOR,This would be a nice feature to have with `datasette publish` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1612296210,`datasette install -r requirements.txt`,
https://github.com/simonw/datasette/issues/1258#issuecomment-1437671409,https://api.github.com/repos/simonw/datasette/issues/1258,1437671409,IC_kwDOBm6k_c5VsR_x,2670795,brandonrobertz,2023-02-20T23:39:58Z,2023-02-20T23:39:58Z,CONTRIBUTOR,"This is pretty annoying for FTS because sqlite throws an error instead of just doing something like returning all or no results. This makes users who are unfamiliar with SQL and Datasette think the canned query page is broken and is a frequent source of confusion.
To anyone dealing with this: My solution is to modify the canned query so that it returns no results which cues people to fill in the blank parameters.
So instead of `emails_fts match escape_fts(:search))`
My canned queries now look like this:
`emails_fts match escape_fts(iif(:search=="""", ""*"", :search))`
There are no asterisks in my data so the result is always blank.
Ultimately it would be nice to be able to handle this in the metadata. Either making some named parameters required or setting some default values.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,Allow canned query params to specify default values,
https://github.com/simonw/datasette/pull/564#issuecomment-1420941334,https://api.github.com/repos/simonw/datasette/issues/564,1420941334,IC_kwDOBm6k_c5UsdgW,82988,psychemedia,2023-02-07T15:14:10Z,2023-02-07T15:14:10Z,CONTRIBUTOR,"Is this feature covered by any more recent updates to `datasette`, or via any plugins that you're aware of?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473288428,First proof-of-concept of Datasette Library,
https://github.com/simonw/datasette/issues/2023#issuecomment-1425974877,https://api.github.com/repos/simonw/datasette/issues/2023,1425974877,IC_kwDOBm6k_c5U_qZd,193185,cldellow,2023-02-10T15:32:41Z,2023-02-10T15:32:41Z,CONTRIBUTOR,"I think this feature was removed in Datasette 0.61 and moved to a plugin. People who want hashed URLs can use the [datasette-hashed-urls](https://docs.datasette.io/en/stable/performance.html#performance-hashed-urls) plugin to achieve the same affect.
It looks like you're trying to disable hashed urls, so I think you can just remove that config setting and things will work.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1579695809,Error: Invalid setting 'hash_urls' in settings.json in 0.64.1,
https://github.com/simonw/datasette/pull/2003#issuecomment-1404065571,https://api.github.com/repos/simonw/datasette/issues/2003,1404065571,IC_kwDOBm6k_c5TsFcj,536941,fgregg,2023-01-25T18:44:42Z,2023-01-25T18:44:42Z,CONTRIBUTOR,see this related discussion to a change in API in sqlite-utils https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1555701851,Show referring tables and rows when the referring foreign key is compound,
https://github.com/simonw/datasette/issues/2001#issuecomment-1403084856,https://api.github.com/repos/simonw/datasette/issues/2001,1403084856,IC_kwDOBm6k_c5ToWA4,193185,cldellow,2023-01-25T04:31:02Z,2023-01-25T04:31:02Z,CONTRIBUTOR,"Aha, it's user error on my part.
Adding
```
sqlite3_db_config.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_int, ctypes.c_int]
```
makes it work reliably both on the CLI and from datasette, and now I can reproduce the errors you mentioned in the issue description.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1553615704,Datasette is not compatible with SQLite's strict quoting compilation option,
https://github.com/simonw/datasette/issues/2001#issuecomment-1403078134,https://api.github.com/repos/simonw/datasette/issues/2001,1403078134,IC_kwDOBm6k_c5ToUX2,193185,cldellow,2023-01-25T04:20:43Z,2023-01-25T04:22:28Z,CONTRIBUTOR,"I'm on Ubuntu, unfortunately. :( Would it still be relevant?
I think I've narrowed things down a bit more.
Even `sqlite3_free(sqlite3_malloc(128))` segfaults -- this suggests to me that it's something about the sqlite3 library that was loaded, vs, say, getting the wrong db handle when I go spelunking in the Connection object.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1553615704,Datasette is not compatible with SQLite's strict quoting compilation option,
https://github.com/simonw/datasette/issues/2001#issuecomment-1403053144,https://api.github.com/repos/simonw/datasette/issues/2001,1403053144,IC_kwDOBm6k_c5ToORY,193185,cldellow,2023-01-25T03:34:53Z,2023-01-25T03:34:53Z,CONTRIBUTOR,"Your comment introduced me to this issue in sqlite and to the `ctypes` module - thanks!
> I also hope that the datasette developers will enable this mode in a test environment [...]
> perhaps we could figure out how to invoke it using `ctypes`
I'm not a Datasette developer, but I _am_ curious to learn more about getting unholy access to the sqlite C APIs inside of Datasette. (Such access could also help #1293, and if done without grovelling inside of pysqlite's Connection object for the db handle, could even be relatively safe.)
I experimented a bit. I came up with https://gist.github.com/cldellow/85bba507c314b127f85563869cd94820
If you run `python3 enable-strict-quoting-sqlite3.py`, it seems to set those flags correctly -- `SELECT ""foo""` fails where it would normally succeed.
But if you put it in a `plugins/` dir and run `datasette --plugins-dir plugins/`, it segfaults when it tries to call `sqlite3_db_config` on the connections created by Datasette.
I am... confused. I'm _pretty_ sure I'm using the same python and the same libsqlite3 in both scenarios, so I would expect it to work.
@gwk do you know anything that might help me debug the segfault? I gather that my approach of going grovelling inside of a `PyObject` is particularly dangerous, but I was thinking (a) it's necessary in order to test Datasette's use of the sqlite3 library and (b) even if it's not portable, it'd be good enough for running the tests on a single machine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1553615704,Datasette is not compatible with SQLite's strict quoting compilation option,
https://github.com/simonw/datasette/issues/1099#issuecomment-1402898291,https://api.github.com/repos/simonw/datasette/issues/1099,1402898291,IC_kwDOBm6k_c5Tnodz,536941,fgregg,2023-01-25T00:55:06Z,2023-01-25T00:55:06Z,CONTRIBUTOR,"I went ahead and spiked something together, in #2003 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/pull/2003#issuecomment-1402898033,https://api.github.com/repos/simonw/datasette/issues/2003,1402898033,IC_kwDOBm6k_c5TnoZx,536941,fgregg,2023-01-25T00:54:41Z,2023-01-25T00:54:41Z,CONTRIBUTOR,"@simonw, let me know what you think about this approach!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1555701851,Show referring tables and rows when the referring foreign key is compound,
https://github.com/simonw/datasette/issues/1099#issuecomment-1402900354,https://api.github.com/repos/simonw/datasette/issues/1099,1402900354,IC_kwDOBm6k_c5Tno-C,536941,fgregg,2023-01-25T00:58:26Z,2023-01-25T00:58:26Z,CONTRIBUTOR,"> My original idea for compound foreign keys was to turn both of those columns into links, but that doesn't fit here because `database_name` is already part of a different foreign key.
it's pretty hard to know what the right thing to do is if a field is part of multiple foreign keys.
but, if that's not the case, what about making each of the columns a link. seems like an improvement over the status quo.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1099#issuecomment-1402563930,https://api.github.com/repos/simonw/datasette/issues/1099,1402563930,IC_kwDOBm6k_c5TmW1a,536941,fgregg,2023-01-24T20:11:11Z,2023-01-24T20:11:11Z,CONTRIBUTOR,"hi @simonw, this bug bit me today.
the UX for linking from a table to the foreign key seems tough!
the design in the other direction seems a lot easier, for a given primary key detail page, add links back to the tables that refer to the row.
would you be open to a PR that solved the second problem but not the first?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/2000#issuecomment-1399847946,https://api.github.com/repos/simonw/datasette/issues/2000,1399847946,IC_kwDOBm6k_c5Tb_wK,193185,cldellow,2023-01-23T06:08:00Z,2023-01-23T06:08:00Z,CONTRIBUTOR,"Actually, I discovered [your post](https://til.simonwillison.net/datasette/register-new-plugin-hooks) showing how a plugin can add a Datasette hook. That's wild! I've released `datasette-rewrite-sql` that adds this ability, albeit via monkey patching.
I had hoped to be able to expose `request` to the hook (or, even better `actor`) when the SQL was being run as a result of a user's HTTP request.
But some spelunking in the code makes me suspect that would actually require co-operation from Datasette itself. I'd be happy to be wrong and pointed in the right direction, though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1552368054,rewrite_sql hook,
https://github.com/simonw/datasette/pull/1159#issuecomment-1399589414,https://api.github.com/repos/simonw/datasette/issues/1159,1399589414,IC_kwDOBm6k_c5TbAom,193185,cldellow,2023-01-22T19:48:41Z,2023-01-22T19:48:41Z,CONTRIBUTOR,"Hey @lovasoa, I hope you don't mind - I pulled this PR into [datasette-ui-extras](https://github.com/cldellow/datasette-ui-extras), a plugin I'm making that collects UI tweaks to Datasette.
You can apply it to your own Datasette instance by running `datasette install datasette-ui-extras`","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",774332247,Improve the display of facets information,
https://github.com/simonw/datasette/issues/1696#issuecomment-1407767434,https://api.github.com/repos/simonw/datasette/issues/1696,1407767434,IC_kwDOBm6k_c5T6NOK,193185,cldellow,2023-01-29T20:56:20Z,2023-01-29T20:56:20Z,CONTRIBUTOR,"I did some horrible things in https://github.com/cldellow/datasette-ui-extras/issues/2 to enable this in my plugin -- example here: https://dux-demo.fly.dev/cooking/posts?_facet=owner_user_id&owner_user_id=67
The implementation relies on two things:
- a `filters_from_request` hook that adds a good human description (unfortunately, without the benefit of the CSS styling you mention)
- doing something evil to hijack the `exact` and `not` operators in the `Filters` class. We can't leave them as is, or we'll get 2 human descriptions -- the built-in Datasette one and the one from my plugin. We can't remove them, or the filters UI will stop supporting the `=` and `!=` operators
This got me thinking: it'd be neat if the list of operators that the filters UI supported wasn't a closed set.
A motivating example: adding a geospatial `NEAR` operator. Ideally it'd take two arguments - a target point and a radius, so you could express a filter like `find me all rows whose lat/lng are within 10km of 43.4516° N, 80.4925° W`. (Optionally, the UI could be enhanced if the geonames database was loaded and queried, so a user could say `find me all rows whose lat/lng are within 10km of Kitchener, ON`, and the city gets translated to a lat/lng for them)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1186696202,Show foreign key label when filtering,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407716963,https://api.github.com/repos/simonw/datasette/issues/2008,1407716963,IC_kwDOBm6k_c5T6A5j,193185,cldellow,2023-01-29T17:04:03Z,2023-01-29T17:04:03Z,CONTRIBUTOR,"Performance tests - I think most places don't have them as a formal gate enforced by CI. TypeScript and scalac seem to have tests that run to capture timings. The timings are included by a bot as a comment or build check, and also stored in a database so you can graph changes over time to spot regressions. Probably overkill for Datasette!
Window functions - oh, good point. Looks like Ubuntu shipped JSON1 support as far back as sqlite 3.11. I'll let this PR linger until there's a way to run against different SQLite versions. For now, I'm shipping this with `datasette-ui-extras`, since I think it's OK for a plugin to enforce a higher minimum requirement.
Tests - there actually did end up being test changes to capture the undercount bug of the current implementation, so the current implementation would fail against the new tests.
Perhaps a non-window function version could be written that uses `random()` instead of `row_number() over ()` in order to get a unique key. It's technically not unique, but in practice, I imagine it'll work well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,array facet: don't materialize unnecessary columns,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407558284,https://api.github.com/repos/simonw/datasette/issues/2008,1407558284,IC_kwDOBm6k_c5T5aKM,193185,cldellow,2023-01-29T04:23:58Z,2023-01-29T04:24:27Z,CONTRIBUTOR,"Ack, this PR is broken. I see now that the `inner.*` is necessary for ensuring the correct count in the face of rows having duplicate values in views.
That fixes the overcounting, but I think can undercount when the rows have the same data, eg a view like:
```sql
SELECT '[""bar""]' tags UNION ALL SELECT '[""bar""]'
```
will produce a count of `{""bar"": 1 }`, when it should be `{""bar"": 2}`. In fact, this could apply in tables without primary keys, too.
If `inner` came from a base table that had a primary key or a rowid, we could use those column(s) to solve that case.
I guess a general solution would be to compute a window function so we have a distinct ID for each row. Will fiddle to see if I can get that working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,array facet: don't materialize unnecessary columns,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407561308,https://api.github.com/repos/simonw/datasette/issues/2008,1407561308,IC_kwDOBm6k_c5T5a5c,193185,cldellow,2023-01-29T04:50:50Z,2023-01-29T04:50:50Z,CONTRIBUTOR,I pushed a revised version which ends up being faster -- the example which currently takes 4 seconds now runs in 500ms.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,array facet: don't materialize unnecessary columns,
https://github.com/simonw/datasette/issues/1973#issuecomment-1407523547,https://api.github.com/repos/simonw/datasette/issues/1973,1407523547,IC_kwDOBm6k_c5T5Rrb,193185,cldellow,2023-01-29T00:40:31Z,2023-01-29T00:40:31Z,CONTRIBUTOR,"A +1 for switching to `CustomRow`: I think you currently only get a `CustomRow` if the result set had a column that was an fkey ([this code](https://github.com/simonw/datasette/blob/3c352b7132ef09b829abb69a0da0ad00be5edef9/datasette/views/table.py#L667-L682))
Otherwise you get vanilla `sqlite3.Row`s, which will fail if you try to access `.columns` or lookup the cell by name, which surprised me recently","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,render_cell plugin hook's row object is not a sqlite.Row,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407470429,https://api.github.com/repos/simonw/datasette/issues/2008,1407470429,IC_kwDOBm6k_c5T5Etd,193185,cldellow,2023-01-28T19:34:29Z,2023-01-28T19:34:29Z,CONTRIBUTOR,"I don't know how/if you do automated tests for performance, so I haven't changed any of the tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,array facet: don't materialize unnecessary columns,
https://github.com/simonw/datasette/pull/1982#issuecomment-1376620851,https://api.github.com/repos/simonw/datasette/issues/1982,1376620851,IC_kwDOBm6k_c5SDZEz,49699333,dependabot[bot],2023-01-10T02:03:18Z,2023-01-10T02:03:18Z,CONTRIBUTOR,"Looks like sphinx is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1525560504,Bump sphinx from 5.3.0 to 6.1.2,
https://github.com/simonw/datasette/issues/1983#issuecomment-1375810027,https://api.github.com/repos/simonw/datasette/issues/1983,1375810027,IC_kwDOBm6k_c5SATHr,25778,eyeseast,2023-01-09T15:35:58Z,2023-01-09T15:35:58Z,CONTRIBUTOR,"Yes please, and thank you. I realized I was maybe getting myself in trouble using that, but I think it's a good way to standardize JSON handling.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1525815985,Make CustomJSONEncoder a documented public API,
https://github.com/simonw/datasette/pull/1974#issuecomment-1372188571,https://api.github.com/repos/simonw/datasette/issues/1974,1372188571,IC_kwDOBm6k_c5Rye-b,49699333,dependabot[bot],2023-01-05T13:02:40Z,2023-01-05T13:02:40Z,CONTRIBUTOR,Superseded by #1976.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516376583,Bump sphinx from 5.3.0 to 6.0.0,
https://github.com/simonw/datasette/issues/1973#issuecomment-1369044959,https://api.github.com/repos/simonw/datasette/issues/1973,1369044959,IC_kwDOBm6k_c5Rmfff,193185,cldellow,2023-01-02T15:41:40Z,2023-01-02T15:41:40Z,CONTRIBUTOR,"Thanks for the response!
Yes, it does seem like a pretty nice developer experience--both the automagical labelling of fkeys, and the ability to index the row by column name in addition to column index.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,render_cell plugin hook's row object is not a sqlite.Row,
https://github.com/simonw/datasette/pull/1977#issuecomment-1375596856,https://api.github.com/repos/simonw/datasette/issues/1977,1375596856,IC_kwDOBm6k_c5R_fE4,49699333,dependabot[bot],2023-01-09T13:06:14Z,2023-01-09T13:06:14Z,CONTRIBUTOR,Superseded by #1982.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1522552817,Bump sphinx from 5.3.0 to 6.1.1,
https://github.com/simonw/datasette/issues/1978#issuecomment-1375708725,https://api.github.com/repos/simonw/datasette/issues/1978,1375708725,IC_kwDOBm6k_c5R_6Y1,25778,eyeseast,2023-01-09T14:30:00Z,2023-01-09T14:30:00Z,CONTRIBUTOR,Totally missed that issue. I can close this as a duplicate.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1522778923,Document datasette.urls.row and row_blob,
https://github.com/simonw/datasette/issues/1614#issuecomment-1364345119,https://api.github.com/repos/simonw/datasette/issues/1614,1364345119,IC_kwDOBm6k_c5RUkEf,536941,fgregg,2022-12-23T21:27:10Z,2022-12-23T21:27:10Z,CONTRIBUTOR,is this issue closed by #1893?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1115435536,Try again with SQLite codemirror support,
https://github.com/simonw/datasette/issues/1796#issuecomment-1364345071,https://api.github.com/repos/simonw/datasette/issues/1796,1364345071,IC_kwDOBm6k_c5RUkDv,536941,fgregg,2022-12-23T21:27:02Z,2022-12-23T21:27:02Z,CONTRIBUTOR,@simonw is this issue closed by #1893?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1355148385,Research an upgrade to CodeMirror 6,
https://github.com/simonw/datasette/pull/1976#issuecomment-1373592231,https://api.github.com/repos/simonw/datasette/issues/1976,1373592231,IC_kwDOBm6k_c5R31qn,49699333,dependabot[bot],2023-01-06T13:02:15Z,2023-01-06T13:02:15Z,CONTRIBUTOR,Superseded by #1977.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1520712722,Bump sphinx from 5.3.0 to 6.1.0,
https://github.com/simonw/datasette/issues/1605#issuecomment-1332310772,https://api.github.com/repos/simonw/datasette/issues/1605,1332310772,IC_kwDOBm6k_c5PaXL0,25778,eyeseast,2022-11-30T15:06:37Z,2022-11-30T15:06:37Z,CONTRIBUTOR,I'll add issues for both and do a documentation PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports,
https://github.com/simonw/datasette/issues/1605#issuecomment-1331187551,https://api.github.com/repos/simonw/datasette/issues/1605,1331187551,IC_kwDOBm6k_c5PWE9f,25778,eyeseast,2022-11-29T19:29:42Z,2022-11-29T19:29:42Z,CONTRIBUTOR,"Interesting. I started a version using metadata like I outlined up top, but I realized that there's no documented way for a plugin to access either metadata or canned queries. Or at least, I couldn't find a way.
There is this method: https://github.com/simonw/datasette/blob/main/datasette/app.py#L472 but I don't want to rely on it if it's not documented. Same with this: https://github.com/simonw/datasette/blob/main/datasette/app.py#L544
If those are safe, I'll build on them. I'm also happy to document them, if that greases the wheels.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports,
https://github.com/simonw/datasette/pull/1931#issuecomment-1339916064,https://api.github.com/repos/simonw/datasette/issues/1931,1339916064,IC_kwDOBm6k_c5P3X8g,3556,davidbgk,2022-12-06T19:42:45Z,2022-12-06T19:42:45Z,CONTRIBUTOR,"The `""return"": true` option is really nice!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert,
https://github.com/simonw/datasette/issues/1929#issuecomment-1339906969,https://api.github.com/repos/simonw/datasette/issues/1929,1339906969,IC_kwDOBm6k_c5P3VuZ,3556,davidbgk,2022-12-06T19:34:20Z,2022-12-06T19:34:20Z,CONTRIBUTOR,I confirm that it works 👍 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191,Incorrect link from the API explorer to the JSON API documentation,
https://github.com/simonw/datasette/issues/1886#issuecomment-1321241426,https://api.github.com/repos/simonw/datasette/issues/1886,1321241426,IC_kwDOBm6k_c5OwItS,536941,fgregg,2022-11-20T20:58:54Z,2022-11-20T20:58:54Z,CONTRIBUTOR,i wrote up a blog post of how i'm using it! https://bunkum.us/2022/11/20/mgdo-stack.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here",
https://github.com/simonw/datasette/issues/1884#issuecomment-1321460293,https://api.github.com/repos/simonw/datasette/issues/1884,1321460293,IC_kwDOBm6k_c5Ow-JF,15178711,asg017,2022-11-21T04:40:55Z,2022-11-21T04:40:55Z,CONTRIBUTOR,"Counting any virtual tables can be pretty tricky. On one hand, counting a [CSV virtual table](https://www.sqlite.org/csv.html) would return the number of rows in the CSV, which is helpful (but can be I/O intensive). Counting a [FTS5 virtual table](https://www.sqlite.org/fts5.html) would return the number of entries in the FTS index, which is kindof helpful, but can be misleading in some cases.
On the other hand, arbitrarily running `COUNT(*)` on some virtual tables can be incredibly expensive. SQLite offers new shortcuts/pushdowns on `COUNT(*)` queries for virtual tables, and instead calls the underlying vtab implementation and iterates through all rows in the table without discretion. For example, a virtual table that's backed by a Postgres table would call `select * from pg_table`, which would use up a lot of network and CPU calls. Or a virtual table backed by a [google sheet](https://github.com/0x6b/libgsqlite) would make network/API requests to get all the rows from the sheet just to make a count.
The [`pragma_table_list`](https://www.sqlite.org/pragma.html#pragma_table_list) pragma tells you when a table is a regular table or virtual (in the `type` column), but was only added in version 3.37.0 (2021-11-27).
Personally, I wouldnt try to `COUNT(*)` virtual tables - it depends on how the virtual table is implemented, it requires that the connection has the proper extensions loaded, and it may accientally cause perf issues for new-age extensions. A few extensions that I'm writing have virtual tables that wouldn't benefit much from `COUNT(*)`, and the fact that SQLite iterates through all rows in a table to count just makes things worse. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect,
https://github.com/simonw/datasette/issues/1886#issuecomment-1321003094,https://api.github.com/repos/simonw/datasette/issues/1886,1321003094,IC_kwDOBm6k_c5OvOhW,9020979,hydrosquall,2022-11-20T00:52:05Z,2022-11-20T00:52:05Z,CONTRIBUTOR,"Happy birthday to datasette and thank you Simon for your continued effort on this project!
I use datasette (python) as a fast layer on top of search for github projects using https://github.com/dogsheep/github-to-sqlite , and use the JSON API it provides to serve sample data to make Vega-Lite graphing workshop examples that don't require authentication/API keys. It's awesome to have a full SQL API support working without needing to develop any custom API middleware for both filtering and grouping.
I've also enjoyed using it as a teaching tool for working with public dataset in [civic data workshops](https://2022.open-data.nyc/event/low-code-visual-data-exploration-with-nyc-public-data/) and as a platform for making visualization [plugins](https://github.com/hydrosquall/datasette-nteract-data-explorer) . I
I'm especially excited about datasette-lite, as it will let people participate in future editions of this workshop without having to install anything to make use of their own tables :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here",
https://github.com/simonw/datasette/issues/1897#issuecomment-1319533445,https://api.github.com/repos/simonw/datasette/issues/1897,1319533445,IC_kwDOBm6k_c5OpnuF,95570,bgrins,2022-11-18T04:38:03Z,2022-11-18T04:38:03Z,CONTRIBUTOR,Are you tracking the change to send the JSON over to the frontend separately or was that part of this? Something like this is probably pretty close https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25#diff-0c93232bfd5477eeac96382e52769108b41433d960d5277ffcccf2f464e60abdR9,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete,
https://github.com/simonw/datasette/issues/1899#issuecomment-1318897922,https://api.github.com/repos/simonw/datasette/issues/1899,1318897922,IC_kwDOBm6k_c5OnMkC,95570,bgrins,2022-11-17T16:32:42Z,2022-11-17T16:32:42Z,CONTRIBUTOR,Another idea would be to just not set a min-height and allow the 1 line input to be 1 line heigh,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452495049,Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused ,
https://github.com/simonw/datasette/issues/1890#issuecomment-1317889323,https://api.github.com/repos/simonw/datasette/issues/1890,1317889323,IC_kwDOBm6k_c5OjWUr,536941,fgregg,2022-11-17T00:47:36Z,2022-11-17T00:47:36Z,CONTRIBUTOR,amazing! thanks @simonw ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets,
https://github.com/simonw/datasette/issues/1899#issuecomment-1317873458,https://api.github.com/repos/simonw/datasette/issues/1899,1317873458,IC_kwDOBm6k_c5OjScy,95570,bgrins,2022-11-17T00:31:07Z,2022-11-17T00:31:07Z,CONTRIBUTOR,"This is one way to fix it
```patch
r.html
diff --git a/datasette/static/cm-editor-6.0.1.js b/datasette/static/cm-editor-6.0.1.js
index c1fd2ab..68cf398 100644
--- a/datasette/static/cm-editor-6.0.1.js
+++ b/datasette/static/cm-editor-6.0.1.js
@@ -22,7 +22,14 @@ export function editorFromTextArea(textarea, conf = {}) {
// https://github.com/codemirror/lang-sql#user-content-sqlconfig.tables
let view = new EditorView({
doc: textarea.value,
+
extensions: [
+ EditorView.theme({
+ "".cm-content"": {
+ // Height on cm-content ensures the editor is focusable by clicking beyond the height of the text
+ minHeight: ""70px"",
+ },
+ }),
keymap.of([
{
key: ""Shift-Enter"",
diff --git a/datasette/templates/_codemirror.html b/datasette/templates/_codemirror.html
index dea4710..c4629ae 100644
--- a/datasette/templates/_codemirror.html
+++ b/datasette/templates/_codemirror.html
@@ -4,7 +4,6 @@
.cm-editor {
resize: both;
overflow: hidden;
- min-height: 70px;
width: 80%;
border: 1px solid #ddd;
}
```
I don't love it but it seems to work for the default case. You can still retrigger the bug by resizing the editor to be > 70px high.
The other approach would be to listen for a click on that empty region and move focus to the editor, or something","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452495049,Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused ,
https://github.com/simonw/datasette/pull/1893#issuecomment-1317834838,https://api.github.com/repos/simonw/datasette/issues/1893,1317834838,IC_kwDOBm6k_c5OjJBW,95570,bgrins,2022-11-16T23:50:58Z,2022-11-16T23:50:58Z,CONTRIBUTOR,"Should we empty out the fixture schema to avoid fixture autocomplete showing up on live databases in the interim, or are you planning to tackle #1897 shortly?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1317805482,https://api.github.com/repos/simonw/datasette/issues/1893,1317805482,IC_kwDOBm6k_c5OjB2q,95570,bgrins,2022-11-16T23:18:17Z,2022-11-16T23:18:17Z,CONTRIBUTOR,Alright with https://github.com/simonw/datasette/pull/1893/commits/f254be4b38936e95e7a7f25866e7c6b0520db96f we should be getting autocomplete on fixture data. Give that a test and see what you think,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1317715580,https://api.github.com/repos/simonw/datasette/issues/1893,1317715580,IC_kwDOBm6k_c5Oir58,95570,bgrins,2022-11-16T21:49:51Z,2022-11-16T21:49:51Z,CONTRIBUTOR,"I think the table completion still has some quirks to work out. Something like
```
schema: {
""[123_starts_with_digits]"": [""content""],
}
```
Seems to work alright, although it will append it after any other numbers you've started typing - so you end up with `select * from 12[123_starts_with_digits]` if you typed ""12"" to get the completion to appear. This might just be an issue with numeric names, I haven't tested it in a lot of detail.
You can do
```
searchable: [
{
label: ""name with . and spaces"",
apply: ""[name with . and spaces]"",
},
""pk"",
""text1"",
""text2"",
],
```
Which is pretty neat and will show the non-escaped string but complete to the escaped one. You can't easily do that with the table names themselves (you can pass a `tables` array like so https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/src/sql.ts#L121 but it will overwrite the columns from the schema ).
It's buggy enough (bad output for these unusual table names) that I'd suggest that work gets moved into a follow up to the upgrade to 6. That would give space to sort out how to deliver that to the view directly, figure out where name escaping should happen, and have overall testing to uncover bugs and fix papercuts before enabling it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1317681193,https://api.github.com/repos/simonw/datasette/issues/1893,1317681193,IC_kwDOBm6k_c5Oijgp,95570,bgrins,2022-11-16T21:19:13Z,2022-11-16T21:19:13Z,CONTRIBUTOR,"Alright, added Cmd+Enter to submit (Ctrl+Enter on Windows as well bc of using Meta-Enter on codemirror). We can make that MacOS only by changing the combo to Cmd+Enter specifically but I think it's probably fine to have both.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1317789308,https://api.github.com/repos/simonw/datasette/issues/1893,1317789308,IC_kwDOBm6k_c5Oi958,95570,bgrins,2022-11-16T22:59:57Z,2022-11-16T22:59:57Z,CONTRIBUTOR,"I can push up a commit that uses the static fixtures schema for testing, but given that the query used to generate it is authed we would still need some work to make that work on live data, right? Ideally it could come down to db and query views directly to avoid waiting on an extra xhr and managing that state change.On Nov 16, 2022, at 2:16 PM, Simon Willison ***@***.***> wrote:
Honestly I'm not too bothered if table names with weird characters don't work correctly here - I care about those in the Datasette fixtures.db database because Datasette aims to support ANY valid SQLite database, so I need stuff in the test suite that includes weird edge cases like this. But I would hope very few people actually create tables with spaces in their names, so it's not a huge concern to me if autocompletion doesn't work properly for those.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1317329157,https://api.github.com/repos/simonw/datasette/issues/1893,1317329157,IC_kwDOBm6k_c5OhNkF,95570,bgrins,2022-11-16T16:46:52Z,2022-11-16T16:46:52Z,CONTRIBUTOR,"> ![]()
>
> UI issue I see on the autocomplete popup with overlapping icon & text. Screenshot's from Firefox, it seems even a little more pronounced on Safari
I checked and if I empty out app.css the bug goes away, so there's some kind of inheritance issue there. It's hard to debug bc the autocomplete popup goes away on blur (i.e. when trying to inspect it in devtools), but at least it's narrowed down a bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1317326406,https://api.github.com/repos/simonw/datasette/issues/1893,1317326406,IC_kwDOBm6k_c5OhM5G,95570,bgrins,2022-11-16T16:45:09Z,2022-11-16T16:45:09Z,CONTRIBUTOR,"For escaped table names it looks like we could pass a Completion object (https://codemirror.net/docs/ref/#autocomplete) instead of a string which would allow the non escaped name to be a label and then the escaped name to actually complete in the editor, which might help with some of the funkiness I was seeing w/ completion","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1317314064,https://api.github.com/repos/simonw/datasette/issues/1893,1317314064,IC_kwDOBm6k_c5OhJ4Q,95570,bgrins,2022-11-16T16:36:46Z,2022-11-16T16:36:46Z,CONTRIBUTOR,"With
```patch
diff --git a/datasette/templates/_codemirror_foot.html b/datasette/templates/_codemirror_foot.html
index ed709b3..74fe18e 100644
--- a/datasette/templates/_codemirror_foot.html
+++ b/datasette/templates/_codemirror_foot.html
@@ -7,7 +7,11 @@
sqlFormat.hidden = false;
}
if (sqlInput) {
- var editor = (window.editor = cm.editorFromTextArea(sqlInput));
+ var editor = (window.editor = cm.editorFromTextArea(sqlInput, {
+ schema: {
+ compound_three_primary_keys: [""pk1"", ""pk2"", ""pk3"", ""content""],
+ },
+ }));
```
we get table autocompletion and column completion if you name the table in the query (see screencast). I do see bugs with escaped table names like `""'123_starts_with_digits'"": [""col1"", ""col2""]` or `""[123_starts_with_digits]"": [""col1"", ""col2""]` where it doesn't seem to pick up the column names though. I think it needs some further testing and debugging.
https://user-images.githubusercontent.com/95570/202238521-e613b4e2-ba92-4418-9068-fc022edaee93.mp4
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1317281292,https://api.github.com/repos/simonw/datasette/issues/1893,1317281292,IC_kwDOBm6k_c5OhB4M,95570,bgrins,2022-11-16T16:19:16Z,2022-11-16T16:19:16Z,CONTRIBUTOR,"Ha, nice idea! Updating the dialect with that list.
I'm thinking of also adding `count` to the list since that's a common thing people would want to autocomplete. I notice BQ console highlights `count` in the same manner as other keywords like `select` as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1317522323,https://api.github.com/repos/simonw/datasette/issues/1893,1317522323,IC_kwDOBm6k_c5Oh8uT,95570,bgrins,2022-11-16T18:59:49Z,2022-11-16T18:59:49Z,CONTRIBUTOR,Or I guess you could return only the escaped table name and then we could derive the unescaped from the client side (removing the outer `[]` when present),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1317520304,https://api.github.com/repos/simonw/datasette/issues/1893,1317520304,IC_kwDOBm6k_c5Oh8Ow,95570,bgrins,2022-11-16T18:58:43Z,2022-11-16T18:58:43Z,CONTRIBUTOR,Nice. And is it possible to include another field which is an escaped table name (only when necessary) - i.e. `[123_starts_with_digits]`. Or is that easy enough to derive on the client? I'm thinking we'd map those to Completion objects so that CM would show the non escaped text but complete to escaped.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1316387382,https://api.github.com/repos/simonw/datasette/issues/1893,1316387382,IC_kwDOBm6k_c5Odno2,95570,bgrins,2022-11-16T05:33:55Z,2022-11-16T05:33:55Z,CONTRIBUTOR,I added a commit to make our own dialect at https://github.com/simonw/datasette/pull/1893/commits/e273fc8ed5341bdf0b622e722d761bd2acc30a90. Pulled in the full list of keywords from https://www.sqlite.org/lang_keywords.html but haven't gone through and pruned it to only include common select keywords. @simonw you'll have better knowledge than me on that - do you want to take a first shot at narrowing that down to the set that people will be using in the editor?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1316339035,https://api.github.com/repos/simonw/datasette/issues/1893,1316339035,IC_kwDOBm6k_c5Odb1b,95570,bgrins,2022-11-16T04:47:11Z,2022-11-16T04:47:11Z,CONTRIBUTOR,"> Have you ever seen CodeMirror correctly auto-completing columns? I'm not entirely sure I believe that the feature works anywhere else.
I was thinking of the BigQuery console, like
![]()
But they must be doing something pretty custom & appears to be using Monaco anyway. I suspect some kind of lower level autocomplete integration could make this work, but if the table completion is a good-enough starting point I think it's not too hard. The main issue is that we don't pass the relevant table data down to QueryView.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1316320521,https://api.github.com/repos/simonw/datasette/issues/1893,1316320521,IC_kwDOBm6k_c5OdXUJ,95570,bgrins,2022-11-16T04:29:23Z,2022-11-16T04:29:23Z,CONTRIBUTOR,"![]()
UI issue I see on the autocomplete popup with overlapping icon & text. Screenshot's from Firefox, it seems even a little more pronounced on Safari","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1316318961,https://api.github.com/repos/simonw/datasette/issues/1893,1316318961,IC_kwDOBm6k_c5OdW7x,95570,bgrins,2022-11-16T04:27:51Z,2022-11-16T04:27:51Z,CONTRIBUTOR,"> The resize handle doesn't appear on Mobile Safari on iPhone - I don't think that particularly matters though.
>
> The textarea does get a weird border around it when focused on iPhone though.
The default focus styles appear to be
```
.c1.cm-editor.cm-focused {
outline: 1px dotted #212121;
}
```
Which I also see on desktop. Would be nice to changed to whatever the default UA textarea styles are to blend in better but I wouldn't recommend removing it entirely - just to keep the visual indication that the element is focused. Maybe followup material to have a theming pass","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1316256386,https://api.github.com/repos/simonw/datasette/issues/1893,1316256386,IC_kwDOBm6k_c5OdHqC,95570,bgrins,2022-11-16T03:18:06Z,2022-11-16T03:18:06Z,CONTRIBUTOR,"> If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values.
This version ""sort of"" works when on the main database page where the template passes the relevant data https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25 by doing this and passing that into the `schema` object:
```
let TABLES_DATA = [];
{% if tables is defined %}
TABLES_DATA = {{ tables | tojson(indent=2) }};
{% endif %}
// Turn into an object, shaped like https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L27.
const TABLES_SCHEMA = Object.fromEntries(
new Map(
TABLES_DATA.map((table) => {
return [table.name, table.columns];
})
).entries()
);
```
But there are a number of papercuts with it - it's not escaping table names with spaces (likely be fixable from the data being passed into the view) but mainly it doesn't seem to autocomplete columns. I think it might only want to do it when you first type the table name from my read of https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L37. It's possible I'm just passing something wrong, but it may end up being something that needs feature work upstream.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1316243602,https://api.github.com/repos/simonw/datasette/issues/1893,1316243602,IC_kwDOBm6k_c5OdEiS,95570,bgrins,2022-11-16T03:11:46Z,2022-11-16T03:11:46Z,CONTRIBUTOR,Was just reviewing the SQL options and there's an [upperCaseKeywords](https://github.com/codemirror/lang-sql#user-content-sqlconfig.uppercasekeywords) if we'd rather have SELECT vs select. Datasette seems to prefer lowercase so probably best to keep it as-is,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1316041828,https://api.github.com/repos/simonw/datasette/issues/1893,1316041828,IC_kwDOBm6k_c5OcTRk,95570,bgrins,2022-11-15T23:51:35Z,2022-11-15T23:51:35Z,CONTRIBUTOR,"I experimented with autocompleting the actual schema in https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25, but it would need some work (current problems with it listed in the commit message there)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1315869946,https://api.github.com/repos/simonw/datasette/issues/1893,1315869946,IC_kwDOBm6k_c5ObpT6,95570,bgrins,2022-11-15T21:12:38Z,2022-11-15T21:12:38Z,CONTRIBUTOR,https://github.com/Sphinxxxx/cm-resize isn't compatible with 6. There's a suggestion to try using CSS resize in https://discuss.codemirror.net/t/resizing-codemirror-6/3265/2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1315869040,https://api.github.com/repos/simonw/datasette/issues/1893,1315869040,IC_kwDOBm6k_c5ObpFw,95570,bgrins,2022-11-15T21:11:42Z,2022-11-15T21:11:42Z,CONTRIBUTOR,"extraKeys is done - Shift+Enter is added in the helper function, and it appears that the Tab behavior now defaults to what the `Tab: false` setting was doing (allowing it to escape to the form)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/pull/1893#issuecomment-1315853097,https://api.github.com/repos/simonw/datasette/issues/1893,1315853097,IC_kwDOBm6k_c5OblMp,95570,bgrins,2022-11-15T20:55:40Z,2022-11-15T20:55:40Z,CONTRIBUTOR,Should also minify the bundled output,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete",
https://github.com/simonw/datasette/issues/1886#issuecomment-1314241058,https://api.github.com/repos/simonw/datasette/issues/1886,1314241058,IC_kwDOBm6k_c5OVboi,25778,eyeseast,2022-11-14T19:06:35Z,2022-11-14T19:06:35Z,CONTRIBUTOR,"This probably counts as a case study: https://github.com/eyeseast/spatial-data-cooking-show. Even has video.
Seriously, though, this workflow has become integral to my work with reporters and editors across USA TODAY Network. Very often, I get sent a folder of data in mixed formats, with a vague ask of how we should communicate some part of it to users. Datasette and its constellation of tools makes it easy to get a quick look at that data, run exploratory queries, map it and ask questions to figure out what's important to show. And then I export a version of the data that's exactly what I need for display.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here",
https://github.com/simonw/datasette/issues/1884#issuecomment-1314066229,https://api.github.com/repos/simonw/datasette/issues/1884,1314066229,IC_kwDOBm6k_c5OUw81,25778,eyeseast,2022-11-14T16:48:35Z,2022-11-14T16:48:35Z,CONTRIBUTOR,"I'm realizing I don't know if a virtual table will ever return a count. Maybe it depends on the implementation. For these three, just checking now, it'll always return zero.
That said, I'm not sure there's any downside to having them return zero and caching that. (They're hidden, too.) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect,
https://github.com/simonw/datasette/issues/1884#issuecomment-1313962183,https://api.github.com/repos/simonw/datasette/issues/1884,1313962183,IC_kwDOBm6k_c5OUXjH,25778,eyeseast,2022-11-14T15:46:32Z,2022-11-14T15:46:32Z,CONTRIBUTOR,"It does work, though I think it's probably still worth excluding virtual tables that will always be zero. Here's the same inspection as before, now with `--load-extension spatialite`:
```json
{
""alltheplaces"": {
""hash"": ""0843cfe414439ab903c22d1121b7ddbc643418c35c7f0edbcec82ef1452411df"",
""size"": 963375104,
""file"": ""alltheplaces.db"",
""tables"": {
""spatial_ref_sys"": {
""count"": 6215
},
""spatialite_history"": {
""count"": 18
},
""sqlite_sequence"": {
""count"": 2
},
""geometry_columns"": {
""count"": 3
},
""spatial_ref_sys_aux"": {
""count"": 6164
},
""views_geometry_columns"": {
""count"": 0
},
""virts_geometry_columns"": {
""count"": 0
},
""geometry_columns_statistics"": {
""count"": 3
},
""views_geometry_columns_statistics"": {
""count"": 0
},
""virts_geometry_columns_statistics"": {
""count"": 0
},
""geometry_columns_field_infos"": {
""count"": 0
},
""views_geometry_columns_field_infos"": {
""count"": 0
},
""virts_geometry_columns_field_infos"": {
""count"": 0
},
""geometry_columns_time"": {
""count"": 3
},
""geometry_columns_auth"": {
""count"": 3
},
""views_geometry_columns_auth"": {
""count"": 0
},
""virts_geometry_columns_auth"": {
""count"": 0
},
""data_licenses"": {
""count"": 10
},
""sql_statements_log"": {
""count"": 0
},
""states"": {
""count"": 56
},
""counties"": {
""count"": 3234
},
""idx_states_geometry_rowid"": {
""count"": 56
},
""idx_states_geometry_node"": {
""count"": 3
},
""idx_states_geometry_parent"": {
""count"": 2
},
""idx_counties_geometry_rowid"": {
""count"": 3234
},
""idx_counties_geometry_node"": {
""count"": 98
},
""idx_counties_geometry_parent"": {
""count"": 97
},
""idx_places_geometry_rowid"": {
""count"": 1236796
},
""idx_places_geometry_node"": {
""count"": 38163
},
""idx_places_geometry_parent"": {
""count"": 38162
},
""places"": {
""count"": 1332609
},
""SpatialIndex"": {
""count"": 0
},
""ElementaryGeometries"": {
""count"": 0
},
""KNN"": {
""count"": 0
},
""idx_states_geometry"": {
""count"": 56
},
""idx_counties_geometry"": {
""count"": 3234
},
""idx_places_geometry"": {
""count"": 1236796
}
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect,
https://github.com/simonw/datasette/issues/1886#issuecomment-1313252879,https://api.github.com/repos/simonw/datasette/issues/1886,1313252879,IC_kwDOBm6k_c5ORqYP,883348,adipasquale,2022-11-14T08:10:23Z,2022-11-14T08:10:23Z,CONTRIBUTOR,"Hi @simonw and thanks for the great tools you're publishing, your dedication is inspiring!
I work for the French Ministry of Culture on a surveying tool for objects protected for their historical value. It is part of a program building modern public services called [beta.gouv.fr](https://beta.gouv.fr/).
In that context I'm using data published by the Ministry that I have ingested into datasette and published on a free Fly instance : https://collectif-objets-datasette.fly.dev . I have also ingested another data set with infos about french cities on this instance so that I can perform joined queries.
The surveying tool synchronizes its data regularly from this datasette instance, and I also use it to perform queries when asked generic questions about the distribution of objects. (The data is not very accessible as it's undocumented and for internal usage mostly)","{""total_count"": 3, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 3, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here",
https://github.com/simonw/datasette/issues/1884#issuecomment-1309735529,https://api.github.com/repos/simonw/datasette/issues/1884,1309735529,IC_kwDOBm6k_c5OEPpp,25778,eyeseast,2022-11-10T03:57:23Z,2022-11-10T03:57:23Z,CONTRIBUTOR,Here's how to get a list of virtual tables: https://stackoverflow.com/questions/46617118/how-to-fetch-names-of-virtual-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect,
https://github.com/simonw/datasette/issues/1871#issuecomment-1309650806,https://api.github.com/repos/simonw/datasette/issues/1871,1309650806,IC_kwDOBm6k_c5OD692,3556,davidbgk,2022-11-10T01:38:58Z,2022-11-10T01:38:58Z,CONTRIBUTOR,"> Realized the API explorer doesn't need the API key piece at all - it can work with standard cookie-based auth.
>
> This also reflects how most plugins are likely to use this API, where they'll be adding JavaScript that uses `fetch()` to call the write API directly.
I agree (that's what I did with the previous insert plugin), maybe a complete example using `fetch()` in the documentation would be valuable as a “Getting started with the API” or similar?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool,
https://github.com/simonw/datasette/issues/1872#issuecomment-1296076803,https://api.github.com/repos/simonw/datasette/issues/1872,1296076803,IC_kwDOBm6k_c5NQJAD,192568,mroswell,2022-10-30T02:50:34Z,2022-10-30T02:50:34Z,CONTRIBUTOR,"should this issue be under https://github.com/simonw/datasette-publish-vercel/issues ?
Perhaps I just need to update:
datasette-publish-vercel==0.11
in requirements.txt?
I'll try that and see what happens...
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428560020,"SITE-BUSTING ERROR: ""render_template() called before await ds.invoke_startup()""",
https://github.com/simonw/datasette/issues/1872#issuecomment-1296080804,https://api.github.com/repos/simonw/datasette/issues/1872,1296080804,IC_kwDOBm6k_c5NQJ-k,192568,mroswell,2022-10-30T03:06:32Z,2022-10-30T03:06:32Z,CONTRIBUTOR,"I updated datasette-publish-vercel to 0.14.2 in requirements.txt
And the site is back up!
Is there a way that we can get some sort of notice when something like this will have critical impact on website function?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428560020,"SITE-BUSTING ERROR: ""render_template() called before await ds.invoke_startup()""",
https://github.com/simonw/datasette/pull/1870#issuecomment-1295667649,https://api.github.com/repos/simonw/datasette/issues/1870,1295667649,IC_kwDOBm6k_c5NOlHB,536941,fgregg,2022-10-29T00:52:43Z,2022-10-29T00:53:43Z,CONTRIBUTOR,"> Are you saying that I can build a container, but then when I run it and it does `datasette serve -i data.db ...` it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that `serve` command?
Somehow, `datasette serve -i data.db` will lead to the `data.db` being modified, which will trigger a [copy-on-write](https://docs.docker.com/storage/storagedriver/#the-copy-on-write-cow-strategy) of `data.db` into the read-write layer of the container.
I don't understand **how** that happens.
it kind of feels like a bug in sqlite, but i can't quite follow the sqlite code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro",
https://github.com/simonw/datasette/pull/1870#issuecomment-1294285471,https://api.github.com/repos/simonw/datasette/issues/1870,1294285471,IC_kwDOBm6k_c5NJTqf,536941,fgregg,2022-10-28T01:06:03Z,2022-10-28T01:06:03Z,CONTRIBUTOR,"as far as i can tell, [this is where the ""immutable"" argument is used](https://github.com/sqlite/sqlite/blob/c97bb14fab566f6fa8d967c8fd1e90f3702d5b73/src/pager.c#L4926-L4931) in sqlite:
```c
pPager->noLock = sqlite3_uri_boolean(pPager->zFilename, ""nolock"", 0);
if( (iDc & SQLITE_IOCAP_IMMUTABLE)!=0
|| sqlite3_uri_boolean(pPager->zFilename, ""immutable"", 0) ){
vfsFlags |= SQLITE_OPEN_READONLY;
goto act_like_temp_file;
}
```
so it does set the read only flag, but then has a goto.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro",
https://github.com/simonw/datasette/pull/1870#issuecomment-1294237783,https://api.github.com/repos/simonw/datasette/issues/1870,1294237783,IC_kwDOBm6k_c5NJIBX,536941,fgregg,2022-10-27T23:42:18Z,2022-10-27T23:42:18Z,CONTRIBUTOR,Relevant sqlite forum thread: https://www.sqlite.org/forum/forumpost/02f7bda329f41e30451472421cf9ce7f715b768ce3db02797db1768e47950d48,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro",
https://github.com/simonw/datasette/issues/1851#issuecomment-1292519956,https://api.github.com/repos/simonw/datasette/issues/1851,1292519956,IC_kwDOBm6k_c5NCkoU,15178711,asg017,2022-10-26T19:20:33Z,2022-10-26T19:20:33Z,CONTRIBUTOR,"> This could use a new plugin hook, too. I don't want to complicate your life too much, but for things like GIS, I'd want a way to turn regular JSON into SpatiaLite geometries or combine X/Y coordinates into point geometries and such. Happy to help however I can.
@eyeseast Maybe you could do this with triggers? Like you can insert JSON-friendly data into a ""raw"" table, and create a trigger that transforms that inserted data into the proper table
Here's an example:
```sql
-- meant to be updated from a Datasette insert
create table points_raw(longitude int, latitude int);
-- the target table with proper spatliate geometries
create table points(point geometry);
CREATE TRIGGER insert_points_raw INSERT ON points_raw
BEGIN
insert into points(point) values (makepoint(new.longitude, new.latitude))
END;
```
You could then POST a new row to `points_raw` like this:
```
POST /db/points_raw
Authorization: Bearer xxx
Content-Type: application/json
{
""row"": {
""longitude"": 27.64356,
""latitude"": -47.29384
}
}
```
Then SQLite with run the trigger and insert a new row in `points` with the correct geometry point. Downside is you'd have duplicated data with `points_raw`, but maybe it could be a `TEMP` table (or have a cron that deletes all rows from that table every so often?)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/datasette/issues/1851#issuecomment-1292592210,https://api.github.com/repos/simonw/datasette/issues/1851,1292592210,IC_kwDOBm6k_c5NC2RS,25778,eyeseast,2022-10-26T20:03:46Z,2022-10-26T20:03:46Z,CONTRIBUTOR,"Yeah, every time I see something cool done with triggers, I remember that I need to start using triggers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/datasette/issues/1851#issuecomment-1291228502,https://api.github.com/repos/simonw/datasette/issues/1851,1291228502,IC_kwDOBm6k_c5M9pVW,25778,eyeseast,2022-10-25T23:02:10Z,2022-10-25T23:02:10Z,CONTRIBUTOR,That's reasonable. Canned queries and custom endpoints are certainly going to give more room for specific needs. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/datasette/issues/1851#issuecomment-1290615599,https://api.github.com/repos/simonw/datasette/issues/1851,1290615599,IC_kwDOBm6k_c5M7Tsv,25778,eyeseast,2022-10-25T14:05:12Z,2022-10-25T14:05:12Z,CONTRIBUTOR,"This could use a new plugin hook, too. I don't want to complicate your life too much, but for things like GIS, I'd want a way to turn regular JSON into SpatiaLite geometries or combine X/Y coordinates into point geometries and such. Happy to help however I can.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/datasette/issues/1301#issuecomment-1271035998,https://api.github.com/repos/simonw/datasette/issues/1301,1271035998,IC_kwDOBm6k_c5Lwnhe,536941,fgregg,2022-10-07T02:38:04Z,2022-10-07T02:38:04Z,CONTRIBUTOR,the only mode that `publish cloudrun` supports right now is immutable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",860722711,Publishing to cloudrun with immutable mode?,
https://github.com/simonw/datasette/issues/1836#issuecomment-1271020193,https://api.github.com/repos/simonw/datasette/issues/1836,1271020193,IC_kwDOBm6k_c5Lwjqh,536941,fgregg,2022-10-07T02:15:05Z,2022-10-07T02:21:08Z,CONTRIBUTOR,"when i hack the connect method to open non mutable files with ""mode=ro"" and not ""immutable=1"" https://github.com/simonw/datasette/blob/eff112498ecc499323c26612d707908831446d25/datasette/database.py#L79
then:
```bash
870 B RUN /bin/sh -c datasette inspect nlrb.db --inspect-file inspect-data.json
```
the `datasette inspect` layer is only the size of the json file!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1271008997,https://api.github.com/repos/simonw/datasette/issues/1836,1271008997,IC_kwDOBm6k_c5Lwg7l,536941,fgregg,2022-10-07T02:00:37Z,2022-10-07T02:00:49Z,CONTRIBUTOR,"yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in `immutable` mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!
running a test of that now. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1271003212,https://api.github.com/repos/simonw/datasette/issues/1836,1271003212,IC_kwDOBm6k_c5LwfhM,536941,fgregg,2022-10-07T01:52:04Z,2022-10-07T01:52:04Z,CONTRIBUTOR,"and if we try immutable mode, which is how things are opened by `datasette inspect` we duplicate the files!!!
```python
# test_sql_immutable.py
import sqlite3
import sys
db_name = sys.argv[1]
conn = sqlite3.connect(f'file:/app/{db_name}?immutable=1', uri=True)
cur = conn.cursor()
cur.execute('select count(*) from filing')
print(cur.fetchone())
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1270992795,https://api.github.com/repos/simonw/datasette/issues/1836,1270992795,IC_kwDOBm6k_c5Lwc-b,536941,fgregg,2022-10-07T01:29:15Z,2022-10-07T01:50:14Z,CONTRIBUTOR,"fascinatingly, telling python to open sqlite in read only mode makes this layer have a size of 0
```python
# test_sql_ro.py
import sqlite3
import sys
db_name = sys.argv[1]
conn = sqlite3.connect(f'file:/app/{db_name}?mode=ro', uri=True)
cur = conn.cursor()
cur.execute('select count(*) from filing')
print(cur.fetchone())
```
that's quite weird because setting the file permissions to read only didn't do anything. (on reflection, that chmod isn't doing anything because the dockerfile commands are run as root)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1270988081,https://api.github.com/repos/simonw/datasette/issues/1836,1270988081,IC_kwDOBm6k_c5Lwb0x,536941,fgregg,2022-10-07T01:19:01Z,2022-10-07T01:27:35Z,CONTRIBUTOR,"okay, some progress!! running some sql against a database file causes that file to get duplicated even if it doesn't apparently change the file.
make a little test script like this:
```python
# test_sql.py
import sqlite3
import sys
db_name = sys.argv[1]
conn = sqlite3.connect(f'file:/app/{db_name}', uri=True)
cur = conn.cursor()
cur.execute('select count(*) from filing')
print(cur.fetchone())
```
then
```docker
RUN python test_sql.py nlrb.db
```
produced a layer that's the same size as `nlrb.db`!!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1270936982,https://api.github.com/repos/simonw/datasette/issues/1836,1270936982,IC_kwDOBm6k_c5LwPWW,536941,fgregg,2022-10-07T00:52:41Z,2022-10-07T00:52:41Z,CONTRIBUTOR,"it's not that the inspect command is somehow changing the db files. if i set them to only read-only, the ""inspect"" layer still has the same very large size.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1270923537,https://api.github.com/repos/simonw/datasette/issues/1836,1270923537,IC_kwDOBm6k_c5LwMER,536941,fgregg,2022-10-07T00:46:08Z,2022-10-07T00:46:08Z,CONTRIBUTOR,"i thought it was maybe to do with reading through all the files, but that does not seem to be the case
if i make a little test file like:
```python
# test_read.py
import hashlib
import sys
import pathlib
HASH_BLOCK_SIZE = 1024 * 1024
def inspect_hash(path):
""""""Calculate the hash of a database, efficiently.""""""
m = hashlib.sha256()
with path.open(""rb"") as fp:
while True:
data = fp.read(HASH_BLOCK_SIZE)
if not data:
break
m.update(data)
return m.hexdigest()
inspect_hash(pathlib.Path(sys.argv[1]))
```
then a line in the Dockerfile like
```docker
RUN python test_read.py nlrb.db && echo ""[]"" > /etc/inspect.json
```
just produes a layer of `3B`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1480#issuecomment-1271101072,https://api.github.com/repos/simonw/datasette/issues/1480,1271101072,IC_kwDOBm6k_c5Lw3aQ,536941,fgregg,2022-10-07T04:39:10Z,2022-10-07T04:39:10Z,CONTRIBUTOR,switching from `immutable=1` to `mode=ro` completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database,
https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651,https://api.github.com/repos/simonw/datasette/issues/1836,1271100651,IC_kwDOBm6k_c5Lw3Tr,536941,fgregg,2022-10-07T04:38:14Z,2022-10-07T04:38:14Z,CONTRIBUTOR,"> yes, and i also think that this is causing the apparent memory problems in #1480. when the container starts up, it will make some operation on the database in `immutable` mode which apparently makes some small change to the db file. if that's so, then the db files will be copied to the read/write layer which counts against cloudrun's memory allocation!
>
> running a test of that now.
this completely addressed #1480 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1836#issuecomment-1271103097,https://api.github.com/repos/simonw/datasette/issues/1836,1271103097,IC_kwDOBm6k_c5Lw355,536941,fgregg,2022-10-07T04:43:41Z,2022-10-07T04:43:41Z,CONTRIBUTOR,"@simonw, should i open up a new issue for investigating the differences between ""immutable=1"" and ""mode=ro"" and possibly switching to ""mode=ro"". Or would you like to keep that conversation in this issue?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/1480#issuecomment-1269847461,https://api.github.com/repos/simonw/datasette/issues/1480,1269847461,IC_kwDOBm6k_c5LsFWl,536941,fgregg,2022-10-06T11:21:49Z,2022-10-06T11:21:49Z,CONTRIBUTOR,"thanks @simonw, i'll spend a little more time trying to figure out why this isn't working on cloudrun, and then will flip over to fly if i can't.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database,
https://github.com/simonw/datasette/issues/1480#issuecomment-1268629159,https://api.github.com/repos/simonw/datasette/issues/1480,1268629159,IC_kwDOBm6k_c5Lnb6n,536941,fgregg,2022-10-05T16:00:55Z,2022-10-05T16:00:55Z,CONTRIBUTOR,"as a next step, i'll fetch the docker image from the google registry, and see what memory and disk usage looks like when i run it locally.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database,
https://github.com/simonw/datasette/issues/1480#issuecomment-1268613335,https://api.github.com/repos/simonw/datasette/issues/1480,1268613335,IC_kwDOBm6k_c5LnYDX,536941,fgregg,2022-10-05T15:45:49Z,2022-10-05T15:45:49Z,CONTRIBUTOR,"running into this as i continue to grow my labor data warehouse.
Here a CloudRun PM says the container size should **not** count against memory: https://stackoverflow.com/a/56570717","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database,
https://github.com/simonw/datasette/issues/370#issuecomment-1261930179,https://api.github.com/repos/simonw/datasette/issues/370,1261930179,IC_kwDOBm6k_c5LN4bD,72577720,MichaelTiemannOSC,2022-09-29T08:17:46Z,2022-09-29T08:17:46Z,CONTRIBUTOR,"Just watched this video which demonstrates the integration of *any* webapp into JupyterLab: https://youtu.be/FH1dKKmvFtc
Maybe this is the answer?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",377155320,Integration with JupyterLab,
https://github.com/simonw/datasette/issues/1062#issuecomment-1260829829,https://api.github.com/repos/simonw/datasette/issues/1062,1260829829,IC_kwDOBm6k_c5LJryF,536941,fgregg,2022-09-28T12:27:19Z,2022-09-28T12:27:19Z,CONTRIBUTOR,"for teaching `register_output_renderer` to stream it seems like the two options are to
1. a [nested query technique ](https://github.com/simonw/datasette/issues/526#issuecomment-505162238)to paginate through
2. a fetching model that looks like something
```python
with sqlite_timelimit(conn, time_limit_ms):
c.execute(query)
for chunk in c.fetchmany(chunk_size):
yield from chunk
```
currently `db.execute` is not a generator, so this would probably need a new method?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/1062#issuecomment-1260909128,https://api.github.com/repos/simonw/datasette/issues/1062,1260909128,IC_kwDOBm6k_c5LJ_JI,536941,fgregg,2022-09-28T13:22:53Z,2022-09-28T14:09:54Z,CONTRIBUTOR,"if you went this route:
```python
with sqlite_timelimit(conn, time_limit_ms):
c.execute(query)
for chunk in c.fetchmany(chunk_size):
yield from chunk
```
then `time_limit_ms` would probably have to be greatly extended, because the time spent in the loop will depend on the downstream processing.
i wonder if this was why you were thinking this feature would need a dedicated connection?
---
reading more, there's no real limit i can find on the number of active cursors (or more precisely active prepared statements objects, because sqlite doesn't really have cursors).
maybe something like this would be okay?
```python
with sqlite_timelimit(conn, time_limit_ms):
c.execute(query)
# step through at least one to evaluate the statement, not sure if this is necessary
yield c.execute.fetchone()
for chunk in c.fetchmany(chunk_size):
yield from chunk
```
this seems quite weird that there's not more of limit of the number of active prepared statements, but i haven't been able to find one.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/526#issuecomment-1259718517,https://api.github.com/repos/simonw/datasette/issues/526,1259718517,IC_kwDOBm6k_c5LFcd1,536941,fgregg,2022-09-27T16:02:51Z,2022-09-27T16:04:46Z,CONTRIBUTOR,"i think that `max_returned_rows` **is** a defense mechanism, just not for connection exhaustion. `max_returned_rows` is a defense mechanism against **memory bombs**.
if you are potentially yielding out hundreds of thousands or even millions of rows, you need to be quite careful about data flow to not run out of memory on the server, or on the client.
you have a lot of places in your code that are protective of that right now, but `max_returned_rows` acts as the final backstop.
so, given that, it makes sense to have removing `max_returned_rows` altogether be a non-goal, but instead allow for for specific codepaths (like streaming csv's) be able to bypass.
that could dramatically lower the surface area for a memory-bomb attack.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/526#issuecomment-1258910228,https://api.github.com/repos/simonw/datasette/issues/526,1258910228,IC_kwDOBm6k_c5LCXIU,536941,fgregg,2022-09-27T03:11:07Z,2022-09-27T03:11:07Z,CONTRIBUTOR,"i think this feature would be safe, as its really only the time limit that can, and imo, should protect against long running queries, as it is pretty easy to make very expensive queries that don't return many rows.
moving away from `max_returned_rows` will requires some thinking about:
1. memory usage and data flows to handle potentially very large result sets
2. how to avoid rendering tens or hundreds of thousands of [html rows](#1655).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/526#issuecomment-1258878311,https://api.github.com/repos/simonw/datasette/issues/526,1258878311,IC_kwDOBm6k_c5LCPVn,536941,fgregg,2022-09-27T02:19:48Z,2022-09-27T02:19:48Z,CONTRIBUTOR,"this sql query doesn't trip up `maximum_returned_rows` but does timeout
```sql
with recursive counter(x) as (
select 0
union
select x + 1 from counter
)
select * from counter LIMIT 10 OFFSET 100000000
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/526#issuecomment-1258871525,https://api.github.com/repos/simonw/datasette/issues/526,1258871525,IC_kwDOBm6k_c5LCNrl,536941,fgregg,2022-09-27T02:09:32Z,2022-09-27T02:14:53Z,CONTRIBUTOR,"thanks @simonw, i learned something i didn't know about sqlite's execution model!
> Imagine if Datasette CSVs did allow unlimited retrievals. Someone could hit the CSV endpoint for that recursive query and tie up Datasette's SQL connection effectively forever.
why wouldn't the `sqlite_timelimit` guard prevent that?
---
on my local version which has the code to [turn off truncations for query csv](#1820), `sqlite_timelimit` does protect me.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/526#issuecomment-1258849766,https://api.github.com/repos/simonw/datasette/issues/526,1258849766,IC_kwDOBm6k_c5LCIXm,536941,fgregg,2022-09-27T01:27:03Z,2022-09-27T01:27:03Z,CONTRIBUTOR,"i agree with that concern! but if i'm understanding the code correctly, `maximum_returned_rows` does not protect against long-running queries in any way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/pull/1820#issuecomment-1258803261,https://api.github.com/repos/simonw/datasette/issues/1820,1258803261,IC_kwDOBm6k_c5LB9A9,536941,fgregg,2022-09-27T00:03:09Z,2022-09-27T00:03:09Z,CONTRIBUTOR,"the pattern in this PR `max_returned_rows` control the maximum rows rendered through html and json, and the csv render bypasses that.
i think it would be better to have each of these different query renderers have more direct control for how many rows to fetch, instead of relying on the internals of the `execute` method.
generally, users will not want to paginate through tens of thousands of results, but often will want to download a full query as json or as csv. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386456717,[SPIKE] Don't truncate query CSVs,
https://github.com/simonw/datasette/issues/526#issuecomment-1258337011,https://api.github.com/repos/simonw/datasette/issues/526,1258337011,IC_kwDOBm6k_c5LALLz,536941,fgregg,2022-09-26T16:49:48Z,2022-09-26T16:49:48Z,CONTRIBUTOR,"i think the smallest change that gets close to what i want is to change the behavior so that `max_returned_rows` is not applied in the `execute` method when we are are asking for a csv of query.
there are some infelicities for that approach, but i'll make a PR to make it easier to discuss.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/1836#issuecomment-1272357976,https://api.github.com/repos/simonw/datasette/issues/1836,1272357976,IC_kwDOBm6k_c5L1qRY,536941,fgregg,2022-10-08T16:56:51Z,2022-10-08T16:56:51Z,CONTRIBUTOR,"when you are running from docker, you **always** will want to run as `mode=ro` because the same thing that is causing duplication in the inspect layer will cause duplication in the final container read/write layer when `datasette serve` runs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400374908,docker image is duplicating db files somehow,
https://github.com/simonw/datasette/issues/526#issuecomment-1254064260,https://api.github.com/repos/simonw/datasette/issues/526,1254064260,IC_kwDOBm6k_c5Kv4CE,536941,fgregg,2022-09-21T18:17:04Z,2022-09-21T18:18:01Z,CONTRIBUTOR,"hi @simonw, this is becoming more of a bother for my [labor data warehouse](https://labordata.bunkum.us/). Is there any research or a spike i could do that would help you investigate this issue?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/1813#issuecomment-1250901367,https://api.github.com/repos/simonw/datasette/issues/1813,1250901367,IC_kwDOBm6k_c5Kjz13,883348,adipasquale,2022-09-19T11:34:45Z,2022-09-19T11:34:45Z,CONTRIBUTOR,"oh and by writing this I just realized the difference: the URL on fly.io is with a custom SQL command whereas the local one is without.
It seems that there is no pagination when using custom SQL commands which makes sense
Sorry for this useless issue, maybe this can be useful for someone else / me in the future.
Thanks again for this wonderful project !","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1377811868,missing next and next_url in JSON responses from an instance deployed on Fly ,
https://github.com/simonw/datasette/issues/1655#issuecomment-1258166572,https://api.github.com/repos/simonw/datasette/issues/1655,1258166572,IC_kwDOBm6k_c5K_hks,536941,fgregg,2022-09-26T14:57:04Z,2022-09-26T14:57:04Z,CONTRIBUTOR,"I think that paginating, even in javascript, could be very helpful. Maybe render json or csv into the page and let javascript loading that into the dom?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data,
https://github.com/simonw/datasette/issues/526#issuecomment-1258167564,https://api.github.com/repos/simonw/datasette/issues/526,1258167564,IC_kwDOBm6k_c5K_h0M,536941,fgregg,2022-09-26T14:57:44Z,2022-09-26T15:08:36Z,CONTRIBUTOR,"reading the database execute method i have a few questions.
https://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/database.py#L229-L242
---
unless i'm missing something (which is very likely!!), the `max_returned_rows` argument doesn't actually offer any protections against running very expensive queries.
It's not like adding a `LIMIT max_rows` argument. it make sense that it isn't because, the query could already have an `LIMIT` argument. Doing something like `select * from (query) limit {max_returned_rows}` **might** be protective but wouldn't always.
Instead the code executes the full original query, and if still has time it fetches out the first `max_rows + 1` rows.
this *does* offer some protection of memory exhaustion, as you won't hydrate a huge result set into python (however, there are [data flow patterns](https://github.com/simonw/datasette/issues/1727#issuecomment-1258129113) that could avoid that too)
given the current architecture, i don't see how creating a new connection would be use?
---
If we just removed the `max_return_rows` limitation, then i think most things would be fine **except** for the QueryViews. Right now rendering, just [5000 rows takes a lot of client-side memory](https://github.com/simonw/datasette/issues/1655) so some form of pagination would be required.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/1727#issuecomment-1258129113,https://api.github.com/repos/simonw/datasette/issues/1727,1258129113,IC_kwDOBm6k_c5K_YbZ,536941,fgregg,2022-09-26T14:30:11Z,2022-09-26T14:48:31Z,CONTRIBUTOR,"from your analysis, it seems like the GIL is blocking on loading of the data from sqlite to python, (particularly in the `fetchmany` call)
this is probably a simplistic idea, but what if you had the python code in the `execute` method iterate over the cursor and yield out rows or small chunks of rows.
something like:
```python
with sqlite_timelimit(conn, time_limit_ms):
try:
cursor = conn.cursor()
cursor.execute(sql, params if params is not None else {})
except:
...
max_returned_rows = self.ds.max_returned_rows
if max_returned_rows == page_size:
max_returned_rows += 1
if max_returned_rows and truncate:
for i, row in enumerate(cursor):
yield row
if i == max_returned_rows - 1:
break
else:
for row in cursor:
yield row
truncated = False
```
this kind of thing works well with a postgres server side cursor, but i'm not sure if it will hold for sqlite.
you would still spend about the same amount of time in python and would be contending for the gil, but it would be could be non blocking.
depending on the data flow, this could also some benefit for memory. (data stays in more compact sqlite-land until you need it)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1217759117,Research: demonstrate if parallel SQL queries are worthwhile,
https://github.com/simonw/datasette/issues/1810#issuecomment-1248204219,https://api.github.com/repos/simonw/datasette/issues/1810,1248204219,IC_kwDOBm6k_c5KZhW7,82988,psychemedia,2022-09-15T14:44:47Z,2022-09-15T14:46:26Z,CONTRIBUTOR,"A couple+ of possible use case examples:
- someone has a collection of articles indexed with FTS; they want to publish a simple search tool over the results;
- someone has an image collection and they want to be able to search over description text to return images;
- someone has a set of locations with descriptions, and wants to run a query over places and descriptions and get results as a listing or on a map;
- someone has a set of audio or video files with titles, descriptions and/or transcripts, and wants to be able to search over them and return playable versions of returned items.
In many cases, I suspect the raw content will be in one table, but the search table will be a second (eg FTS) table. Generally, the search may be over one or more joined tables, and the results constructed from one or more tables (which may or may not be distinct from the search tables).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374626873,Featured table(s) on the homepage,
https://github.com/simonw/datasette/issues/1817#issuecomment-1256781274,https://api.github.com/repos/simonw/datasette/issues/1817,1256781274,IC_kwDOBm6k_c5K6PXa,50527,jefftriplett,2022-09-23T22:59:46Z,2022-09-23T22:59:46Z,CONTRIBUTOR,"While you are adding features, would you be future-proofing your APIs if you switched over some arguments over to keyword-only arguments or would that be too disruptive?
Thinking out loud:
```
async def render_template(
self, templates, *, context=None, plugin_context=None, request=None, view_name=None
):
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1384273985,Expose `sql` and `params` arguments to various plugin hooks,
https://github.com/simonw/datasette/pull/1799#issuecomment-1237381569,https://api.github.com/repos/simonw/datasette/issues/1799,1237381569,IC_kwDOBm6k_c5JwPHB,49699333,dependabot[bot],2022-09-05T18:36:42Z,2022-09-05T18:36:42Z,CONTRIBUTOR,"Looks like aiofiles is no longer updatable, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1362242558,"Update aiofiles requirement from <0.9,>=0.4 to >=0.4,<22.2",
https://github.com/simonw/datasette/pull/1685#issuecomment-1237381620,https://api.github.com/repos/simonw/datasette/issues/1685,1237381620,IC_kwDOBm6k_c5JwPH0,49699333,dependabot[bot],2022-09-05T18:36:47Z,2022-09-05T18:36:47Z,CONTRIBUTOR,"Looks like jinja2 is no longer updatable, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180778860,"Update jinja2 requirement from <3.1.0,>=2.10.3 to >=2.10.3,<3.2.0",
https://github.com/simonw/datasette/pull/1789#issuecomment-1221576460,https://api.github.com/repos/simonw/datasette/issues/1789,1221576460,IC_kwDOBm6k_c5Iz8cM,15178711,asg017,2022-08-21T16:16:42Z,2022-08-21T16:16:42Z,CONTRIBUTOR,"Rebased, Read the docs failure should now now fixed
Re docs - ya that's a pretty ambitious page, I'm still not 100% sure what the best practices are/should be... Would be happy to make that page in a future PR","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1344823170,Add new entrypoint option to `--load-extension`,
https://github.com/simonw/datasette/issues/1779#issuecomment-1214437408,https://api.github.com/repos/simonw/datasette/issues/1779,1214437408,IC_kwDOBm6k_c5IYtgg,536941,fgregg,2022-08-14T19:42:58Z,2022-08-14T19:42:58Z,CONTRIBUTOR,thanks @simonw!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1334628400,google cloudrun updated their limits on maxscale based on memory and cpu count,
https://github.com/simonw/datasette/issues/1779#issuecomment-1210675046,https://api.github.com/repos/simonw/datasette/issues/1779,1210675046,IC_kwDOBm6k_c5IKW9m,536941,fgregg,2022-08-10T13:28:37Z,2022-08-10T13:28:37Z,CONTRIBUTOR,maybe a simpler solution is to set the maxscale to like 2? since datasette is not set up to make use of container scaling anyway?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1334628400,google cloudrun updated their limits on maxscale based on memory and cpu count,
https://github.com/simonw/datasette/pull/1789#issuecomment-1223347322,https://api.github.com/repos/simonw/datasette/issues/1789,1223347322,IC_kwDOBm6k_c5I6sx6,15178711,asg017,2022-08-23T00:03:20Z,2022-08-23T00:03:20Z,CONTRIBUTOR,"@simonw to build the extension on ubuntu, you can run:
```
apt-get update && apt-get install libsqlite3-dev gcc
gcc ext.c -fPIC -shared -o ext.so
```
I'm not the best with Actions, but if you set the cache key to `ext.c`, run those two commands to download dependencies + compile to `ext.so`, then the unit test should pick it up and run it correctly. Let me know if you want me to update the PR with that added","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1344823170,Add new entrypoint option to `--load-extension`,
https://github.com/simonw/datasette/issues/1191#issuecomment-1200732975,https://api.github.com/repos/simonw/datasette/issues/1191,1200732975,IC_kwDOBm6k_c5Hkbsv,2670795,brandonrobertz,2022-08-01T05:39:27Z,2022-08-01T05:39:27Z,CONTRIBUTOR,I've got a URL shortening plugin that I would like to embed on the query page but I'd like avoid capturing the entire `query.html` template. A feature like this would solve it. Where's this at and how can I help?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/pull/1693#issuecomment-1168704157,https://api.github.com/repos/simonw/datasette/issues/1693,1168704157,IC_kwDOBm6k_c5FqQKd,49699333,dependabot[bot],2022-06-28T13:11:36Z,2022-06-28T13:11:36Z,CONTRIBUTOR,Superseded by #1763.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1184850337,Bump black from 22.1.0 to 22.3.0,
https://github.com/simonw/datasette/pull/1753#issuecomment-1163091750,https://api.github.com/repos/simonw/datasette/issues/1753,1163091750,IC_kwDOBm6k_c5FU18m,49699333,dependabot[bot],2022-06-22T13:22:34Z,2022-06-22T13:22:34Z,CONTRIBUTOR,Superseded by #1760.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1261826957,Bump furo from 2022.4.7 to 2022.6.4.1,
https://github.com/simonw/datasette/issues/1713#issuecomment-1173358747,https://api.github.com/repos/simonw/datasette/issues/1713,1173358747,IC_kwDOBm6k_c5F8Aib,2670795,brandonrobertz,2022-07-04T05:16:35Z,2022-07-04T05:16:35Z,CONTRIBUTOR,"This feature is pretty important and would be nice if it would be all within Datasette (no separate CLI/deploy required). My workflow now is to basically just copy the result and paste into a Google Sheet, which works, but then it's not discoverable to other journalists browsing the Datasette instance. I started building a plugin similar to [datasette-saved-queries](https://datasette.io/plugins/datasette-saved-queries) but one that maintains its own DB (required if you're working with all immutable DBs), but got bogged down in details.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203943272,Datasette feature for publishing snapshots of query results,
https://github.com/simonw/datasette/issues/1528#issuecomment-1151887842,https://api.github.com/repos/simonw/datasette/issues/1528,1151887842,IC_kwDOBm6k_c5EqGni,25778,eyeseast,2022-06-10T03:23:08Z,2022-06-10T03:23:08Z,CONTRIBUTOR,I just put together a version of this in a plugin: https://github.com/eyeseast/datasette-query-files. Happy to have any feedback.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1060631257,"Add new `""sql_file""` key to Canned Queries in metadata?",
https://github.com/simonw/datasette/issues/1742#issuecomment-1128064864,https://api.github.com/repos/simonw/datasette/issues/1742,1128064864,IC_kwDOBm6k_c5DPOdg,25778,eyeseast,2022-05-16T19:42:13Z,2022-05-16T19:42:13Z,CONTRIBUTOR,"Just to add a wrinkle here, this loads fine: https://alltheplaces-datasette.fly.dev/alltheplaces/places.geojson?_trace=1
But also, this doesn't add any trace data: https://alltheplaces-datasette.fly.dev/alltheplaces/places.json?_trace=1
What am I missing?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1237586379,?_trace=1 fails with datasette-geojson for some reason,
https://github.com/simonw/datasette/issues/1742#issuecomment-1128049716,https://api.github.com/repos/simonw/datasette/issues/1742,1128049716,IC_kwDOBm6k_c5DPKw0,25778,eyeseast,2022-05-16T19:24:44Z,2022-05-16T19:24:44Z,CONTRIBUTOR,"Where is `_trace` getting injected? And is it something a plugin should be able to handle? (If it is, I guess I should handle it in this case.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1237586379,?_trace=1 fails with datasette-geojson for some reason,
https://github.com/simonw/datasette/issues/741#issuecomment-1125342229,https://api.github.com/repos/simonw/datasette/issues/741,1125342229,IC_kwDOBm6k_c5DE1wV,25778,eyeseast,2022-05-12T19:21:16Z,2022-05-12T19:21:16Z,CONTRIBUTOR,"Came here to check if this had been flagged already. Was helping a colleague get something on Cloud Run and had to dig to find `--extra-options=""--setting sql_time_limit_ms 2500""`.
If I get some time next week, maybe I'll try to tackle it. Would definitely make things easier to be able to do something like this:
```sh
datasette publish cloudrun something.db --setting sql_time_limit_ms 2500
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/1728#issuecomment-1111751734,https://api.github.com/repos/simonw/datasette/issues/1728,1111751734,IC_kwDOBm6k_c5CQ_w2,127565,wragge,2022-04-28T05:09:59Z,2022-04-28T05:09:59Z,CONTRIBUTOR,"Thanks, I'll give it a try!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111752676,https://api.github.com/repos/simonw/datasette/issues/1728,1111752676,IC_kwDOBm6k_c5CQ__k,127565,wragge,2022-04-28T05:11:54Z,2022-04-28T05:11:54Z,CONTRIBUTOR,"And in terms of the bug, yep I agree that option 2 would be the most useful and least frustrating.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111712953,https://api.github.com/repos/simonw/datasette/issues/1728,1111712953,IC_kwDOBm6k_c5CQ2S5,127565,wragge,2022-04-28T03:48:36Z,2022-04-28T03:48:36Z,CONTRIBUTOR,"I don't think that'd work for this project. The db is very big, and my aim was to have an environment where researchers could be making use of the data, but be easily able to add corrections to the HTR/OCR extracted data when they came across problems. It's in its immutable (!) form here: https://sydney-stock-exchange-xqtkxtd5za-ts.a.run.app/stock_exchange/stocks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1728#issuecomment-1111705323,https://api.github.com/repos/simonw/datasette/issues/1728,1111705323,IC_kwDOBm6k_c5CQ0br,127565,wragge,2022-04-28T03:32:06Z,2022-04-28T03:32:06Z,CONTRIBUTOR,"Ah, that would be it! I have a core set of data which doesn't change to which I want authorised users to be able to submit corrections. I was going to deal with the persistence issue by just grabbing the user corrections at regular intervals and saving to GitHub. I might need to rethink. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1218133366,Writable canned queries fail with useless non-error against immutable databases,
https://github.com/simonw/datasette/issues/1713#issuecomment-1103312860,https://api.github.com/repos/simonw/datasette/issues/1713,1103312860,IC_kwDOBm6k_c5Bwzfc,536941,fgregg,2022-04-20T00:52:19Z,2022-04-20T00:52:19Z,CONTRIBUTOR,feels related to #1402 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203943272,Datasette feature for publishing snapshots of query results,
https://github.com/simonw/datasette/issues/1713#issuecomment-1099540225,https://api.github.com/repos/simonw/datasette/issues/1713,1099540225,IC_kwDOBm6k_c5BiacB,25778,eyeseast,2022-04-14T19:09:57Z,2022-04-14T19:09:57Z,CONTRIBUTOR,"I wonder if this overlaps with what I outlined in #1605. You could run something like this:
```sh
datasette freeze -d exports/
aws s3 cp exports/ s3://my-export-bucket/$(date)
```
And maybe that does what you need. Of course, that plugin isn't built yet. But that's the idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203943272,Datasette feature for publishing snapshots of query results,
https://github.com/simonw/datasette/issues/1699#issuecomment-1094453751,https://api.github.com/repos/simonw/datasette/issues/1699,1094453751,IC_kwDOBm6k_c5BPAn3,25778,eyeseast,2022-04-11T01:32:12Z,2022-04-11T01:32:12Z,CONTRIBUTOR,"Was looking through old issues and realized a bunch of this got discussed in #1101 (including by me!), so sorry to rehash all this. Happy to help with whatever piece of it I can. Would be very excited to be able to use format plugins with exports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,Proposal: datasette query,
https://github.com/simonw/datasette/issues/1699#issuecomment-1092386254,https://api.github.com/repos/simonw/datasette/issues/1699,1092386254,IC_kwDOBm6k_c5BHH3O,25778,eyeseast,2022-04-08T02:39:25Z,2022-04-08T02:39:25Z,CONTRIBUTOR,"And just to think this through a little more, here's what `stream_geojson` might look like:
```python
async def stream_geojson(datasette, columns, rows, database, stream):
db = datasette.get_database(database)
for row in rows:
feature = await row_to_geojson(row, db)
stream.write(feature + ""\n"") # just assuming newline mode for now
```
Alternately, that could be an async generator, like this:
```python
async def stream_geojson(datasette, columns, rows, database):
db = datasette.get_database(database)
for row in rows:
feature = await row_to_geojson(row, db)
yield feature
```
Not sure which makes more sense, but I think this pattern would open up a lot of possibility. If you had your [stream_indented_json](https://til.simonwillison.net/python/output-json-array-streaming) function, you could do `yield from stream_indented_json(rows, 2)` and be one your way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,Proposal: datasette query,
https://github.com/simonw/datasette/issues/1699#issuecomment-1092370880,https://api.github.com/repos/simonw/datasette/issues/1699,1092370880,IC_kwDOBm6k_c5BHEHA,25778,eyeseast,2022-04-08T02:07:40Z,2022-04-08T02:07:40Z,CONTRIBUTOR,"So maybe `render_output_render` returns something like this:
```python
@hookimpl
def register_output_renderer(datasette):
return {
""extension"": ""geojson"",
""render"": render_geojson,
""stream"": stream_geojson,
""can_render"": can_render_geojson,
}
```
And stream gets an iterator, instead of a list of rows, so it can efficiently handle large queries. Maybe it also gets passed a destination stream, or it returns an iterator. I'm not sure what makes more sense. Either way, that might cover both CLI exports and streaming responses.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,Proposal: datasette query,
https://github.com/simonw/datasette/issues/1699#issuecomment-1092357672,https://api.github.com/repos/simonw/datasette/issues/1699,1092357672,IC_kwDOBm6k_c5BHA4o,25778,eyeseast,2022-04-08T01:39:40Z,2022-04-08T01:39:40Z,CONTRIBUTOR,"> My best thought on how to differentiate them so far is plugins: if Datasette plugins that provide alternative outputs - like .geojson and .yml and suchlike - also work for the datasette query command that would make a lot of sense to me.
That's my thinking, too. It's really the thing I've been wanting since writing `datasette-geojson`, since I'm always exporting with `datasette --get`. The workflow I'm always looking for is something like this:
```sh
cd alltheplaces-datasette
datasette query dunkin_in_suffolk -f geojson -o dunkin_in_suffolk.geojson
```
I think this probably needs either a new plugin hook separate from `register_output_renderer` or a way to use that without going through the HTTP stack. Or maybe a render mode that writes to a stream instead of a response. Maybe there's a new key in the dictionary that `register_output_renderer` returns that handles CLI exports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,Proposal: datasette query,
https://github.com/simonw/datasette/issues/1101#issuecomment-1105642187,https://api.github.com/repos/simonw/datasette/issues/1101,1105642187,IC_kwDOBm6k_c5B5sLL,25778,eyeseast,2022-04-21T18:59:08Z,2022-04-21T18:59:08Z,CONTRIBUTOR,"Ha! That was your idea (and a good one).
But it's probably worth measuring to see what overhead it adds. It did require both passing in the database and making the whole thing `async`.
Just timing the queries themselves:
1. [Using `AsGeoJSON(geometry) as geometry`](https://alltheplaces-datasette.fly.dev/alltheplaces?sql=select%0D%0A++id%2C%0D%0A++properties%2C%0D%0A++AsGeoJSON%28geometry%29+as+geometry%2C%0D%0A++spider%0D%0Afrom%0D%0A++places%0D%0Aorder+by%0D%0A++id%0D%0Alimit%0D%0A++1000) takes 10.235 ms
2. [Leaving as binary](https://alltheplaces-datasette.fly.dev/alltheplaces?sql=select%0D%0A++id%2C%0D%0A++properties%2C%0D%0A++geometry%2C%0D%0A++spider%0D%0Afrom%0D%0A++places%0D%0Aorder+by%0D%0A++id%0D%0Alimit%0D%0A++1000) takes 8.63 ms
Looking at the network panel:
1. Takes about 200 ms for the `fetch` request
2. Takes about 300 ms
I'm not sure how best to time the GeoJSON generation, but it would be interesting to check. Maybe I'll write a plugin to add query times to response headers.
The other thing to consider with async streaming is that it might be well-suited for a slower response. When I have to get the whole result and send a response in a fixed amount of time, I need the most efficient query possible. If I can hang onto a connection and get things one chunk at a time, maybe it's ok if there's some overhead.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1101#issuecomment-1105588651,https://api.github.com/repos/simonw/datasette/issues/1101,1105588651,IC_kwDOBm6k_c5B5fGr,25778,eyeseast,2022-04-21T18:15:39Z,2022-04-21T18:15:39Z,CONTRIBUTOR,"What if you split rendering and streaming into two things:
- `render` is a function that returns a response
- `stream` is a function that sends chunks, or yields chunks passed to an ASGI `send` callback
That way current plugins still work, and streaming is purely additive. A `stream` function could get a cursor or iterator of rows, instead of a list, so it could more efficiently handle large queries.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1688#issuecomment-1079806857,https://api.github.com/repos/simonw/datasette/issues/1688,1079806857,IC_kwDOBm6k_c5AXIuJ,9020979,hydrosquall,2022-03-27T01:01:14Z,2022-03-27T01:01:14Z,CONTRIBUTOR,"Thank you! I went through the cookiecutter template, and published my first package here: https://github.com/hydrosquall/datasette-nteract-data-explorer","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1181432624,[plugins][documentation] Is it possible to serve per-plugin static folders when writing one-off (single file) plugins?,
https://github.com/simonw/datasette/issues/1688#issuecomment-1079550754,https://api.github.com/repos/simonw/datasette/issues/1688,1079550754,IC_kwDOBm6k_c5AWKMi,9020979,hydrosquall,2022-03-26T01:27:27Z,2022-03-26T03:16:29Z,CONTRIBUTOR,"> Is there a way to serve a static assets when using the plugins/ directory method instead of installing plugins as a new python package?
As a workaround, I found I can serve my statics from a non-plugin specific folder using the [--static](https://docs.datasette.io/en/stable/custom_templates.html#serving-static-files) CLI flag.
```bash
datasette ~/Library/Safari/History.db \
--plugins-dir=plugins/ \
--static assets:dist/
```
It's not ideal because it means I'll change the cache pattern path depending on how the plugin is running (via pip install or as a one off script), but it's usable as a workaround.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1181432624,[plugins][documentation] Is it possible to serve per-plugin static folders when writing one-off (single file) plugins?,
https://github.com/simonw/datasette/issues/1684#issuecomment-1078126065,https://api.github.com/repos/simonw/datasette/issues/1684,1078126065,IC_kwDOBm6k_c5AQuXx,536941,fgregg,2022-03-24T20:08:56Z,2022-03-24T20:13:19Z,CONTRIBUTOR,"would be nice if the behavior was
1. try to facet all the columns
2. for bigger tables try to facet the indexed columns
3. for the biggest tables, turn off autofacetting completely
This is based on my assumption that what determines autofaceting is the rarity of unique values. Which may not be true!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1179998071,Mechanism for disabling faceting on large tables only,
https://github.com/simonw/datasette/issues/1581#issuecomment-1077047295,https://api.github.com/repos/simonw/datasette/issues/1581,1077047295,IC_kwDOBm6k_c5AMm__,536941,fgregg,2022-03-24T04:08:18Z,2022-03-24T04:08:18Z,CONTRIBUTOR,this has been addressed by the datasette-hashed-urls plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1089529555,"when hashed urls are turned on, the _memory db has improperly long-lived cache expiry",
https://github.com/simonw/datasette/pull/1582#issuecomment-1077047152,https://api.github.com/repos/simonw/datasette/issues/1582,1077047152,IC_kwDOBm6k_c5AMm9w,536941,fgregg,2022-03-24T04:07:58Z,2022-03-24T04:07:58Z,CONTRIBUTOR,this has been obviated by the datasette-hashed-urls plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1090055810,don't set far expiry if hash is '000',
https://github.com/simonw/datasette/issues/1549#issuecomment-1087428593,https://api.github.com/repos/simonw/datasette/issues/1549,1087428593,IC_kwDOBm6k_c5A0Nfx,536941,fgregg,2022-04-04T11:17:13Z,2022-04-04T11:17:13Z,CONTRIBUTOR,"another way to get the behavior of downloading the file is to use the download attribute of the anchor tag
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#attr-download","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability,
https://github.com/simonw/datasette/issues/1384#issuecomment-1066222323,https://api.github.com/repos/simonw/datasette/issues/1384,1066222323,IC_kwDOBm6k_c4_jULz,2670795,brandonrobertz,2022-03-14T00:36:42Z,2022-03-14T00:36:42Z,CONTRIBUTOR,"> Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :)
All good. Report back any issues you find with this stuff. Metadata/dynamic config hasn't been tested widely outside of what I've done AFAIK. If you find a strong use case for async meta, it's going to be better to know sooner rather than later!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-1066169718,https://api.github.com/repos/simonw/datasette/issues/1384,1066169718,IC_kwDOBm6k_c4_jHV2,2670795,brandonrobertz,2022-03-13T19:48:49Z,2022-03-13T19:48:49Z,CONTRIBUTOR,"> For my reference, did you include a `render_cell` plugin calling `get_metadata` in those tests?
You shouldn't need to do this, as I mentioned previously. The code inside `render_cell` hook already has access to the most recently sync'd metadata via `datasette._metadata_local`. Refreshing the metadata for every cell seems ... excessive.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-1066006292,https://api.github.com/repos/simonw/datasette/issues/1384,1066006292,IC_kwDOBm6k_c4_ifcU,2670795,brandonrobertz,2022-03-13T02:09:44Z,2022-03-13T02:09:44Z,CONTRIBUTOR,"> If I'm understanding your plugin code correctly, you query the db using the sync handle every time `get_metdata` is called, right? Won't this become a pretty big bottleneck if a hook into `render_cell` is trying to read metadata / plugin config?
Reading from sqlite DBs is pretty quick and I didn't notice significant performance issues when I was benchmarking. I tested on very large Datasette deployments (hundreds of DBs, millions of rows). See [""Many small queries are efficient in sqlite""](https://sqlite.org/np1queryprob.html) for more information on the rationale here. Also note that in the [datasette-live-config](https://github.com/next-LI/datasette-live-config) reference plugin, the DB connection is cached, so that eliminated most of the performance worries we had.
If you need to ensure fresh metadata is being read inside of a `render_cell` hook specifically, you don't need to do anything further! `get_metadata` gets called before `render_cell` every request, so it already has access to the synced meta. There shouldn't be a need to call `get_metadata(...)` or `metadata(...)` inside `render_cell`, you can just use `datasette._metadata_local` if you're really worried about performance.
> The plugin is close, but looks like it only grabs remote metadata, is that right? Instead what I'm wanting is to grab metadata embedded in the attached databases.
Yes correct, the datadette-remote-metadata plugin doesn't do that. But the datasette-live-config plugin does. [It supports a `__metadata` table](https://github.com/next-LI/datasette-live-config/blob/main/datasette_live_config/__init__.py#L107-L138) that, when it exists on an attached DB, gets pulled into the Datasette internal `_metadata` and is also accessible via `get_metadata`. Updating is instantaneous so there's no gotchas for users or security issues for users relying on the metadata-based permissions. Simon talked about eventually making something like this a standard feature of Datasette, but I'm not sure what the status is on that!
Good luck!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-1065940779,https://api.github.com/repos/simonw/datasette/issues/1384,1065940779,IC_kwDOBm6k_c4_iPcr,2670795,brandonrobertz,2022-03-12T18:49:29Z,2022-03-12T18:50:07Z,CONTRIBUTOR,"Hello! Just wanted to chime in and note that there's a plugin to have Datasette [watch for updates to an external metadata.yaml/json and update the internal settings accordingly](https://datasette.io/plugins/datasette-remote-metadata), so I think the cache/poll use case is already covered. @khusmann If you don't need truly dynamic metadata then what you've come up with or the plugin ought to work fine.
Making the get_metadata async won't improve the situation by itself as only some of the code paths accessing metadata use that hook. The other paths use the internal metadata dict. Trying to force all paths through a async hook would have performance ramifications and making everything use the internal meta will cause problems for users that need changes to take effect immediately. This is why I came to the non-async solution as it was the path of least change within Datasette. As always, open to new ideas, etc!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1655#issuecomment-1062450649,https://api.github.com/repos/simonw/datasette/issues/1655,1062450649,IC_kwDOBm6k_c4_U7XZ,536941,fgregg,2022-03-09T01:10:46Z,2022-03-09T01:10:46Z,CONTRIBUTOR,"i increased the max_returned_row, because I have some scripts that get CSVs from this site, and this makes doing pagination of CSVs less annoying for many cases. i know that's streaming csvs is something you are hoping to address in 1.0. let me know if there's anything i can do to help with that.
as for what if anything can be done about the size of the dom, I don't have any ideas right now, but i'll poke around.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data,
https://github.com/simonw/datasette/pull/1593#issuecomment-1031455498,https://api.github.com/repos/simonw/datasette/issues/1593,1031455498,IC_kwDOBm6k_c49esMK,49699333,dependabot[bot],2022-02-07T13:13:22Z,2022-02-07T13:13:22Z,CONTRIBUTOR,Superseded by #1631.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1101705012,"Update pytest-asyncio requirement from <0.17,>=0.10 to >=0.10,<0.18",
https://github.com/simonw/datasette/issues/1605#issuecomment-1018778667,https://api.github.com/repos/simonw/datasette/issues/1605,1018778667,IC_kwDOBm6k_c48uVQr,25778,eyeseast,2022-01-21T19:00:01Z,2022-01-21T19:00:01Z,CONTRIBUTOR,"Let me know if you want help prototyping any of this, because I'm thinking about it and trying stuff out. Happy to be a sounding board, if it helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports,
https://github.com/simonw/datasette/issues/1605#issuecomment-1018741262,https://api.github.com/repos/simonw/datasette/issues/1605,1018741262,IC_kwDOBm6k_c48uMIO,25778,eyeseast,2022-01-21T18:05:09Z,2022-01-21T18:05:09Z,CONTRIBUTOR,"Thinking about this more, as well as #1356 and various other tickets related to output formats, I think there's a missing plugin hook for formatting results, separate from `register_output_renderer` (or maybe part of it, depending on #1101).
Right now, as I understand it, getting output in any format goes through the normal view stack -- a table, a row or a query -- and so by the time `register_output_renderer` gets it, the results have already been truncated or paginated. What I'd want, I think, is to be able to register ways to format results independent of where those results are sent.
It's possible this could be done using [`conn.row_factory`](https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.row_factory) (maybe in the `prepare_connection` hook), but I'm not sure that's where it belongs.
Another option is some kind of registry of serializers, which `register_output_renderer` and other plugin hooks could use. What I'm trying to avoid here is writing a plugin that also needs plugins for formats I haven't thought of yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports,
https://github.com/simonw/datasette/issues/1605#issuecomment-1016994329,https://api.github.com/repos/simonw/datasette/issues/1605,1016994329,IC_kwDOBm6k_c48nhoZ,25778,eyeseast,2022-01-20T00:27:17Z,2022-01-20T00:27:17Z,CONTRIBUTOR,"Right now, I usually have a line in a Makefile like this:
```make
combined.geojson: project.db
pipenv run datasette project.db --get /project/combined.geojson \
--load-extension spatialite \
--setting sql_time_limit_ms 5000 \
--setting max_returned_rows 20000 \
-m metadata.yml > $@
```
That all assumes I've loaded whatever I need into `project.db` and created a canned query called `combined` (and then uses `datasette-geojson` for geojson output).
It works, but as you can see, it's a lot to manage, a lot of boilerplate, and it wasn't obvious how to get there. If there's an error in the canned query, I get an HTML error page, so that's hard to debug. And it's only one query, so each output needs a line like this. Make isn't ideal, either, for that reason.
The thing I really liked with `datafreeze` was doing templated filenames. I have a project now where I need to export a bunch of litttle geojson files, based on queries, and it would be awesome to be able to do something like this:
```yml
databases:
project:
queries:
boundaries:
sql: ""SELECT * FROM boundaries""
filename: ""boundaries/{id}.geojson""
mode: ""item""
format: geojson
```
And then do:
```sh
datasette freeze -m metadata.yml project.db
```
For HTML export, maybe there's a `template` argument, or `format: template` or something. And that gets you a static site generator, kinda for free.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports,
https://github.com/simonw/datasette/issues/1601#issuecomment-1016651485,https://api.github.com/repos/simonw/datasette/issues/1601,1016651485,IC_kwDOBm6k_c48mN7d,25778,eyeseast,2022-01-19T16:39:03Z,2022-01-19T16:39:03Z,CONTRIBUTOR,"I think both of these are Spatialite specific. They get generated when you first initialize the extension. KNN is actually deprecated in favor of [KNN2](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN2), as I understand it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list,
https://github.com/simonw/datasette/issues/1591#issuecomment-1010947634,https://api.github.com/repos/simonw/datasette/issues/1591,1010947634,IC_kwDOBm6k_c48QdYy,82988,psychemedia,2022-01-12T11:32:17Z,2022-01-12T11:32:17Z,CONTRIBUTOR,Is it possible to parse things like `--ext-{plugin}-{arg} VALUE` ?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?,
https://github.com/simonw/datasette/pull/1574#issuecomment-1007844190,https://api.github.com/repos/simonw/datasette/issues/1574,1007844190,IC_kwDOBm6k_c48Ente,536941,fgregg,2022-01-08T00:42:12Z,2022-01-08T00:42:12Z,CONTRIBUTOR,is there a reason to not always use the slim option?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,introduce new option for datasette package to use a slim base image,
https://github.com/simonw/datasette/issues/1612#issuecomment-1021497165,https://api.github.com/repos/simonw/datasette/issues/1612,1021497165,IC_kwDOBm6k_c484s9N,639012,jsfenfen,2022-01-25T18:44:23Z,2022-01-25T18:44:23Z,CONTRIBUTOR,"OMG, this might be the fastest OS ticket I've ever filed, thanks so much @simonw ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area,
https://github.com/simonw/datasette/issues/1583#issuecomment-1002825217,https://api.github.com/repos/simonw/datasette/issues/1583,1002825217,IC_kwDOBm6k_c47xeYB,536941,fgregg,2021-12-30T00:34:16Z,2021-12-30T00:34:16Z,CONTRIBUTOR,"if that is not desirable, it might be good to document that users might want to set up a lifecycle rule to automatically delete these build artifacts. something like https://stackoverflow.com/questions/59937542/can-i-delete-container-images-from-google-cloud-storage-artifacts-bucket","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1090810196,consider adding deletion step of cloudbuild artifacts to gcloud publish,
https://github.com/simonw/datasette/issues/1547#issuecomment-997519202,https://api.github.com/repos/simonw/datasette/issues/1547,997519202,IC_kwDOBm6k_c47dO9i,127565,wragge,2021-12-20T01:36:58Z,2021-12-20T01:36:58Z,CONTRIBUTOR,"Yep, that works -- thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates,
https://github.com/simonw/datasette/issues/1547#issuecomment-997511968,https://api.github.com/repos/simonw/datasette/issues/1547,997511968,IC_kwDOBm6k_c47dNMg,127565,wragge,2021-12-20T01:21:59Z,2021-12-20T01:21:59Z,CONTRIBUTOR,"I've installed the alpha version but get an error when starting up Datasette:
```
Traceback (most recent call last):
File ""/Users/tim/.pyenv/versions/stock-exchange/bin/datasette"", line 5, in
from datasette.cli import cli
File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/cli.py"", line 15, in
from .app import Datasette, DEFAULT_SETTINGS, SETTINGS, SQLITE_LIMIT_ATTACHED, pm
File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/app.py"", line 31, in
from .views.database import DatabaseDownload, DatabaseView
File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/views/database.py"", line 25, in
from datasette.plugins import pm
File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/plugins.py"", line 29, in
mod = importlib.import_module(plugin)
File ""/Users/tim/.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py"", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/filters.py"", line 9, in
@hookimpl(specname=""filters_from_request"")
TypeError: __call__() got an unexpected keyword argument 'specname'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates,
https://github.com/simonw/datasette/issues/1561#issuecomment-997128712,https://api.github.com/repos/simonw/datasette/issues/1561,997128712,IC_kwDOBm6k_c47bvoI,536941,fgregg,2021-12-18T02:35:48Z,2021-12-18T02:35:48Z,CONTRIBUTOR,interesting! i love this feature. this + full caching with cloudflare is really super!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on",
https://github.com/simonw/datasette/issues/1552#issuecomment-996229007,https://api.github.com/repos/simonw/datasette/issues/1552,996229007,IC_kwDOBm6k_c47YT-P,3556,davidbgk,2021-12-16T22:04:39Z,2021-12-16T22:04:39Z,CONTRIBUTOR,"Wow, that was fast, thank you so much @simonw !
> I'm also not convinced that this configuration syntax is right. It's a bit weird having a `""facets""` list that can either by column-name-strings or `{""type-of-facet"": ""column-name""}` objects. Maybe there's a better design for this?
I agree that it's not ideal, my initial naive approach was to detect if it's an array, like what is done here:
https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/facets.py#L312-L313
But it requires an extra query to determine the type, which is a bit problematic, especially for big tables I guess.
Taking a look at #510, I wonder if a `facet_delimiter` should be defined for that kind of columns (that would help our team not to have an intermediary conversion step from `foo|bar` to `[""foo"",""bar""]` for instance).
To be consistent with the `--extract-column` parameter, maybe an explicit casting/delimiter would be useful: `--set-column 'Foo:Array:|'`.
Throwing a lot of ideas without knowing the big picture… but sometimes newcomers have superpowers :).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/1552#issuecomment-995296725,https://api.github.com/repos/simonw/datasette/issues/1552,995296725,IC_kwDOBm6k_c47UwXV,3556,davidbgk,2021-12-15T23:29:32Z,2021-12-15T23:29:32Z,CONTRIBUTOR,"@simonw thank you for your fast answer and your guidance!
While digging into the code, I found an undocumented way of doing it:
```yaml
facets: [""Facet for a column"", {""array"": ""Facet for an array""}]
```
The only remaining problem with that solution is here: https://github.com/simonw/datasette/blob/250db8192cb8aba5eb8cd301ccc2a49525bc3d24/datasette/facets.py#L33
We have:
```python
type, metadata_config = metadata_config.items()[0]
```
But it requires to cast the `dict_items` as a list prior to access the first element:
```python
type, metadata_config = list(metadata_config.items())[0]
```
I guess it's an unspotted bug? (I mean, independently of the facets-with-arrays issue.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`),
https://github.com/simonw/datasette/issues/526#issuecomment-993078038,https://api.github.com/repos/simonw/datasette/issues/526,993078038,IC_kwDOBm6k_c47MSsW,536941,fgregg,2021-12-14T01:46:52Z,2021-12-14T01:46:52Z,CONTRIBUTOR,"the nested query idea is very nice, and i stole if for [my client side paginator](https://observablehq.com/d/1d5da3a3c3f2f347#DatasetteClient). However, it won't do the right thing if the original query orders by random().
If you go the nested query route, maybe raise a 4XX status code if the query has such a clause?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/1553#issuecomment-993014772,https://api.github.com/repos/simonw/datasette/issues/1553,993014772,IC_kwDOBm6k_c47MDP0,536941,fgregg,2021-12-13T23:46:18Z,2021-12-13T23:46:18Z,CONTRIBUTOR,these headers would also be relevant for json exports of custom queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header,
https://github.com/simonw/datasette/issues/1553#issuecomment-992986587,https://api.github.com/repos/simonw/datasette/issues/1553,992986587,IC_kwDOBm6k_c47L8Xb,536941,fgregg,2021-12-13T22:57:04Z,2021-12-13T22:57:04Z,CONTRIBUTOR,would also be good if the header said the what the max row limit was,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header,
https://github.com/simonw/datasette/issues/526#issuecomment-992971072,https://api.github.com/repos/simonw/datasette/issues/526,992971072,IC_kwDOBm6k_c47L4lA,536941,fgregg,2021-12-13T22:29:34Z,2021-12-13T22:29:34Z,CONTRIBUTOR,just came by to open this issue. would make my data analysis in observable a lot better!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/1549#issuecomment-991754237,https://api.github.com/repos/simonw/datasette/issues/1549,991754237,IC_kwDOBm6k_c47HPf9,536941,fgregg,2021-12-11T19:14:39Z,2021-12-11T19:14:39Z,CONTRIBUTOR,"that option is not available on [custom queries](https://labordata.bunkum.us/odpr-962a140?sql=with+local_union_filings+as+%28%0D%0A++select+*+from+lm_data+%0D%0A++where%0D%0A++++yr_covered+%3E+cast%28strftime%28%27%25Y%27%2C+%27now%27%2C+%27-5+years%27%29+as+int%29%0D%0A++++and+desig_name+%3D+%27LU%27%0D%0A++order+by+yr_covered+desc%0D%0A%29%2C%0D%0Amost_recent_filing+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from+local_union_filings%0D%0A++group+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++most_recent_filing%0D%0Awhere%0D%0A++next_election+%3E%3D+strftime%28%27%25Y-%25m%27%2C+%27now%27%29%0D%0A++and+next_election+%3C+strftime%28%27%25Y-%25m%27%2C+%27now%27%2C+%27%2B1+year%27%29%0D%0Aorder+by%0D%0A++members+desc%3B).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability,
https://github.com/simonw/datasette/issues/1522#issuecomment-976117989,https://api.github.com/repos/simonw/datasette/issues/1522,976117989,IC_kwDOBm6k_c46LmDl,813732,glasnt,2021-11-23T03:00:34Z,2021-11-23T03:00:34Z,CONTRIBUTOR,"I tried deploying the most recent version of the Dockerfile in this thread ([link to comment](https://github.com/simonw/datasette/issues/1522#issuecomment-974605128)), and after trying a few different different combinations, I was only successful when I used `--no-cpu-throttling` (""CPU Is always allocated"" in the UI)
Using this method, I got a very similar issue to you: The first time I'd load the site I'd get a 503. But after that first load, I didn't get the issue again. It would re-occur if the service started from cold boot.
I suspect this is a race condition in the supervisord configuration. The errors I got were the same `Connection refused: AH00957: http: attempt to connect to 127.0.0.1:8001 (127.0.0.1) failed`, and that seems to indicate that `datasette` hadn't yet started.
Looking at the order of logs getting back, the processes reported successfully completing loading after the first 503 was returned, so that makes me think race condition.
I can replicate this locally, if I `docker run` and request `localhost:5000/prefix` _before_ I get the `datasette entered RUNNING state` message. Cloud Run wakes up when requests are received, so this test would semi-replicate that, but local docker would be the equivalent of a persistent process, hence it doesn't normally exhibit the same issues.
Unfortunately supervisor/supervisor issue 122 (not linking as to prevent cross-project link spam) seems to say that dependency chaining is a feature that's been asked for for a long time, but hasn't been implemented. You could try some suggestions in that thread. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy,
https://github.com/simonw/datasette/issues/1528#issuecomment-975955589,https://api.github.com/repos/simonw/datasette/issues/1528,975955589,IC_kwDOBm6k_c46K-aF,15178711,asg017,2021-11-22T22:00:30Z,2021-11-22T22:00:30Z,CONTRIBUTOR,"Oh, another thing to consider: I believe this would be the first `""_file""` key in datasette's metadata, compared to other `""_url""` keys like `""license_url""` or `""about_url""`. Not too sure what considerations to include with this (ex should missing files cause Datasette to stop before starting, should build scripts bundle these sql files somewhere during `datasette package`, etc.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1060631257,"Add new `""sql_file""` key to Canned Queries in metadata?",
https://github.com/simonw/datasette/pull/1495#issuecomment-974108455,https://api.github.com/repos/simonw/datasette/issues/1495,974108455,IC_kwDOBm6k_c46D7cn,192568,mroswell,2021-11-19T14:14:35Z,2021-11-19T14:14:35Z,CONTRIBUTOR,A nudge on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options,
https://github.com/simonw/datasette/issues/1380#issuecomment-967747190,https://api.github.com/repos/simonw/datasette/issues/1380,967747190,IC_kwDOBm6k_c45rqZ2,813732,glasnt,2021-11-13T00:47:26Z,2021-11-13T00:47:26Z,CONTRIBUTOR,"Would it make sense to run datasette with a fswatch/inotifywait on a folder, then? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder,
https://github.com/simonw/datasette/pull/1514#issuecomment-972852184,https://api.github.com/repos/simonw/datasette/issues/1514,972852184,IC_kwDOBm6k_c45_IvY,49699333,dependabot[bot],2021-11-18T13:11:15Z,2021-11-18T13:11:15Z,CONTRIBUTOR,Superseded by #1516.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1056117435,Bump black from 21.9b0 to 21.11b0,
https://github.com/simonw/datasette/pull/1495#issuecomment-960420237,https://api.github.com/repos/simonw/datasette/issues/1495,960420237,IC_kwDOBm6k_c45PtmN,192568,mroswell,2021-11-04T03:12:01Z,2021-11-04T03:12:01Z,CONTRIBUTOR,"This all looks promising! I will need detailed documentation on how to upgrade datasette once it's available, and how to implement. (@fgregg example looks very straightforward on the plugin front.)
I'll be so excited if I can get:
https://list.saferdisinfectants.org/
instead of
https://list.saferdisinfectants.org/disinfectants/listN
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options,
https://github.com/simonw/datasette/pull/1500#issuecomment-971568829,https://api.github.com/repos/simonw/datasette/issues/1500,971568829,IC_kwDOBm6k_c456Pa9,49699333,dependabot[bot],2021-11-17T13:13:58Z,2021-11-17T13:13:58Z,CONTRIBUTOR,Superseded by #1514.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1041158024,Bump black from 21.9b0 to 21.10b0,
https://github.com/simonw/datasette/issues/1012#issuecomment-970266123,https://api.github.com/repos/simonw/datasette/issues/1012,970266123,IC_kwDOBm6k_c451RYL,45380,bollwyvl,2021-11-16T13:18:36Z,2021-11-16T13:18:36Z,CONTRIBUTOR,"Congratulations, looks like it went through! There was a bit of a hold-up
on the JupyterLab ones, but it's semi automated: a dependabot pr to
warehouse and a CI deploy, with a click in between.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py,
https://github.com/simonw/datasette/issues/1401#issuecomment-950150483,https://api.github.com/repos/simonw/datasette/issues/1401,950150483,IC_kwDOBm6k_c44oiVT,418191,jaywgraves,2021-10-23T13:09:10Z,2021-10-23T13:09:10Z,CONTRIBUTOR,"I think it's because of this in `app.css`
```
ol,
ul {
list-style: none;
}
```
https://github.com/simonw/datasette/blame/main/datasette/static/app.css#L35-L38
You could probably reinstate that by providing your own CSS.
https://docs.datasette.io/en/0.24/custom_templates.html#custom-css-and-javascript","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",950664971,unordered list is not rendering bullet points in description_html on database page,
https://github.com/simonw/datasette/issues/1284#issuecomment-949604763,https://api.github.com/repos/simonw/datasette/issues/1284,949604763,IC_kwDOBm6k_c44mdGb,536941,fgregg,2021-10-22T12:54:34Z,2021-10-22T12:54:34Z,CONTRIBUTOR,i'm going to take a swing at this today. we'll see.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",845794436,Feature or Documentation Request: Individual table as home page template,
https://github.com/simonw/datasette/issues/1480#issuecomment-947203725,https://api.github.com/repos/simonw/datasette/issues/1480,947203725,IC_kwDOBm6k_c44dS6N,110420,ghing,2021-10-20T00:21:54Z,2021-10-20T00:21:54Z,CONTRIBUTOR,"This StackOverflow post, [sqlite - Cloud Run: Why does my instance need so much RAM?](https://stackoverflow.com/questions/59812405/cloud-run-why-does-my-instance-need-so-much-ram), points to [this section of the Cloud Run docs](https://cloud.google.com/run/docs/troubleshooting) that says:
> Note that the Cloud Run container instances run in an environment where the files written to the local filesystem count towards the available memory. This also includes any log files that are not written to /var/log/* or /dev/log.
Does datasette write any large files when starting?
Or does the [`COPY` command in the Dockerfile](https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L349) count as writing to the local filesystem?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database,
https://github.com/simonw/datasette/issues/1480#issuecomment-947196177,https://api.github.com/repos/simonw/datasette/issues/1480,947196177,IC_kwDOBm6k_c44dRER,110420,ghing,2021-10-20T00:05:10Z,2021-10-20T00:05:10Z,CONTRIBUTOR,"I was looking through the Dockerfile-generation code to see if there was anything that would cause memory usage to be a lot during deployment.
I noticed that the Dockerfile [runs `datasette --inspect`](https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L354). Is it possible that this is using a lot of memory usage?
Or would that come into play when running `gcloud builds submit`, not when it's actually deployed?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database,
https://github.com/simonw/datasette/issues/1396#issuecomment-946467547,https://api.github.com/repos/simonw/datasette/issues/1396,946467547,IC_kwDOBm6k_c44afLb,72577720,MichaelTiemannOSC,2021-10-19T08:10:26Z,2021-10-19T08:10:26Z,CONTRIBUTOR,"Now that 0.59 has excellent annotated release notes, you can re-confirm this is fixed by updating the published Docker image and checking that these fixes still work ;-)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1432#issuecomment-946287922,https://api.github.com/repos/simonw/datasette/issues/1432,946287922,IC_kwDOBm6k_c44ZzUy,192568,mroswell,2021-10-19T01:16:41Z,2021-10-19T01:16:41Z,CONTRIBUTOR,"Resolved, with assistance from @ashishdotme (Thank you!)
Updated requirements.txt to include:
```
datasette==0.59
datasette-publish-vercel==0.11
sqlite-utils==3.6
```
Ran:
```
$ pip3 install -r requirements.txt
```
The site is back at work! Yay!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=,
https://github.com/simonw/datasette/issues/1432#issuecomment-946255239,https://api.github.com/repos/simonw/datasette/issues/1432,946255239,IC_kwDOBm6k_c44ZrWH,192568,mroswell,2021-10-18T23:55:25Z,2021-10-18T23:55:25Z,CONTRIBUTOR,"I am getting this when I visit my live Datasette page:
```
This Serverless Function has crashed.
Your connection is working correctly.
Vercel is working correctly.
500: INTERNAL_SERVER_ERROR
Code: FUNCTION_INVOCATION_FAILED
ID: ...
```
And in the server logs, I'm getting
```
[GET] /disinfectants/listN
19:53:14:23
module initialization error: __init__() got an unexpected keyword argument 'config'
module initialization error
__init__() got an unexpected keyword argument 'config'
```
Which is the same error that @ashishdotme reported above.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=,
https://github.com/simonw/datasette/pull/1489#issuecomment-943594738,https://api.github.com/repos/simonw/datasette/issues/1489,943594738,IC_kwDOBm6k_c44Phzy,49699333,dependabot[bot],2021-10-14T18:04:13Z,2021-10-14T18:04:13Z,CONTRIBUTOR,"OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. You can also ignore all major, minor, or patch releases for a dependency by adding an [`ignore` condition](https://docs.github.com/en/code-security/supply-chain-security/configuration-options-for-dependency-updates#ignore) with the desired `update_types` to your config file.
If you change your mind, just re-open this PR and I'll resolve any conflicts on it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026379132,"Update pyyaml requirement from ~=5.3 to >=5.3,<7.0",
https://github.com/simonw/datasette/pull/1489#issuecomment-943594735,https://api.github.com/repos/simonw/datasette/issues/1489,943594735,IC_kwDOBm6k_c44Phzv,49699333,dependabot[bot],2021-10-14T18:04:12Z,2021-10-14T18:04:12Z,CONTRIBUTOR,Looks like this PR is closed. If you re-open it I'll rebase it as long as no-one else has edited it (you can use `@dependabot reopen` if the branch has been deleted).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026379132,"Update pyyaml requirement from ~=5.3 to >=5.3,<7.0",
https://github.com/simonw/datasette/pull/1495#issuecomment-954384496,https://api.github.com/repos/simonw/datasette/issues/1495,954384496,IC_kwDOBm6k_c444sBw,536941,fgregg,2021-10-29T03:07:13Z,2021-10-29T03:07:13Z,CONTRIBUTOR,"okay @simonw, made the requested changes. tests are running locally. i think this is ready for you to look at again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options,
https://github.com/simonw/datasette/issues/1380#issuecomment-953366110,https://api.github.com/repos/simonw/datasette/issues/1380,953366110,IC_kwDOBm6k_c440zZe,813732,glasnt,2021-10-27T22:48:55Z,2021-10-27T22:48:55Z,CONTRIBUTOR,"It looks like if the files argument is a directory, `config_dir` is set, but files in that folder are only loaded into `self.files` at the `Datasette` class initialisation.
I tried seeing if I could get `--reload` to work, but I'm getting issues trying to use that command when specifying a directory, as the command `serve` ends up in the files list(?):
```
datasette serve . --reload
Error: Invalid value for '[FILES]...': Path 'serve' does not exist.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder,
https://github.com/simonw/datasette/issues/1380#issuecomment-953334718,https://api.github.com/repos/simonw/datasette/issues/1380,953334718,IC_kwDOBm6k_c440ru-,813732,glasnt,2021-10-27T21:45:04Z,2021-10-27T21:45:04Z,CONTRIBUTOR,"I am also getting this issue, using the currently most recent version of datasette
```
$ datasette --version
datasette, version 0.59.1
```
If I run `datasette` within just a folder of files,
```
$ datasette serve .
```
Adding new files while datasette is running shows no new files, and removing files causes datasette to return 500 errors.
```
home
Error 500
[Errno 2] No such file or directory: 'mydatabase.db'
Powered by Datasette
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder,
https://github.com/simonw/datasette/issues/1480#issuecomment-938171377,https://api.github.com/repos/simonw/datasette/issues/1480,938171377,IC_kwDOBm6k_c4361vx,110420,ghing,2021-10-07T21:33:12Z,2021-10-07T21:33:12Z,CONTRIBUTOR,"Thanks for the reply @simonw. What services have you had better success with than Cloud Run for larger database?
Also, what about my issue description makes you think there may be a workaround?
Is there any instrumentation I could add to see at which point in the deploy the memory usage spikes? Should I be able to see this whether it's running under Docker locally, or do you suspect this is Cloud Run-specific?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database,
https://github.com/simonw/datasette/pull/1453#issuecomment-919135732,https://api.github.com/repos/simonw/datasette/issues/1453,919135732,IC_kwDOBm6k_c42yOX0,49699333,dependabot[bot],2021-09-14T13:10:38Z,2021-09-14T13:10:38Z,CONTRIBUTOR,Superseded by #1471.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",982780906,Bump black from 21.7b0 to 21.8b0,
https://github.com/simonw/datasette/issues/1464#issuecomment-918621705,https://api.github.com/repos/simonw/datasette/issues/1464,918621705,IC_kwDOBm6k_c42wQ4J,7476523,bobwhitelock,2021-09-13T22:17:17Z,2021-09-13T22:17:17Z,CONTRIBUTOR,"> haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?
Not certain, but I think tests in CI run on Ubuntu and don't appear to install any additional Sqlite-related dependencies, and so my guess is the version of Sqlite installed by default on Ubuntu has the `SQLITE_ENABLE_FTS3_PARENTHESIS` option enabled and so doesn't run into this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/datasette/issues/1464#issuecomment-917642487,https://api.github.com/repos/simonw/datasette/issues/1464,917642487,IC_kwDOBm6k_c42shz3,51016,ctb,2021-09-12T14:03:09Z,2021-09-12T14:03:09Z,CONTRIBUTOR,"haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/datasette/issues/1464#issuecomment-915343886,https://api.github.com/repos/simonw/datasette/issues/1464,915343886,IC_kwDOBm6k_c42jwoO,7476523,bobwhitelock,2021-09-08T15:32:06Z,2021-09-08T15:32:06Z,CONTRIBUTOR,"Thanks, that does look similar!
> Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.
Ah that makes sense, I guess that's why this isn't just always installed already. I wonder if a possible solution to this issue could be doing feature detection on whether this feature is supported by the current Sqlite version, and if not these tests could be disabled locally? But possibly there's a better way to handle this, will see what @simonw thinks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/datasette/issues/1464#issuecomment-915302885,https://api.github.com/repos/simonw/datasette/issues/1464,915302885,IC_kwDOBm6k_c42jmnl,51016,ctb,2021-09-08T14:44:50Z,2021-09-08T14:44:50Z,CONTRIBUTOR,"thanks for the response! full errors attached; excerpt:
```
...
def test_searchmode(table_metadata, querystring, expected_rows):
with make_app_client(
metadata={""databases"": {""fixtures"": {""tables"": {""searchable"": table_metadata}}}}
) as client:
response = client.get(""/fixtures/searchable.json?"" + querystring)
> assert expected_rows == response.json[""rows""]
E AssertionError: assert [[1, 'barry c...sel', 'puma']] == []
E Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']
E Use -v to get the full diff
/Users/t/dev/datasette/tests/test_api.py:1115: AssertionError
```
[errors.txt](https://github.com/simonw/datasette/files/7129719/errors.txt)
A quick scan of #1223 suggests you're right. Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/datasette/issues/1464#issuecomment-915299013,https://api.github.com/repos/simonw/datasette/issues/1464,915299013,IC_kwDOBm6k_c42jlrF,7476523,bobwhitelock,2021-09-08T14:40:28Z,2021-09-08T14:40:28Z,CONTRIBUTOR,"What are the full errors you're getting?
This *may* be the same issue as described in https://github.com/simonw/datasette/pull/1223 - essentially the test suite (and corresponding Datasette features I assume) are by default implicitly dependent on your Sqlite installation having been compiled with the `SQLITE_ENABLE_FTS3_PARENTHESIS` option. If this is the same issue then I think this can be fixed either by recompiling with that option or (probably more easily) by running `pip install pysqlite3-binary`, which will be used in preference to your system Sqlite installation and has this option enabled.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/datasette/issues/1464#issuecomment-915279711,https://api.github.com/repos/simonw/datasette/issues/1464,915279711,IC_kwDOBm6k_c42jg9f,51016,ctb,2021-09-08T14:16:49Z,2021-09-08T14:16:49Z,CONTRIBUTOR,on commit d57ab156b35ec642,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/datasette/pull/1455#issuecomment-913001282,https://api.github.com/repos/simonw/datasette/issues/1455,913001282,IC_kwDOBm6k_c42a0tC,51016,ctb,2021-09-04T16:31:24Z,2021-09-04T16:31:24Z,CONTRIBUTOR,I love it! maybe 'researchers' instead? Or 'scientists and researchers'?,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988325628,Add scientists to target groups,
https://github.com/simonw/datasette/issues/1473#issuecomment-922394999,https://api.github.com/repos/simonw/datasette/issues/1473,922394999,IC_kwDOBm6k_c42-qF3,192568,mroswell,2021-09-19T00:44:39Z,2021-09-19T00:45:32Z,CONTRIBUTOR,"I replaced:
```
![]()
```
with:
```
![]()
```
I'd still love to know what caused this (and how to troubleshoot to figure it out), so I'll leave it open for a bit, but I do have a functional logo linking to the Hugo home page, at least locally. I'll likely push tomorrow.
(Before trying this, I tried to apply a background image to the `a` tag. That didn't work.)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",999902754,base logo link visits `undefined` rather than href url,
https://github.com/simonw/datasette/issues/1473#issuecomment-922363640,https://api.github.com/repos/simonw/datasette/issues/1473,922363640,IC_kwDOBm6k_c42-ib4,192568,mroswell,2021-09-18T19:45:47Z,2021-09-18T19:45:47Z,CONTRIBUTOR,"An update, if I remove the `img` tag and replace it with the text, ""Safer or Toxic?"" it links to the right place.
Also, if I keep things exactly as they are, and it improperly, but consistently goes to the `undefined` page, on THAT 404 page, a click on the image properly clicks through to the www.SaferOrToxic.org page.
Weird stuff.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",999902754,base logo link visits `undefined` rather than href url,
https://github.com/simonw/datasette/issues/859#issuecomment-905904540,https://api.github.com/repos/simonw/datasette/issues/859,905904540,IC_kwDOBm6k_c41_wGc,2670795,brandonrobertz,2021-08-25T21:59:14Z,2021-08-25T21:59:55Z,CONTRIBUTOR,"I did two tests: one with 1000 5-30mb DBs and a second with 20 multi gig DBs. For the second, I created them like so:
`for i in {1..20}; do sqlite-generate db$i.db --tables ${i}00 --rows 100,2000 --columns 5,100 --pks 0 --fks 0; done`
This was for deciding whether to use lots of small DBs or to group things into a smaller number of bigger DBs. The second strategy wins.
By simply persisting the `_internal` DB to disk, I was able to avoid most of the performance issues I was experiencing previously. (To do this, I changed the `datasette/internal_db.py:init_internal_db` creates to if not exists, and changed the `_internal` DB instantiation in `datasette/app.py:Datasette.__init__` to a path with `is_mutable=True`.) Super rough, but the pages now load so I can continue testing ideas.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-905899177,https://api.github.com/repos/simonw/datasette/issues/859,905899177,IC_kwDOBm6k_c41_uyp,2670795,brandonrobertz,2021-08-25T21:48:00Z,2021-08-25T21:48:00Z,CONTRIBUTOR,"Upon first stab, there's two issues here:
- DB/table/row counts (as discussed above). This isn't too bad if the DBs are actually above the MAX limit check.
- Populating the internal DB. On first load of a giant set of DBs, it can take 10-20 mins to populate. By altering datasette and persisting the internal DB to disk, this problem is vastly improved, but I'm sure this will cause problems elsewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/1425#issuecomment-895003796,https://api.github.com/repos/simonw/datasette/issues/1425,895003796,IC_kwDOBm6k_c41WKyU,3243482,abdusco,2021-08-09T07:14:35Z,2021-08-09T07:14:35Z,CONTRIBUTOR,"I believe this also provides a workaround for the problem I face in https://github.com/simonw/datasette/issues/1300.
Now I should be able to get table PKs and generate a row URL. I'll test this out and report my findings.
```py
from datasette.utils import path_from_row_pks
pks = await db.primary_keys(table)
url = self.ds.urls.row_blob(
database,
table,
path_from_row_pks(row, pks, not pks),
column,
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1419#issuecomment-893114612,https://api.github.com/repos/simonw/datasette/issues/1419,893114612,IC_kwDOBm6k_c41O9j0,536941,fgregg,2021-08-05T02:29:06Z,2021-08-05T02:29:06Z,CONTRIBUTOR,"there's a lot of complexity here, that's probably not worth addressing. i got what i needed by patching the dockerfile that cloudrun uses to install a newer version of sqlite.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959710008,`publish cloudrun` should deploy a more recent SQLite version,
https://github.com/simonw/datasette/issues/1419#issuecomment-892276385,https://api.github.com/repos/simonw/datasette/issues/1419,892276385,IC_kwDOBm6k_c41Lw6h,536941,fgregg,2021-08-04T00:58:49Z,2021-08-04T00:58:49Z,CONTRIBUTOR,"yes, [filter clause on aggregate queries were added to sqlite3 in 3.30](https://www.sqlite.org/releaselog/3_30_1.html)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959710008,`publish cloudrun` should deploy a more recent SQLite version,
https://github.com/simonw/datasette/issues/859#issuecomment-904982056,https://api.github.com/repos/simonw/datasette/issues/859,904982056,IC_kwDOBm6k_c418O4o,2670795,brandonrobertz,2021-08-24T21:15:04Z,2021-08-24T21:15:30Z,CONTRIBUTOR,"I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/1401#issuecomment-884910320,https://api.github.com/repos/simonw/datasette/issues/1401,884910320,IC_kwDOBm6k_c40vqjw,536941,fgregg,2021-07-22T13:26:01Z,2021-07-22T13:26:01Z,CONTRIBUTOR,"ordered lists didn't work either, btw","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",950664971,unordered list is not rendering bullet points in description_html on database page,
https://github.com/simonw/datasette/issues/1641#issuecomment-1049879118,https://api.github.com/repos/simonw/datasette/issues/1641,1049879118,IC_kwDOBm6k_c4-k-JO,536941,fgregg,2022-02-24T13:49:26Z,2022-02-24T13:49:26Z,CONTRIBUTOR,"maybe worth considering adding buttons for paren, asterisk, etc. under the input text box on mobile?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149310456,Tweak mobile keyboard settings,
|