html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/1884#issuecomment-1313962183,https://api.github.com/repos/simonw/datasette/issues/1884,1313962183,IC_kwDOBm6k_c5OUXjH,25778,2022-11-14T15:46:32Z,2022-11-14T15:46:32Z,CONTRIBUTOR,"It does work, though I think it's probably still worth excluding virtual tables that will always be zero. Here's the same inspection as before, now with `--load-extension spatialite`:
```json
{
""alltheplaces"": {
""hash"": ""0843cfe414439ab903c22d1121b7ddbc643418c35c7f0edbcec82ef1452411df"",
""size"": 963375104,
""file"": ""alltheplaces.db"",
""tables"": {
""spatial_ref_sys"": {
""count"": 6215
},
""spatialite_history"": {
""count"": 18
},
""sqlite_sequence"": {
""count"": 2
},
""geometry_columns"": {
""count"": 3
},
""spatial_ref_sys_aux"": {
""count"": 6164
},
""views_geometry_columns"": {
""count"": 0
},
""virts_geometry_columns"": {
""count"": 0
},
""geometry_columns_statistics"": {
""count"": 3
},
""views_geometry_columns_statistics"": {
""count"": 0
},
""virts_geometry_columns_statistics"": {
""count"": 0
},
""geometry_columns_field_infos"": {
""count"": 0
},
""views_geometry_columns_field_infos"": {
""count"": 0
},
""virts_geometry_columns_field_infos"": {
""count"": 0
},
""geometry_columns_time"": {
""count"": 3
},
""geometry_columns_auth"": {
""count"": 3
},
""views_geometry_columns_auth"": {
""count"": 0
},
""virts_geometry_columns_auth"": {
""count"": 0
},
""data_licenses"": {
""count"": 10
},
""sql_statements_log"": {
""count"": 0
},
""states"": {
""count"": 56
},
""counties"": {
""count"": 3234
},
""idx_states_geometry_rowid"": {
""count"": 56
},
""idx_states_geometry_node"": {
""count"": 3
},
""idx_states_geometry_parent"": {
""count"": 2
},
""idx_counties_geometry_rowid"": {
""count"": 3234
},
""idx_counties_geometry_node"": {
""count"": 98
},
""idx_counties_geometry_parent"": {
""count"": 97
},
""idx_places_geometry_rowid"": {
""count"": 1236796
},
""idx_places_geometry_node"": {
""count"": 38163
},
""idx_places_geometry_parent"": {
""count"": 38162
},
""places"": {
""count"": 1332609
},
""SpatialIndex"": {
""count"": 0
},
""ElementaryGeometries"": {
""count"": 0
},
""KNN"": {
""count"": 0
},
""idx_states_geometry"": {
""count"": 56
},
""idx_counties_geometry"": {
""count"": 3234
},
""idx_places_geometry"": {
""count"": 1236796
}
}
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,
https://github.com/simonw/datasette/issues/1886#issuecomment-1313271719,https://api.github.com/repos/simonw/datasette/issues/1886,1313271719,IC_kwDOBm6k_c5ORu-n,124274,2022-11-14T08:25:12Z,2022-11-14T08:25:12Z,NONE,"Nothing spectacular yet but I think this falls under ""cool/cute application of datasette"": [improving fakedata performance for fun](https://lucapette.me/writing/improving-fakedata-performance-for-fun/). tl;dr I used datasette to visualize benchmarking data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,
https://github.com/simonw/datasette/issues/1886#issuecomment-1313252879,https://api.github.com/repos/simonw/datasette/issues/1886,1313252879,IC_kwDOBm6k_c5ORqYP,883348,2022-11-14T08:10:23Z,2022-11-14T08:10:23Z,CONTRIBUTOR,"Hi @simonw and thanks for the great tools you're publishing, your dedication is inspiring!
I work for the French Ministry of Culture on a surveying tool for objects protected for their historical value. It is part of a program building modern public services called [beta.gouv.fr](https://beta.gouv.fr/).
In that context I'm using data published by the Ministry that I have ingested into datasette and published on a free Fly instance : https://collectif-objets-datasette.fly.dev . I have also ingested another data set with infos about french cities on this instance so that I can perform joined queries.
The surveying tool synchronizes its data regularly from this datasette instance, and I also use it to perform queries when asked generic questions about the distribution of objects. (The data is not very accessible as it's undocumented and for internal usage mostly)","{""total_count"": 3, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 3, ""rocket"": 0, ""eyes"": 0}",1447050738,
https://github.com/simonw/datasette/issues/1850#issuecomment-1313156167,https://api.github.com/repos/simonw/datasette/issues/1850,1313156167,IC_kwDOBm6k_c5ORSxH,9599,2022-11-14T06:23:39Z,2022-11-14T06:23:39Z,OWNER,The API explorer is now live here: https://latest-1-0-dev.datasette.io/-/api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,
https://github.com/simonw/datasette/issues/1850#issuecomment-1313155712,https://api.github.com/repos/simonw/datasette/issues/1850,1313155712,IC_kwDOBm6k_c5ORSqA,9599,2022-11-14T06:22:57Z,2022-11-14T06:22:57Z,OWNER,"I think the ability to create tokens should be protected by a `create-tokens` permission, not just a global setting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,
https://github.com/simonw/datasette/issues/1888#issuecomment-1313139657,https://api.github.com/repos/simonw/datasette/issues/1888,1313139657,IC_kwDOBm6k_c5OROvJ,9599,2022-11-14T06:04:48Z,2022-11-14T06:04:48Z,OWNER,Demo: https://latest-1-0-dev.datasette.io/-/api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447439985,
https://github.com/simonw/datasette/issues/1866#issuecomment-1313128913,https://api.github.com/repos/simonw/datasette/issues/1866,1313128913,IC_kwDOBm6k_c5ORMHR,9599,2022-11-14T05:48:22Z,2022-11-14T05:48:22Z,OWNER,"I changed my mind about the `""return_rows"": true` option - I'm going to rename it to `""return"": true`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,
https://github.com/simonw/datasette/issues/1874#issuecomment-1313127054,https://api.github.com/repos/simonw/datasette/issues/1874,1313127054,IC_kwDOBm6k_c5ORLqO,9599,2022-11-14T05:45:00Z,2022-11-14T05:45:00Z,OWNER,"Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=&method=POST
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,
https://github.com/simonw/datasette/issues/1871#issuecomment-1313125870,https://api.github.com/repos/simonw/datasette/issues/1871,1313125870,IC_kwDOBm6k_c5ORLXu,9599,2022-11-14T05:42:50Z,2022-11-14T05:42:50Z,OWNER,Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=%7B%22confirm%22%3A+true%7D&method=POST,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,
https://github.com/simonw/datasette/issues/1874#issuecomment-1313125123,https://api.github.com/repos/simonw/datasette/issues/1874,1313125123,IC_kwDOBm6k_c5ORLMD,9599,2022-11-14T05:41:20Z,2022-11-14T05:42:23Z,OWNER,"I also changed the confirmation JSON returned by this endpoint to add the `database` and `table` like so:
```json
{
""ok"": true,
""database"": ""data"",
""table"": ""docs"",
""row_count"": 1,
""message"": ""Pass \""confirm\"": true to confirm""
}
```
Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,
https://github.com/simonw/datasette/issues/1874#issuecomment-1313119558,https://api.github.com/repos/simonw/datasette/issues/1874,1313119558,IC_kwDOBm6k_c5ORJ1G,9599,2022-11-14T05:30:27Z,2022-11-14T05:30:27Z,OWNER,Found a bug: you get a 500 error if you try this against an immutable database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,
https://github.com/simonw/datasette/issues/1850#issuecomment-1313115059,https://api.github.com/repos/simonw/datasette/issues/1850,1313115059,IC_kwDOBm6k_c5ORIuz,9599,2022-11-14T05:21:30Z,2022-11-14T05:21:30Z,OWNER,New documentation for these features currently lives here: https://docs.datasette.io/en/1.0-dev/json_api.html#the-json-write-api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,
https://github.com/simonw/datasette/issues/1875#issuecomment-1313114283,https://api.github.com/repos/simonw/datasette/issues/1875,1313114283,IC_kwDOBm6k_c5ORIir,9599,2022-11-14T05:20:00Z,2022-11-14T05:20:00Z,OWNER,"I started a conversation about JSON error standards on Mastodon here: https://fedi.simonwillison.net/web/@simon/109338725610487457
Quite a few people pointed to this RFC independently.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,
https://github.com/simonw/datasette/issues/1887#issuecomment-1313113642,https://api.github.com/repos/simonw/datasette/issues/1887,1313113642,IC_kwDOBm6k_c5ORIYq,9599,2022-11-14T05:18:51Z,2022-11-14T05:18:51Z,OWNER,Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447388809,
https://github.com/simonw/datasette/issues/1887#issuecomment-1313097713,https://api.github.com/repos/simonw/datasette/issues/1887,1313097713,IC_kwDOBm6k_c5OREfx,9599,2022-11-14T05:00:54Z,2022-11-14T05:00:54Z,OWNER,"I'm going to add a `""confirm"": true` option to the API. Without that, it returns a note about how many rows will be deleted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447388809,
https://github.com/simonw/datasette/issues/1871#issuecomment-1313097057,https://api.github.com/repos/simonw/datasette/issues/1871,1313097057,IC_kwDOBm6k_c5OREVh,9599,2022-11-14T04:59:28Z,2022-11-14T04:59:28Z,OWNER,In playing with the API explorer just now I realized it's way too easy to accidentally drop a table using it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,
https://github.com/simonw/datasette/issues/1871#issuecomment-1313072900,https://api.github.com/repos/simonw/datasette/issues/1871,1313072900,IC_kwDOBm6k_c5OQ-cE,9599,2022-11-14T04:15:50Z,2022-11-14T04:15:50Z,OWNER,"For the example links - I'm going to have these at the bottom of the page so you don't have to scroll past them.
Ideally these would take the user's permissions into account. This could make the page expensive to load, but I'm going to risk it for the moment.
Something like this then:
> - data
> - /data/-/create - create table
> - /data/table1/-/insert - insert into table1
> - /data/table1/-/drop - drop table1
I won't bother with per-row demo links (for update and delete) because there could be thousands of them for each table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,
https://github.com/simonw/datasette/issues/1871#issuecomment-1313062699,https://api.github.com/repos/simonw/datasette/issues/1871,1313062699,IC_kwDOBm6k_c5OQ78r,9599,2022-11-14T04:03:29Z,2022-11-14T04:12:41Z,OWNER,"Two things left before I close this issue:
- [x] I want to preserve the state of the forms in the URL - probably after a `#`
- [ ] Instead of hard-coding the current examples, I want to provide a list of links which populate the forms","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,
https://github.com/simonw/datasette/issues/1886#issuecomment-1313052863,https://api.github.com/repos/simonw/datasette/issues/1886,1313052863,IC_kwDOBm6k_c5OQ5i_,9599,2022-11-14T03:40:50Z,2022-11-14T03:40:50Z,OWNER,"Tim Sherratt on Twitter: https://twitter.com/wragge/status/1591930345469153282
> Where do I start? The [#GLAMWorkbench](https://twitter.com/hashtag/GLAMWorkbench?src=hashtag_click) now includes a number of examples where GLAM data is harvested, processed, and then made available for exploration via Datasette.
>
> https://glam-workbench.net/
>
> For example the GLAM Name Index Search brings together 10+ million entries from 240 indexes and provides an aggregated search using the Datasette search-all plugin:
>
> https://glam-workbench.net/name-search/
>
> Most recently I converted PDFs of the Tasmanian Postal Directories to a big Datasette instance: https://updates.timsherratt.org/2022/09/15/from-pdfs-to.html the process is documented and reusable.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1447050738,
https://github.com/simonw/datasette/issues/1886#issuecomment-1312898318,https://api.github.com/repos/simonw/datasette/issues/1886,1312898318,IC_kwDOBm6k_c5OQT0O,19851673,2022-11-14T00:52:16Z,2022-11-14T00:52:16Z,NONE,"I'm a cryptic crossword enthusiast and have spent a lot of time scraping and parsing cryptic crossword clues from various blogs, forums and publications. The result is over **half a million clues from cryptic crosswords over the past twelve years**, including the clue, answer, puzzle date, puzzle name and a link to the original source. This is all hosted using Datasette, which has been a delight to use: https://cryptics.georgeho.org/
This dataset is a significant work of crossword archivism and scholarship, as acquiring historical crosswords and structuring their contents require focused effort and tedious cleaning that few are willing to do for such trivial data - for example, according to [this 2004 selection guide](https://cryptics.georgeho.org/static/documents/Selection_AppendixE_v2.pdf), the Library of Congress explicitly does not collect crossword puzzles. Anecdotally, I know that many constructors/setters of cryptic crosswords use this dataset as a resource, and some even simply call it ""the database"" - this is probably one of the most impactful data projects I've worked on!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,
https://github.com/simonw/datasette/issues/1871#issuecomment-1312822353,https://api.github.com/repos/simonw/datasette/issues/1871,1312822353,IC_kwDOBm6k_c5OQBRR,9599,2022-11-13T21:07:40Z,2022-11-13T21:07:40Z,OWNER,I'm going to need extra code to toggle POST closed when GET opens and vice-versa.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,
https://github.com/simonw/datasette/issues/1871#issuecomment-1312821031,https://api.github.com/repos/simonw/datasette/issues/1871,1312821031,IC_kwDOBm6k_c5OQA8n,9599,2022-11-13T21:02:06Z,2022-11-13T21:03:11Z,OWNER,"Actually no, I'm going to add a class of `details-menu` to the other details elements that SHOULD be closed. That way custom templates using `` won't close in a surprising way.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,
https://github.com/simonw/datasette/issues/1871#issuecomment-1312816451,https://api.github.com/repos/simonw/datasette/issues/1871,1312816451,IC_kwDOBm6k_c5OP_1D,9599,2022-11-13T20:39:26Z,2022-11-13T20:39:34Z,OWNER,I'm going to add a special `no-auto-close` class to these and teach that code not to close them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,
https://github.com/simonw/datasette/issues/1871#issuecomment-1312816292,https://api.github.com/repos/simonw/datasette/issues/1871,1312816292,IC_kwDOBm6k_c5OP_yk,9599,2022-11-13T20:38:42Z,2022-11-13T20:38:42Z,OWNER,"The current API explorer uses details/summary elements for the GET and POST dialogs.
I only want one of these to be open at a time, to reflect that you can make either a GET or a POST.
I just noticed that clicking anywhere else on the page closes both elements, which isn't what I want to happen. Turns out that's because of this code I added as part of Datasette's menu implementation!
https://github.com/simonw/datasette/blob/9f54f00a50a4d950cfd69a0ff3526ae82c858826/datasette/templates/_close_open_menus.html#L2-L15","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,
https://github.com/simonw/datasette/issues/1886#issuecomment-1312814245,https://api.github.com/repos/simonw/datasette/issues/1886,1312814245,IC_kwDOBm6k_c5OP_Sl,2090382,2022-11-13T20:28:26Z,2022-11-13T20:28:26Z,NONE,"I work at The Wall Street Journal as a computational journalist and serve as our self-appointed Datasette evangelist. They say that to a hammer everything looks like a nail, but the reality is newsrooms find themselves in a sea of nails!
I've only got a couple public projects that I can share, but happy to offer you a look at some of the internal projects.
More often than not the internal projects stay internal because the reporting doesn't lead anywhere or I can't convince an editor to greenlight it. But imho that's the beauty of datasette: a (relatively) painless mechanism to see if there's any there there.
- [WSJ Inflation Tracker](wsj.com/inflationtracker)
- I scraped the oscars website and turned it into a datasette instance and ran the numbers on [best actress/best picture overlap ](https://www.wsj.com/livecoverage/oscars-academy-awards-2022/card/the-best-actress-nominees-aren-t-in-any-best-pictures-contenders-when-is-the-last-time-that-happened--mDxvbLug3rq84pxLE8gY)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,
https://github.com/simonw/datasette/issues/1882#issuecomment-1312582512,https://api.github.com/repos/simonw/datasette/issues/1882,1312582512,IC_kwDOBm6k_c5OPGtw,9599,2022-11-12T22:11:18Z,2022-11-12T22:11:18Z,OWNER,"I like this:
```json
{
""ok"": true,
""database"": ""data"",
""table"": ""agai2n"",
""table_url"": ""http://127.0.0.1:8001/data/agai2n"",
""schema"": ""CREATE TABLE [agai2n] (\n [hello] INTEGER\n)""
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,
https://github.com/simonw/datasette/issues/1882#issuecomment-1312581121,https://api.github.com/repos/simonw/datasette/issues/1882,1312581121,IC_kwDOBm6k_c5OPGYB,9599,2022-11-12T22:01:32Z,2022-11-12T22:01:32Z,OWNER,I'm going to change it to `table` in the output AND the input.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,
https://github.com/simonw/datasette/issues/1882#issuecomment-1312581008,https://api.github.com/repos/simonw/datasette/issues/1882,1312581008,IC_kwDOBm6k_c5OPGWQ,9599,2022-11-12T22:00:52Z,2022-11-12T22:00:52Z,OWNER,"Tried out my prototype in the API explorer:
The `""name""` on the output is bothering me a bit - should it be `table_name` or `table` instead?
Problem is I really like `name` for the input, so should it be consistent to have the same name on the output here, or should I aim for consistency with other endpoints that might return `table` rather than the ambiguous `name` elsewhere?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,
https://github.com/simonw/datasette/issues/1882#issuecomment-1312580348,https://api.github.com/repos/simonw/datasette/issues/1882,1312580348,IC_kwDOBm6k_c5OPGL8,9599,2022-11-12T21:55:54Z,2022-11-12T21:56:45Z,OWNER,"What should this API return?
I think the name of the table (`name`), the URL to that table (`table_url` - for consistency with how faceting API works already) and the schema of the table (`schema`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,
https://github.com/simonw/datasette/issues/1882#issuecomment-1312575048,https://api.github.com/repos/simonw/datasette/issues/1882,1312575048,IC_kwDOBm6k_c5OPE5I,9599,2022-11-12T21:22:58Z,2022-11-12T21:22:58Z,OWNER,Need to validate the table name. SQLite supports almost any table name - but they can't contain a newline character and cannot start with `sqlite_` - according to https://stackoverflow.com/questions/3694276/what-are-valid-table-names-in-sqlite/43049720#43049720,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,
https://github.com/simonw/datasette/issues/1882#issuecomment-1312556044,https://api.github.com/repos/simonw/datasette/issues/1882,1312556044,IC_kwDOBm6k_c5OPAQM,9599,2022-11-12T19:29:11Z,2022-11-12T19:29:11Z,OWNER,"Thought of an edge-case: with `sqlite-utils` one feature I really like is that I can pipe data into it without caring if the table already exists or not:
cat data.json | sqlite-utils insert my.db mytable -
How could this new API support that?
I thought about adding a `""create"": true` option to `/db/table/-/insert` which creates the table if it doesn't already exist, but if I do that I'll need to start adding other options to that endpoint - to set the primary key, add foreign keys and suchlike - which would be ignored except for the cases where the table was being created from scratch.
This doesn't feel right to me - I want to keep those options here, on `/db/-/create`.
One idea I had was to implement it such that you can call `/db/-/create` multiple times for the same table, but only if you are using the `""rows""` option. If so, and if the rows can be safely inserted, it would let you do that.
But instead, I'm going to outsource this to the CLI tool I plan to write that feeds data into this API. I'm already planning to use that tool for CSV inserts (so the API doesn't need to accept CSV directly). I think it's a good place for other usability enhancements like ""insert this, creating the table if it does not exist"" as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,
https://github.com/simonw/datasette/issues/1878#issuecomment-1312534826,https://api.github.com/repos/simonw/datasette/issues/1878,1312534826,IC_kwDOBm6k_c5OO7Eq,18738650,2022-11-12T17:34:58Z,2022-11-12T17:34:58Z,NONE,"Hi Simon. I have just started experimenting with datasette in earnest, looking at it's suitability for air quality open data. A bulk upsert \ upsert_all would be very useful for me in enabling real time data to be pushed from a sql server database with FME server to a datasette db.
An hourly process queries the last 2 hours of data and pushes that to my database, inserting new data and updating existing combinations of pk siteid and date_time. This is already implemented on our current [open data portal](https://opendata.bristol.gov.uk/explore/dataset/air-quality-data-continuous/table/?disjunctive.location&sort=date_time). Excited to see your progress with this! Thank you for this amazing software.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311437901,https://api.github.com/repos/simonw/datasette/issues/1883,1311437901,IC_kwDOBm6k_c5OKvRN,31312775,2022-11-11T09:20:21Z,2022-11-11T09:20:21Z,NONE,Amazing - thank you for fixing and releasing that so quickly and for showing your process! <3 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311314981,https://api.github.com/repos/simonw/datasette/issues/1883,1311314981,IC_kwDOBm6k_c5OKRQl,9599,2022-11-11T07:15:48Z,2022-11-11T07:15:48Z,OWNER,I released that fix in Datasette 0.63.1: https://docs.datasette.io/en/stable/changelog.html#v0-63-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311299535,https://api.github.com/repos/simonw/datasette/issues/1883,1311299535,IC_kwDOBm6k_c5OKNfP,9599,2022-11-11T06:54:58Z,2022-11-11T06:54:58Z,OWNER,"This time deployed with:
```
cd demos/apache-proxy
fly deploy --build-arg DATASETTE_REF=8d9a957c6329d26cc1e417b5d6911640d74765eb
```
To ensure the exact commit with the fix.
And that fixed it!
```
% curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid'
HTTP/2 302
date: Fri, 11 Nov 2022 06:54:45 GMT
server: Fly/b1863e2e7 (2022-11-09)
location: /prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1
link: ; rel=preload
content-type: text/plain
x-proxied-by: Apache2 Debian
via: 2 fly.io
fly-request-id: 01GHJQGBSXBR7E53TY0EKMQ9PA-sjc
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311292463,https://api.github.com/repos/simonw/datasette/issues/1883,1311292463,IC_kwDOBm6k_c5OKLwv,9599,2022-11-11T06:44:24Z,2022-11-11T06:44:24Z,OWNER,"Modifying that test to the following does indeed cause a failure:
```python
def test_base_url_affects_filter_redirects(app_client_base_url_prefix):
response = app_client_base_url_prefix.get(
""/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid""
)
assert response.status == 302
assert (
response.headers[""location""]
== ""/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1""
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311291632,https://api.github.com/repos/simonw/datasette/issues/1883,1311291632,IC_kwDOBm6k_c5OKLjw,9599,2022-11-11T06:43:00Z,2022-11-11T06:43:00Z,OWNER,"https://datasette-apache-proxy-demo.datasette.io/prefix/-/asgi-scope is useful:
It confirms that `/prefix/` is nowhere to be seen in the incoming request data:
```
'path': '/-/asgi-scope',
'query_string': b'',
'raw_path': b'/-/asgi-scope',
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311290115,https://api.github.com/repos/simonw/datasette/issues/1883,1311290115,IC_kwDOBm6k_c5OKLMD,9599,2022-11-11T06:40:14Z,2022-11-11T06:41:56Z,OWNER,"I modified that config file to have this line instead:
```
ProxyPass /prefix/ http://127.0.0.1:8001/ nocanon
```
And then deployed it by running:
flyctl deploy --build-arg DATASETTE_REF=main
This does NOT seem to have fixed the bug:
```
~ % curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid'
HTTP/2 302
date: Fri, 11 Nov 2022 06:40:01 GMT
server: Fly/b1863e2e7 (2022-11-09)
location: /fixtures/binary_data?_sort=rowid&rowid__exact=1
link: ; rel=preload
content-type: text/plain
x-proxied-by: Apache2 Debian
via: 2 fly.io
fly-request-id: 01GHJPNCF51CJ626EWZEHK2CH9-sjc
```
https://datasette-apache-proxy-demo.datasette.io/prefix/-/versions seems to confirm that this is the latest deployed version (0.63), so it looks like the deploy worked.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311286593,https://api.github.com/repos/simonw/datasette/issues/1883,1311286593,IC_kwDOBm6k_c5OKKVB,9599,2022-11-11T06:34:09Z,2022-11-11T06:34:09Z,OWNER,"https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass includes this note:
> Normally, mod_proxy will canonicalise ProxyPassed URLs. But this may be incompatible with some backends, particularly those that make use of *PATH_INFO*. The optional *nocanon* keyword suppresses this and passes the URL path ""raw"" to the backend.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311284537,https://api.github.com/repos/simonw/datasette/issues/1883,1311284537,IC_kwDOBm6k_c5OKJ05,9599,2022-11-11T06:30:38Z,2022-11-11T06:30:38Z,OWNER,"Is there a chance that it's Apache that's messing with that `location:` header here, not Datasette?
https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/demos/apache-proxy/000-default.conf#L7-L13","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311283301,https://api.github.com/repos/simonw/datasette/issues/1883,1311283301,IC_kwDOBm6k_c5OKJhl,9599,2022-11-11T06:28:38Z,2022-11-11T06:29:33Z,OWNER,"`path_with_added_args(request, redirect_params)` should be preserving the current path from the request.
https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/utils/__init__.py#L273-L286","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311282970,https://api.github.com/repos/simonw/datasette/issues/1883,1311282970,IC_kwDOBm6k_c5OKJca,9599,2022-11-11T06:28:05Z,2022-11-11T06:28:05Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/views/table.py#L227-L249,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311280709,https://api.github.com/repos/simonw/datasette/issues/1883,1311280709,IC_kwDOBm6k_c5OKI5F,9599,2022-11-11T06:25:27Z,2022-11-11T06:25:27Z,OWNER,"I tried adding this test but it passed! I expected it to fail:
```python
def test_base_url_affects_filter_redirects(app_client_base_url_prefix):
response = app_client_base_url_prefix.get(
""/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid""
)
assert response.status == 302
assert (
response.headers[""location""]
== ""/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1""
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,
https://github.com/simonw/datasette/issues/1883#issuecomment-1311278678,https://api.github.com/repos/simonw/datasette/issues/1883,1311278678,IC_kwDOBm6k_c5OKIZW,9599,2022-11-11T06:22:37Z,2022-11-11T06:22:37Z,OWNER,"If you view source on that page the HTML looks correct:
```html