github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906646452 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 | 906646452 | IC_kwDOEhK-wc42ClO0 | 9599 | 2021-08-26T18:34:34Z | 2021-08-26T18:35:20Z | MEMBER | I tried this ampersand fix: https://regex101.com/r/ojU2H9/1 ```python # https://regex101.com/r/ojU2H9/1 _invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)') def fix_bad_xml(xml): # More fixes for things like '&' not as part of an entity return _invalid_ampersand_re.sub('&', xml) ``` Even with that I'm still getting total garbage in the `<en-note>` content - it's just HTML, not even trying to be XML. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
978743426 | |
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906635938 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 | 906635938 | IC_kwDOEhK-wc42Ciqi | 9599 | 2021-08-26T18:18:27Z | 2021-08-26T18:18:27Z | MEMBER | It looks like I was using the round-trip to dump the `<?xml version="1.0" encoding="UTF-8" standalone="no"?>` and `<!DOCTYPE` prefixes. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
978743426 | |
https://github.com/simonw/datasette/issues/859#issuecomment-905900807 | https://api.github.com/repos/simonw/datasette/issues/859 | 905900807 | IC_kwDOBm6k_c41_vMH | 9599 | 2021-08-25T21:51:10Z | 2021-08-25T21:51:10Z | OWNER | 10-20 minutes to populate `_internal`! How many databases and tables is that for? I may have to rethink the `_internal` mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed. I need to get my own stress testing rig setup for this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
642572841 | |
https://github.com/simonw/sqlite-utils/issues/323#issuecomment-905886797 | https://api.github.com/repos/simonw/sqlite-utils/issues/323 | 905886797 | IC_kwDOCGYnMM41_rxN | 9599 | 2021-08-25T21:25:18Z | 2021-08-25T21:25:18Z | OWNER | As far as I can tell the Python `sqlite3` module doesn't actually have a mechanism for de-registering a custom SQL function. This means that if I implement a mechanism whereby each call to `.convert()` registers a new SQL function with a random suffix (`convert_value_23424()` for example) those functions will stay registered - and if `.convert()` is called a large number of times the number of obsolete custom function registrations will grow without bounds. For that reason, I'm going to `wontfix` this issue. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
979627285 | |
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905206234 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 | 905206234 | IC_kwDOEhK-wc419Fna | 9599 | 2021-08-25T05:58:42Z | 2021-08-25T05:58:42Z | MEMBER | https://github.com/dogsheep/evernote-to-sqlite/blob/36a466f142e5bad52719851c2fbda0c05cd35b99/evernote_to_sqlite/utils.py#L34-L42 Not sure why I was round-tripping the `content_xml` like that - I will try not doing that. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
978743426 | |
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905203570 | https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 | 905203570 | IC_kwDOEhK-wc419E9y | 9599 | 2021-08-25T05:51:22Z | 2021-08-25T05:53:27Z | MEMBER | The debugger showed me that it broke on a string that looked like this: ```xml <?xml version="1.0" encoding="UTF-8" standalone="no"?> <!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd"> <en-note> <h1 title="Q3 2018 Reflection & Development"> <span title=Q3 2018 Reflection & Development"> Q3 2018 Reflection & Development </span> </h1> ... ``` Yeah that is not valid XML! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
978743426 | |
https://github.com/simonw/datasette/pull/1447#issuecomment-905097468 | https://api.github.com/repos/simonw/datasette/issues/1447 | 905097468 | IC_kwDOBm6k_c418rD8 | 9599 | 2021-08-25T01:28:53Z | 2021-08-25T01:28:53Z | OWNER | Good catch, thanks! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
978614898 | |
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905043974 | https://api.github.com/repos/simonw/sqlite-utils/issues/319 | 905043974 | IC_kwDOCGYnMM418eAG | 9599 | 2021-08-24T23:33:44Z | 2021-08-24T23:33:44Z | OWNER | Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-data-from-files | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
976399638 | |
https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040902 | https://api.github.com/repos/simonw/sqlite-utils/issues/321 | 905040902 | IC_kwDOCGYnMM418dQG | 9599 | 2021-08-24T23:25:03Z | 2021-08-24T23:25:03Z | OWNER | I'm going to skip this test on windows. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
978537855 | |
https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040307 | https://api.github.com/repos/simonw/sqlite-utils/issues/321 | 905040307 | IC_kwDOCGYnMM418dGz | 9599 | 2021-08-24T23:23:36Z | 2021-08-24T23:23:36Z | OWNER | https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says: > Currently, `TextIOWrapper` uses `locale.getpreferredencoding(False)` (hereinafter called “locale encoding”) when encoding is not specified. > ... > Package authors using macOS or Linux may forget that the default encoding is not always UTF-8. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
978537855 | |
https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905039576 | https://api.github.com/repos/simonw/sqlite-utils/issues/321 | 905039576 | IC_kwDOCGYnMM418c7Y | 9599 | 2021-08-24T23:21:29Z | 2021-08-24T23:21:29Z | OWNER | Hah, the error here is actually: ``` > assert result.exit_code == 1, result.output E AssertionError: E E assert 0 == 1 E + where 0 = <Result okay>.exit_code ``` So I was expecting an error, but instead the command worked. I suspect this is because on Windows the default character set may not be UTF-8? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
978537855 | |
https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905037323 | https://api.github.com/repos/simonw/sqlite-utils/issues/321 | 905037323 | IC_kwDOCGYnMM418cYL | 9599 | 2021-08-24T23:15:29Z | 2021-08-24T23:15:29Z | OWNER | Huh, tests are failing but only on Windows! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
978537855 | |
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021933 | https://api.github.com/repos/simonw/sqlite-utils/issues/319 | 905021933 | IC_kwDOCGYnMM418Ynt | 9599 | 2021-08-24T22:36:04Z | 2021-08-24T22:36:04Z | OWNER | > Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode. I thought about supporting those different policies (with something like `--errors ignore`) but I feel like that's getting a little bit too deep into the weeds. Right now if you try to import an invalid file the behaviour is the same as for the `sqlite-utils insert` command (I added the same detailed error message): ``` Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text 'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte The input you provided uses a character encoding other than utf-8. You can fix this by passing the --encoding= option with the encoding of the file. If you do not know the encoding, running 'file filename.csv' may tell you. It's often worth trying: --encoding=latin-1 ``` If someone has data that can't be translated to valid text using a known encoding, I'm happy leaving them to have to insert it into a `BLOB` column instead. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
976399638 | |
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021047 | https://api.github.com/repos/simonw/sqlite-utils/issues/319 | 905021047 | IC_kwDOCGYnMM418YZ3 | 9599 | 2021-08-24T22:33:48Z | 2021-08-24T22:33:48Z | OWNER | I had a few doubts about the design just now. Since `content_text` is supported as a special argument, an alternative way of handling the above would be: sqlite-utils insert-files /tmp/text.db files *.txt -c path -c content_text -c size This does exactly the same thing as just using `--text` and not specifying any columns, because the actual implementation of `--text` is as follows: https://github.com/simonw/sqlite-utils/blob/0c796cd945b146b7395ff5f553861400be503867/sqlite_utils/cli.py#L1851-L1855 But actually I think that's OK - ``--text`` is a useful shorthand that avoids you having to remember how to manually specify those columns with `-c`. So I'm going to leave the design as is. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
976399638 | |
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013183 | https://api.github.com/repos/simonw/sqlite-utils/issues/319 | 905013183 | IC_kwDOCGYnMM418We_ | 9599 | 2021-08-24T22:15:34Z | 2021-08-24T22:15:34Z | OWNER | Here's the error message I have working for invalid unicode: ``` sqlite-utils insert-files /tmp/text.db files *.txt --text [------------------------------------] 0% Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text 'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
976399638 | |
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013162 | https://api.github.com/repos/simonw/sqlite-utils/issues/319 | 905013162 | IC_kwDOCGYnMM418Weq | 9599 | 2021-08-24T22:15:31Z | 2021-08-24T22:15:31Z | OWNER | I'm going to assume utf-8 but allow `--encoding` to be used to specify something different, since that option is already supported by other commands. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
976399638 | |
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905001586 | https://api.github.com/repos/simonw/sqlite-utils/issues/319 | 905001586 | IC_kwDOCGYnMM418Tpy | 9599 | 2021-08-24T21:52:50Z | 2021-08-24T21:52:50Z | OWNER | Will need to re-title this section of the documentation: https://sqlite-utils.datasette.io/en/3.16/cli.html#inserting-binary-data-from-files - "Inserting binary data from files" will become "Inserting data from files" I'm OK with keeping the default as `BLOB` but I could add a `--text` option which stores the content as text instead. If the text can't be stored as `utf-8` I'll probably raise an error. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
976399638 | |
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-904999850 | https://api.github.com/repos/simonw/sqlite-utils/issues/319 | 904999850 | IC_kwDOCGYnMM418TOq | 9599 | 2021-08-24T21:49:08Z | 2021-08-24T21:49:08Z | OWNER | This is a good idea. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
976399638 | |
https://github.com/simonw/datasette/issues/1446#issuecomment-904954530 | https://api.github.com/repos/simonw/datasette/issues/1446 | 904954530 | IC_kwDOBm6k_c418IKi | 9599 | 2021-08-24T20:32:47Z | 2021-08-24T20:32:47Z | OWNER | Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
978357984 | |
https://github.com/simonw/datasette/issues/1446#issuecomment-904866495 | https://api.github.com/repos/simonw/datasette/issues/1446 | 904866495 | IC_kwDOBm6k_c417yq_ | 9599 | 2021-08-24T18:13:49Z | 2021-08-24T18:13:49Z | OWNER | OK, now the following optional CSS gives us a sticky footer: ```css html, body { height: 100%; } body { display: flex; flex-direction: column; } .not-footer { flex: 1 0 auto; } footer { flex-shrink: 0; } ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
978357984 | |
https://github.com/simonw/datasette/issues/1445#issuecomment-904037087 | https://api.github.com/repos/simonw/datasette/issues/1445 | 904037087 | IC_kwDOBm6k_c414oLf | 9599 | 2021-08-23T19:10:17Z | 2021-08-23T19:10:17Z | OWNER | Rather than trying to run that monstrosity in a single `union all` query, a better approach may be to use `fetch()` requests as seen in https://datasette.io/plugins/datasette-search-all | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
977323133 | |
https://github.com/simonw/datasette/issues/1445#issuecomment-904036200 | https://api.github.com/repos/simonw/datasette/issues/1445 | 904036200 | IC_kwDOBm6k_c414n9o | 9599 | 2021-08-23T19:08:54Z | 2021-08-23T19:08:54Z | OWNER | Figured out a query for searching across every column in every table! https://til.simonwillison.net/datasette/search-all-columns-trick#user-content-same-trick-for-the-entire-database ```sql with tables as ( select name as table_name from sqlite_master where type = 'table' ), queries as ( select 'select ''' || tables.table_name || ''' as _table, rowid from "' || tables.table_name || '" where ' || group_concat( '"' || name || '" like ''%'' || :search || ''%''', ' or ' ) as query from pragma_table_info(tables.table_name), tables group by tables.table_name ) select group_concat(query, ' union all ') from queries ``` The SQL query this generates for larger databases is _extremely_ long - but it does seem to work for smaller databases. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
977323133 | |
https://github.com/simonw/datasette/issues/1445#issuecomment-904027166 | https://api.github.com/repos/simonw/datasette/issues/1445 | 904027166 | IC_kwDOBm6k_c414lwe | 9599 | 2021-08-23T18:56:20Z | 2021-08-23T18:56:20Z | OWNER | A related but potentially even more useful ability would be running a search across every column of every table in a whole database. For anything less than a few 100MB this could be incredibly useful. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
977323133 | |
https://github.com/simonw/datasette/issues/1445#issuecomment-904026253 | https://api.github.com/repos/simonw/datasette/issues/1445 | 904026253 | IC_kwDOBm6k_c414liN | 9599 | 2021-08-23T18:54:49Z | 2021-08-23T18:54:49Z | OWNER | The bigger problem here is UI design. This feels like a pretty niche requirement to me, so adding a prominent search box to the table page (which already has the filters interface, plus the full-text search box for tables that have FTS configured) feels untidy. I could tuck it away in the table cog menu, but that's a weird place for something like this to live. Maybe add it as a new type of filter? Filters apply to specific columns though, so this would be the first filter that applied to _all_ columns - which doesn't really fit the existing filter interface very well. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
977323133 | |
https://github.com/simonw/datasette/issues/1445#issuecomment-904024939 | https://api.github.com/repos/simonw/datasette/issues/1445 | 904024939 | IC_kwDOBm6k_c414lNr | 9599 | 2021-08-23T18:52:35Z | 2021-08-23T18:52:35Z | OWNER | The downside of the current implementation of this trick is that it only works for exact LIKE partial matches in a specific table - if you search for `dog cat` and `dog` appears in `title` but `cat` appears in `description` you won't get back that result. I think that's fine though. If you want more advanced search there are other mechanisms you can use. This is meant to be a very quick and dirty starting point for exploring a brand new table. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
977323133 | |
https://github.com/simonw/sqlite-utils/issues/320#issuecomment-903288691 | https://api.github.com/repos/simonw/sqlite-utils/issues/320 | 903288691 | IC_kwDOCGYnMM411xdz | 9599 | 2021-08-22T15:46:56Z | 2021-08-22T15:46:56Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#schema-analyze-dump-and-save | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
976405225 | |
https://github.com/simonw/sqlite-utils/issues/320#issuecomment-903288430 | https://api.github.com/repos/simonw/sqlite-utils/issues/320 | 903288430 | IC_kwDOCGYnMM411xZu | 9599 | 2021-08-22T15:44:55Z | 2021-08-22T15:45:52Z | OWNER | ``` curl 'https://api.github.com/users/dogsheep/repos' | sqlite-utils memory - --analyze ``` ``` stdin.id: (1/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.node_id: (2/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.name: (3/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.full_name: (4/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.private: (5/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 1 Most common: 13: 0 stdin.owner: (6/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 1 Most common: 13: {"login": "dogsheep", "id": 53015001, "node_id": "MDEyOk9yZ2FuaXphdGlvbjUzMDE1MD... stdin.html_url: (7/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.description: (8/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.fork: (9/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 1 Most common: 13: 0 stdin.url: (10/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.forks_url: (11/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.keys_url: (12/73) Total rows: 13 ... ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
976405225 | |
https://github.com/simonw/datasette/issues/894#issuecomment-902375388 | https://api.github.com/repos/simonw/datasette/issues/894 | 902375388 | IC_kwDOBm6k_c41ySfc | 9599 | 2021-08-20T02:07:53Z | 2021-08-20T02:07:53Z | OWNER | I could add these sorting links to the cog menu for any `TEXT` columns. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
657572753 | |
https://github.com/simonw/datasette/issues/894#issuecomment-902375088 | https://api.github.com/repos/simonw/datasette/issues/894 | 902375088 | IC_kwDOBm6k_c41ySaw | 9599 | 2021-08-20T02:07:13Z | 2021-08-20T02:07:26Z | OWNER | Maybe `?_sort_numeric=col` and `?_sort_numeric_desc=col` would be better here. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
657572753 | |
https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902356871 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20 | 902356871 | IC_kwDOC8tyDs41yN-H | 9599 | 2021-08-20T01:12:48Z | 2021-08-20T01:12:48Z | MEMBER | Also on `workout_points.workout_id` to speed up queries to show all points in a specific workout. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
975166271 | |
https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902355471 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20 | 902355471 | IC_kwDOC8tyDs41yNoP | 9599 | 2021-08-20T01:09:07Z | 2021-08-20T01:09:07Z | MEMBER | Workaround: sqlite-utils create-index healthkit.db workout_points -- -date See https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
975166271 | |
https://github.com/dogsheep/twitter-to-sqlite/pull/49#issuecomment-902330301 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49 | 902330301 | IC_kwDODEm0Qs41yHe9 | 9599 | 2021-08-20T00:01:56Z | 2021-08-20T00:01:56Z | MEMBER | Thanks! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
681575714 | |
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329884 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57 | 902329884 | IC_kwDODEm0Qs41yHYc | 9599 | 2021-08-20T00:01:05Z | 2021-08-20T00:01:05Z | MEMBER | Maybe Click changed something which meant that this broke things when it didn't used to? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
907645813 | |
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329455 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57 | 902329455 | IC_kwDODEm0Qs41yHRv | 9599 | 2021-08-19T23:59:56Z | 2021-08-19T23:59:56Z | MEMBER | This looks like the bug to me: https://github.com/dogsheep/twitter-to-sqlite/blob/197e69cec40052c423a5ed071feb5f7cccea41b9/twitter_to_sqlite/cli.py#L239-L241 `type=str, default=False` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
907645813 | |
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328760 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57 | 902328760 | IC_kwDODEm0Qs41yHG4 | 9599 | 2021-08-19T23:57:41Z | 2021-08-19T23:57:41Z | MEMBER | Weird, added debug code and got this: `{'screen_name': 'simonw', 'count': 200, 'since_id': 'False', 'tweet_mode': 'extended'}` - so maybe it's a `twitter-to-sqlite` bug where somehow the string `False` is being passed somewhere. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
907645813 | |
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328369 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57 | 902328369 | IC_kwDODEm0Qs41yHAx | 9599 | 2021-08-19T23:56:26Z | 2021-08-19T23:56:26Z | MEMBER | https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-user_timeline says the API has been replaced by the new v2 one, but it should still work - and the `since_id` parameter is still documented on that page. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
907645813 | |
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902327457 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57 | 902327457 | IC_kwDODEm0Qs41yGyh | 9599 | 2021-08-19T23:53:25Z | 2021-08-19T23:53:25Z | MEMBER | I'm getting this too. Looking into it now. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
907645813 | |
https://github.com/simonw/datasette/issues/1426#issuecomment-902263367 | https://api.github.com/repos/simonw/datasette/issues/1426 | 902263367 | IC_kwDOBm6k_c41x3JH | 9599 | 2021-08-19T21:33:51Z | 2021-08-19T21:36:28Z | OWNER | I was worried about if it's possible to allow access to `/fixtures` but deny access to `/fixtures?sql=...` From various answers on Stack Overflow it looks like this should handle that: ``` User-agent: * Disallow: /fixtures? ``` I could use this for tables too - it may well be OK to access table index pages while still avoiding pagination, facets etc. I think this should block both query strings and row pages while allowing the table page itself: ``` User-agent: * Disallow: /fixtures/searchable? Disallow: /fixtures/searchable/* ``` Could even accompany that with a `sitemap.xml` that explicitly lists all of the tables - which would mean adding sitemaps to Datasette core too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
964322136 | |
https://github.com/simonw/datasette/issues/1426#issuecomment-902260338 | https://api.github.com/repos/simonw/datasette/issues/1426 | 902260338 | IC_kwDOBm6k_c41x2Zy | 9599 | 2021-08-19T21:28:25Z | 2021-08-19T21:29:40Z | OWNER | Actually it looks like you can send a `sitemap.xml` to Google using an unauthenticated GET request to: https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP According to https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
964322136 | |
https://github.com/simonw/datasette/issues/1426#issuecomment-902260799 | https://api.github.com/repos/simonw/datasette/issues/1426 | 902260799 | IC_kwDOBm6k_c41x2g_ | 9599 | 2021-08-19T21:29:13Z | 2021-08-19T21:29:13Z | OWNER | Bing's equivalent is: https://www.bing.com/webmasters/help/Sitemaps-3b5cf6ed http://www.bing.com/ping?sitemap=FULL_URL_OF_SITEMAP | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
964322136 | |
https://github.com/simonw/datasette/issues/1443#issuecomment-902258509 | https://api.github.com/repos/simonw/datasette/issues/1443 | 902258509 | IC_kwDOBm6k_c41x19N | 9599 | 2021-08-19T21:25:07Z | 2021-08-19T21:25:07Z | OWNER | https://docs.datasette.io/en/latest/internals.html#databases | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
974995592 | |
https://github.com/simonw/datasette/pull/1434#issuecomment-902254712 | https://api.github.com/repos/simonw/datasette/issues/1434 | 902254712 | IC_kwDOBm6k_c41x1B4 | 9599 | 2021-08-19T21:18:31Z | 2021-08-19T21:18:57Z | OWNER | I deployed a demo to https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app using the mechanism from #1442. e.g. demo here: https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app/fixtures?sql=select+*+from+searchable | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
970463436 | |
https://github.com/simonw/datasette/issues/1415#issuecomment-902251316 | https://api.github.com/repos/simonw/datasette/issues/1415 | 902251316 | IC_kwDOBm6k_c41x0M0 | 9599 | 2021-08-19T21:14:15Z | 2021-08-19T21:14:15Z | OWNER | https://github.com/ahmetb/cloud-run-faq#how-do-i-continuously-deploy-to-cloud-run suggests the following: > - `roles/run.admin` to deploy applications > - `roles/iam.serviceAccountUser` on the service account that your app will use It also links to https://cloud.google.com/run/docs/reference/iam/roles | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
959137143 | |
https://github.com/simonw/datasette/issues/1415#issuecomment-902250361 | https://api.github.com/repos/simonw/datasette/issues/1415 | 902250361 | IC_kwDOBm6k_c41xz95 | 9599 | 2021-08-19T21:12:28Z | 2021-08-19T21:12:28Z | OWNER | I would love to know this too! I always find figuring out minimal permissions to be really difficult. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
959137143 | |
https://github.com/simonw/datasette/issues/1442#issuecomment-902243498 | https://api.github.com/repos/simonw/datasette/issues/1442 | 902243498 | IC_kwDOBm6k_c41xySq | 9599 | 2021-08-19T21:04:01Z | 2021-08-19T21:04:01Z | OWNER | That successfully deployed to https://datasette-latest-deploy-this-branch-j7hipcg4aq-uc.a.run.app/ even though the tests failed. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
974987856 | |
https://github.com/simonw/datasette/issues/1442#issuecomment-902239215 | https://api.github.com/repos/simonw/datasette/issues/1442 | 902239215 | IC_kwDOBm6k_c41xxPv | 9599 | 2021-08-19T20:56:46Z | 2021-08-19T20:56:46Z | OWNER | I'm going to only run the tests if it's a push to `main` - that way I can ship demo branches really quickly, even if they don't yet have passing tests. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
974987856 | |
https://github.com/simonw/datasette/issues/1442#issuecomment-902235714 | https://api.github.com/repos/simonw/datasette/issues/1442 | 902235714 | IC_kwDOBm6k_c41xwZC | 9599 | 2021-08-19T20:50:38Z | 2021-08-19T20:50:38Z | OWNER | Would this allow anyone to push a PR to this repo that would result in their code being deployed against my Cloud Run account? I'm reasonably confident that it would not, since the secrets would not be visible to their PR branch. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
974987856 | |
https://github.com/simonw/datasette/issues/1442#issuecomment-902231018 | https://api.github.com/repos/simonw/datasette/issues/1442 | 902231018 | IC_kwDOBm6k_c41xvPq | 9599 | 2021-08-19T20:42:08Z | 2021-08-19T20:42:08Z | OWNER | If I get this working I should document it on https://docs.datasette.io/en/stable/contributing.html | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
974987856 | |
https://github.com/simonw/datasette/issues/1442#issuecomment-902217726 | https://api.github.com/repos/simonw/datasette/issues/1442 | 902217726 | IC_kwDOBm6k_c41xr_- | 9599 | 2021-08-19T20:21:47Z | 2021-08-19T20:21:47Z | OWNER | I think the neatest way to implement this would be for the `on -> push -> branches` list to be the list of branches that should be deployed in this way. The rest of the code can react to that. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
974987856 | |
https://github.com/simonw/datasette/issues/1442#issuecomment-902191150 | https://api.github.com/repos/simonw/datasette/issues/1442 | 902191150 | IC_kwDOBm6k_c41xlgu | 9599 | 2021-08-19T19:43:05Z | 2021-08-19T19:43:59Z | OWNER | Maybe as simple as teaching https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml to run on pushes to ALL branches: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L3-L6 And then quit early if the branch is not in some allow-list. If it IS in the allow-list, use the name of the branch to dynamically construct the name of the Cloud Run service here: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L60 Need to skip the documentation build and deployment stuff for other branches though. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
974987856 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-901475812 | https://api.github.com/repos/simonw/datasette/issues/1293 | 901475812 | IC_kwDOBm6k_c41u23k | 9599 | 2021-08-18T22:41:19Z | 2021-08-18T22:41:19Z | OWNER | > Maybe I split this out into a separate Python library that gets tested against _every_ SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself? I'm going to do this, and call the Python library `sqlite-explain`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/sqlite-utils/issues/37#issuecomment-901452199 | https://api.github.com/repos/simonw/sqlite-utils/issues/37 | 901452199 | IC_kwDOCGYnMM41uxGn | 9599 | 2021-08-18T21:48:57Z | 2021-08-18T21:48:57Z | OWNER | I did a bunch of work on this in #266. The library is now pretty thoroughly typed, and I even found a couple of bugs using `mypy` along the way: #313 and #315. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
465815372 | |
https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440752 | https://api.github.com/repos/simonw/sqlite-utils/issues/318 | 901440752 | IC_kwDOCGYnMM41uuTw | 9599 | 2021-08-18T21:25:30Z | 2021-08-18T21:25:30Z | OWNER | Some questions: - Should this support compression formats other than gzip? - Should `memory` learn to auto-detect gzipped data? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
974067156 | |
https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440207 | https://api.github.com/repos/simonw/sqlite-utils/issues/318 | 901440207 | IC_kwDOCGYnMM41uuLP | 9599 | 2021-08-18T21:24:28Z | 2021-08-18T21:24:49Z | OWNER | Something like this then: sqlite-utils file.db "select * from t" --csv --gz > t.csv.gz Maybe add a `-o t.csv.gz` option too so you don't have to use a `>`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
974067156 | |
https://github.com/simonw/sqlite-utils/issues/295#issuecomment-901403298 | https://api.github.com/repos/simonw/sqlite-utils/issues/295 | 901403298 | IC_kwDOCGYnMM41ulKi | 9599 | 2021-08-18T20:19:04Z | 2021-08-18T20:19:04Z | OWNER | Thanks, this was a bug. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
934123448 | |
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901399139 | https://api.github.com/repos/simonw/sqlite-utils/issues/296 | 901399139 | IC_kwDOCGYnMM41ukJj | 9599 | 2021-08-18T20:12:34Z | 2021-08-18T20:13:12Z | OWNER | Documentation for `table.search(..., quote=True)`: https://sqlite-utils.datasette.io/en/latest/python-api.html#searching-with-table-search In the API reference: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.search And for the CLI `--quote` option: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-searches | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
944326512 | |
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901398216 | https://api.github.com/repos/simonw/sqlite-utils/issues/296 | 901398216 | IC_kwDOCGYnMM41uj7I | 9599 | 2021-08-18T20:11:01Z | 2021-08-18T20:11:01Z | OWNER | ``` % sqlite-utils search fixtures.db searchable 'dog"' Error: malformed MATCH expression: [dog"] Try running this again with the --quote option ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
944326512 | |
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901390635 | https://api.github.com/repos/simonw/sqlite-utils/issues/296 | 901390635 | IC_kwDOCGYnMM41uiEr | 9599 | 2021-08-18T19:58:53Z | 2021-08-18T19:58:53Z | OWNER | ``` sqlite-utils search fixtures.db searchable 'dog"' Error: malformed MATCH expression: [dog"] ``` This error message could suggest retrying with `--quote`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
944326512 | |
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901379930 | https://api.github.com/repos/simonw/sqlite-utils/issues/296 | 901379930 | IC_kwDOCGYnMM41ufda | 9599 | 2021-08-18T19:40:38Z | 2021-08-18T19:40:38Z | OWNER | Also add `sqlite-utils search ... --quote` option. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
944326512 | |
https://github.com/simonw/sqlite-utils/issues/246#issuecomment-901353345 | https://api.github.com/repos/simonw/sqlite-utils/issues/246 | 901353345 | IC_kwDOCGYnMM41uY-B | 9599 | 2021-08-18T18:57:13Z | 2021-08-18T18:57:13Z | OWNER | More documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#quoting-characters-for-use-in-search | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
831751367 | |
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901338841 | https://api.github.com/repos/simonw/sqlite-utils/issues/296 | 901338841 | IC_kwDOCGYnMM41uVbZ | 9599 | 2021-08-18T18:33:26Z | 2021-08-18T18:45:12Z | OWNER | I think I'll do this as an optional `table.search(..., escape=True)` parameter. Actually I'll do `quote=True` for consistency with the new `db.quote_fts()` method. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
944326512 | |
https://github.com/simonw/sqlite-utils/issues/246#issuecomment-901345800 | https://api.github.com/repos/simonw/sqlite-utils/issues/246 | 901345800 | IC_kwDOCGYnMM41uXII | 9599 | 2021-08-18T18:44:48Z | 2021-08-18T18:44:48Z | OWNER | The `db.quote_fts(value)` method from #247 can now be used for this - documentation here: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.quote_fts I'll be adding further improvements relating to this (a `table.search(q, quote=True)` parameter) in #296. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
831751367 | |
https://github.com/simonw/sqlite-utils/pull/247#issuecomment-901338988 | https://api.github.com/repos/simonw/sqlite-utils/issues/247 | 901338988 | IC_kwDOCGYnMM41uVds | 9599 | 2021-08-18T18:33:39Z | 2021-08-18T18:33:39Z | OWNER | This was also requested in #296. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
832687563 | |
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901338356 | https://api.github.com/repos/simonw/sqlite-utils/issues/296 | 901338356 | IC_kwDOCGYnMM41uVT0 | 9599 | 2021-08-18T18:32:39Z | 2021-08-18T18:32:39Z | OWNER | This is a good call. I have a fix for this in Datasette but it's not in `sqlite-utils` yet: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L824-L835 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
944326512 | |
https://github.com/simonw/sqlite-utils/issues/317#issuecomment-901337305 | https://api.github.com/repos/simonw/sqlite-utils/issues/317 | 901337305 | IC_kwDOCGYnMM41uVDZ | 9599 | 2021-08-18T18:30:59Z | 2021-08-18T18:30:59Z | OWNER | I'm just going to remove this - I added it when the library was mostly undocumented, but it has comprehensive documentation now. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
972827346 | |
https://github.com/simonw/datasette/issues/1439#issuecomment-900715375 | https://api.github.com/repos/simonw/datasette/issues/1439 | 900715375 | IC_kwDOBm6k_c41r9Nv | 9599 | 2021-08-18T00:15:28Z | 2021-08-18T00:15:28Z | OWNER | Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
973139047 | |
https://github.com/simonw/datasette/issues/1439#issuecomment-900714630 | https://api.github.com/repos/simonw/datasette/issues/1439 | 900714630 | IC_kwDOBm6k_c41r9CG | 9599 | 2021-08-18T00:13:33Z | 2021-08-18T00:13:33Z | OWNER | The documentation should definitely cover how table names become URLs, in case any third party code needs to be able to calculate this themselves. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
973139047 | |
https://github.com/simonw/datasette/issues/1439#issuecomment-900712981 | https://api.github.com/repos/simonw/datasette/issues/1439 | 900712981 | IC_kwDOBm6k_c41r8oV | 9599 | 2021-08-18T00:09:59Z | 2021-08-18T00:12:32Z | OWNER | So given the original examples, a table called `table.csv` would have the following URLs: - `/db/table-.csv` - the HTML version - `/db/table-.csv.csv` - the CSV version - `/db/table-.csv.json` - the JSON version And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this: - `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version - `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version - `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
973139047 | |
https://github.com/simonw/datasette/issues/1439#issuecomment-900711967 | https://api.github.com/repos/simonw/datasette/issues/1439 | 900711967 | IC_kwDOBm6k_c41r8Yf | 9599 | 2021-08-18T00:08:09Z | 2021-08-18T00:08:09Z | OWNER | Here's an alternative I just made up which I'm calling "dot dash" encoding: ```python def dot_dash_encode(s): return s.replace("-", "--").replace(".", "-.") def dot_dash_decode(s): return s.replace("-.", ".").replace("--", "-") ``` And some examples: ```python for example in ( "hello", "hello.csv", "hello-and-so-on.csv", "hello-.csv", "hello--and--so--on-.csv", "hello.csv.", "hello.csv.-", "hello.csv.--", ): print(example) print(dot_dash_encode(example)) print(example == dot_dash_decode(dot_dash_encode(example))) print() ``` Outputs: ``` hello hello True hello.csv hello-.csv True hello-and-so-on.csv hello--and--so--on-.csv True hello-.csv hello---.csv True hello--and--so--on-.csv hello----and----so----on---.csv True hello.csv. hello-.csv-. True hello.csv.- hello-.csv-.-- True hello.csv.-- hello-.csv-.---- True ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
973139047 | |
https://github.com/simonw/datasette/issues/1439#issuecomment-900709703 | https://api.github.com/repos/simonw/datasette/issues/1439 | 900709703 | IC_kwDOBm6k_c41r71H | 9599 | 2021-08-18T00:03:09Z | 2021-08-18T00:03:09Z | OWNER | But... what if I invent my own escaping scheme? I actually did this once before, in https://github.com/simonw/datasette/commit/9fdb47ca952b93b7b60adddb965ea6642b1ff523 - while I was working on porting Datasette to ASGI in https://github.com/simonw/datasette/issues/272#issuecomment-494192779 because ASGI didn't yet have the `raw_path` mechanism. I could bring that back - it looked like this: ``` "table/and/slashes" => "tableU+002FandU+002Fslashes" "~table" => "U+007Etable" "+bobcats!" => "U+002Bbobcats!" "U+007Etable" => "UU+002B007Etable" ``` But I didn't particularly like it - it was quite verbose. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
973139047 | |
https://github.com/simonw/datasette/issues/1439#issuecomment-900705226 | https://api.github.com/repos/simonw/datasette/issues/1439 | 900705226 | IC_kwDOBm6k_c41r6vK | 9599 | 2021-08-17T23:50:32Z | 2021-08-17T23:50:47Z | OWNER | An alternative solution would be to use some form of escaping for the characters that form the name of the table. The obvious way to do this would be URL-encoding - but it doesn't hold for `.` characters. The hex for that is `%2E` but watch what happens with that in a URL: ``` # Against Cloud Run: curl -s 'https://datasette.io/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path 'path': '/-/asgi-scope/foo/bar/baz.', 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz.', 'root_path': '', # Against Vercel: curl -s 'https://til.simonwillison.net/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path 'path': '/-/asgi-scope/foo/bar%2Fbaz%2E', 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz%2E', 'root_path': '', ``` Surprisingly in this case Vercel DOES keep it intact, but Cloud Run does not. It's still no good though: I need a solution that works on Vercel, Cloud Run and every other potential hosting provider too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
973139047 | |
https://github.com/simonw/datasette/issues/1439#issuecomment-900699670 | https://api.github.com/repos/simonw/datasette/issues/1439 | 900699670 | IC_kwDOBm6k_c41r5YW | 9599 | 2021-08-17T23:34:23Z | 2021-08-17T23:34:23Z | OWNER | The challenge comes down to telling the difference between the following: - `/db/table` - an HTML table page - `/db/table.csv` - the CSV version of `/db/table` - `/db/table.csv` - no this one is actually a database table called `table.csv` - `/db/table.csv.csv` - the CSV version of `/db/table.csv` - `/db/table.csv.csv.csv` and so on... | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
973139047 | |
https://github.com/simonw/datasette/issues/1438#issuecomment-900690998 | https://api.github.com/repos/simonw/datasette/issues/1438 | 900690998 | IC_kwDOBm6k_c41r3Q2 | 9599 | 2021-08-17T23:11:16Z | 2021-08-17T23:12:25Z | OWNER | I have completely failed to replicate this initial bug - but it's still there on the `thesession.vercel.app` deployment (even though my own deployments to Vercel do not exhibit it). Here's a one-liner to replicate it against that deployment: `curl -s 'https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22' | rg '.csv'` Whit outputs this: `<p class="export-links">This data as <a href="/thesession.json?sql=select * from tunes where name like "%wise maid%"">json</a>, <a href="/thesession.csv?sql=select * from tunes where name like "%wise maid%"&_size=max">CSV</a></p>` It looks like, rather than being URL-encoded, the original query string is somehow making it through to Jinja and then being auto-escaped there. The weird thing is that the equivalent query executed against my `til.simonwillison.net` Vercel instance does this: `curl -s 'https://til.simonwillison.net/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25a%25%22' | rg '.csv'` `<p class="export-links">This data as <a href="/fixtures.json?sql=select%20*%20from%20searchable%20where%20text1%20like%20%22%25a%25%22">json</a>, <a href="/fixtures.csv?sql=select%20*%20from%20searchable%20where%20text1%20like%20%22%25a%25%22&_size=max">CSV</a></p>` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
972918533 | |
https://github.com/simonw/datasette/issues/1438#issuecomment-900681413 | https://api.github.com/repos/simonw/datasette/issues/1438 | 900681413 | IC_kwDOBm6k_c41r07F | 9599 | 2021-08-17T22:47:44Z | 2021-08-17T22:47:44Z | OWNER | I deployed another copy of `fixtures.db` on Vercel at https://til.simonwillison.net/fixtures so I can compare it with `fixtures.db` on Cloud Run at https://latest.datasette.io/fixtures | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
972918533 | |
https://github.com/simonw/datasette/issues/1438#issuecomment-900518343 | https://api.github.com/repos/simonw/datasette/issues/1438 | 900518343 | IC_kwDOBm6k_c41rNHH | 9599 | 2021-08-17T18:04:42Z | 2021-08-17T18:04:42Z | OWNER | Here's how `request.query_string` works: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/asgi.py#L86-L88 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
972918533 | |
https://github.com/simonw/datasette/issues/1438#issuecomment-900516826 | https://api.github.com/repos/simonw/datasette/issues/1438 | 900516826 | IC_kwDOBm6k_c41rMva | 9599 | 2021-08-17T18:02:27Z | 2021-08-17T18:02:27Z | OWNER | The key difference I can spot between Vercel and Cloud Run is that `+` in a query string gets converted to `%20` by Vercel before it gets to my app, but does not for Cloud Run: ``` # Vercel ~ % curl -s 'https://til.simonwillison.net/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'sql=select%20*%20from%20tunes%20where%20name%20like%20%22%25' b'wise%20maid%25%22%0D%0A', 'raw_path': b'/-/asgi-scope', # Cloud Run ~ % curl -s 'https://latest-with-plugins.datasette.io/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%2' b'2%0D%0A', 'raw_path': b'/-/asgi-scope', ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
972918533 | |
https://github.com/simonw/datasette/issues/1438#issuecomment-900513267 | https://api.github.com/repos/simonw/datasette/issues/1438 | 900513267 | IC_kwDOBm6k_c41rL3z | 9599 | 2021-08-17T17:57:05Z | 2021-08-17T17:57:05Z | OWNER | I'm having trouble replicating this bug outside of Vercel. Against Cloud Run: view-source:https://latest.datasette.io/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22 The HTML here is: ```html <p class="export-links">This data as <a href="/fixtures.json?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22">json</a>, ... <a href="/fixtures.csv?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22&_size=max">CSV</a> </p> ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
972918533 | |
https://github.com/simonw/datasette/issues/1438#issuecomment-900502364 | https://api.github.com/repos/simonw/datasette/issues/1438 | 900502364 | IC_kwDOBm6k_c41rJNc | 9599 | 2021-08-17T17:40:41Z | 2021-08-17T17:40:41Z | OWNER | Bug is likely in `path_with_format` itself: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L710-L729 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
972918533 | |
https://github.com/simonw/datasette/issues/1438#issuecomment-900500824 | https://api.github.com/repos/simonw/datasette/issues/1438 | 900500824 | IC_kwDOBm6k_c41rI1Y | 9599 | 2021-08-17T17:38:16Z | 2021-08-17T17:38:16Z | OWNER | Relevant template code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/templates/query.html#L71 `renderers` comes from here: https://github.com/simonw/datasette/blob/2883098770fc66e50183b2b231edbde20848d4d6/datasette/views/base.py#L593-L608 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
972918533 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-899915829 | https://api.github.com/repos/simonw/datasette/issues/1293 | 899915829 | IC_kwDOBm6k_c41o6A1 | 9599 | 2021-08-17T01:02:35Z | 2021-08-17T01:02:35Z | OWNER | New approach: this time I'm building a simplified executor for the bytecode operations themselves. ```python def execute_operations(operations, max_iterations = 100, trace=None): trace = trace or (lambda *args: None) registers: Dict[int, Any] = {} cursors: Dict[int, Tuple[str, Dict]] = {} instruction_pointer = 0 iterations = 0 result_row = None while True: iterations += 1 if iterations > max_iterations: break operation = operations[instruction_pointer] trace(instruction_pointer, dict(operation)) opcode = operation["opcode"] if opcode == "Init": if operation["p2"] != 0: instruction_pointer = operation["p2"] continue else: instruction_pointer += 1 continue elif opcode == "Goto": instruction_pointer = operation["p2"] continue elif opcode == "Halt": break elif opcode == "OpenRead": cursors[operation["p1"]] = ("database_table", { "rootpage": operation["p2"], "connection": operation["p3"], }) elif opcode == "OpenEphemeral": cursors[operation["p1"]] = ("ephemeral", { "num_columns": operation["p2"], "index_keys": [], }) elif opcode == "MakeRecord": registers[operation["p3"]] = ("MakeRecord", { "registers": list(range(operation["p1"] + operation["p2"])) }) elif opcode == "IdxInsert": record = registers[operation["p2"]] cursors[operation["p1"]][1]["index_keys"].append(record) elif opcode == "Rowid": registers[operation["p2"]] = ("rowid", { "table": operation["p1"] }) elif opcode == "Sequence": registers[operation["p2"]] = ("sequence", { "next_from_cursor": operat… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1423#issuecomment-899749881 | https://api.github.com/repos/simonw/datasette/issues/1423 | 899749881 | IC_kwDOBm6k_c41oRf5 | 9599 | 2021-08-16T19:07:02Z | 2021-08-16T19:07:02Z | OWNER | Demo: https://latest.datasette.io/fixtures/compound_three_primary_keys?_facet=content&_facet_size=max&_facet=pk1&_facet=pk2 <img width="686" alt="fixtures__compound_three_primary_keys__1_001_rows" src="https://user-images.githubusercontent.com/9599/129616596-cc51b668-7cb8-482f-9e20-e0d8ca4b71be.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
962391325 | |
https://github.com/simonw/datasette/issues/1423#issuecomment-899744109 | https://api.github.com/repos/simonw/datasette/issues/1423 | 899744109 | IC_kwDOBm6k_c41oQFt | 9599 | 2021-08-16T18:58:29Z | 2021-08-16T18:58:29Z | OWNER | I didn't bother with the tooltip, just the visible display if `?_facet_size=max`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
962391325 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898961535 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898961535 | IC_kwDOBm6k_c41lRB_ | 9599 | 2021-08-14T21:37:24Z | 2021-08-14T21:37:24Z | OWNER | Did some more research into building SQLite custom versions via `pysqlite3` - here's what I figured out for macOS (which should hopefully work for Linux too): https://til.simonwillison.net/sqlite/build-specific-sqlite-pysqlite-macos | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898936068 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898936068 | IC_kwDOBm6k_c41lK0E | 9599 | 2021-08-14T17:44:54Z | 2021-08-14T17:44:54Z | OWNER | Another interesting query to consider: https://latest.datasette.io/fixtures?sql=explain+select+*+from++pragma_table_info%28+%27123_starts_with_digits%27%29 That one shows `VColumn` instead of `Column`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898933865 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898933865 | IC_kwDOBm6k_c41lKRp | 9599 | 2021-08-14T17:27:16Z | 2021-08-14T17:28:29Z | OWNER | Maybe I split this out into a separate Python library that gets tested against *every* SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself? Datasette could depend on that library. The library could be released independently of Datasette any time a new SQLite version comes out. I could even run a separate git scraper repo that checks for new SQLite releases and submits PRs against the library when a new release comes out. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898913629 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898913629 | IC_kwDOBm6k_c41lFVd | 9599 | 2021-08-14T16:14:12Z | 2021-08-14T16:14:12Z | OWNER | I would feel a lot more comfortable about all of this if I had a robust mechanism for running the Datasette test suite against multiple versions of SQLite itself. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898913554 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898913554 | IC_kwDOBm6k_c41lFUS | 9599 | 2021-08-14T16:13:40Z | 2021-08-14T16:13:40Z | OWNER | I think I need to care about the following: - `ResultRow` and `Column` for the final result - `OpenRead` for opening tables - `OpenEphemeral` then `MakeRecord` and `IdxInsert` for writing records into ephemeral tables `Column` may reference either a table (from `OpenRead`) or an ephemeral table (from `OpenEphemeral`). That *might* be enough. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/sqlite-utils/issues/316#issuecomment-898824020 | https://api.github.com/repos/simonw/sqlite-utils/issues/316 | 898824020 | IC_kwDOCGYnMM41kvdU | 9599 | 2021-08-14T05:12:23Z | 2021-08-14T05:12:23Z | OWNER | No visible backticks on https://sqlite-utils.datasette.io/en/latest/reference.html any more. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
970320615 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898788262 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898788262 | IC_kwDOBm6k_c41kmum | 9599 | 2021-08-14T01:22:26Z | 2021-08-14T01:51:08Z | OWNER | Tried a more complicated query: ```sql explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101 ``` Here's the explain: ``` sqlite> explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101 ...> ; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 41 0 00 Start at 41 1 OpenEphemeral 2 6 0 k(1,-B) 00 nColumn=6 2 Integer 101 1 0 00 r[1]=101; LIMIT counter 3 OpenRead 0 32 0 4 00 root=32 iDb=0; searchable 4 Integer 16 3 0 00 r[3]=16; return address 5 Once 0 16 0 00 6 OpenEphemeral 3 1 0 k(1,) 00 nColumn=1; Result of SELECT 1 7 VOpen 1 0 0 vtab:7FCBCA72BE80 00 8 Function0 1 7 6 unknown(-1) 01 r[6]=func(r[7]) 9 Integer 5 4 0 00 r[4]=5 10 Integer 1 5 0 00 r[5]=1 11 VFilter 1 16 4 00 iplan=r[4] zplan='' 12 Rowid 1 8 0 00 r[8]=rowid 13 MakeRecord 8 1 9 C 00 r[9]=mkrec(r[8]) 14 IdxInsert 3 9 8 1 00 key=r[9] 15 VNext 1 12 0 00 16 Return 3 0 0 00 17 Rewind 3 33 0 00 18 Column 3… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898760808 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898760808 | IC_kwDOBm6k_c41kgBo | 9599 | 2021-08-13T23:03:01Z | 2021-08-13T23:03:01Z | OWNER | Another idea: strip out any `order by` clause to try and keep this simpler. I doubt that's going to cope with complex nested queries though. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898760020 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898760020 | IC_kwDOBm6k_c41kf1U | 9599 | 2021-08-13T23:00:28Z | 2021-08-13T23:01:27Z | OWNER | New theory: this is all about `SorterOpen` and `SorterInsert`. Consider the following with extra annotations at the end of the lines after the `--`: ``` addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 25 0 00 Start at 25 1 SorterOpen 2 5 0 k(1,B) 00 -- New SORTER in r2 with 5 slots 2 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable 3 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities 4 Rewind 0 16 0 00 5 Column 0 6 3 00 r[3]=facetable.neighborhood 6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 7 IfNot 1 15 1 00 8 Column 0 5 4 00 r[4]=facetable.city_id 9 SeekRowid 1 15 4 00 intkey=r[4] 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 Column 0 6 5 00 r[5]=facetable.neighborhood 13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7]) 14 SorterInsert 2 9 5 3 00 key=r[9]-- WRITES record from r9 (line above) into sorter in r2 15 Next 0 5 0 01 16 OpenPseudo 3 10 5 00 5 columns in r[10] 17 SorterSort 2 24 0 00 -- runs the sort, not relevant to my goal 18 SorterData 2 10 3 00 r[10]=data -- "Write into register P2 (r10) the current sorter data for sorter cursor P1 (sorter 2)" 19 Column 3 2 8 … | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898576097 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898576097 | IC_kwDOBm6k_c41jy7h | 9599 | 2021-08-13T16:19:57Z | 2021-08-13T16:19:57Z | OWNER | I think I need to look out for `OpenPseudo` and, when that occurs, take a look at the most recent `SorterInsert` and use that to find the `MakeRecord` and then use the `MakeRecord` to figure out the columns that went into it. After all of that I'll be able to resolve that "table 3" reference. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898572065 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898572065 | IC_kwDOBm6k_c41jx8h | 9599 | 2021-08-13T16:13:16Z | 2021-08-13T16:13:16Z | OWNER | Aha! That `MakeRecord` line says `r[5..7]` - and r5 = neighborhood, r6 = facet_cities.name, r7 = facetable.state So if the `MakeRecord` defines what goes into that pseudo-table column 2 of that pseudo-table would be `state` - which is what we want. This is really convoluted. I'm no longer confident I can get this to work in a sensible way, especially since I've not started exploring what complex nested tables with CTEs and sub-selects do yet. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898569319 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898569319 | IC_kwDOBm6k_c41jxRn | 9599 | 2021-08-13T16:09:01Z | 2021-08-13T16:10:48Z | OWNER | Need to figure out what column 2 of that pseudo-table is. I think the answer is here: ``` 4 Rewind 0 16 0 00 5 Column 0 6 3 00 r[3]=facetable.neighborhood 6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 7 IfNot 1 15 1 00 8 Column 0 5 4 00 r[4]=facetable.city_id 9 SeekRowid 1 15 4 00 intkey=r[4] 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 Column 0 6 5 00 r[5]=facetable.neighborhood 13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7]) 14 SorterInsert 2 9 5 3 00 key=r[9] 15 Next 0 5 0 01 16 OpenPseudo 3 10 5 00 5 columns in r[10] ``` I think the `OpenPseduo` line puts five columns in `r[10]` - and those five columns are the five from the previous block - maybe the five leading up to the `MakeRecord` call on line 13. In which case column 2 would be `facet_cities.name` - assuming we start counting from 0. But the debug code said "r[8]=state". | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898567974 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898567974 | IC_kwDOBm6k_c41jw8m | 9599 | 2021-08-13T16:07:00Z | 2021-08-13T16:07:00Z | OWNER | So this line: ``` 19 Column 3 2 8 00 r[8]=state ``` Means "Take column 2 of table 3 (the pseudo-table) and store it in register 8" | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898564705 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898564705 | IC_kwDOBm6k_c41jwJh | 9599 | 2021-08-13T16:02:12Z | 2021-08-13T16:04:06Z | OWNER | More debug output: ``` table_rootpage_by_register={0: 43, 1: 42} names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_id=0 cid=6 column_register=3 table_id=0 cid=5 column_register=4 table_id=1 cid=1 column_register=6 table_id=0 cid=4 column_register=7 table_id=0 cid=6 column_register=5 table_id=3 cid=2 column_register=8 table_id=3 cid=2 column_register=8 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=8 = (table='facetable', cid=2) table_id=3 cid=1 column_register=7 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=7 = (table='facetable', cid=1) table_id=3 cid=0 column_register=6 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=6 = (table='facetable', cid=0) result_registers=[6, 7, 8] columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)} all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('faceta… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898554859 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898554859 | IC_kwDOBm6k_c41jtvr | 9599 | 2021-08-13T15:46:18Z | 2021-08-13T15:46:18Z | OWNER | So it looks like the bug is in the code that populates `columns_by_column_register`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898554427 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898554427 | IC_kwDOBm6k_c41jto7 | 9599 | 2021-08-13T15:45:32Z | 2021-08-13T15:45:32Z | OWNER | Some useful debug output: ``` table_rootpage_by_register={0: 43, 1: 42} names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} result_registers=[6, 7, 8] columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)} all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('facetable', 8): 'complex_array', ('facetable', 9): 'distinct_some_null'} ``` The `result_registers` should each correspond to the correct entry in `columns_by_column_register` but they do not. Python code: ```python def columns_for_query(conn, sql, params=None): """ Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of ``(table_name, column_name)`` pairs corresponding to the columns that would be returned by that SQL query. Each pair indicates the source table and column for the returned column, or ``(None, None)`` if no table and column could be derived (e.g. for "select 1") """ if sql.lower().strip().startswith("explain"): return [] opcodes = conn.execute("explain " + sql, params).fetchall() table_rootpage_by_register = { r["p1"]: r["p2"] for r in opcodes if r["opcode"] == "OpenRead" } print(f"{table_rootpage_by_register=}") names_and_types_by_rootpage = dict( [(r[0], (r[1], r[2])) for r in conn.execute( "select rootpage, name, type from sqlite_master where rootpage in ({})".format( ", ".join(map(str, table_rootpage_by_register.values())) ) )] ) print(f"{names_and_types_by_rootpage=}") columns_by_column_register = {} for opcode in opcodes: if opcode["opcode"] in ("Rowid", "Column"): … | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898545815 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898545815 | IC_kwDOBm6k_c41jriX | 9599 | 2021-08-13T15:31:53Z | 2021-08-13T15:31:53Z | OWNER | My hunch here is that registers or columns are being reused in a way that makes my code break - my code is pretty dumb, there are places in it where maybe the first mention of a register wins instead of the last one? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898541972 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898541972 | IC_kwDOBm6k_c41jqmU | 9599 | 2021-08-13T15:26:06Z | 2021-08-13T15:29:06Z | OWNER | ResultRow: > The registers P1 through P1+P2-1 contain a single row of results. This opcode causes the sqlite3_step() call to terminate with an SQLITE_ROW return code and it sets up the sqlite3_stmt structure to provide access to the r(P1)..r(P1+P2-1) values as the result row. Column: > Interpret the data that cursor P1 points to as a structure built using the MakeRecord instruction. (See the MakeRecord opcode for additional information about the format of the data.) Extract the P2-th column from this record. If there are less that (P2+1) values in the record, extract a NULL. > > The value extracted is stored in register P3. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 | |
https://github.com/simonw/datasette/issues/1293#issuecomment-898541543 | https://api.github.com/repos/simonw/datasette/issues/1293 | 898541543 | IC_kwDOBm6k_c41jqfn | 9599 | 2021-08-13T15:25:26Z | 2021-08-13T15:25:26Z | OWNER | But the debug output here seems to be saying what we want it to say: ``` 17 SorterSort 2 24 0 00 18 SorterData 2 10 3 00 r[10]=data 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] ``` We want to get back `neighborhood`, `facet_cities.name`, `state`. Why then are we seeing `[('facet_cities', 'name'), ('facetable', 'state'), (None, None)]`? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
849978964 |