github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/sqlite-utils/issues/163#issuecomment-696465788 | https://api.github.com/repos/simonw/sqlite-utils/issues/163 | 696465788 | MDEyOklzc3VlQ29tbWVudDY5NjQ2NTc4OA== | 9599 | 2020-09-22T01:33:04Z | 2020-09-22T01:33:04Z | OWNER | This would apply to `.transform()` in #114 too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
706001517 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696454485 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696454485 | MDEyOklzc3VlQ29tbWVudDY5NjQ1NDQ4NQ== | 9599 | 2020-09-22T00:42:35Z | 2020-09-22T00:42:35Z | OWNER | The reason I'm working on this now is that I'd like to support many more options for data cleanup in the Datasette ecosystem - so being able to do things like convert the type of existing columns becomes increasingly important. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696454084 | https://api.github.com/repos/simonw/sqlite-utils/issues/162 | 696454084 | MDEyOklzc3VlQ29tbWVudDY5NjQ1NDA4NA== | 9599 | 2020-09-22T00:40:44Z | 2020-09-22T00:40:44Z | OWNER | Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#registering-custom-sql-functions | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705995722 | |
https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696449345 | https://api.github.com/repos/simonw/sqlite-utils/issues/162 | 696449345 | MDEyOklzc3VlQ29tbWVudDY5NjQ0OTM0NQ== | 9599 | 2020-09-22T00:22:46Z | 2020-09-22T00:22:46Z | OWNER | Inspired by the idea of adding `conversions=` to #114 - since this would make it easy to register custom Python functions that can be used to convert the values in a table. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705995722 | |
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696446658 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | 696446658 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NjY1OA== | 9599 | 2020-09-22T00:13:55Z | 2020-09-22T00:14:21Z | OWNER | Idea: allow a `conversions=` parameter, as seen on `.insert_all()` and friends, which lets you apply a SQL transformation function as part of the operation. E.g.: ```python table.transform({"age": int}, conversions={"name": "upper(?)"}) ``` https://sqlite-utils.readthedocs.io/en/stable/python-api.html#converting-column-values-using-sql-functions | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705975133 | |
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696445766 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | 696445766 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NTc2Ng== | 9599 | 2020-09-22T00:10:50Z | 2020-09-22T00:11:12Z | OWNER | A less horrible interface might be the following: ```python # Ensure the 'age' column is not null: table.transform(not_null={"age"}) # The 'age' column is not null but I don't want it to be: table.transform(not_null={"age": False}) ``` So if the argument is a set it means "make sure these are all not null" - if the argument is a dictionary it means "set these to be null or not null depending on if their dictionary value is true or false". | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705975133 | |
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444842 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | 696444842 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NDg0Mg== | 9599 | 2020-09-22T00:07:43Z | 2020-09-22T00:09:05Z | OWNER | Syntax challenge: I could use `.transform(defaults={"age": None})` to indicate that the `age` column should have its default removed, but how would I tell `.transform()` that the `age` column, currently `not null`, should have the `not null` removed from it? I could do this: `.transform(not_not_null={"age"})` - it's a bit gross but it's also kind of funny. I actually like it! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705975133 | |
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444353 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | 696444353 | MDEyOklzc3VlQ29tbWVudDY5NjQ0NDM1Mw== | 9599 | 2020-09-22T00:06:12Z | 2020-09-22T00:06:12Z | OWNER | I should support `not_null=` and `default=` arguments to the `.transform()` method because it looks like you can't use `ALTER TABLE` to change those. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705975133 | |
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443845 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | 696443845 | MDEyOklzc3VlQ29tbWVudDY5NjQ0Mzg0NQ== | 9599 | 2020-09-22T00:04:31Z | 2020-09-22T00:04:44Z | OWNER | Good news: the `.columns` introspection does tell me those things: ``` >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db.create_table("foo", {"id": int, "name": str, "age": int}, defaults={"age": 1}, not_null={"name", "age"}) <Table foo (id, name, age)> >>> db["foo"] <Table foo (id, name, age)> >>> print(db["foo"].schema) CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT NOT NULL, [age] INTEGER NOT NULL DEFAULT 1 ) >>> db["foo"].columns [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value=None, is_pk=0), Column(cid=2, name='age', type='INTEGER', notnull=1, default_value='1', is_pk=0)] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705975133 | |
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443190 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | 696443190 | MDEyOklzc3VlQ29tbWVudDY5NjQ0MzE5MA== | 9599 | 2020-09-22T00:02:22Z | 2020-09-22T00:02:22Z | OWNER | How would I detect which columns are `not_null` and what their defaults are? I don`t think my introspection logic handles that yet. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705975133 | |
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443042 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | 696443042 | MDEyOklzc3VlQ29tbWVudDY5NjQ0MzA0Mg== | 9599 | 2020-09-22T00:01:50Z | 2020-09-22T00:01:50Z | OWNER | When you transform a table, it should keep its primary key, foreign keys, not_null and defaults. I don't think it needs to care about `hash_id` or `extracts=` since those don't affect the structure of the table as it is being created - well, `hash_id` does but if we are transforming an existing table we will get the `hash_id` column for free. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705975133 | |
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696442621 | https://api.github.com/repos/simonw/sqlite-utils/issues/161 | 696442621 | MDEyOklzc3VlQ29tbWVudDY5NjQ0MjYyMQ== | 9599 | 2020-09-22T00:00:23Z | 2020-09-22T00:00:23Z | OWNER | I still need to figure out what to do about these various other table properties: https://github.com/simonw/sqlite-utils/blob/b34c9b40c206d7a9d7ee57a8c1f198ff1f522735/sqlite_utils/db.py#L775-L787 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705975133 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696435194 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696435194 | MDEyOklzc3VlQ29tbWVudDY5NjQzNTE5NA== | 9599 | 2020-09-21T23:34:14Z | 2020-09-21T23:35:00Z | OWNER | I think the fiddliest part of the implementation here is code that takes the existing `columns_dict` of the table and the incoming `columns=` and `drop=` and `rename=` parameters and produces the columns dictionary for the new table, ready to be fed to `.create_table()`. This logic probably also needs to return a structure that can be used to build the `INSERT INTO ... SELECT ... FROM` query. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434638 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696434638 | MDEyOklzc3VlQ29tbWVudDY5NjQzNDYzOA== | 9599 | 2020-09-21T23:32:26Z | 2020-09-21T23:32:26Z | OWNER | A test that confirms that this mechanism can turn a `rowid` into a non-rowid table would be good too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434237 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696434237 | MDEyOklzc3VlQ29tbWVudDY5NjQzNDIzNw== | 9599 | 2020-09-21T23:31:07Z | 2020-09-21T23:31:57Z | OWNER | Does it make sense to support the `pk=` argument for changing the primary key? If the user requests a primary key that doesn't make sense I think an integrity error will be raised when the SQL is being executed, which should hopefully cancel the transaction and raise an error. Need to check that this is what happens. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434097 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696434097 | MDEyOklzc3VlQ29tbWVudDY5NjQzNDA5Nw== | 9599 | 2020-09-21T23:30:40Z | 2020-09-21T23:30:40Z | OWNER | Since I have a `column_order=None` argument already, maybe I can ignore the order of the columns in that first argument and use that instead? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433778 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696433778 | MDEyOklzc3VlQ29tbWVudDY5NjQzMzc3OA== | 9599 | 2020-09-21T23:29:39Z | 2020-09-21T23:29:39Z | OWNER | The `columns=` argument is optional - so you can do just a rename operation like so: ``` table.transform(rename={"age": "dog_age"}) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433542 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696433542 | MDEyOklzc3VlQ29tbWVudDY5NjQzMzU0Mg== | 9599 | 2020-09-21T23:28:58Z | 2020-09-21T23:28:58Z | OWNER | If you want to both change the type of a column AND rename it in the same operation, how would you do that? I think like this: ```python table.transform({"age": int}, rename={"age": "dog_age"}) ``` So any rename logic is applied at the end, after the type transformation or re-ordering logic. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696432690 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696432690 | MDEyOklzc3VlQ29tbWVudDY5NjQzMjY5MA== | 9599 | 2020-09-21T23:26:32Z | 2020-09-21T23:27:38Z | OWNER | To expand on what that first argument - the `columns` argument - does. Say you have a table like this: ``` id integer name text age text ``` Any columns omitted from the `columns=` argument are left alone - they have to be explicitly dropped using `drop=` if you want to drop them. Any new columns are added (at the end of the table): ``` table.tranform({"size": float}) ``` Any columns that have their type changed will have their type changed: ``` table.tranform({"age": int}) ``` Should I also re-order columns if the order doesn't match? I think so. Open question as to what happens to columns that aren't mentioned at all in the dictionary though - what order should they go in? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696431058 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696431058 | MDEyOklzc3VlQ29tbWVudDY5NjQzMTA1OA== | 9599 | 2020-09-21T23:21:37Z | 2020-09-21T23:21:37Z | OWNER | I may need to do something special for `rowid` tables to ensure that the `rowid` values in the transformed table match those from the old table. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696430843 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696430843 | MDEyOklzc3VlQ29tbWVudDY5NjQzMDg0Mw== | 9599 | 2020-09-21T23:21:00Z | 2020-09-21T23:21:00Z | OWNER | For FTS tables associated with the table that is being transformed, should I automatically drop the old FTS table and recreate it against the new one or will it just magically continue to work after the table is renamed? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696423138 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696423138 | MDEyOklzc3VlQ29tbWVudDY5NjQyMzEzOA== | 9599 | 2020-09-21T22:59:17Z | 2020-09-21T23:01:06Z | OWNER | I'm going to sketch out a prototype of this new API design in that branch. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696423066 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696423066 | MDEyOklzc3VlQ29tbWVudDY5NjQyMzA2Ng== | 9599 | 2020-09-21T22:59:01Z | 2020-09-21T22:59:01Z | OWNER | I'm rethinking the API design now. Maybe it could look like this: To change the type of the `author_id` column from `text` to `int`: ```python books.transform({"author_id": int}) ``` This would leave the existing columns alone, but would change the type of this column. To rename `author_id` to `author_identifier`: ```python books.transform(rename={"author_id": "author_identifier"}) ``` To drop a column: ```python books.transform(drop=["author_id"]) ``` Since the parameters all operate on columns they don't need to be called `drop_column` and `rename_column`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696421240 | https://api.github.com/repos/simonw/sqlite-utils/issues/114 | 696421240 | MDEyOklzc3VlQ29tbWVudDY5NjQyMTI0MA== | 9599 | 2020-09-21T22:53:48Z | 2020-09-21T22:53:48Z | OWNER | I've decided to call this `table.transform()` - I was over-thinking whether people would remember that `.transform()` actually transforms the table, but that's what documentation is for. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
621989740 | |
https://github.com/simonw/datasette/issues/972#issuecomment-696308847 | https://api.github.com/repos/simonw/datasette/issues/972 | 696308847 | MDEyOklzc3VlQ29tbWVudDY5NjMwODg0Nw== | 9599 | 2020-09-21T19:01:25Z | 2020-09-21T19:01:25Z | OWNER | I did a bunch of initial work for this in #427. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705840673 | |
https://github.com/simonw/datasette/issues/971#issuecomment-696307922 | https://api.github.com/repos/simonw/datasette/issues/971 | 696307922 | MDEyOklzc3VlQ29tbWVudDY5NjMwNzkyMg== | 9599 | 2020-09-21T18:59:52Z | 2020-09-21T19:00:02Z | OWNER | Given `dbstat` isn't as widely available as I thought I'm going to let people who want to use `dbstat` run their own `select * from dbstat` queries rather than bake support directly into Datasette. The experience of exploring `dbstat` will improve if I land support for running facets against arbitrary custom SQL queries, which is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646 https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705827457 | |
https://github.com/simonw/datasette/issues/971#issuecomment-696304108 | https://api.github.com/repos/simonw/datasette/issues/971 | 696304108 | MDEyOklzc3VlQ29tbWVudDY5NjMwNDEwOA== | 9599 | 2020-09-21T18:52:50Z | 2020-09-21T18:52:50Z | OWNER | Looks like the `pysqlite3-binary` package doesn't support `dbstat` either. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705827457 | |
https://github.com/simonw/datasette/issues/971#issuecomment-696302868 | https://api.github.com/repos/simonw/datasette/issues/971 | 696302868 | MDEyOklzc3VlQ29tbWVudDY5NjMwMjg2OA== | 9599 | 2020-09-21T18:50:40Z | 2020-09-21T18:50:40Z | OWNER | Easiest way to get this may be to run `create view dbstat_view as select * from dbstat` on databases that support it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705827457 | |
https://github.com/simonw/datasette/issues/971#issuecomment-696302020 | https://api.github.com/repos/simonw/datasette/issues/971 | 696302020 | MDEyOklzc3VlQ29tbWVudDY5NjMwMjAyMA== | 9599 | 2020-09-21T18:49:09Z | 2020-09-21T18:49:09Z | OWNER | ... made harder to work on because I apparently don't have the `DBSTAT_VTAB` module on macOS. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705827457 | |
https://github.com/simonw/datasette/issues/971#issuecomment-696298614 | https://api.github.com/repos/simonw/datasette/issues/971 | 696298614 | MDEyOklzc3VlQ29tbWVudDY5NjI5ODYxNA== | 9599 | 2020-09-21T18:43:07Z | 2020-09-21T18:43:07Z | OWNER | Or, do this: ```sql SELECT 1 FROM dbstat limit 1; ``` And see if it returns a "table does not exist" error. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705827457 | |
https://github.com/simonw/datasette/issues/971#issuecomment-696297930 | https://api.github.com/repos/simonw/datasette/issues/971 | 696297930 | MDEyOklzc3VlQ29tbWVudDY5NjI5NzkzMA== | 9599 | 2020-09-21T18:41:47Z | 2020-09-21T18:41:47Z | OWNER | https://www.sqlite.org/dbstat.html > The DBSTAT virtual table is an eponymous virtual table, meaning that is not necessary to run CREATE VIRTUAL TABLE to create an instance of the dbstat virtual table before using it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705827457 | |
https://github.com/simonw/datasette/issues/971#issuecomment-696297601 | https://api.github.com/repos/simonw/datasette/issues/971 | 696297601 | MDEyOklzc3VlQ29tbWVudDY5NjI5NzYwMQ== | 9599 | 2020-09-21T18:41:07Z | 2020-09-21T18:41:07Z | OWNER | How to detect it? Looks like it's visible in SQLite compile time options: https://latest.datasette.io/-/versions ``` "compile_options": [ "COMPILER=gcc-8.3.0", "ENABLE_COLUMN_METADATA", "ENABLE_DBSTAT_VTAB", ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705827457 | |
https://github.com/simonw/datasette/issues/670#issuecomment-696163452 | https://api.github.com/repos/simonw/datasette/issues/670 | 696163452 | MDEyOklzc3VlQ29tbWVudDY5NjE2MzQ1Mg== | 652285 | 2020-09-21T14:46:10Z | 2020-09-21T14:46:10Z | NONE | I'm currently using PostgREST to serve OpenAPI APIs off Postgresql databases. I would like to try out datasette once this becomes available on Postgres. | { "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
564833696 | |
https://github.com/simonw/datasette/issues/970#issuecomment-695896557 | https://api.github.com/repos/simonw/datasette/issues/970 | 695896557 | MDEyOklzc3VlQ29tbWVudDY5NTg5NjU1Nw== | 9599 | 2020-09-21T04:40:12Z | 2020-09-21T04:40:12Z | OWNER | The Python standard library has a module for this: https://docs.python.org/3/library/webbrowser.html | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705108492 | |
https://github.com/simonw/datasette/issues/970#issuecomment-695895960 | https://api.github.com/repos/simonw/datasette/issues/970 | 695895960 | MDEyOklzc3VlQ29tbWVudDY5NTg5NTk2MA== | 9599 | 2020-09-21T04:36:45Z | 2020-09-21T04:36:45Z | OWNER | I like this. It could work with the `--root` option too and automatically sign you in as the root user. | { "total_count": 1, "+1": 0, "-1": 0, "laugh": 1, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705108492 | |
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879531 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | 695879531 | MDEyOklzc3VlQ29tbWVudDY5NTg3OTUzMQ== | 9599 | 2020-09-21T02:55:28Z | 2020-09-21T02:55:54Z | MEMBER | Actually for the tie-breaker it should be something like https://latest.datasette.io/fixtures?sql=select+pk%2C+created%2C+planet_int%2C+on_earth%2C+state%2C+city_id%2C+neighborhood%2C+tags%2C+complex_array%2C+distinct_some_null+from+facetable+where+%28created+%3E+%3Ap1+or+%28created+%3D+%3Ap1+and+%28%28pk+%3E+%3Ap0%29%29%29%29+order+by+created%2C+pk+limit+11&p0=10&p1=2019-01-16+08%3A00%3A00 ```sql where ( created > :p1 or ( created = :p1 and ((pk > :p0)) ) ) ``` But with `rowid` and `timestamp` in place of `pk` and `created`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705215230 | |
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879237 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | 695879237 | MDEyOklzc3VlQ29tbWVudDY5NTg3OTIzNw== | 9599 | 2020-09-21T02:53:29Z | 2020-09-21T02:53:29Z | MEMBER | If previous page ended at `2018-02-11T16:32:53+00:00`: ```sql select search_index.rowid, search_index.type, search_index.key, search_index.title, search_index.category, search_index.timestamp, search_index.search_1 from search_index where date("timestamp") = '2018-02-11' and timestamp < '2018-02-11T16:32:53+00:00' order by search_index.timestamp desc, rowid limit 41 ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705215230 | |
https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695877627 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16 | 695877627 | MDEyOklzc3VlQ29tbWVudDY5NTg3NzYyNw== | 9599 | 2020-09-21T02:42:29Z | 2020-09-21T02:42:29Z | MEMBER | Fun twist: assuming `timestamp` is always stored as UTC, I need the interface to be timezone aware so I can see e.g. everything from 4th July 2020 in the San Francisco timezone definition of 4th July 2020. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
694493566 | |
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695875274 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | 695875274 | MDEyOklzc3VlQ29tbWVudDY5NTg3NTI3NA== | 9599 | 2020-09-21T02:28:58Z | 2020-09-21T02:28:58Z | MEMBER | Datasette's implementation is complex because it has to support compound primary keys: https://github.com/simonw/datasette/blob/a258339a935d8d29a95940ef1db01e98bb85ae63/datasette/utils/__init__.py#L88-L114 - but that's not something that's needed for dogsheep-beta. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705215230 | |
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856967 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | 695856967 | MDEyOklzc3VlQ29tbWVudDY5NTg1Njk2Nw== | 9599 | 2020-09-21T00:26:59Z | 2020-09-21T00:26:59Z | MEMBER | It's a shame Datasette doesn't currently have an easy way to implement sorted-by-rank keyset-paginated using a TableView or QueryView. I'll have to do this using the custom SQL query constructed in the plugin: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L8-L43 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705215230 | |
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856398 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | 695856398 | MDEyOklzc3VlQ29tbWVudDY5NTg1NjM5OA== | 9599 | 2020-09-21T00:22:20Z | 2020-09-21T00:22:20Z | MEMBER | I'm going to try for keyset pagination sorted by relevance just as a learning exercise. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705215230 | |
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855723 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | 695855723 | MDEyOklzc3VlQ29tbWVudDY5NTg1NTcyMw== | 9599 | 2020-09-21T00:16:52Z | 2020-09-21T00:17:53Z | MEMBER | It feels a bit weird to implement keyset pagination against results sorted by `rank` because the ranks could change substantially if the search index gets updated while the user is paginating. I may just ignore that though. If you want reliable pagination you can get it by sorting by date. Maybe it doesn't even make sense to offer pagination if you sort by relevance? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705215230 | |
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855646 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26 | 695855646 | MDEyOklzc3VlQ29tbWVudDY5NTg1NTY0Ng== | 9599 | 2020-09-21T00:16:11Z | 2020-09-21T00:16:11Z | MEMBER | Should I do this with offset/limit or should I do proper keyset pagination? I think keyset because then it will work well for the full search interface with no filters or search string. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705215230 | |
https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695851036 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16 | 695851036 | MDEyOklzc3VlQ29tbWVudDY5NTg1MTAzNg== | 9599 | 2020-09-20T23:34:57Z | 2020-09-20T23:34:57Z | MEMBER | Really basic starting point is to add facet by date. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
694493566 | |
https://github.com/simonw/sqlite-utils/issues/160#issuecomment-695839557 | https://api.github.com/repos/simonw/sqlite-utils/issues/160 | 695839557 | MDEyOklzc3VlQ29tbWVudDY5NTgzOTU1Nw== | 9599 | 2020-09-20T21:37:03Z | 2020-09-20T21:37:03Z | OWNER | Should this support `ignore=True` as well? I'm tempted to skip that - I think `replace=True` is more useful because it implies "ignore if the options are already the same, but replace if they are different". | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
705190723 | |
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-695698227 | https://api.github.com/repos/simonw/sqlite-utils/issues/42 | 695698227 | MDEyOklzc3VlQ29tbWVudDY5NTY5ODIyNw== | 9599 | 2020-09-20T04:27:26Z | 2020-09-20T04:28:26Z | OWNER | This is going to need #114 (the `transform_table()` method) in order to convert string columns into integer foreign key columns. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
470345929 | |
https://github.com/simonw/sqlite-utils/issues/68#issuecomment-695695776 | https://api.github.com/repos/simonw/sqlite-utils/issues/68 | 695695776 | MDEyOklzc3VlQ29tbWVudDY5NTY5NTc3Ng== | 9599 | 2020-09-20T04:25:47Z | 2020-09-20T04:25:47Z | OWNER | This is a dupe of #130 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
531583658 | |
https://github.com/simonw/datasette/issues/943#issuecomment-695133768 | https://api.github.com/repos/simonw/datasette/issues/943 | 695133768 | MDEyOklzc3VlQ29tbWVudDY5NTEzMzc2OA== | 9599 | 2020-09-19T00:06:56Z | 2020-09-19T00:07:35Z | OWNER | [dogsheep-beta](https://github.com/dogsheep/dogsheep-beta) could do with this too. It currently [makes a call](https://github.com/dogsheep/dogsheep-beta/blob/ab36101bdae69b11af7c6bd7edee838d052e6ecf/dogsheep_beta/__init__.py#L216-L225) to `TableView` in a similar way to `datasette-graphql` in order to calculate facets. `dogsheep-beta` would benefit with a mechanism for changing the facet timeout setting during that call (as would `datasette-graphql`, see the [DatasetteSpecialConfig mechanism](https://github.com/simonw/datasette-graphql/blob/f9dc5c518b7cdc94b93873ef20069a7ea2882a95/datasette_graphql/utils.py#L516-L519) it uses). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
681375466 | |
https://github.com/dogsheep/dogsheep-beta/issues/15#issuecomment-695124698 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15 | 695124698 | MDEyOklzc3VlQ29tbWVudDY5NTEyNDY5OA== | 9599 | 2020-09-18T23:17:38Z | 2020-09-18T23:17:38Z | MEMBER | This can be part of the demo instance in #6. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
694136490 | |
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-695113871 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | 695113871 | MDEyOklzc3VlQ29tbWVudDY5NTExMzg3MQ== | 9599 | 2020-09-18T22:30:17Z | 2020-09-18T22:30:17Z | MEMBER | I think I know what's going on here: https://github.com/dogsheep/dogsheep-beta/blob/0f1b951c5131d16f3c8559a8e4d79ed5c559e3cb/dogsheep_beta/__init__.py#L166-L171 This is a logic bug - the `compiled` variable could be the template from the previous loop! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703970814 | |
https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695109140 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25 | 695109140 | MDEyOklzc3VlQ29tbWVudDY5NTEwOTE0MA== | 9599 | 2020-09-18T22:12:20Z | 2020-09-18T22:12:20Z | MEMBER | Documented here: https://github.com/dogsheep/dogsheep-beta/blob/534fc9689227eba70e69a45da0cee5820bbda9e1/README.md#datasette-plugin | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
704685890 | |
https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695108895 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25 | 695108895 | MDEyOklzc3VlQ29tbWVudDY5NTEwODg5NQ== | 9599 | 2020-09-18T22:11:32Z | 2020-09-18T22:11:32Z | MEMBER | I'm going to make this a new plugin configuration setting, `template_debug`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
704685890 | |
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694557425 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | 694557425 | MDEyOklzc3VlQ29tbWVudDY5NDU1NzQyNQ== | 9599 | 2020-09-17T23:41:01Z | 2020-09-17T23:41:01Z | MEMBER | I removed all of the `json.loads()` calls and I'm still getting that `Undefined` error. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703970814 | |
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | 694554584 | MDEyOklzc3VlQ29tbWVudDY5NDU1NDU4NA== | 9599 | 2020-09-17T23:31:25Z | 2020-09-17T23:31:25Z | MEMBER | I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703970814 | |
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694553579 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | 694553579 | MDEyOklzc3VlQ29tbWVudDY5NDU1MzU3OQ== | 9599 | 2020-09-17T23:28:37Z | 2020-09-17T23:28:37Z | MEMBER | More investigation in pdb: ``` (dogsheep-beta) dogsheep-beta % datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb > /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads() -> raise TypeError(f'the JSON object must be str, bytes or bytearray, ' (Pdb) list 336 if s.startswith('\ufeff'): 337 raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)", 338 s, 0) 339 else: 340 if not isinstance(s, (bytes, bytearray)): 341 -> raise TypeError(f'the JSON object must be str, bytes or bytearray, ' 342 f'not {s.__class__.__name__}') 343 s = s.decode(detect_encoding(s), 'surrogatepass') 344 345 if "encoding" in kw: 346 import warnings (Pdb) bytes <class 'bytes'> (Pdb) locals()['s'] Undefined (Pdb) type(locals()['s']) <class 'jinja2.runtime.Undefined'> ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703970814 | |
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552681 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | 694552681 | MDEyOklzc3VlQ29tbWVudDY5NDU1MjY4MQ== | 9599 | 2020-09-17T23:25:54Z | 2020-09-17T23:25:54Z | MEMBER | This is the template fragment it's rendering: ```html+jinja <div style="overflow: hidden;"> <p>Tweet by <a href="https://twitter.com/{{ display.screen_name }}">@{{ display.screen_name }}</a> ({{ display.user_name }}, {{ "{:,}".format(display.followers_count or 0) }} followers) on <a href="https://twitter.com/{{ display.screen_name }}/status/{{ display.tweet_id }}">{{ display.created_at }}</a></p> </p> <blockquote>{{ display.full_text }}</blockquote> {% if display.media_urls and json.loads(display.media_urls) %} {% for url in json.loads(display.media_urls) %} <img src="{{ url }}" style="height: 200px;"> {% endfor %} {% endif %} </div> ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703970814 | |
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552393 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | 694552393 | MDEyOklzc3VlQ29tbWVudDY5NDU1MjM5Mw== | 9599 | 2020-09-17T23:25:01Z | 2020-09-17T23:25:17Z | MEMBER | Ran `locals()` In the debugger: `{'range': <class 'range'>, 'dict': <class 'dict'>, 'lipsum': <function generate_lorem_ipsum at 0x10aeff430>, 'cycler': <class 'jinja2.utils.Cycler'>, 'joiner': <class 'jinja2.utils.Joiner'>, 'namespace': <class 'jinja2.utils.Namespace'>, 'rank': -9.383801886431414, 'rowid': 14297, 'type': 'twitter.db/tweets', 'key': '312658917933076480', 'title': 'Tweet by @chrisstreeter', 'category': 2, 'timestamp': '2013-03-15T20:17:49+00:00', 'search_1': '@simonw are you at pycon? Would love to meet you.', 'display': {'avatar_url': 'https://pbs.twimg.com/profile_images/806275088597204993/38yLHfJi_normal.jpg', 'user_name': 'Chris Streeter', 'screen_name': 'chrisstreeter', 'followers_count': 280, 'tweet_id': 312658917933076480, 'created_at': '2013-03-15T20:17:49+00:00', 'full_text': '@simonw are you at pycon? Would love to meet you.', 'media_urls_2': '[]', 'media_urls': '[]'}, 'json': <module 'json' from '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py'>}` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703970814 | |
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551646 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | 694551646 | MDEyOklzc3VlQ29tbWVudDY5NDU1MTY0Ng== | 9599 | 2020-09-17T23:22:48Z | 2020-09-17T23:22:48Z | MEMBER | Looks like its happening in a Jinja fragment template for one of the results: ``` /Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/__init__.py(169)process_results() -> output = compiled.render({**result, **{"json": json}}) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/asyncsupport.py(71)render() -> return original_render(self, *args, **kwargs) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(1090)render() -> self.environment.handle_exception() /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(832)handle_exception() -> reraise(*rewrite_traceback_stack(source=source)) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/_compat.py(28)reraise() -> raise value.with_traceback(tb) <template>(5)top-level template code() > /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads() -> raise TypeError(f'the JSON object must be str, bytes or bytearray, ' ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703970814 | |
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551406 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24 | 694551406 | MDEyOklzc3VlQ29tbWVudDY5NDU1MTQwNg== | 9599 | 2020-09-17T23:22:07Z | 2020-09-17T23:22:07Z | MEMBER | Neat, I can debug this with the new `--pdb` option: datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703970814 | |
https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-694548909 | https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16 | 694548909 | MDEyOklzc3VlQ29tbWVudDY5NDU0ODkwOQ== | 9599 | 2020-09-17T23:15:09Z | 2020-09-17T23:15:09Z | MEMBER | I have sort by date now, #21. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
694493566 | |
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693794700 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | 693794700 | MDEyOklzc3VlQ29tbWVudDY5Mzc5NDcwMA== | 9599 | 2020-09-17T04:02:39Z | 2020-09-17T04:02:39Z | MEMBER | It would be useful if you could pass an `--accept` option to this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703218756 | |
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693789129 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | 693789129 | MDEyOklzc3VlQ29tbWVudDY5Mzc4OTEyOQ== | 9599 | 2020-09-17T03:40:01Z | 2020-09-17T03:40:01Z | MEMBER | Bug with endpoints that return dictionaries rather than arrays: ``` github-to-sqlite get /users/simonw [ "login", "id", "node_id", "avatar_url", "gravatar_id", "url", "html_url", "followers_url", "following_url", "gists_url", "starred_url", "subscriptions_url", "organizations_url", "repos_url", "events_url", "received_events_url", "type", "site_admin", "name", "company", "blog", "location", "email", "hireable", "bio", "twitter_username", "public_repos", "public_gists", "followers", "following", "created_at", "updated_at" ] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703218756 | |
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788387 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | 693788387 | MDEyOklzc3VlQ29tbWVudDY5Mzc4ODM4Nw== | 9599 | 2020-09-17T03:36:47Z | 2020-09-17T03:36:58Z | MEMBER | Fun demo of the `--nl` option: github-to-sqlite get /users/simonw/repos --paginate --nl | sqlite-utils insert simonw.db repos - --nl | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703218756 | |
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788032 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | 693788032 | MDEyOklzc3VlQ29tbWVudDY5Mzc4ODAzMg== | 9599 | 2020-09-17T03:35:22Z | 2020-09-17T03:35:22Z | MEMBER | Documentation: https://github.com/dogsheep/github-to-sqlite/blob/b02bf135485c0a7a3768868967f45a6b5e515289/README.md#making-authenticated-api-calls | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703218756 | |
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693775622 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | 693775622 | MDEyOklzc3VlQ29tbWVudDY5Mzc3NTYyMg== | 9599 | 2020-09-17T02:48:34Z | 2020-09-17T02:48:34Z | MEMBER | I'd like a `--paginate` option that does the same thing as https://github.com/simonw/paginate-json | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703218756 | |
https://github.com/simonw/datasette/issues/507#issuecomment-693775390 | https://api.github.com/repos/simonw/datasette/issues/507 | 693775390 | MDEyOklzc3VlQ29tbWVudDY5Mzc3NTM5MA== | 9599 | 2020-09-17T02:47:35Z | 2020-09-17T02:47:35Z | OWNER | I have a pattern for creating screenshots using Puppeteer running in a GitHub Action now, see https://simonwillison.net/2020/Sep/3/weeknotes-airtable-screenshots-dogsheep/#weeknotes-2020-09-03-social-media-cards-tils | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
455852801 | |
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693773191 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50 | 693773191 | MDEyOklzc3VlQ29tbWVudDY5Mzc3MzE5MQ== | 9599 | 2020-09-17T02:39:26Z | 2020-09-17T02:39:26Z | MEMBER | I'm going to start with `github-to-sqlite get` and `github-to-sqlite post` - I may add `put` and suchlike later on. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
703218756 | |
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693695177 | https://api.github.com/repos/simonw/sqlite-utils/issues/159 | 693695177 | MDEyOklzc3VlQ29tbWVudDY5MzY5NTE3Nw== | 9599 | 2020-09-16T22:17:53Z | 2020-09-16T22:17:53Z | OWNER | @spdkils can you share a minimal code example that exhibits the behavior you're seeing? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
702386948 | |
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693694968 | https://api.github.com/repos/simonw/sqlite-utils/issues/159 | 693694968 | MDEyOklzc3VlQ29tbWVudDY5MzY5NDk2OA== | 9599 | 2020-09-16T22:17:19Z | 2020-09-16T22:17:19Z | OWNER | That's strange... this test here doesn't manually commit a transaction and passes: https://github.com/simonw/sqlite-utils/blob/7805d53bcf11199bd1f2b07e05ae90151f9d0eb0/tests/test_delete.py#L17-L23 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
702386948 | |
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693694343 | https://api.github.com/repos/simonw/sqlite-utils/issues/159 | 693694343 | MDEyOklzc3VlQ29tbWVudDY5MzY5NDM0Mw== | 9599 | 2020-09-16T22:15:39Z | 2020-09-16T22:15:39Z | OWNER | Independent of the transaction changes in #121 I may be able to check `self.conn.in_transaction` to see if a transaction is active and, if one is NOT active, execute the delete inside of one. https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.in_transaction | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
702386948 | |
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693589321 | https://api.github.com/repos/simonw/sqlite-utils/issues/159 | 693589321 | MDEyOklzc3VlQ29tbWVudDY5MzU4OTMyMQ== | 9599 | 2020-09-16T18:41:42Z | 2020-09-16T18:41:42Z | OWNER | Yeah I'm going to class this as a bug - that's definitely confusing. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
702386948 | |
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693486183 | https://api.github.com/repos/simonw/sqlite-utils/issues/159 | 693486183 | MDEyOklzc3VlQ29tbWVudDY5MzQ4NjE4Mw== | 11712349 | 2020-09-16T15:34:13Z | 2020-09-16T15:34:13Z | NONE | I appreciate the response, it's just unexpected. If I insert, it commits, if I update it commits, if I upsert it commits... if I delete.. it doesn't??? Confused me... I did just db commit it... But it's confusing. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
702386948 | |
https://github.com/simonw/sqlite-utils/pull/158#issuecomment-693199392 | https://api.github.com/repos/simonw/sqlite-utils/issues/158 | 693199392 | MDEyOklzc3VlQ29tbWVudDY5MzE5OTM5Mg== | 9599 | 2020-09-16T06:21:29Z | 2020-09-16T06:21:29Z | OWNER | Thanks! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
697203800 | |
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693199049 | https://api.github.com/repos/simonw/sqlite-utils/issues/159 | 693199049 | MDEyOklzc3VlQ29tbWVudDY5MzE5OTA0OQ== | 9599 | 2020-09-16T06:20:26Z | 2020-09-16T06:20:26Z | OWNER | See #121 - I need to think harder about how this all interacts with transactions. You can do this: ```python with db.conn: db["mytable"].delete_where() ``` But that should be documented and maybe rethought. | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
702386948 | |
https://github.com/simonw/datasette/issues/943#issuecomment-693010291 | https://api.github.com/repos/simonw/datasette/issues/943 | 693010291 | MDEyOklzc3VlQ29tbWVudDY5MzAxMDI5MQ== | 9599 | 2020-09-15T22:20:55Z | 2020-09-15T22:20:55Z | OWNER | Should I instantiate a single `Client` and reuse it for all internal requests, or can I instantiate a new `Client` for each request? https://www.python-httpx.org/advanced/#why-use-a-client says that the main benefit of a Client instance is HTTP connection pooling - which isn't an issue for these internal requests since they won't be using the HTTP protocol at all, they'll be calling the ASGI application directly. So I'm leaning towards instantiating a fresh client for every internal request. I'll run a microbenchmark to check that this doesn't have any unpleasant performance implications. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
681375466 | |
https://github.com/simonw/datasette/issues/943#issuecomment-693008540 | https://api.github.com/repos/simonw/datasette/issues/943 | 693008540 | MDEyOklzc3VlQ29tbWVudDY5MzAwODU0MA== | 9599 | 2020-09-15T22:16:07Z | 2020-09-15T22:16:07Z | OWNER | I think I can use `async with httpx.AsyncClient(base_url="http://localhost/") as client:` to ensure I don't need to use `http://localhost/` on every call. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
681375466 | |
https://github.com/simonw/datasette/issues/943#issuecomment-693007512 | https://api.github.com/repos/simonw/datasette/issues/943 | 693007512 | MDEyOklzc3VlQ29tbWVudDY5MzAwNzUxMg== | 9599 | 2020-09-15T22:13:30Z | 2020-09-15T22:13:30Z | OWNER | I could solve streaming using something like this: ```python async with datasette.stream("GET", "/fixtures/compound_three_primary_keys.csv?_stream=on&_size=max") as response: async for chunk in response.aiter_bytes(): print(chunk) ``` Which would be a wrapper around `AsyncClient.stream(method, url, ...)` from https://www.python-httpx.org/async/#streaming-responses | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
681375466 | |
https://github.com/simonw/datasette/issues/943#issuecomment-693005033 | https://api.github.com/repos/simonw/datasette/issues/943 | 693005033 | MDEyOklzc3VlQ29tbWVudDY5MzAwNTAzMw== | 9599 | 2020-09-15T22:06:58Z | 2020-09-15T22:10:58Z | OWNER | What if `datasette.get()` was an alias for `httpx.get()`, pre-configured to route to the correct application? And with some sugar that added `http://localhost/` to the beginning of the path if it was missing? This would make `httpx` a dependency of core Datasette, which I think is OK. It would also solve the return type problem: I would return whatever `httpx` returns. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
681375466 | |
https://github.com/simonw/datasette/issues/943#issuecomment-693004770 | https://api.github.com/repos/simonw/datasette/issues/943 | 693004770 | MDEyOklzc3VlQ29tbWVudDY5MzAwNDc3MA== | 9599 | 2020-09-15T22:06:13Z | 2020-09-15T22:06:13Z | OWNER | I'm tempted to create a `await datasette.request()` method which can take any HTTP verb - then have `datasette.get()` and `datasette.post()` as thin wrappers around it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
681375466 | |
https://github.com/simonw/datasette/issues/943#issuecomment-693004572 | https://api.github.com/repos/simonw/datasette/issues/943 | 693004572 | MDEyOklzc3VlQ29tbWVudDY5MzAwNDU3Mg== | 9599 | 2020-09-15T22:05:39Z | 2020-09-15T22:05:39Z | OWNER | Maybe these methods become the way most Datasette tests are written, replacing the existing `TestClient` mechanism? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
681375466 | |
https://github.com/simonw/datasette/issues/943#issuecomment-693004296 | https://api.github.com/repos/simonw/datasette/issues/943 | 693004296 | MDEyOklzc3VlQ29tbWVudDY5MzAwNDI5Ng== | 9599 | 2020-09-15T22:04:54Z | 2020-09-15T22:04:54Z | OWNER | So what should I do about streaming responses? I could deliberately ignore them - through an exception if you attempt to run `await datasette.get(...)` against a streaming URL. I could load the entire response into memory and return it as a wrapped object. I could support some kind of asynchronous iterator mechanism. This would be pretty elegant if I could decide the right syntax for it - it would allow plugins to take advantage of other internal URLs that return streaming content without needing to load that content entirely into memory in order to process it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
681375466 | |
https://github.com/simonw/datasette/issues/943#issuecomment-693003652 | https://api.github.com/repos/simonw/datasette/issues/943 | 693003652 | MDEyOklzc3VlQ29tbWVudDY5MzAwMzY1Mg== | 9599 | 2020-09-15T22:03:08Z | 2020-09-15T22:03:08Z | OWNER | I'm not going to mess around with formats - you'll get back the exact response that a web client would receive. Question: what should the response object look like? e.g. if you do: response = await datasette.get("/db/table.json") What should `response` be? I could reuse the Datasette `Response` class from `datasette.utils.asgi`. This would work well for regular responses which just have a status code, some headers and a response body. It wouldn't be great for streaming responses though such as you get back from `?_stream=1` CSV exports. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
681375466 | |
https://github.com/simonw/datasette/issues/891#issuecomment-693001937 | https://api.github.com/repos/simonw/datasette/issues/891 | 693001937 | MDEyOklzc3VlQ29tbWVudDY5MzAwMTkzNw== | 9599 | 2020-09-15T21:58:56Z | 2020-09-15T21:58:56Z | OWNER | Here's what that looks like: ``` Traceback (most recent call last): File "/Users/simon/Dropbox/Development/datasette/plugins/sql_error.py", line 5, in oh_no_error return 100 / 0 ZeroDivisionError: division by zero ERROR: conn=<sqlite3.Connection object at 0x10bce0030>, sql = 'select oh_no_error()', params = {}: user-defined function raised exception INFO: 127.0.0.1:54066 - "GET /data?sql=select+oh_no_error%28%29 HTTP/1.1" 400 Bad Request ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
653529088 | |
https://github.com/simonw/datasette/issues/891#issuecomment-693000522 | https://api.github.com/repos/simonw/datasette/issues/891 | 693000522 | MDEyOklzc3VlQ29tbWVudDY5MzAwMDUyMg== | 9599 | 2020-09-15T21:55:11Z | 2020-09-15T21:55:11Z | OWNER | I'm going to turn this on. If people complain about it I can turn it off again (or make it a configuration setting). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
653529088 | |
https://github.com/simonw/datasette/issues/891#issuecomment-692999893 | https://api.github.com/repos/simonw/datasette/issues/891 | 692999893 | MDEyOklzc3VlQ29tbWVudDY5Mjk5OTg5Mw== | 9599 | 2020-09-15T21:53:36Z | 2020-09-15T21:53:36Z | OWNER | Here's the commit (from 15 years ago) where `enable_callback_tracebacks` was first added: https://github.com/ghaering/pysqlite/commit/1e8bd36be93b7d7425910642b72e4152c77b0dfd > - Exceptions in callbacks lead to the query being aborted now instead of silently leading to generating values. > - Exceptions in callbacks can be echoed to stderr if you call the module level function enable_callback_tracebacks: enable_callback_tracebacks(1). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
653529088 | |
https://github.com/simonw/datasette/issues/891#issuecomment-692998061 | https://api.github.com/repos/simonw/datasette/issues/891 | 692998061 | MDEyOklzc3VlQ29tbWVudDY5Mjk5ODA2MQ== | 9599 | 2020-09-15T21:49:03Z | 2020-09-15T21:49:03Z | OWNER | I've been trying to figure out why this is an optional setting that defaults to off. I think it's because it writes directly to `stderr`, so the maintainers of `sqlite3` reasonably decided that people should be able to opt in to that rather than having weird stuff show up on `stderr` that they weren't expecting. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
653529088 | |
https://github.com/simonw/datasette/issues/891#issuecomment-692968792 | https://api.github.com/repos/simonw/datasette/issues/891 | 692968792 | MDEyOklzc3VlQ29tbWVudDY5Mjk2ODc5Mg== | 9599 | 2020-09-15T20:44:15Z | 2020-09-15T20:44:15Z | OWNER | https://github.com/peter-wangxu/persist-queue/issues/74 warns that this might not work with PyPy. I could solve that with: ```python if hasattr(sqlite3, "enable_callback_tracebacks"): sqlite3.enable_callback_tracebacks(True) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
653529088 | |
https://github.com/simonw/datasette/issues/877#issuecomment-692967733 | https://api.github.com/repos/simonw/datasette/issues/877 | 692967733 | MDEyOklzc3VlQ29tbWVudDY5Mjk2NzczMw== | 9599 | 2020-09-15T20:42:04Z | 2020-09-15T20:42:04Z | OWNER | I'm not going to drop CSRF protection - it's still needed for older browsers - but I have relaxed the circumstances under which it is applied. It only applies to requests that include cookies for example, so API clients that don't send cookies don't need to worry about it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
648421105 | |
https://github.com/simonw/datasette/issues/889#issuecomment-692967123 | https://api.github.com/repos/simonw/datasette/issues/889 | 692967123 | MDEyOklzc3VlQ29tbWVudDY5Mjk2NzEyMw== | 9599 | 2020-09-15T20:40:52Z | 2020-09-15T20:40:52Z | OWNER | Thanks - I've fixed this in `datasette-media` and the other plugins that use that hook now I think. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
649907676 | |
https://github.com/simonw/datasette/issues/888#issuecomment-692966625 | https://api.github.com/repos/simonw/datasette/issues/888 | 692966625 | MDEyOklzc3VlQ29tbWVudDY5Mjk2NjYyNQ== | 9599 | 2020-09-15T20:39:49Z | 2020-09-15T20:39:49Z | OWNER | Thanks, I've fixed that now. It only affected the GitHub release notes - the ones at https://docs.datasette.io/en/stable/changelog.html#v0-45 had the correct links. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
649702801 | |
https://github.com/simonw/datasette/issues/634#issuecomment-692965761 | https://api.github.com/repos/simonw/datasette/issues/634 | 692965761 | MDEyOklzc3VlQ29tbWVudDY5Mjk2NTc2MQ== | 9599 | 2020-09-15T20:37:58Z | 2020-09-15T20:37:58Z | OWNER | I fixed this in 5e0b72247ecab4ce0fcec599b77a83d73a480872 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
522352520 | |
https://github.com/simonw/datasette/issues/849#issuecomment-692965391 | https://api.github.com/repos/simonw/datasette/issues/849 | 692965391 | MDEyOklzc3VlQ29tbWVudDY5Mjk2NTM5MQ== | 9599 | 2020-09-15T20:37:14Z | 2020-09-15T20:37:14Z | OWNER | I've been running on `main` for a while now with no issues. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
639072811 | |
https://github.com/simonw/datasette/issues/956#issuecomment-692965022 | https://api.github.com/repos/simonw/datasette/issues/956 | 692965022 | MDEyOklzc3VlQ29tbWVudDY5Mjk2NTAyMg== | 9599 | 2020-09-15T20:36:34Z | 2020-09-15T20:36:34Z | OWNER | https://hub.docker.com/r/datasetteproject/datasette/tags - 0.49.1 was successfully pushed to Docker Hub by https://github.com/simonw/datasette/runs/1119815175?check_suite_focus=true | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
688427751 | |
https://github.com/simonw/datasette/issues/956#issuecomment-692955850 | https://api.github.com/repos/simonw/datasette/issues/956 | 692955850 | MDEyOklzc3VlQ29tbWVudDY5Mjk1NTg1MA== | 9599 | 2020-09-15T20:17:49Z | 2020-09-15T20:17:49Z | OWNER | I think I've fixed this with recent changes I made as part of #941 - but I won't know until I release the next version. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
688427751 | |
https://github.com/simonw/datasette/issues/946#issuecomment-692955379 | https://api.github.com/repos/simonw/datasette/issues/946 | 692955379 | MDEyOklzc3VlQ29tbWVudDY5Mjk1NTM3OQ== | 9599 | 2020-09-15T20:16:50Z | 2020-09-15T20:16:50Z | OWNER | Can't reproduce this bug now. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
682184050 | |
https://github.com/simonw/datasette/issues/492#issuecomment-692953174 | https://api.github.com/repos/simonw/datasette/issues/492 | 692953174 | MDEyOklzc3VlQ29tbWVudDY5Mjk1MzE3NA== | 9599 | 2020-09-15T20:12:29Z | 2020-09-15T20:12:29Z | OWNER | I fixed this in ea340cf320a2566d24517fb4a0c9852c5059e771 for #963 (a duplicate of this issue). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
449854604 | |
https://github.com/simonw/datasette/issues/967#issuecomment-692951144 | https://api.github.com/repos/simonw/datasette/issues/967 | 692951144 | MDEyOklzc3VlQ29tbWVudDY5Mjk1MTE0NA== | 9599 | 2020-09-15T20:08:12Z | 2020-09-15T20:08:12Z | OWNER | I think the easiest fix is for me to ensure that calls to `__len__` on the `MagicParameters` class always return at least 1. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
702069429 | |
https://github.com/simonw/datasette/issues/967#issuecomment-692946616 | https://api.github.com/repos/simonw/datasette/issues/967 | 692946616 | MDEyOklzc3VlQ29tbWVudDY5Mjk0NjYxNg== | 9599 | 2020-09-15T19:59:21Z | 2020-09-15T19:59:21Z | OWNER | I wish I could call https://www.sqlite.org/c3ref/bind_parameter_count.html and https://www.sqlite.org/c3ref/bind_parameter_name.html from Python. Might be possible to do that using `ctypes` - see this example code: https://mail.python.org/pipermail//pypy-commit/2013-February/071372.html ```python param_count = lib.sqlite3_bind_parameter_count(self.statement) for idx in range(1, param_count + 1): param_name = lib.sqlite3_bind_parameter_name(self.statement, idx) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
702069429 | |
https://github.com/simonw/datasette/issues/967#issuecomment-692945504 | https://api.github.com/repos/simonw/datasette/issues/967 | 692945504 | MDEyOklzc3VlQ29tbWVudDY5Mjk0NTUwNA== | 9599 | 2020-09-15T19:57:10Z | 2020-09-15T19:57:10Z | OWNER | So the problem actually occurs when the `MagicParameters` class wraps an empty dictionary. Relevant code: https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L228-L236 And: https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L364-L383 I'm passing a special magic parameters dictionary for the Python `sqlite3` module to look up parameters in. When that dictionary is `{}` a `__len__` check is performed on that dictionary, the result comes back as 0 and as a result it assumes there are no parameters. I tracked down the relevant C code: https://github.com/python/cpython/blob/81715808716198471fbca0a3db42ac408468dbc5/Modules/_sqlite/statement.c#L218-L237 ```c Py_BEGIN_ALLOW_THREADS num_params_needed = sqlite3_bind_parameter_count(self->st); Py_END_ALLOW_THREADS if (PyTuple_CheckExact(parameters) || PyList_CheckExact(parameters) || (!PyDict_Check(parameters) && PySequence_Check(parameters))) { /* parameters passed as sequence */ if (PyTuple_CheckExact(parameters)) { num_params = PyTuple_GET_SIZE(parameters); } else if (PyList_CheckExact(parameters)) { num_params = PyList_GET_SIZE(parameters); } else { num_params = PySequence_Size(parameters); } if (num_params != num_params_needed) { PyErr_Format(pysqlite_ProgrammingError, "Incorrect number of bindings supplied. The current " "statement uses %d, and there are %zd supplied.", num_params_needed, num_params); return; } ``` It looks to me like this should fail if the number of keys known to be in the dictionary differs from the number of named parameters in the query. But if those numbers fail to match it still works as far as I can tell - it's only dictionary length of 0 that is causing the problems. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
702069429 | |
https://github.com/simonw/datasette/issues/967#issuecomment-692940375 | https://api.github.com/repos/simonw/datasette/issues/967 | 692940375 | MDEyOklzc3VlQ29tbWVudDY5Mjk0MDM3NQ== | 9599 | 2020-09-15T19:47:09Z | 2020-09-15T19:47:09Z | OWNER | Yes! The tests all pass if I update the test function to do this: ```python response = magic_parameters_client.post( "/data/runme_post{}".format(qs), {"ignore_me": "1"}, csrftoken_from=use_csrf or None, allow_redirects=False, ) ``` So the bug only occurs if the POST body is completely empty. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
702069429 | |
https://github.com/simonw/datasette/issues/967#issuecomment-692938935 | https://api.github.com/repos/simonw/datasette/issues/967 | 692938935 | MDEyOklzc3VlQ29tbWVudDY5MjkzODkzNQ== | 9599 | 2020-09-15T19:44:21Z | 2020-09-15T19:44:41Z | OWNER | While I'm running the above test, in the rounds that work the `receive()` awaitable returns `{'type': 'http.request', 'body': b'csrftoken=IlpwUGlSMFVVa3Z3ZlVoamQi.uY2U1tF4i0M-5M6x34vnBCmJgr0'}` In the rounds that fails it returns `{'type': 'http.request'}` So it looks like the `csrftoken_from=True` parameter may be helping just by ensuring the `body` key is present and not missing. I wonder if it would work if a body of `b''` was present there? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
702069429 |