issue_comments
23 rows where reactions = "{"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}" and "updated_at" is on date 2020-02-24 sorted by updated_at descending
This data as json, CSV (advanced)
Suggested facets: issue_url, created_at (date), updated_at (date)
user 4
- simonw 19
- aviflax 2
- clausjuhl 1
- tunguyenatwork 1
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
590608228 | https://github.com/simonw/datasette/pull/683#issuecomment-590608228 | https://api.github.com/repos/simonw/datasette/issues/683 | MDEyOklzc3VlQ29tbWVudDU5MDYwODIyOA== | simonw 9599 | 2020-02-24T23:52:35Z | 2020-02-24T23:52:35Z | OWNER | I'm going to punt on the ability to introspect the write queue and poll for completion using a UUID for the moment. Can add those later. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.execute_write() and .execute_write_fn() methods on Database 570101428 | |
590607385 | https://github.com/simonw/datasette/pull/683#issuecomment-590607385 | https://api.github.com/repos/simonw/datasette/issues/683 | MDEyOklzc3VlQ29tbWVudDU5MDYwNzM4NQ== | simonw 9599 | 2020-02-24T23:49:37Z | 2020-02-24T23:49:37Z | OWNER | Here's the class UploadApp(HTTPEndpoint): def init(self, scope, receive, send, datasette): self.datasette = datasette super().init(scope, receive, send)
@hookimpl def asgi_wrapper(datasette): def wrap_with_asgi_auth(app): async def wrapped_app(scope, recieve, send): if scope["path"] == "/-/upload-csv": await UploadApp(scope, recieve, send, datasette) else: await app(scope, recieve, send)
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.execute_write() and .execute_write_fn() methods on Database 570101428 | |
590606825 | https://github.com/simonw/datasette/pull/683#issuecomment-590606825 | https://api.github.com/repos/simonw/datasette/issues/683 | MDEyOklzc3VlQ29tbWVudDU5MDYwNjgyNQ== | simonw 9599 | 2020-02-24T23:47:38Z | 2020-02-24T23:47:38Z | OWNER | Another demo plugin: class DeleteTableApp(HTTPEndpoint): def init(self, scope, receive, send, datasette): self.datasette = datasette super().init(scope, receive, send)
@hookimpl def asgi_wrapper(datasette): def wrap_with_asgi_auth(app): async def wrapped_app(scope, recieve, send): if scope["path"] == "/-/delete-table": await DeleteTableApp(scope, recieve, send, datasette) else: await app(scope, recieve, send)
{% block content %} <form action="/-/delete-table" method="POST"></form> {{ super() }} {% endblock %} ``` (Needs CSRF protection added) I ran Datasette like this:
Result: I can delete tables! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.execute_write() and .execute_write_fn() methods on Database 570101428 | |
590599257 | https://github.com/simonw/datasette/pull/683#issuecomment-590599257 | https://api.github.com/repos/simonw/datasette/issues/683 | MDEyOklzc3VlQ29tbWVudDU5MDU5OTI1Nw== | simonw 9599 | 2020-02-24T23:21:56Z | 2020-02-24T23:22:35Z | OWNER | Also: are UUIDs really necessary here or could I use a simpler form of task identifier? Like an in-memory counter variable that starts at 0 and increments every time this instance of Datasette issues a new task ID? The neat thing about UUIDs is that I don't have to worry if there are multiple Datasette instances accepting writes behind a load balancer. That seems pretty unlikely (especially considering SQLite databases encourage only one process to be writing at a time)... but I am experimenting with PostgreSQL support in #670 so it's probably worth ensuring these task IDs really are globally unique. I'm going to stick with UUIDs. They're short-lived enough that their size doesn't really matter. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.execute_write() and .execute_write_fn() methods on Database 570101428 | |
590598689 | https://github.com/simonw/datasette/pull/683#issuecomment-590598689 | https://api.github.com/repos/simonw/datasette/issues/683 | MDEyOklzc3VlQ29tbWVudDU5MDU5ODY4OQ== | simonw 9599 | 2020-02-24T23:20:11Z | 2020-02-24T23:20:11Z | OWNER | I think But is it weird having a function that returns different types depending on if you passed I'm OK with the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.execute_write() and .execute_write_fn() methods on Database 570101428 | |
590598248 | https://github.com/simonw/datasette/pull/683#issuecomment-590598248 | https://api.github.com/repos/simonw/datasette/issues/683 | MDEyOklzc3VlQ29tbWVudDU5MDU5ODI0OA== | simonw 9599 | 2020-02-24T23:18:50Z | 2020-02-24T23:18:50Z | OWNER | I'm not convinced by the return value of the Do I really need that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.execute_write() and .execute_write_fn() methods on Database 570101428 | |
590593247 | https://github.com/simonw/datasette/issues/675#issuecomment-590593247 | https://api.github.com/repos/simonw/datasette/issues/675 | MDEyOklzc3VlQ29tbWVudDU5MDU5MzI0Nw== | aviflax 141844 | 2020-02-24T23:02:52Z | 2020-02-24T23:02:52Z | NONE |
Excellent, thanks!
Hmm, well, anyone running
I think I like it. Easy to remember! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--cp option for datasette publish and datasette package for shipping additional files and directories 567902704 | |
590593120 | https://github.com/simonw/datasette/pull/683#issuecomment-590593120 | https://api.github.com/repos/simonw/datasette/issues/683 | MDEyOklzc3VlQ29tbWVudDU5MDU5MzEyMA== | simonw 9599 | 2020-02-24T23:02:30Z | 2020-02-24T23:02:30Z | OWNER | I'm going to muck around with a couple more demo plugins - in particular one derived from datasette-upload-csvs - to make sure I'm comfortable with this API - then add a couple of tests and merge it with documentation that warns "this is still an experimental feature and may change". |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.execute_write() and .execute_write_fn() methods on Database 570101428 | |
590592581 | https://github.com/simonw/datasette/pull/683#issuecomment-590592581 | https://api.github.com/repos/simonw/datasette/issues/683 | MDEyOklzc3VlQ29tbWVudDU5MDU5MjU4MQ== | simonw 9599 | 2020-02-24T23:00:44Z | 2020-02-24T23:01:09Z | OWNER | I've been testing this out by running one-off demo plugins. I saved the following in a file called class AsgiLogToSqliteViaWriteQueue: lookup_columns = ( "path", "user_agent", "referer", "accept_language", "content_type", "query_string", )
def header(d, name): return d.get(name.encode("utf8"), b"").decode("utf8") or None def lookup(db, table, value): return db[table].lookup({"name": value}) if value else None @hookimpl def asgi_wrapper(datasette): def wrap_with_class(app): return AsgiLogToSqliteViaWriteQueue( app, next(iter(datasette.databases.values())) )
``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.execute_write() and .execute_write_fn() methods on Database 570101428 | |
590543398 | https://github.com/simonw/datasette/issues/681#issuecomment-590543398 | https://api.github.com/repos/simonw/datasette/issues/681 | MDEyOklzc3VlQ29tbWVudDU5MDU0MzM5OA== | clausjuhl 2181410 | 2020-02-24T20:53:56Z | 2020-02-24T20:53:56Z | NONE | Excellent. I'll implement the simple plugin-solution now. And will have a go at a more mature plugin later. Thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Cashe-header missing in http-response 569317377 | |
590539805 | https://github.com/simonw/datasette/issues/675#issuecomment-590539805 | https://api.github.com/repos/simonw/datasette/issues/675 | MDEyOklzc3VlQ29tbWVudDU5MDUzOTgwNQ== | simonw 9599 | 2020-02-24T20:44:59Z | 2020-02-24T20:45:08Z | OWNER | Design looks great to me. I'm not keen on two letter short versions ( |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--cp option for datasette publish and datasette package for shipping additional files and directories 567902704 | |
590518182 | https://github.com/simonw/datasette/pull/683#issuecomment-590518182 | https://api.github.com/repos/simonw/datasette/issues/683 | MDEyOklzc3VlQ29tbWVudDU5MDUxODE4Mg== | simonw 9599 | 2020-02-24T19:53:12Z | 2020-02-24T19:53:12Z | OWNER | Next steps are from comment https://github.com/simonw/datasette/issues/682#issuecomment-590517338
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.execute_write() and .execute_write_fn() methods on Database 570101428 | |
590517744 | https://github.com/simonw/datasette/issues/682#issuecomment-590517744 | https://api.github.com/repos/simonw/datasette/issues/682 | MDEyOklzc3VlQ29tbWVudDU5MDUxNzc0NA== | simonw 9599 | 2020-02-24T19:52:16Z | 2020-02-24T19:52:16Z | OWNER | Moving further development to a pull request: https://github.com/simonw/datasette/pull/683 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for writing to database via a queue 569613563 | |
590511601 | https://github.com/simonw/datasette/issues/682#issuecomment-590511601 | https://api.github.com/repos/simonw/datasette/issues/682 | MDEyOklzc3VlQ29tbWVudDU5MDUxMTYwMQ== | simonw 9599 | 2020-02-24T19:38:27Z | 2020-02-24T19:38:27Z | OWNER | I tested this using the following code in a view (after |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for writing to database via a queue 569613563 | |
590436368 | https://github.com/simonw/datasette/issues/682#issuecomment-590436368 | https://api.github.com/repos/simonw/datasette/issues/682 | MDEyOklzc3VlQ29tbWVudDU5MDQzNjM2OA== | simonw 9599 | 2020-02-24T17:00:21Z | 2020-02-24T17:00:21Z | OWNER | Interesting challenge: I would like to be able to "await" on Problem is: https://stackoverflow.com/a/32894169 suggests using Janus, a thread-aware asyncio queue: https://github.com/aio-libs/janus |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for writing to database via a queue 569613563 | |
590430988 | https://github.com/simonw/datasette/issues/682#issuecomment-590430988 | https://api.github.com/repos/simonw/datasette/issues/682 | MDEyOklzc3VlQ29tbWVudDU5MDQzMDk4OA== | simonw 9599 | 2020-02-24T16:50:48Z | 2020-02-24T16:50:48Z | OWNER | I'm dropping the progress bar idea. This mechanism is supposed to guarantee exclusive access to the single write connection, which means it should be targeted by operations that are as short as possible. An operation running long enough to need a progress bar is too long! Any implementation of progress bars for long running write operations needs to happen elsewhere in the stack. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for writing to database via a queue 569613563 | |
590417619 | https://github.com/simonw/datasette/issues/682#issuecomment-590417619 | https://api.github.com/repos/simonw/datasette/issues/682 | MDEyOklzc3VlQ29tbWVudDU5MDQxNzYxOQ== | simonw 9599 | 2020-02-24T16:27:36Z | 2020-02-24T16:27:36Z | OWNER | Error handling could be tricky. Exceptions thrown in threads don't show up anywhere by default - I would need to explicitly catch them and decide what to do with them. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for writing to database via a queue 569613563 | |
590417366 | https://github.com/simonw/datasette/issues/682#issuecomment-590417366 | https://api.github.com/repos/simonw/datasette/issues/682 | MDEyOklzc3VlQ29tbWVudDU5MDQxNzM2Ng== | simonw 9599 | 2020-02-24T16:27:10Z | 2020-02-24T16:27:10Z | OWNER | I wonder if I even need the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for writing to database via a queue 569613563 | |
590405736 | https://github.com/simonw/datasette/issues/675#issuecomment-590405736 | https://api.github.com/repos/simonw/datasette/issues/675 | MDEyOklzc3VlQ29tbWVudDU5MDQwNTczNg== | aviflax 141844 | 2020-02-24T16:06:27Z | 2020-02-24T16:06:27Z | NONE |
Great! I’ll give it a go.
Makes a ton of sense, will do.
Great point. I double checked the docs for The usage string is:
so in fact it’ll be more consistent to use a space to delimit the source and destination paths, like so:
and I suppose the short-form version of the option should be
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--cp option for datasette publish and datasette package for shipping additional files and directories 567902704 | |
590399600 | https://github.com/simonw/datasette/issues/682#issuecomment-590399600 | https://api.github.com/repos/simonw/datasette/issues/682 | MDEyOklzc3VlQ29tbWVudDU5MDM5OTYwMA== | simonw 9599 | 2020-02-24T15:56:10Z | 2020-02-24T15:56:23Z | OWNER | Implementation planMethod on Database class called Which calls Throws an error of database isn't mutable. Add We write to that queue with Have a This should be enough to get it all working. MVP can skip the .5s timeout entirely But... what about that progress bar supporting stretch goal? For that let's have each write operation that's currently in progress have total and done integer properties. So I guess we can add those to the Should we have the ability to see what the currently executing write is? Seems useful. Hopefully I can integrate https://github.com/tqdm/tqdm such that it calculates ETAs without actually trying to print to the console. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for writing to database via a queue 569613563 | |
590209074 | https://github.com/simonw/datasette/issues/676#issuecomment-590209074 | https://api.github.com/repos/simonw/datasette/issues/676 | MDEyOklzc3VlQ29tbWVudDU5MDIwOTA3NA== | tunguyenatwork 58088336 | 2020-02-24T08:20:15Z | 2020-02-24T08:20:15Z | NONE | Awesome, thank you so much. I’ll try it out and let you know. On Sun, Feb 23, 2020 at 1:44 PM Simon Willison notifications@github.com wrote:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
?_searchmode=raw option for running FTS searches without escaping characters 568091133 | |
590154309 | https://github.com/simonw/datasette/issues/682#issuecomment-590154309 | https://api.github.com/repos/simonw/datasette/issues/682 | MDEyOklzc3VlQ29tbWVudDU5MDE1NDMwOQ== | simonw 9599 | 2020-02-24T03:14:10Z | 2020-02-24T03:14:10Z | OWNER | Some prior art: Charles Leifer implemented a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for writing to database via a queue 569613563 | |
590153892 | https://github.com/simonw/datasette/issues/682#issuecomment-590153892 | https://api.github.com/repos/simonw/datasette/issues/682 | MDEyOklzc3VlQ29tbWVudDU5MDE1Mzg5Mg== | simonw 9599 | 2020-02-24T03:10:45Z | 2020-02-24T03:13:03Z | OWNER | Some more detailed notes I made earlier: Datasette would run a single write thread per database. That thread gets an exclusive connection, and a queue. Plugins can add functions to the queue which will be called and given access to that connection. The write thread for that database is created the first time a write is attempted. Question: should that thread have its own asyncio loop so that async techniques like httpx can be used within the thread? I think not at first - only investigate this if it turns out to be necessary in the future. This thread will run as part of the Datasette process. This means there is always a risk that the thread will die in the middle of something because the server got restarted - so use transactions to limit risk of damage to database should that happen. I don’t want web responses blocking waiting for stuff to happen here - so every task put on that queue will have a task ID, and that ID will be returned such that client code can poll for its completion. Could the request block for up to 0.5s just in case the write is really fast, then return a polling token if it isn't finished yet? Looks possible - There will be a Stretch goal: It would be cool if write operations could optionally handle their own progress reports. That way I can do some really nice UI around what’s going on with these things. This mechanism has a ton of potential. It may even be how we handle things like Twitter imports and suchlike - queued writing tasks. One catch with this approach: if a plugin is reading from APIs etc it shouldn't block writes to the database while it is doing so. So sticking a function in the queue that does additional time consuming stuff is actually an anti pattern. Instead, plugins should schedule their API access in the main event loop and occasionally write just the updates they need to make to that write queue. Implementation notesMaybe each item in the queue is a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for writing to database via a queue 569613563 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
issue 5