{"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/19#issuecomment-688626037", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19", "id": 688626037, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODYyNjAzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-08T05:27:07Z", "updated_at": "2020-09-08T05:27:07Z", "author_association": "MEMBER", "body": "A really clever way to do this would be with triggers. The indexer script would add triggers to each of the database tables that it is indexing - each in their own database.\r\n\r\nThose triggers would then maintain a `_index_queue_` table. This table would record the primary key of rows that are added, modified or deleted. The indexer could then work by reading through the `_index_queue_` table, re-indexing (or deleting) just the primary keys listed there, and then emptying the queue once it has finished.\r\n\r\nThis would add a small amount of overhead to insert/update/delete queries run against the table. My hunch is that the overhead would be miniscule, but I could still allow people to opt-out for tables that are so high traffic that this would matter.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 695556681, "label": "Figure out incremental re-indexing"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/19#issuecomment-688625430", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19", "id": 688625430, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODYyNTQzMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-08T05:24:50Z", "updated_at": "2020-09-08T05:24:50Z", "author_association": "MEMBER", "body": "I thought about allowing tables to define a incremental indexing SQL query - maybe something that can return just records touched in the past hour, or records since a recorded \"last indexed record\" value.\r\n\r\nThe problem with this is deletes - if you delete a record, how does the indexer know to remove it? See #18  - that's already caused problems.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 695556681, "label": "Figure out incremental re-indexing"}, "performed_via_github_app": null}