@Phil_Duby the JSON I’m working from is from the Friends+Me Google+ Exporter. I haven’t written anything to start from the google community takeout that Google finally got around to releasing a few days ago. That was too little too late for me.
I installed a Discourse development environment on my laptop and I do test imports into it with the importer I wrote. I dump the discourse_development database before importing, then restore it back before each subsequent import attempt. That helps only so much with filtering; I go through the category into which I have imported, and for each possible spam topic, I pull up the Discourse profile for the spammer and each possible astroturf poster, and check their topic and post history.
If they clearly spam or clearly astroturf, I really don’t care if they have one or two possibly-useful posts, I blacklist them. I click on “show email” in their profile preferences and it shows something like 12376914238747120000@gplus.invalid — I copy the 12376914238747120000 to the blacklist. Occasionally I find somene who seems questionable and I’m not sure, and then I append their ID to https://plus.google.com/u/0/ and go look at their C+ post history across communities, and I almost always find that they post the same drivel in 5 or 10 or more communities, and I add them to the blacklist. I’ve identified 437 spam/astroturf IDs primarily using this process. A few of them I identified by right-clicking on profiles of obvious spammers in G+ and copying the number from the end of their user URL (see above).
Looking through the Microcontroller Based Projects stream, it looks like it’s more like 80% or 90% spam. Not very many real projects, mostly just linkfarm posts from “SEO consultants” trying to game pagerank. It’s not that links aren’t valuable here, but if it’s just a link and what it points to is valuable, a web search will find the original source anyway, no point gumming up this forum with the link. I’m trying to focus the import here on OC rather than enabling linkfarming for SEO.
An alternative would be to identify the few actually useful posts, and bring in their content. That might be easier. There are fewer than 3000 posts all told in that community…
The json I have is from Google+ Exporter as well. I had a look at takeout a month ago, but all it seems to do was create links back to existing google+ content. Which of course is going away. I have not seen the recent release.
I have been working with the json content in a (json aware) text editor side-by-side with the current google+ content. Match the id of the spam content author with the json content, then search the json for other content by the same id for comparisons. The text editor lets me open a url directly from the json content. For other cases, I do much the same as you suggest, looking at profiles and related posts.
@funinthefalls it’s not obvious where to import tinyG — feels like we should have a controllers category with a sub-category for each specific controller; regardless of whether we’re pulling in old G+ content; tinyG, smoothie, etc. thoughts?
Smoothie created, along with link to official support forum. @Arthur_Wolf clearly hasn’t had time to publish the static archive I made for him, and why should he waste his time on that? I think if he doesn’t notice that I asked about pulling the content into that category soon I should just import it…
@Arthur_Wolf exactly, you are busy! Do you strenuously object to me importing the old G+ content here, along with the description that exists already pointing people to the official support forum? I’d love to make your life just a little easier!
@anon57870006 recently posted with a screenshot of a post from the old quadrap community, which I had archived but not yet had a chance to examine. Looks like there’s lots of useful stuff in there. The community content stopped before July 2017 and the spammers descended (with comment spam) after that, so I’m planning to drop posts and comments by date.
OpenRC Project [Update: archived for future investigation; noticed only one spam post afte reviewing two years of history, amazingly clean! @Daniel_Noree approved this port on Google+. Port complete ]
DIY Flying Robots / UAVs [Update: archived for future investigation; needs spam cleaning prior to import as it looks like mostly spam at least back through mid 2016. Perhaps a whitelist would be more appropriate for this import? Moderator @Marc_MERLIN approved the port, looking for volunteers to provide whitelist of people who created useful content.]
Do those two especially spark more ideas of useful maker content to capture? We have a couple more weeks in which to archive, but I don’t want to bet on G+ being fully available until the last possible day. Get those nominations in!
Thank you! Looks like a great idea, and Marc MERLIN responded there with thanks, so that import is underway. I haven’t looked for spam yet, but there is clearly real content there.
I’ll plan to archive it, but it has substantial spam in it, so porting the content here will have to wait for someone who is willing to filter it for spam; we’re trying to avoid porting lots of spam into makerforums.
As long as I get content archived before the Google+ shutdown, the port to makerforums can happen after the shutdown, so I’m focusing first on archive for nominated communities.
Any chance you could help me archive a non-maker community? Sorry if it’s taboo to ask such a thing here. It’s a 25k member G+ group. I’m pretty technical, but not a coder it that helps anything. Have plenty of websites up and running, but not as skilled as you to do such a task.
Semi related: I just got a new Ender 3, and plan on joining the community soon enough. (relatively)
In any case, awesome work. Thanks for using your powers for good!
@Jesse_Kramasz My family have been extremely tolerant of my passion for saving maker communities, but I don’t think I can ask for more. I can make suggestions at least. The Friends+Me Google+ Exporter is now I think $25nope still $20 as of this writing and lets you export to WordPress and blogger as well as the JSON format I’m using. There are others who will do the discourse import process for a fee as it’s their consulting day job, and my discourse importer is open source. Google takeout is now supposed to give owners and moderators a static site which you could deploy. There are several open source tools for rescuing information. For example, you can look at https://github.com/FiXato/Plexodus-Tools
I have a sneaking suspicion that at least some new posts here might be thanks to the hackaday post so I’ll wave hello to new people, and mention that I posted there about some of the alternatives as well.
