Lizard, the 8ch monitor

Hi Holla Forums

I made a CLI 8ch thread monitor: git.teknik.io/abrax/lizard

Anyone want to try it out?

Other urls found in this thread:

git.teknik.io/abrax/lizard/archive/master.zip
docs.python.org/3/library/hashlib.html
docs.python.org/release/3.4.4/library/hashlib.html
git.teknik.io/abrax/lizard/src/master/dist/lizard-0.1.linux-x86_64.tar.gz
8ch.net/tech/res/703632.html
packaging.python.org/distributing/
twitter.com/SFWRedditGifs

looks interesting. will try it.

wtf?, whats the point of using it when you don't know what's the thread is all about.

It looks neat. It's a pity 8ch sucks so much. Your software would be cool to use with a better imageboard.

If your software just slurps up images indiscriminately, you're going to download CP eventually. Which is pretty hot tbh. Hopefully it works through Tor.

I just pushed an update, so make sure you have the new version. I realized after making the thread that it wasn't adding threads properly.


People don't seem to be using titles that often. Also, when you use the comment text as a fallback in cases of no title, you it's not very useful because not all threads mention what they're about in the first half dozen words.

I'm not averse to adding it if there's a nice way of solving that issue. or you can add it yourself


Dunno, I like it. The only problem is that it moves so slow, it's a pain to keep track of what happened in what thread... Hence this.

Yeah, but so does your browser through thumbnails, so I figure it's no big deal.

You poor bastard.


Your software appears to download the full images, rather than thumbnails. Apples and oranges, m80.

Don't get me wrong, I'm with you: full size CP is way easier to fap to. I like the cut of your child-rapin' jib.

Anyway, I like this software. But have you considered rewriting it in Rust?

Yes, after considering it I decided to use Python. ;-)

...

Have you considered licensing it under GPL, BSD or some other free software license?

lol what for? just use it if you want to man, i promise i won't sue

It's like a promise not to sue, but legally binding enough that people are much more likely to be comfortable hosting it and modifying it. Technically, I wouldn't be allowed to mirror/fork your repository to host it on notabug.org as long as you don't use a (libre) license, for example. It can save a lot of trouble.

Licensing it is easy as hell, too. If you want something that's equivalent to "do what you want, I won't sue", put this at the top of each Python file, under the shebang:
# Copyright 2017 abrax# Copying and distribution of this file, with or without modification, are# permitted in any medium without royalty provided the copyright notice and# this notice are preserved. This file is offered as-is, without any warranty.
That's all it takes. It also means that people won't be able to sue you if they claim your script rm -rf /ed them or something.

I know how licensing works, I just didn't want to bother with this since I don't see how it would matter in practice. I've never seen anyone try to enforce these things with such a small project.

I'll add one just because you asked, though.

Well, I managed to fix most of the issues in my tracker, leaving only 3.


If anyone is actually using this and has ideas/suggestions, let me know.

Pretty neat. I was getting an error in the db_methods package, creating an empty folder named "data" will fix it.

That's weird, I wrote a bunch of code to make sure it creates the dir if it doesn't exist. Python is really annoying about this, all the file creation operations just crash instead of generating the necessary directory tree.

I'll see if I can fix it.

i have to run python3 ./lizard i to make anything show up... but i cant get past the help menu

what do?

im in ubuntu and have to run python3 in front of ./lizard to make it not give the error no module found: logic.db_methods

Does lizard.py have permission to execute?

Also try to cd into the same directory as lizard.py and run it from there. It should be setting the working directory automatically so it can find db_methods and stuff, but perhaps that doesn't work on Ubuntu for some reason. Works for me on Arch.

if python3 ./lizard shows the help screen what is the result of python3 ./lizard c ?

a bunch of shit about peewee module

which i also had to install previously using:
pip3 install peewee

Yeah, that's my ORM.

Do you mind posting the stack trace (including the command you used)? Search&replace your username out of it if you care about that, I don't think there would be any other identifying information.

[email protected]/* */:/home/bob/Downloads/lizard# sudo python3 ./lizard.py c

Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3600, in connect **self.connect_kwargs) File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3869, in _connect conn = sqlite3.connect(database, **kwargs)sqlite3.OperationalError: unable to open database fileDuring handling of the above exception, another exception occurred:Traceback (most recent call last): File "./lizard.py", line 61, in logic.db_methods.create_new_database() File "/home/bob/Downloads/lizard/logic/db_methods.py", line 27, in create_new_database db.connect() File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3602, in connect self.initialize_connection(self._local.conn) File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3514, in __exit__ reraise(new_type, new_type(*exc_args), traceback) File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 134, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3600, in connect **self.connect_kwargs) File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3869, in _connect conn = sqlite3.connect(database, **kwargs)

That looks like the problem was talking about. Creating a directory called "data" in the lizard folder should fix it, but commit cc7fe98018 takes care of it so if you just download the most recent version from the repo that should take care of matters.

that fixed the peewee error when i do: python3 ./lizard.py c

however it still doesnt like that i have to use python3 before hand or something i dunno.. shit

What do you mean?

"python3 script.py" just invokes the py3 interpreter directly to run the script. "./script.py" should do same thing if I'm not mistaken.

Where do you have python installed? Maybe that's the problem.

i used dos2unix and now i can run ./script.py just fine

it just shows the help info still

im not sure where pythons installed

nvm i guess its working now

OP, after installing the requirements (via pip & textfile) and changing permissions of lizard.py, I get this error when attempting to run the script from the terminal.

lizard ) sudo ./lizard.py Traceback (most recent call last): File "./lizard.py", line 4, in import logic.db_methodsImportError: No module named logic.db_methods

interestingly, when running the interpreter I get this:
lizard ) python3 lizard.py Traceback (most recent call last): File "lizard.py", line 4, in import logic.db_methods File "/home/maps/lizard/logic/db_methods.py", line 5, in import humanizeImportError: No module named 'humanize'

i'm too tired and drunk to give any more effort to this, so if this is relevant, there you go, otherwise, wow idiot

top box is accidental copy of second attempt as root, same error as first non-superuser attempt, ignore sudo

I had that happen once, the line endings are sometimes corrupted when you pull from the repo.

lizard.py just takes the arguments from sys.argv and does an if-else loop to see what applies. If the arguments given don't match any expected pattern then it prints the help info.

Either you're not giving the correct arguments ("./lizard.py c" looks correct) or somehow the arguments are getting lost.

Put this in a .py file and run it with some random arguments:
import sysprint "This is the name of the script: ", sys.argv[0]print "Number of arguments: ", len(sys.argv)print "The arguments are: " , str(sys.argv)
It should print back the arguments you gave it.

not op but.. you drunk
you have to use python3 will solve that i think

the second you have to install the humanize module using pip3

lol, don't run it with sudo

How come it's still asking for humanize? That's in requirements.txt. Not sure about the first block, logic.db_methods is part of my shit, it shouldn't have trouble importing that.

i did, read "lizard )" as $

I then installed humanize via pip. I tried to install other missing modules and they weren't immediately recognized and I'm not looking for their analogue right now.

i'm not putting aside error on my part, but i'm not seeing it at the moment inb4 fish and i followed op's directions, so figured if it is a bug, it may be helpful.

so this is mainly used to download pictures
what other functions does it serve?

wait my answer of "i did" makes no formal sense in terms of responding to your first comment, but then again, your first comment was kind of irrelevant anyway in the context of your second

but anyway that's my story. if i am missing something blatantly obvious i'm ok with that, i've been up for 33 hours.

looks like a cool tui program, would like to help make it better, even if that means making it shitbrain proof on my behalf

I use it to keep track of replies to interesting threads. This way I don't have to keep the browser open, and also it's easier to follow very slow boards.

Eh, fucked if I know tbh m8. Seems like distribution is a fucking pain in Python... Maybe I can package it better for people who don't want to fuck around with source just to get it to run. I really wasn't expecting it to be this fragile.

*not op*
1. Are you sure you downloaded all the files and not only "lizard.py"?
if not download here: git.teknik.io/abrax/lizard/archive/master.zip

2. You have to install the dependencies with pip3, pip is that manager of python 2.7 only.
sudo pip3 install humanize
sudo pip3 install peewee

3. Be sure to to run with like python3 lizard.py. Running as ./lizard.py is not working for some reason.

4. If you still get errors, create an empty folder called data alongside "logic".

That's because you use "#!/usr/bin/env python" as a shebang, which defaults to Python 2 on most distros. Replace "python" by "python3" and it'll work.

That would explain it. I'll change it in the repo next time I'm committing.

I was trying to figure out how to package the whole thing into a single file but haven't gotten anywhere.

Thanks for the tip.

I'm on ubuntu and the python installation is 3.4.4. I was having a bug
Downloading HTML for Holla Forums703632...Downloading 3 files: screenshot1.png - a22bc09c1f7397d3595a522e61e7acd0964b59ae3a57a2e4e5a5c3098211c70a.pngTraceback (most recent call last): File "lizard.py", line 53, in logic.db_methods.refresh_all_threads() File "/home/BigGuy/Downloads/lizard/logic/db_methods.py", line 133, in refresh_all_threads dump_thread_files(get_cache_path(t.board, t.thread_no), json) File "/home/BigGuy/Downloads/lizard/logic/web_methods.py", line 44, in dump_thread_files os.path.join(thread_cache_path, 'files', f['original_name'])) File "/home/BigGuy/Downloads/lizard/logic/web_methods.py", line 54, in download_file save_file(file_path, r.content) File "/home/BigGuy/Downloads/lizard/logic/file_io.py", line 36, in save_file existing_hash = hash_file(file_path) File "/home/BigGuy/Downloads/lizard/logic/file_io.py", line 25, in hash_file return hash_bytes(bytes) File "/home/BigGuy/Downloads/lizard/logic/file_io.py", line 15, in hash_bytes h = hashlib.blake2b(digest_size=5)AttributeError: 'module' object has no attribute 'blake2b'
The problem is that BLAKE2 algorithm is not available on version 3.4.4 and if I update manually everything can break, compare versions:
docs.python.org/3/library/hashlib.html
docs.python.org/release/3.4.4/library/hashlib.html

Alright guys, I figured out how to build it into a normal program, give me a minute or two here.


Yes, seems like it but I've removed BLAKE2 in a recent commit anyway.

Okay, the package is done. Look at the repo for install instructions. tl;dr: download git.teknik.io/abrax/lizard/src/master/dist/lizard-0.1.linux-x86_64.tar.gz and extract to root.

this tarball was not recognized by my tar command for some reason. (tar -zxvf) I tried manually copying the files to their respective files and now I have a botched af python installation.
my fault for messing with things (i copied the relevant files into where they would have gone via the extration) but i'd be interested in hearing if other folks have had success with the tarball.

It's weird that you use the exact same command that worked for me.

Do you have logs or something? You should be able to look at the contents of the tar and go clean up easily, it's not that many files. Also could just search for files created/modified after x date/time to easily find the crap.

Also what distro are you again? I can make an rpm.

I was on Ubuntu. The python fuck-up (self-caused by attempting to fix something without knowing what I was doing see: Ubuntu) led me to wipe and reinstall (it was time, anyway.)

The error that I got was that the tar command didn't recognize it as a tarball. There was no other output even with verbose turned on, and the logs are now gone.

I'm fairly certain it had something to do with the build requiring Python3.6 while my system insisted 3.5 was at fresh as it could install.

just tried on the fresh install. Tar command worked without -C / (though I can't find the lizard binary anywhere,) With it I get the same error.

/Downloads> tar -xvzf lizard-0.1.linux-x86_64.tar.gz -C /tar: This does not look like a tar archivetar: Skipping to next headertar: Exiting with failure status due to previous errors

FYI, tarball unzips perfectly via GUI or without -C argument.

Alright, I redownloaded for the third time (something weird appears to be wrecking half of the downloads, so the non-tarball error is on my end) and the extraction went through correctly. However, now I get this error after running the binary:
Traceback (most recent call last): File "/usr/bin/lizard", line 6, in from pkg_resources import load_entry_point File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2927, in @_call_aside File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2913, in _call_aside f(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 635, in _build_master ws.require(__requires__) File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 943, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 829, in resolve raise DistributionNotFound(req, requirers)pkg_resources.DistributionNotFound: The 'lizard==0.1' distribution was not found and is required by the application

Which, to my know-nothing eye, looks like confusion between the Python version the program wants and the one my computer defaults to.

Using a workaround I updated to 3.6 and it's still calling back to 2.7 when I try to execute.

Using Python3.6 or Python3 with a path to the file gets the same error, just updated to reflect Python3, with Python3 warranting a System Error dialog.

Open to suggestions, I'd find this program really useful.

alright, after some pip work (missing humanize on 3.6, and peewee on 3) I got it to work by running it with the interpreter via python3.


... just kidding:
python3 lizard.py l Thread URL Last reply Last seen Refreshed -----------------------------------------------------------------------------------------------------------------Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3683, in execute_sql cursor.execute(sql, params or ())sqlite3.OperationalError: no such table: threadDuring handling of the above exception, another exception occurred:Traceback (most recent call last): File "lizard.py", line 92, in main() File "lizard.py", line 58, in main db_methods.list_threads() File "/home/nope/FOSS/lizard/db_methods.py", line 84, in list_threads for t in threads: File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3156, in __iter__ return iter(self.execute()) File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3149, in execute self._qr = ResultWrapper(model_class, self._execute(), query_meta) File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 2833, in _execute return self.database.execute_sql(sql, params, self.require_commit) File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3690, in execute_sql self.commit() File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3514, in __exit__ reraise(new_type, new_type(*exc_args), traceback) File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 134, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.5/dist-packages/peewee.py", line 3683, in execute_sql cursor.execute(sql, params or ())peewee.OperationalError: no such table: thread

The command to create a new database works, but nothing works from that.

(putting this on top so you can skip to the important part)
Alright, whatever you did you got the environment set up right! That error is a bug that I need to (and can) fix.

Let me explain what I think is going on:
Every other command besides lizard c and lizard b assumes a database exists and has the correct schema (I need to add some error-handling logic here). If there is no threads.orm.db, then when something like lizard l tries to access it it will end up creating a defective threads.orm.db instead, and if you try to run lizard c after it will refuse to work (since it thinks the db exists and doesn't realize it's corrupted).

What I would do to get the version you have to work is:
1. Search for threads.orm.db (should be under /home/nope/lizard_data/threads.orm.db) and delete them all (check size/time/path to make sure you don't delete some unrelated file with coincidentally the same name but I doubt it)
2. Run "python3 lizard.py c" which will create a new, blank database. Search for threads.orm.db and note what path it's created under.
3. Run "python3 lizard.py l" which should print the thread table header and no threads (since you haven't added any) without errors. If you get errors, search for threads.orm.db and see if "lizard l" created a second threads.orm.db elsewhere, which would be a bug that you should tell me about.
4. Try "python3 lizard.py a 8ch.net/tech/res/703632.html#q704662" to add this thread and see what happens.

Or you can just wait for me to make a new release in 1-2 days which actually checks for the database and give a sane error message instead of shitting itself if the db is missing.


Yeah, I work with 3.6 since I always update it. The code itself isn't using any state of the art shit, I just have to figure out how to configure the package to not be so strict with Python version.

-C / tells it to extract to root, so you don't have to manually move them. If you look in the package it puts the code in /usr/lib/python3.6/site-packages/ and then a launcher in /usr/bin/ .

Why your tar is shitting itself, I can't begin to guess. The tar.gz I make is generated by Python distutils but I don't suppose such a mature utility would fuck up a tar. Weird. Since you can get a different archive manager to open it I'm tempted to say the problem is with your system. BTW you can replace "-C /" with "--directory=/", it's the same command.


Here's what a successful "install" looks like for me. `/usr/bin/lizard` would be the path to the program, runnable by just typing `lizard`.

Yeah, it's trying to run it with 2.7 which won't work.

[7 (You)'s] whoa nelly
Yep, your workaround fixed it. I noticed that the previous, erroneous database was blank, but that a directory named thread, containing a file named after the thread, was created.

All is well at the moment.

The tarball error was definitely on my end, not sure why so many of the installations got botched. It was weird. I was able to download and extract the files if I downloaded an instance to the archiver, but downloading the file directly didn't work. However, saving the temp instance from the archiver worked.

Thanks for the follow up, looking forward to seeing how this post interacts with the extant working database.

[code]
b&[email protected]/* */&'d ~/F/lizard> python3 lizard.py l
Thread URL Last reply Last seen Refreshed
-----------------------------------------------------------------------------------------------------------------
8ch.net/tech/res/703632.html 2 hours a minute 34 seconds

Cool, glad to hear it worked! Thanks for sticking with it.

The database thing is, in retrospect my fault. My logic was "well the user shouldn't try to do things that obviously require the database before creating it first so it's pebkac not a bug" but on further reflection the code should handle it. It's planned for v0.2 now.

I've only been using this for about a week so it needs testing. If you manage to break anything else and I'm not here submit an issue to the tracker in the repo (don't forget a stack trace). You'll probably need a teknik.io account but it only requires a username/pass to create.

I think you can email me at [email protected]/* */ but I haven't checked that mailbox yet and who knows when I will.

What's the point to this?

What's the point of anything, man?

Just released v0.2, see readme in repo for changelog.

it's shit. i'm going to rewrite it in rust.

See you in 5 years... Don't worry, I'll be keeping on eye on this thread ;-)

i'm also going to write an imageboard in rust.

Is rust your favorite language?

yes. it could be better though.

What language(s) did you replace with Rust? (inb4 Python)

So, would anyone happen to have some feedback or something?

I'm planning a big change for v0.3: Instead of just a DB of threads, I will keep three separate tables for threads, board and posts.

Threads will work like they do now when you add a thread without an anchored reply.

Posts will be similar to the current anchored reply tracking/>(You) counting. The use of these is very fast threads where you only care about replies to you specifically. For instance I've noticed the Holla Forums sticky can't be tracked easily currently, since you can't ignore the non-You replies. When you call lizard o, it often opens the sticky as well even if there is obviously nothing of interest to you there.

Boards will be a new feature, for very slow boards. It will watch for any activity on the board as a whole. Again, I've noticed that current lizard doesn't make it easy to track slow boards, you end up adding a dozen threads that hardly ever have activity, and miss new threads.

What do you guys think about this idea?

I already have the URL parser which can detect threads vs. anchored replies, adding board URL detection is easy. DB compatibility will break, but I think that's ok thanks to lizard e. The big issue I can see is that if I implement the above model, then it becomes impossible to have the current thread-with-anchored-reply functionality: You can get notified about all replies in a thread, you can get notified about replies quoting a specific post, but how would the user show that they want to get notified about replies in the thread AND get separately notified if the replies are quoting a specific post?

The only solution I can think of is to add two items for such a thread: One in the form of 8ch.net/tech/res/1234.html which lizard will put into the threads table, and one in the form of 8ch.net/tech/res/1234.html#5678 which lizard will put in the posts table. Seems a bit silly to add the same thing twice but it's pretty coherent logically.

And the other issue is packaging. I don't know shit about distributing python apps, and my current "build system" (using the term very loosely) is a horrid hack job. I only did it in the hopes of giving you fags another way of running it. But I've been using the "installed" version for some time myself and it's pretty convenient, so I think packages are the way forward. But the question is how do I into making packages right?

Oh, forgot to say: For the new tables, I think I will do so by creating a superclass base model, and pull some members like url, last_refreshed, last_reply and so on in there. I suspect the refresh logic will barely change, the difference will be in how "last reply/oldest new reply" is identified and how JSON is parsed. Luckily I've already encapsulated those as methods, so now I'll just add two more methods named in a suitable way.

packaging.python.org/distributing/

Hi guy who made lizard

hello


Have you ever followed this guide?

Dubs demand it

So has anyone tried this? Does it suck, or did you just not get it to work and gave up? Any comments/suggestions?

I've been using for 2 weeks and seems to work okay.

8ch has a watchlist

Yeah and it sucks shit, like every other function of this website.

dont worry in going to rewrite it in rust

This, also my browser doesn't keep cookies.

You ever considerd just putting it in the public domain?

yeah